Oozie Jobs Do Not Run When a Large Number of Jobs Are Submitted Concurrently

Issue

When a large number of Oozie jobs are submitted concurrently, the jobs do not run.

Symptom

When a large number of Oozie jobs are submitted concurrently, the jobs do not run.

Cause Analysis

When Oozie submits a job, an oozie-launcher job is started first, and then the oozie-launcher job submits the real job for execution. By default, the oozie-launcher job and the real job are in the same queue.

When a large number of Oozie jobs are submitted concurrently, a large number of oozie-launcher jobs may be started, exhausting the resources of the queue. As a result, no more resources are available to start real jobs, and the jobs are not executed.

Procedure

1. Create a queue for Oozie. For details, see User Guide > Managing an Existing Cluster > Tenant Management > Creating a Tenant. You can also use the launcher-job queue generated during MRS cluster creation.
2. On Manager, choose Cluster > Services > Oozie > Configurations, search for oozie.site.configs, and add oozie.launcher.default.queue as the parameter name and launcher-job as the value.