Oozie Jobs Do Not Run When a Large Number of Jobs Are Submitted Concurrently

Symptom

When a large number of Oozie jobs are submitted concurrently, the jobs do not run.

Cause Analysis

When Oozie submits a job, an oozie-launcher job is started first, and then the oozie-launcher job submits the real job for execution. By default, the oozie-launcher job and the real job are in the same queue.

When a large number of Oozie jobs are submitted concurrently, a large number of oozie-launcher jobs may be started, exhausting the resources of the queue. As a result, no more resources are available to start real jobs, and the jobs are not executed.

Procedure

1. Create a queue for Oozie by referring to Creating a Tenant in the User Guide. Alternatively, you can directly use the launcher-job queue generated during MRS cluster creation.
2. On Manager, choose Cluster > Services > Oozie > Configurations, search for oozie.site.configs, and add oozie.launcher.default.queue as the parameter name and the queue name as the value, for example, launcher-job.