Using JobGateway from Scratch
JobGateway is a gateway service that mimics big data clients (like MapReduce, Spark, Flink, and Hive clients). These clients are typically used to submit big data jobs. JobGateway operates over HTTPS and provides REST APIs for users to submit and manage big data jobs. Users can also query the status of submitted jobs and terminate them if needed. It features ease of use, high performance, high availability, and high scalability, as well as separated monitoring, alarm, and configuration, significantly shortening job submission links and simplifying the big data job submission process.
The JobGateway service consists of two roles: JobBalancer and JobServer. JobBalancer acts as the gateway for forwarding REST job requests and distributing them to different JobServer instances for load balancing. JobServer is the main executor that constructs submission contexts based on job types and submits jobs to YARN.

This section applies only to MRS 3.3.1-LTS or later.
Example
For example, a user submits a large number of jobs through different big data component clients, which is complex and time-consuming. JobGateway provides a much easier way to do this. You only need to install JobGateway and construct HTTP/HTTPS-based RESTful URLs.
In MRS 3.3.1-LTS or later, JobGateway provides a job submission API for tenants. The following uses Hive job submission as an example:
curl --location --request POST 'https://{host}:{port}/v2/job-executions --header 'JobServerAuthorization: {AuthorizationInfo}' --header 'Content-Type: application/json' --data-raw '{ "job_name":"{job-name}", "job_type":"HiveSql", "arguments":["SHOW TABLES"] }'
Return value:
{ "job_submit_result": { "job_id":"84261052-c47b-4691-ae0b-95bff3191c90", "state":"COMPLETE" } }
Parameter |
Description |
Remarks |
---|---|---|
host |
IP address of the JobBalancer server |
- |
port |
JobBalancer HTTPS port |
The default HTTPS port is 29974. |
user.name |
Name of the user who submits a job |
- |
JobServerAuthorization |
Authorization information |
- |
job_name |
Job name |
- |
job_type |
Job type |
HiveSql indicates a Hive SQL job. |
arguments |
Hive SQL job content |
- |

- User constraints: JobGateway does not support job submission by AD users who use clusters. AD users cannot submit jobs on the job management page of the management plane.
- If the configurations of other components such as ZooKeeper are modified on the server and the JobGateway configuration expires, you need to restart the JobGateway component and update the client configuration of the node where the JobServer role is used. To refresh the client configuration of the node where JobServer is located, perform the following steps:
- Log in to the client installation node as user root and run the following command to switch to user omm:
su - omm
- Go to the client installation directory, for example /opt/Bigdata/client, and run the following commands to update the configuration file:
cd /opt/Bigdata/client
sh autoRefreshConfig.sh
- Enter the username and password of the FusionInsight Manager administrator and the floating IP address of FusionInsight Manager.
- Enter the names of the components whose configuration needs to be updated. Use commas (,) to separate the component names. Press Enter to update the configurations of all components if necessary.
If the following information is displayed, the configurations have been updated successfully.
Succeed to refresh components client config.
- Log in to the client installation node as user root and run the following command to switch to user omm:
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot