Using JobGateway from Scratch
JobGateway is a gateway service that simulates big data component (such as Flink, Hive, and HBase) clients to allow you to submit big data jobs through HTTP/HTTPS-based REST APIs. It features ease of use, high performance, availability, and scalability, as well as separated monitoring, alarm, and configuration, significantly shortening job submission links and simplifying the big data job submission process.
Context
For example, a group submits a large number of jobs through different big data component clients, which is complex and time-consuming. JobGateway provides a much easier way. You only need to construct HTTP/HTTPS-based REST APIs.
The following is an example of submitting a Hive job:
curl --location --request POST 'https://{host}:{port}/mrsjob/submit?user.name={username}' --header 'JobServerAuthorization: {AuthorizationInfo}' --header 'Content-Type: application/json' --data-raw '{ "job_name":"{job-name}", "job_type":"HiveSql", "arguments":["SHOW TABLES"] }'
Return value:
{ "id": null, "state": "COMPLETE", "errorCode": 0, "errorCodeDescription": null, "errorDescription": null, "failedNodeList": null, "totalProgress": "0", "job_id": "466710d2-b1ff-4a98-805b-4675292e5cc8" }
Hive job submission parameters |
||
---|---|---|
Parameter |
Description |
Remarks |
host |
IP address of the Nginx server |
- |
port |
Nginx monitoring port |
The default value is 29970 for HTTPS and 29971 for HTTP. |
user.name |
Name of the user who submits a job |
- |
JobServerAuthorization |
Authorization information |
For details, see the JobGateway API document. |
job_name |
Job name |
- |
job_type |
Job type |
HiveSql indicates a HiveSql job. |
arguments |
HiveSql job content |
- |
![](https://support.huaweicloud.com/intl/en-us/cmpntguide-lts-mrs/public_sys-resources/note_3.0-en-us.png)
If the configuration of other components such as ZooKeeper is modified on the server and the JobGateway configuration expires, you need to restart the JobGateway component and update the client configuration of the node where the JobServer role is used. To refresh the client configuration of the node where JobServer is, perform the following steps:
- Log in to the node where the client is installed as user root using PuTTY and run the following command to switch to user omm:
su - omm
- Go to the client installation directory, for example, /opt/Bigdata/client and run the following commands to update the configuration file:
cd /opt/Bigdata/client
sh autoRefreshConfig.sh
- Enter the username and password of the FusionInsight Manager administrator and the floating IP address of FusionInsight Manager.
- Enter the components whose configuration needs to be updated. Use commas (,) to separate the component names. Press Enter to update the configurations of all components if necessary.
If the following information is displayed, the configurations have been updated successfully.
Succeed to refresh components client config.
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot