Using JobGateway from Scratch

JobGateway is a gateway service that simulates big data component (such as Flink, Hive, and HBase) clients to allow you to submit big data jobs through HTTP/HTTPS-based REST APIs. It features ease of use, high performance, availability, and scalability, as well as separated monitoring, alarm, and configuration, significantly shortening job submission links and simplifying the big data job submission process.

Context

For example, a group submits a large number of jobs through different big data component clients, which is complex and time-consuming. JobGateway provides a much easier way. You only need to construct HTTP/HTTPS-based REST APIs.

The following is an example of submitting a Hive job (for details about how to submit other big data jobs, see the JobGateway API document):

curl --location --request POST 'https://{host}:{port}/mrsjob/submit?user.name={username}'
--header 'JobServerAuthorization: {AuthorizationInfo}'
--header 'Content-Type: application/json'
--data-raw '{
"job_name":"{job-name}",
"job_type":"HiveSql",
"arguments":["SHOW TABLES"]
}'

Return value:

{
    "id": null,
    "state": "COMPLETE",
    "errorCode": 0,
    "errorCodeDescription": null,
    "errorDescription": null,
    "failedNodeList": null,
    "totalProgress": "0",
    "job_id": "466710d2-b1ff-4a98-805b-4675292e5cc8"
}

Hive job submission parameters
Parameter	Description	Remarks
host	IP address of the Nginx server	-
port	Nginx monitoring port	The default value is 29970 for HTTPS and 29971 for HTTP.
user.name	Name of the user who submits a job	-
JobServerAuthorization	Authorization information	For details, see the JobGateway API document.
job_name	Job name	-
job_type	Job type	HiveSql indicates a HiveSql job.
arguments	HiveSql job content	-

If the configuration of other components such as ZooKeeper and KMS is modified on the server and the JobGateway configuration expires, you need to restart the JobGateway component and update the client configuration of the node where the JobServer role is used. To refresh the client configuration of the node where JobServer is, perform the following steps:

Log in to the node where the client is installed as user root and run the following command to switch to user omm:
su - omm
Go to the client installation directory, for example, /opt/Bigdata/client and run the following commands to update the configuration file:
cd /opt/Bigdata/client

sh autoRefreshConfig.sh
Enter the username and password of the FusionInsight Manager administrator and the floating IP address of FusionInsight Manager.
Enter the components whose configuration needs to be updated. Use commas (,) to separate the component names. Press Enter to update the configurations of all components if necessary.
If the following information is displayed, the configurations have been updated successfully.
```
Succeed to refresh components client config.
```