Using JobGateway from Scratch

JobGateway is a gateway service that mimics big data clients (like MapReduce, Spark, Flink, and Hive clients). These clients are typically used to submit big data jobs. JobGateway operates over HTTPS and provides REST APIs for users to submit and manage big data jobs. Users can also query the status of submitted jobs and terminate them if needed. It features ease of use, high performance, high availability, and high scalability, as well as separated monitoring, alarm, and configuration, significantly shortening job submission links and simplifying the big data job submission process.

The JobGateway service consists of two roles: JobBalancer and JobServer. JobBalancer acts as the gateway for forwarding REST job requests and distributing them to different JobServer instances for load balancing. JobServer is the main executor that constructs submission contexts based on job types and submits jobs to YARN.

This section applies only to MRS 3.3.1-LTS or later.

Example

For example, a user submits a large number of jobs through different big data component clients, which is complex and time-consuming. JobGateway provides a much easier way to do this. You only need to install JobGateway and construct HTTP/HTTPS-based RESTful URLs.

In MRS 3.3.1-LTS or later, JobGateway provides a job submission API for tenants. The following uses Hive job submission as an example:

curl --location --request POST 'https://{host}:{port}/v2/job-executions
--header 'JobServerAuthorization: {AuthorizationInfo}'
--header 'Content-Type: application/json'
--data-raw '{
"job_name":"{job-name}",
"job_type":"HiveSql",
"arguments":["SHOW TABLES"]
}'

Return value:

{
    "job_submit_result":
        {
           "job_id":"84261052-c47b-4691-ae0b-95bff3191c90",
           "state":"COMPLETE"
        }
}

Parameter	Description	Remarks
host	IP address of the JobBalancer server	-
port	JobBalancer HTTPS port	The default HTTPS port is 29974.
user.name	Name of the user who submits a job	-
JobServerAuthorization	Authorization information	-
job_name	Job name	-
job_type	Job type	HiveSql indicates a Hive SQL job.
arguments	Hive SQL job content	-

User constraints: JobGateway does not support job submission by AD users who use clusters. AD users cannot submit jobs on the job management page of the management plane.
If the configurations of other components such as ZooKeeper are modified on the server and the JobGateway configuration expires, you need to restart the JobGateway component and update the client configuration of the node where the JobServer role is used. To refresh the client configuration of the node where JobServer is located, perform the following steps:
1. Log in to the client installation node as user root and run the following command to switch to user omm:
  su - omm
2. Go to the client installation directory, for example /opt/Bigdata/client, and run the following commands to update the configuration file:
  cd /opt/Bigdata/client
  
  sh autoRefreshConfig.sh
3. Enter the username and password of the FusionInsight Manager administrator and the floating IP address of FusionInsight Manager.
4. Enter the names of the components whose configuration needs to be updated. Use commas (,) to separate the component names. Press Enter to update the configurations of all components if necessary.
  If the following information is displayed, the configurations have been updated successfully.
```
Succeed to refresh components client config.
```