Creating a Job

Updated on 2024-11-29 GMT+08:00

View PDF

You can submit programs developed by yourself to MRS to execute them, and obtain the results.

This section describes how to submit a job (take a MapReduce job as an example) on the MRS console. MapReduce jobs are used to submit JAR programs to quickly process massive amounts of data in parallel and create a distributed data processing and execution environment.

If the job and file management functions are not supported on the cluster details page, submit the jobs in the background.

Before creating a job, you need to upload local data to OBS for data computing and analyzing. MRS allows exporting data from OBS to HDFS for computing and analyzing. After the data analysis and computing are completed, you can store the data in HDFS or export them to OBS. HDFS and OBS can also store the compressed data in the format of bz2 or gz.

NOTE:

If the IAM username contains spaces (for example, admin 01), a job cannot be created.

Submitting a Job on the GUI

Log in to the MRS console.
Choose Clusters > Active Clusters, select a running cluster, and click its name to access the cluster details page.
If Kerberos authentication is enabled for the cluster, perform the following steps. If Kerberos authentication is not enabled for the cluster, skip this step.

In the Basic Information area on the Dashboard page, click Synchronize on the right side of IAM User Sync to synchronize IAM users.
Click the Jobs tab.
Click Create. The Create Job dialog box is displayed.

In Type, select MapReduce. Configure other job information.

**Table 1** Job parameters
Parameter	Description
Name	Job name. It contains 1 to 64 characters. Only letters, digits, hyphens (-), and underscores (_) are allowed. NOTE: You are advised to set different names for different jobs.
Program Path	Path of the program package to be executed. The following requirements must be met: Contains a maximum of 1,023 characters, excluding special characters such as ;\|&><'$. The parameter value cannot be empty or full of spaces. The path of the program to be executed can be stored in HDFS or OBS. The path varies depending on the file system. OBS: The path starts with obs://. Example: *obs://wordcount/program/xxx.jar* HDFS: The path must start with /user. For SparkScript and HiveScript, the path must end with .sql. For MapReduce, the path must end with .jar. For Flink and SparkSubmit, the path must end with .jar or .py. The .sql, .jar, and .py are case-insensitive.
Parameters	(Optional) It is the key parameter for program execution. Separate multiple parameters with space. Configuration method: Program class name Data input path Data output path Program class name: It is specified by a function in your program. MRS is responsible for transferring parameters only. Data input path: Click HDFS or OBS to select a path or manually enter a correct path. Data output path: Enter a directory that does not exist. The parameter contains a maximum of 150,000 characters. It cannot contain special characters ;\|&><'$, but can be left blank. CAUTION: If you enter a parameter with sensitive information (such as the login password), the parameter may be exposed in the job details display and log printing. Exercise caution when performing this operation.
Service Parameters	(Optional) Used to modify service configuration parameters for the job to be executed. The parameter modification applies only to the job to be executed. To add multiple parameters, click on the right. To delete a parameter, click Delete on the right. Table 2 describes the common parameters of a service.
Command Reference	Command submitted to the background for execution when a job is submitted.

**Table 2** Service configuration parameters
Parameter	Description	Example Value
fs.obs.access.key	Key ID for accessing OBS.	-
fs.obs.secret.key	Key corresponding to the key ID for accessing OBS.	-

Confirm job configuration information and click OK.

After the job is created, you can manage it.

Parent topic: Getting Started

Previous topic: Uploading Data

Next topic: Terminating a Cluster

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot