Creating a FlinkServer Job

Scenarios

Define Flink jobs, including Flink SQL and Flink JAR jobs.

The statements of a SQL job submitted on FlinkServer are saved to the DBServer. In MRS 3.5.0 and later versions, FlinkServer encrypts SQL storage by default to protect information. The password field in Flink SQL statements is left blank when displayed in the command output on the FlinkServer web UI. Before submitting a job, you must manually re-enter the password. For a custom connector, the password field name must contain the keyword password to prevent the password being displayed on the page.

Creating a Job

1. Access the Flink web UI. For details, see Accessing the FlinkServer Web UI.
2. Click Job Management. The job management page is displayed.
3. Click Create Job. Create a Flink SQL job or Flink Jar job, enter job information, and click OK. The job is created and the job development page is displayed.
4. (Optional) To develop a job immediately, configure the job on the job development page.

   The system allows you to add a lock to a job. The user who locks the job has all permissions on the job. Other users do not have the permissions to develop, start, or delete the locked job. However, they can forcibly acquire the lock to obtain all permissions on the job. After this function is enabled, you can click Lock or Unlock to lock or unlock a job, or click Acquire Lock to obtain job permissions.

   

   Job locks are enabled by default. You can view the status of this function on FusionInsight Manager. This function is available in MRS 3.3.0 or later only.

   Log in to FusionInsight Manager, choose Cluster > Service > Flink, click Configuration and then All Configurations, and search for the job.edit.lock.enable parameter. If the parameter value is true, the function is enabled. If the parameter value is false, the function is disabled.

   • Creating a Flink SQL job
     1. Develop the job on the job development page.
        Figure 1 FlinkServer job development page
        

     2. Click Check Semantic to check the input content and click Format SQL to format SQL statements.
     3. After the job SQL statements are developed, set basic and customized parameters as required by referring to Table 1 and click Save.

        Table 1 Basic parameters

        Parameter	Description
        Parallelism	Number of parallel jobs
        Maximum Operator Parallelism	Maximum degree of parallelism of operators
        JobManager Memory (MB)	Memory of JobManager The minimum value is 4096.
        Submit Queue	Queue to which a job is submitted. If this parameter is not set, the default queue is used.
        taskManager	taskManager running parameters include: Slots: If this parameter is not specified, the default value 1 is used. You are advised to set this parameter to the number of CPU cores. Memory (MB): The minimum value is 4096.
        Enable CheckPoint	Whether to enable CheckPoint. After CheckPoint is enabled, you need to configure the following information: Time Interval (ms): This parameter is mandatory. Mode: This parameter is mandatory. The options are EXACTLY_ONCE and AT_LEAST_ONCE. Minimum Interval (ms): The minimum value is 10. Timeout Duration: The minimum value is 10. Maximum Parallelism: The value must be a positive integer containing a maximum of 64 characters. Whether to clean up: This parameter can be set to Yes or No. Whether to enable incremental checkpoints: This parameter can be set to Yes or No.
        Failure Recovery Policy	Failure recovery policy of a job. The options are as follows. For details, see Configuring the Flink Job Restart Policy. fixed-delay: You need to configure Retry Times and Retry Interval (s). failure-rate: You need to configure Max Retry Times, Interval (min), and Retry Interval (s). none

     4. Click Submit in the upper left corner to submit the job.
   • Creating a Flink JAR job
     1. Click Select to upload a local JAR file and set parameters by referring to Table 2 or add customized parameters.

        Table 2 Parameter configuration

        Parameter	Description
        Local .jar File	Upload a local JAR file. Upload a local file smaller than the threshold specified by flinkserver.upload.jar.max.size. The default value is 500 MB. Log in to FusionInsight Manager, choose Cluster > Services > Flink, click Configurations and then All Configurations, search for flinkserver.upload.jar.max.size, and set the JAR file threshold. The value ranges from 100 MB to 5120 MB.
        Main Class	Main-Class type. Default: By default, the class name is specified based on the Manifest file in the JAR file. Specify: Manually specify the class name.
        Type	Class name. This parameter is available when Main Class is set to Specify.
        Class Parameter	Class parameters of Main-Class (parameters are separated by spaces).
        Parallelism	Number of parallel jobs Concurrent tasks per job operator. Increasing this value can improve the overall computing performance of a job. However, because adding more threads introduces switchover overhead, the maximum recommended value is four times the number of SPUs in the computing unit. For best performance, set the value to one to two times the number of SPUs.
        JobManager Memory (MB)	Memory of JobManager The minimum value is 4096.
        Submit Queue	Queue to which a job is submitted. If this parameter is not set, the default queue is used.
        taskManager	taskManager running parameters include: Slots: If this parameter is not specified, the default value 1 is used. You are advised to set this parameter to the number of CPU cores. Memory (MB): The minimum value is 4096.

     2. Click Save to save the configuration and click Submit to submit the job.

5. Return to the job management page. You can view information about the created job, including job name, type, status, kind, and description.

   After a job is created, you can start, develop, stop, edit, and delete the job, view job details, and rectify checkpoint faults in the Operation column of the job.

   

   • To read files related to a submitted job on the node as another user, ensure that both the user and the job submitter belong to the same user group and that both have been assigned the FlinkServer application management role. For example, application view is selected as described in Creating a FlinkServer Role.
   • You can view details about jobs in the Running state.
   • You can rectify checkpoint faults for jobs in the Running failed, Running succeeded, or Stop state.
   • To set whether the checkpoints of failed or canceled jobs can be retained, log in to FusionInsight Manager and choose Cluster > Services > Flink, click Configurations and then All Configurations, search for and set the execution.checkpointing.externalized-checkpoint-retention parameter of FlinkServer.
     • DELETE_ON_CANCELLATION: Only checkpoints of failed jobs will be retained.
     • RETAIN_ON_CANCELLATION (default value in MRS3.5.0 or later): Checkpoints of failed or canceled jobs will be retained.
     • NO_EXTERNALIZED_CHECKPOINTS(default value for MRS versions earlier than 3.5.0): Checkpoints of failed or canceled jobs will not be retained.