Creating a Flink SQL job

Updated on 2022-07-04 GMT+08:00

View PDF

This section describes how to create a Flink streaming SQL job. Flink SQL provides users a method for compiling jobs based on their logic requirements. SQL-based business logic expression facilitates service implementation. Currently, Flink job supports compiling Flink SQL statements by using the SQL editor and visual editor. This section describes how to use the SQL editor to compile Flink streaming SQL jobs.

Prerequisites

You have prepared the data input and data output channels. For details, see Preparing Data.

Creating a Flink SQL job

In the left navigation pane of the DLI management console, choose Job Management > Flink Jobs. The Flink Jobs page is displayed.
In the upper right corner of the Flink Jobs page, click Create Job.

Specify job parameters.

**Table 1** Job configuration information
Parameter	Description
Type	Set Type to Flink SQL. You will need to start jobs by compiling SQL statements.
Name	Name of a job. Enter 1 to 57 characters. Only letters, digits, hyphens (-), and underscores (_) are allowed. NOTE: The job name must be globally unique.
Description	Description of a job. It can be up to 512 characters long.
Editor	SQL Editor and Visual Editor are both available. SQL Editor is used by default.
Template Name	This parameter is only used when Editor is set to SQL Editor. You can select a sample template or a custom job template. For details about templates, see Flink Template Management.

Click OK to enter the Edit page.
Edit a Flink SQL job

Enter details SQL statements in the SQL statement edit area. For details about SQL statements, see the Data Lake Insight SQL Syntax Reference.
Click Check Semantics.
- You can Debug or Start a job only after the semantic verification is successful.
- If verification is successful, the message "The SQL semantic verification is complete. No error." will be displayed.
- If verification fails, a red "X" mark will be displayed in front of each SQL statement that produced an error. You can move the cursor to the "X" mark to view error details and change the SQL statement as prompted.

Set job running parameters.

**Table 2** Running parameters
Parameter	Description
CUs	Sum of the number of compute units and job manager CUs of DLI. One CU equals one vCPU and 4 GB.
Job Manager CUs	Number of CUs of the management unit.
Max Concurrent Jobs	Maximum number of Flink SQL jobs that run at the same time. NOTE: This value cannot be greater than four times the compute units (number of CUs minus the number of job manager CUs).
Task Manager Configuration	Whether to set Task Manager resource parameters If this option is selected, you need to set the following parameters: CU(s) per TM: Number of resources occupied by each Task Manager. Slot(s) per TM: Number of slots contained in each Task Manager.
Save Job Log	Whether to save the job running logs to OBS If this option is selected, you need to set the following parameters: OBS Bucket: Select an OBS bucket to store user job logs. If the selected OBS bucket is not authorized, click Authorize. NOTE: If both Enable Checkpointing and Save Job Log are selected, you only need to authorize OBS once.
Alarm Generation upon Job Exception	Whether to report job exceptions, for example, abnormal job running or exceptions due to an insufficient balance, to users via SMS or email If this option is selected, you need to set the following parameters: SMN topic Select a user-defined SMN topic. For details about how to customize SMN topics, see Creating a Topic in the Simple Message Notification User Guide.
Enable Checkpointing	Whether to enable job snapshots. If this function is enabled, jobs can be restored based on the checkpoints. If this option is selected, you need to set the following parameters: Checkpoint Interval is the number of seconds between when new checkpoints are created. The parameter value ranges from 1 to 999999, and the default value is 10. Checkpoint Mode can be set to either of the following values: At least once: Events are processed at least once. Exactly once: Events are processed only once. OBS Bucket: Select an OBS bucket to store your checkpoints. If the selected OBS bucket is not authorized, click Authorize. NOTE: If both Enable Checkpointing and Save Job Log are selected, you only need to authorize OBS once.
Auto Restart upon Exception	Whether to enable automatic restart. If this function is enabled, any job that has become abnormal will be automatically restarted. If this option is selected, you need to set the following parameters: Restore Job from Checkpoint: This parameter is available only when Enable Checkpointing is selected.
Idle State Retention Time	Defines for how long the state of a key is retained without being updated before it is removed in GroupBy or Window. The default value is 1 hour.
Dirty Data Policy	Select a policy for processing dirty data. The following policies are supported: Ignore, Trigger a job exception, and Save. NOTE: Save indicates that the dirty data is stored to the OBS bucket selected above.
Dirty Data Dump Address	Set this parameter when Dirty data policy is set to Save. Click the address box to select the OBS path for storing dirty data.
Queue	Shared queue is selected by default. You can also select a dedicated custom queue. If you selected custom queues, you need to configure the following parameters: UDF Jar: Before selecting a JAR file to be inserted, upload the corresponding JAR package to the OBS bucket and choose Data Management > Package Management to create a package. For details, see Creating a Package. In SQL, you can call a user-defined function that is inserted into a JAR file. NOTE: During job creation, a sub-user can only select a queue that has been allocated to the user. If the remaining capacity of the selected queue cannot meet the job requirements, the system automatically scales up the capacity and you will be billed based on the increased capacity. When a queue is idle, the system automatically scales in the queue.
UDF Jar	User-defined UDF files can be uploaded to DLI using the Package Management function.

Click Save.
Click Start. On the displayed Start Flink Job page, confirm the job specifications, and click Start Now to start the job.

After the job is started, the system automatically switches to the Flink Jobs page, and the created job is displayed in the job list. You can view the job status in the Status column. After a job is successfully submitted, the job status will change from Submitting to Running. After the execution is complete, the message Completed is displayed.

If the job status is Submission failed or Running exception, the job submission failed or the job did not execute successfully. In this case, you can move the cursor over the status icon in the Status column of the job list to view the error details. You can click to copy these details. After handling the fault based on the provided information, resubmit the job.
NOTE:
Other buttons are as follows:
- Save As: Save the created job as a new job.
- Debug: Perform job debugging. For details, see Debugging a Job.
- Format: Format the SQL statements in the editing box.
- Set as Template: Set the created SQL statements as a job template.
- Theme Settings: Set the theme related parameters, including Font Size, Wrap, and Page Style.