Procedure

The following procedure describes how to create a Flink streaming SQL job by using the visual editor in a DIS-CS (Window)-DIS scenario.

1. Log in to the CS management console.
2. In the navigation tree on the left pane of the CS management console, choose Job Management to switch to the Job Management page.
3. On the Job Management page, click Create to switch to the Create Job dialog box.
4. Specify job parameters as required.
   Figure 1 Creating a job
   

   Table 1 Parameters related to job creation

   Parameter	Description
   Type	Select Flink Streaming SQL Job. NOTE: The visual editor supports only Flink Streaming SQL Job.
   Name	Name of a job, which has 1 to 57 characters and only contains letters, digits, hyphens (-), and underlines (_). NOTE: The job name must be unique.
   Description	Description of a job. It contains 0 to 512 characters.
   Editor	Select Visual Editor. Options SQL Editor and Visual Editor are available.

5. Click OK to enter the Edit page.
6. Drag desired elements, such as DIS, Window, and DIS, to the canvas area.
   Figure 2 Dragging elements to the canvas area
   
   

   You can double-click an element to delete it.

7. Connect each element according to the business logic.

   Starting from the egress port of an element, drag on the canvas to the ingress port of another element. You cannot directly connect the egress port of a source element to the ingress port of the sink element. In normal cases, the ingress port of the desired element turns green, rather than remain unchanged.

   

   You can double-click a connection to delete it.

8. Configure the element parameters in the canvas area.
   1. Click the source element, for example, source_dis_1. In the displayed area at the right side, configure parameters related to the element, including parameters involved in Data Stream Attribute Settings and Element Parameter Settings.

      Table 2 Parameters to be configured when DIS serves as the source element

      Parameter	Description
      Data Stream Attribute Settings	Click Add Attribute, and specify Attribute Name and Attribute Type. Attribute Name starts with an English letter and only consists of English letters, digits, and underscores (_). A maximum of 20 characters are allowed. Supported attribute types include STRING, INT, BIGINT, BOOLEAN, DOUBLE, FLOAT, and TIMESTAMP. Click Insert Test Data to insert the test data of the attribute. Click Delete Test Data to delete the test data of the attribute. In the attribute list, click Delete in a row, where the attribute you want to delete resides, to delete the attribute.
      Element Parameter Settings
      Type	Element type. Available options include the following: source-dis source-obs source-cloudtable
      Region	Region where a user resides.
      DIS Stream	This parameter is valid only when DIS is selected under Source Element. Select a DIS stream.
      Partitions	This parameter is valid only when DIS is selected under Source Element. Partitions are the base throughput unit of a DIS stream. Each partition supports a read speed of up to 2 MB/s and a write speed of up to 1000 records/s and 1 MB/s.
      Encoding	This parameter is valid only when DIS is selected under Source Element. Data encoding mode, which can be CSV or JSON.
      Field Delimiter	This parameter is valid only when DIS is selected under Source Element and Encode is set to CSV or when OBS is selected under Source Element. This parameter indicates the delimiter between attributes. The default value is a comma (,).
      JSON config	This parameter is valid only when DIS is selected under Source Element and Encoding is set to JSON. Configure the mapping between the JSON field and the stream definition field, for example, attr1=student.name;attr2=student.age;.

      Table 3 Parameters to be configured when OBS serves as the source element

      Parameter	Description
      Data Stream Attribute Settings	Click Add Attribute, and specify Attribute Name and Attribute Type. Attribute Name starts with an English letter and only consists of English letters, digits, and underscores (_). A maximum of 20 characters are allowed. Supported attribute types include STRING, INT, BIGINT, BOOLEAN, DOUBLE, FLOAT, and TIMESTAMP. Click Insert Test Data to insert the test data of the attribute. Click Delete Test Data to delete the test data of the attribute. In the attribute list, click Delete in a row, where the attribute you want to delete resides, to delete the attribute.
      Element Parameter Settings
      Type	Element type. The options are as follows depending on various source elements: source-dis source-obs source-cloudtable
      Region	Region where a user resides.
      OBS Bucket	This parameter is valid only when OBS is selected under Source Element. Select the OBS bucket where the selected source element is located.
      Object Name	This parameter is valid only when OBS is selected under Source Element. Name of the object stored in the OBS bucket where source data is located.
      Row Delimiter	This parameter is valid only when OBS is selected under Source Element. Delimiter between rows, such as: "\n".
      Field Delimiter	This parameter indicates the delimiter between attributes. The default value is a comma (,).

      Table 4 Parameters to be configured when CloudTable serves as the source element

      Parameter	Description
      Data Stream Attribute Settings	Click Add Attribute, and specify Attribute Name and Attribute Type. Attribute Name starts with an English letter and only consists of English letters, digits, and underscores (_). A maximum of 20 characters are allowed. Supported attribute types include STRING, INT, BIGINT, BOOLEAN, DOUBLE, FLOAT, and TIMESTAMP. Click Insert Test Data to insert the test data of the attribute. Click Delete Test Data to delete the test data of the attribute. In the attribute list, click Delete in a row, where the attribute you want to delete resides, to delete the attribute.
      Element Parameter Settings
      Type	Element type. Available options include the following: source-dis source-obs source-cloudtable
      Region	Region where a user resides.
      Table Name	This parameter is valid only when CloudTable is selected under Source Element. Name of the data table to be read.
      Cluster ID	This parameter is valid only when CloudTable is selected under Source Element. ID of the cluster to which the data table to be read belongs.
      Table Columns	This parameter is valid only when CloudTable is selected under Source Element. This parameter value is in the format of rowKey,f1:c1,f1:c2,f2:c1. Ensure that the column quantity is the same as the number of attributes added in Data Stream Attribute Settings.

   2. Click an operator element, for example, operator_window_1. In the displayed area at the right side, configure parameters related to the element.
      The Window operator can be used for the following two time types: Event Time and Processing Time. For each time type, the following three window types are supported: tumbling window (TUMBLE), sliding window (HOP), and session window (SESSION). You can perform calculation, such as summing up and averaging, on the data in the window.

      Table 5 Window operator element parameter configuration

      Parameter	Description
      Source Attributes	Displays the data source, attribute name, and type that are specified in Source Element.
      Window Aggregation Parameter Configuration
      Time Type	The Window operator supports the following two time types: Event Time and Processing Time.
      Time Attribute	If Time Type is set to Event Time, this parameter indicates the user-provided event time, which is the data with Type of timestamp in Source Attributes. If Time Type is set to Processing Time, this parameter indicates the local system time proctime when events are handled.
      WaterMark	This parameter is valid only when Time Type is set to Event Time. If Time Type is set to Event Time, you must specify this parameter. This is because user data is usually disordered. If a watermark is not configured to properly delay user data, the data aggregation result may be greatly different from the expected. This parameter can be set to By time period or By number of events.
      Delay Period	Maximum delay time. The default value is 20 Seconds.
      Send Period	This parameter is valid only when WaterMark is set to By time period. Watermark sending interval. The default value is 10 Seconds.
      Event Number	This parameter is valid only when WaterMark is set to By number of events. Number of data packets, upon which the watermark is sent. The default value is 10.
      GroupBy
      Window Type	For each time type, the following three window types are supported: Tumbling window Sliding window Session window
      Window Period	The Window operator assigns each element to a window of a specified window size. The specified window size is the window period. The default window period is 1 day.
      Group Attribute	This parameter is optional. Grouping can be performed by time window or attribute. This parameter indicates the attribute specified in Source Element. Multiple attributes can be selected.
      Sliding Period	This parameter is valid only when Window Type is set to Hop Window. The sliding window has two parameters: size and slide. The size parameter is indicated by Window Period and refers to the window size. The slide parameter is indicated by Sliding Period and refers to each slide step. If the slide value is smaller than the size value, sliding windows can be overlapping. In this case, elements are assigned to multiple windows. If the slide value is equal to the size value, the window can be considered as a tumbling window. If the slide value is greater than the size value, the window is considered a jump window. In this case, windows are not overlapping and there are no gaps between windows.
      Select Attribute	Click Add Select Attribute, and specify Function Type and Type. Function Type can be set to Window, Aggregate, or No Function. Various window functions can be selected depending on your Window Type setting: If Window Type is set to Tumble Window, window functions TUMBLE_START and TUMBLE_END are available. If Window Type is set to Hop Window, window functions HOP_START and HOP_END are available. If Window Type is set to Session Window, window functions SESSION_START and SESSION_END are available. The aggregate functions of following types are supported: Count, AVG, SUM, MAX, and MIN. Type can be set to the following: STRING, INT, BIGINT, BOOLEAN, DOUBLE, FLOAT, and TIMESTAMP. Click to display the function parameter setting area and set parameters as required. Click Delete to delete the corresponding function type.

      The Select operator corresponds to the SQL statement Select used for selecting data from data streams. Attribute Name in Source Attributes must be the existing attribute name specified in the source element connected to the Select operator.

      Table 6 Select operator element parameter configuration

      Parameter	Description
      Source Attributes	Displays the data source, attribute name, and type that are specified in Source Element.
      Output Attributes	Click Add Attribute, and specify Select Field and Type. The Select operator is used to select the sink stream. Each output attribute can be: Input attribute of the data source Logical collection of data source attributes, such as addition or subtraction of attributes Function calculation on the source attribute Others Type can be set to the following: STRING, INT, BIGINT, BOOLEAN, DOUBLE, FLOAT, and TIMESTAMP. Click Delete to delete the corresponding attribute.

      The Filter operator corresponds to the SQL statement WHERE used for filtering data from data streams. The filter rules support arithmetic operators, relational operators, and logical operators.

      Table 7 Filter operator element parameter configuration

      Parameter	Description
      Source Attributes	Displays the data source, attribute name, and type that are specified in Source Element.
      Filter Rules	Click Add Rule to specify a filter rule. You can add multiple rules. Click Delete to delete the corresponding filter rule.
      Output Attributes	Displays the attribute name and type.

      The Union operator is used to combine multiple streams. Ensure that the streams have the same attribute, including the attribute type and attribute sequence. Specifically, the attribute in the same row of each source element must have the same Type setting.

      Table 8 Union operator element parameter configuration

      Parameter	Description
      Output Attributes	Displays the attribute name and type.

   3. Click a sink element, for example, sink_dis_1. In the displayed area at the right side, configure parameters related to the element, including parameters involved in Data Stream Attribute Settings and Element Parameter Settings.

      Table 9 Parameters to be configured when DIS serves as the sink element

      Parameter	Description
      Data Stream Attribute Settings	Click Add Attribute, and specify Attribute Name and Attribute Type. Attribute Name starts with an English letter and only consists of English letters, digits, and underscores (_). A maximum of 20 characters are allowed. Supported attribute types include STRING, INT, BIGINT, BOOLEAN, DOUBLE, FLOAT, and TIMESTAMP. In the attribute list, click Delete in a row, where the attribute you want to delete resides, to delete the attribute.
      Element Parameter Settings
      Type	Sink element type. The options are as follows depending on various sink elements: sink-dis sink-cloudtable sink-smn sink-rds
      Region	Region where a user resides.
      DIS Stream	This parameter is valid only when DIS is selected under Sink Element. Select a DIS stream.
      Partition Key	This parameter is valid only when DIS is selected under Sink Element. This parameter refers to the key used for data grouping when DIS serves as the sink stream. Key used for data grouping when there are multiple partitions in a DIS stream. Multiple keys are separated by using commas (,).
      Encoding	This parameter is valid only when DIS is selected under Sink Element. Data encoding mode, which can be CSV or JSON.
      Field Delimiter	This parameter is valid only when DIS is selected under Sink Element. This parameter indicates the delimiter between attributes. The default value is a comma (,).

      Table 10 Parameters to be configured when CloudTable serves as the sink element

      Parameter	Description
      Data Stream Attribute Settings	Click Add Attribute, and specify Attribute Name and Attribute Type. Attribute Name starts with an English letter and only consists of English letters, digits, and underscores (_). A maximum of 20 characters are allowed. Supported attribute types include STRING, INT, BIGINT, BOOLEAN, DOUBLE, FLOAT, and TIMESTAMP. In the attribute list, click Delete in a row, where the attribute you want to delete resides, to delete the attribute.
      Element Parameter Settings
      Type	Sink element type. The options are as follows depending on various sink elements: sink-dis sink-cloudtable sink-smn sink-rds
      Region	Region where a user resides.
      Table Name	This parameter is valid only when CloudTable is selected under Sink Element. Name of the data table to be read.
      Cluster ID	This parameter is valid only when CloudTable is selected under Sink Element. ID of the cluster to which the data table to be read belongs.
      Table Columns	This parameter is valid only when CloudTable is selected under Sink Element. The format is rowKey,f1:c1,f1:c2,f2:c1. The number of columns must be the same as the number of attributes specified in the source element.
      Abnormal Table	This parameter is valid only when CloudTable is selected under Sink Element. Table for dumping abnormal data. This table is used to store data that cannot be written into HBase according to specified configuration. If this field is specified, abnormal data will be written into the specified table. If unspecified, abnormal data will be abandoned.
      Empty Table	This parameter is valid only when CloudTable is selected under Sink Element. Whether to create a table if the target table or column family where data is to be written does not exist. The default value is FALSE.
      Data Records	This parameter is valid only when CloudTable is selected under Sink Element. Amount of data to be written in batches at a time. The value must be a positive integer. The upper limit is 100. The default value is 10.

      Table 11 Parameters to be configured when SMN serves as the sink element

      Parameter	Description
      Data Stream Attribute Settings	Click Add Attribute, and specify Attribute Name and Attribute Type. Attribute Name starts with an English letter and only consists of English letters, digits, and underscores (_). A maximum of 20 characters are allowed. Supported attribute types include STRING, INT, BIGINT, BOOLEAN, DOUBLE, FLOAT, and TIMESTAMP. In the attribute list, click Delete in a row, where the attribute you want to delete resides, to delete the attribute.
      Element Parameter Settings
      Type	Sink element type. The options are as follows depending on various sink elements: sink-dis sink-cloudtable sink-smn sink-rds
      Region	Region where a user resides.
      Topic URN	This parameter is valid only when SMN is selected under Sink Element. Topic URN.
      Message Subject	This parameter is valid only when SMN is selected under Sink Element. Title of the message sent to SMN.
      Column Name	This parameter is valid only when SMN is selected under Sink Element. Column name of the output stream whose content is the content of the message

      Table 12 Parameters to be configured when RDS serves as the sink element

      Parameter	Description
      Data Stream Attribute Settings	Click Add Attribute, and specify Attribute Name and Attribute Type. Attribute Name starts with an English letter and only consists of English letters, digits, and underscores (_). A maximum of 20 characters are allowed. Supported attribute types include STRING, INT, BIGINT, BOOLEAN, DOUBLE, FLOAT, and TIMESTAMP. In the attribute list, click Delete in a row, where the attribute you want to delete resides, to delete the attribute.
      Element Parameter Settings
      Type	Sink element type. The options are as follows depending on various sink elements: sink-dis sink-cloudtable sink-smn sink-rds
      Region	Region where a user resides.
      Username	This parameter is valid only when RDS is selected under Sink Element. Username root used for creating the RDS database instance.
      Password	This parameter is valid only when RDS is selected under Sink Element. Password that is specified during RDS database instance creation.
      DB URL	This parameter is valid only when RDS is selected under Sink Element. The parameter value is a combination of the private network IP address, port number, and DB name of the node where the DB is located. The format is as follows: mysql://192.168.0.12:8635/dbName
      Table Name	This parameter is valid only when RDS is selected under Sink Element. Name of the table created in the node where the DB is located.

9. (Optional) Click SQL Editor to convert the information in the visual editor into SQL statements.
10. (Optional) Click Save to save the job parameter settings.
11. If you need to run the job, click Submit.