DataGen Source Table

Function

DataGen is used to generate random data for debugging and testing.

Prerequisites

None

Precautions

• When you create a DataGen table, the table field type cannot be Array, Map, or Row. You can use COMPUTED COLUMN in CREATE TABLE to construct similar functions.
• When creating a Flink OpenSource SQL job, you need to set Flink Version to 1.12 on the Running Parameters tab of the job editing page, select Save Job Log, and set the OBS bucket for saving job logs.

Syntax

create table dataGenSource(
  attr_name attr_type 
  (',' attr_name attr_type)* 
  (',' WATERMARK FOR rowtime_column_name AS watermark-strategy_expression)
)
with (
  'connector' = 'datagen'
);

Parameters

Example

Create a Flink OpenSource SQL job. Run the following script to generate random data through the DataGen table and output the data to the Print result table.

When you create a job, set Flink Version to 1.12 on the Running Parameters tab. Select Save Job Log, and specify the OBS bucket for saving job logs.

create table dataGenSource(
  user_id string,
  amount int
) with (
  'connector' = 'datagen',
  'rows-per-second' = '1', --Generates a piece of data per second.
  'fields.user_id.kind' = 'random', --Specifies a random generator for the user_id field.
  'fields.user_id.length' = '3' --Limits the length of user_id to 3.
);

create table printSink(
  user_id string,
  amount int
) with (
  'connector' = 'print'
);

insert into printSink select * from dataGenSource;

After the job is submitted, the job status changes to Running. You can perform the following operations to view the output result:

• Method 1:
  1. Log in to the DLI console. In the navigation pane, choose Job Management > Flink Jobs.
  2. Locate the row that contains the target Flink job, and choose More > FlinkUI in the Operation column.
  3. On the Flink UI, choose Task Managers, click the task name, and select Stdout to view job logs.
• Method 2: If you select Save Job Log on the Running Parameters tab before submitting the job, perform the following operations:
  1. Log in to the DLI console. In the navigation pane, choose Job Management > Flink Jobs.
  2. Click the name of the corresponding Flink job, choose Run Log, click OBS Bucket, and locate the folder of the log you want to view according to the date.
  3. Go to the folder of the date, find the folder whose name contains taskmanager, download the taskmanager.out file, and view result logs.