Dumping Data to OBS

Task Name

Name of the dump task. The names of dump tasks created for the same stream must be unique. A dump task name is 1 to 64 characters long. Only letters, digits, hyphens (-), and underscores (_) are allowed.

-

Dump Bucket

Name of the OBS bucket used to store data from the DIS stream. The bucket name is created when you create a bucket in OBS.

-

File Directory

Directory created in OBS to store files from the DIS stream.

This directory name is 0 to 50 characters long.

By default, this parameter is left unspecified.

-

Time Directory Format

Data will be saved according to the time format in the file directory of the OBS bucket.

For example, if the time directory is accurate to day, the save path will be in the format of bucket name/file directory/year/month/day.

You can only select but not enter a value in this field.

-

Record Delimiter

Delimiter used to separate different dump records.

You can only select but not enter a value in this field.

-

Offset

• Latest: Maximum offset, indicating that the latest data will be read.
• Earliest: Minimum offset, indicating that the earliest data will be read.

Latest

Dump Interval (s)

Interval at which data from the DIS stream will be imported into dump destination, such as OBS, MRS, DLI, and DWS. If no data was pushed to the DIS stream during the time specified here, the dump file will not be generated.

Value range: 30s to 900s

Unit: second

Default value: 300s

-

Task Name

Name of the dump task. The names of dump tasks created for the same stream must be unique. A dump task name is 1 to 64 characters long. Only letters, digits, hyphens (-), and underscores (_) are allowed.

-

Dump Bucket

Name of the OBS bucket used to store data from the DIS stream. The bucket name is created when you create a bucket in OBS.

-

File Directory

Directory created in OBS to store files from the DIS stream.

This directory name is 0 to 50 characters long.

By default, this parameter is left unspecified.

-

Time Directory Format

Data will be saved according to the time format in the file directory of the OBS bucket.

For example, if the time directory is accurate to day, the save path will be in the format of bucket name/file directory/year/month/day.

You can only select but not enter a value in this field.

-

Offset

• Latest: Maximum offset, indicating that the latest data will be read.
• Earliest: Minimum offset, indicating that the earliest data will be read.

Latest

Dump Interval (s)

User-defined interval at which data is imported from the current DIS stream into the target place. If no data is pushed to the DIS stream during the current interval, no dump file package will be generated.

Value range: 30s to 900s

Unit: second

Default value: 300s

-

Source Data Schema

JSON or CSV data example, used to describe the JSON or CSV data format. DIS can generate an Avro schema based on the JSON or CSV data sample and convert the uploaded JSON or CSV data to the Parquet format.

-

File Directory

Directory created in OBS to store files from the DIS stream.

This directory name is 0 to 100 characters long.

By default, this parameter is left unspecified.

NOTE:

When the source data type is JSON, EL and built-in functions are supported.

• EL example

Source data:

{"name":"Andy","city":"","time":1556323141582}

The dump file directory is configured as follows:

dis/basePath/app_key_p=${name}

The final directory structure is as follows:

{$Bucket name}/dis/basePath/app_key_p=Andy

• Example of built-in functions

Source data:

{"name":"Andy","city":"","time":1556323141582}

The dump file directory is configured as follows:

dis/basePath/date_p=toDate(${time}, "yyMMdd")

The final directory structure is as follows:

{$Bucket name}/dis-basePath/date_p=20190427

Supported built-in functions

toDate(timestamp, format): Convert the time stamp to a specified time format. For example: toDate (1556323141582, 'yymmdd')

Custom Time Directory

You can click or to disable or enable the custom time directory.

• If the custom timestamp is disabled, the directory where the object file written to OBS resides is named after the creation time of the dump file.

  For example, if a dump file was created on October 16, 2018 and the time directory format is accurate to day, the file will be saved in OBS bucket name/dump file directory/2018/10/16.

• If the custom timestamp is disabled, the directory where the object file written to OBS resides is named in the time format specified in the source data.

  For example, if a dump file was created on October 16, 2018, the time directory format is accurate to day, and the time format specified in the data source is 2017/09/08 11:01:01, the file will be saved in OBS bucket name/dump file directory/2017/09/08. The storage directory is defined based on the time format defined in the source data instead of the time when the dump file is created.

• Example 1: Dump simple JSON data.

Source data:

{ "id":"1", "date":"2018/10/16 11:00:05"}

The configuration is as follows:

Set the timestamp attribute name to date, data type to String, and timestamp format to yyyy/MM/dd HH:mm:ss based on the source data type of the data to be dumped.

After the data is dumped successfully, the storage directory structure depends on the source data timestamp and the time directory format. In this example, the time directory format is accurate to day. Therefore, the final data storage directory is OBS bucket name/dump file directory/2018/10/16.

• Example 2: Dump multiply layers of nested JSON data.

  Source data:

  { "id":"1", "detail":{ "detID":"05790110000000000103#567fd3cb13a4493eaa43076953253eed", "endTime":"2018/10/07 13:26:35" }}

The configuration is as follows:

Set the timestamp attribute name to detail.endTime, data type to String, and timestamp format to yyyy/MM/dd HH:mm:ss based on the source data type of the data to be dumped.

After the data is dumped successfully, the storage directory structure depends on the source data timestamp and the time directory format. In this example, the time directory format is accurate to day. Therefore, the final data storage directory is OBS bucket name/dump file directory/2018/10/07.

• Example 3: Dump CSV data.

Source data:

a,2010-10-12 11:00:00,b,2011-10-12 11:00:10

The configuration is as follows:

Select timestamp 2010-10-12 11:00:00 based on the source data to be dumped. After data is converted to the Parquet format, the corresponding attribute field name is field_1. When creating a dump task, set the timestamp attribute to field_1, data type to String, and timestamp format to yyyy/MM/dd HH:mm:ss.

After the data is dumped successfully, the storage directory structure depends on the source data timestamp and the time directory format. In this example, the time directory format is accurate to day. Therefore, the final data storage directory is OBS bucket name/dump file directory/2010/10/12.

Source Data Timestamp

• Attribute name of the timestamp.
  NOTE:

  Enter the field name corresponding to the timestamp defined in the source data to be uploaded.

• Timestamp format. Possible values can be:

  yyyy/MM/dd HH:mm:ss

  MM/dd/yyyy HH:mm:ss

  dd/MM/yyyy HH:mm:ss

  yyyy-MM-dd HH:mm:ss

  MM-dd-yyyy HH:mm:ss

  dd-MM-yyyy HH:mm:ss

• Data type. Possible values can be:
  • String
  • Timestamp
    NOTE:

    If the type of the source data to be uploaded is Timestamp, the value must be accurate to milliseconds.

Source Data Schema

JSON or CSV data sample, used to describe the JSON or CSV data format. DIS can generate an Avro schema based on the JSON or CSV data sample and convert the uploaded JSON or CSV data to the CarbonData format.

-

CarbonData Retrieval Attribute

Attribute of the carbon table, used to create a carbon writer.

The following keys are supported:

• table_blocksize: Size of a table block. The value ranges from 1 MB to 2048 MB. The default value is 1024 MB.
• table_blocklet_size: Size of the blocklet in a file. The default value is 64 MB.
• local_dictionary_enable: Possible values can be true or false. The default value is false.
• sort_columns: Specifies the index column. Multi-level index columns are separated by commas (,).
• sort_scope: Specifies the scope where data is sorted during loading. Currently, the following types are supported:
  • local_sort: Default value, indicating that data is sorted in a node.
  • no_sort: Data is not sorted. It is used when data needs to be saved to a database quickly. After the data is saved to the database, you can use the Compaction command to create an index when the system is idle.
  • batch_sort: A CarbonData file is generated after the memory is sorted in a node and no full sorting is performed on the node. This configuration improves the loading speed, but the query performance is inferior to that of LOCAL_SORT.

-