Dumping Data to MRS

Task Name

Name of the dump task. The names of dump tasks created for the same stream must be unique. A dump task name is 1 to 64 characters long. Only letters, digits, hyphens (-), and underscores (_) are allowed.

-

MRS Cluster

Click Select. In the Select MRS Cluster dialog box, select an MRS cluster. Data is dumped only to an MRS cluster that is not authenticated by Kerberos.

You can only select but not enter a value in this field.

-

HDFS Path

Click Select. In the Select HDFS Path dialog box, select an HDFS path.

You can only select but not enter a value in this field.

This parameter is available only after you select an MRS cluster.

File Directory

Directory created in MRS to store files from the DIS stream.

This directory name is 0 to 50 characters long.

By default, this parameter is left unspecified.

-

Offset

• Latest: Maximum offset, indicating that the latest data will be read.
• Earliest: Minimum offset, indicating that the earliest data will be read.

Latest

Dump Interval (s)

Interval at which data from the DIS stream will be imported into dump destination, such as OBS, MRS, DLI, and DWS. If no data was pushed to the DIS stream during the time specified here, the dump file will not be generated.

Value range: 30s to 900s

Unit: second

Default value: 300s

-

Temporary Bucket

OBS bucket in which a directory is created for temporarily storing user data. The data in the directory is deleted after being dumped to a specific destination.

-

Temporary Directory

Directory in the chosen Temporary Bucket for temporarily storing data. The data in the directory is deleted after being dumped to a specific destination.

If this field is left blank, the data is stored directly to the Temporary Bucket.

-

Source Data Schema

JSON or CSV data example, used to describe the JSON or CSV data format. DIS can generate an Avro schema based on the JSON or CSV data sample and convert the uploaded JSON or CSV data to the Parquet format.

-

Source Data Schema

JSON or CSV data example, used to describe the JSON or CSV data format. DIS can generate an Avro schema based on the JSON or CSV data sample and convert the uploaded JSON or CSV data to the CarbonData format.

-

CarbonData Retrieval Attribute

Attribute of the carbon table, used to create a carbon writer.

The following keys are supported:

• table_blocksize: Size of a table block. The value ranges from 1 MB to 2048 MB. The default value is 1024 MB.
• table_blocklet_size: Size of the blocklet in a file. The default value is 64 MB.
• local_dictionary_enable: Possible values can be true or false. The default value is false.
• sort_columns: Specifies the index column. Multi-level index columns are separated by commas (,).
• sort_scope: Specifies the scope where data is sorted during loading. Currently, the following types are supported:
  • local_sort: Default value, indicating that data is sorted in a node.
  • no_sort: Data is not sorted. It is used when data needs to be saved to a database quickly. After the data is saved to the database, you can use the Compaction command to create an index when the system is idle.
  • batch_sort: A CarbonData file is generated after the memory is sorted in a node and no full sorting is performed on the node. This configuration improves the loading speed, but the query performance is inferior to that of LOCAL_SORT.

-