Adding a Dump Task_Data Ingestion Service

Function

This API is used to add a dump task.

When adding a dump task, you need to specify the stream name.

URI

URI format
POST /v2/{project_id}/streams/{stream_name}/transfer-tasks
Parameter description
None

Request

Example request

Add an OBS dump task. The dump file format is Text.

POST https://{endpoint}/v2/{project_id}/streams/{stream_name}/transfer-tasks
{
	"destination_type": "OBS",
	"obs_destination_descriptor": {
		"task_name": "task_DtDO",
		"consumer_strategy": "LATEST",
		"agency_name": "dis-global",
		"destination_file_type": "text",
		"obs_bucket_path": "111111111",
		"file_prefix": "0823",
		"partition_format": "yyyy/MM/dd/HH/mm",
		"record_delimiter": "|",
		"deliver_time_interval": 30
	}
}

Add an OBS dump task. The dump file format is Parquet.

POST https://{endpoint}/v2/{project_id}/streams/{stream_name}/transfer-tasks
{
	"destination_type": "OBS",
	"obs_destination_descriptor": {
		"task_name": "task_DtDO",
		"consumer_strategy": "LATEST",
		"agency_name": "dis-global",
		"destination_file_type": "parquet",
		"obs_bucket_path": "111111111",
		"file_prefix": "0823",
		"partition_format": "yyyy/MM/dd/HH/mm",
		"record_delimiter": "|",
		"deliver_time_interval": 30,
		"processing_schema": {
			"timestamp_name": "date",
			"timestamp_type": "String",
			"timestamp_format": "yyyy/MM/dd HH:mm:ss"
		}
	}
}

Add an OBS dump task. The dump file format is CarbonData.

POST https://{endpoint}/v2/{project_id}/streams/{stream_name}/transfer-tasks
     
{
  "destination_type": "OBS",
   "obs_destination_descriptor":
     { 
        "task_name":"task_DtDO",
         "consumer_strategy":"LATEST",
         "agency_name":"dis-global",
"destination_file_type":"CarbonData",
         "obs_bucket_path":"111111111",
         "file_prefix":"0823",
         "partition_format":"yyyy/MM/dd/HH/mm",
         "record_delimiter":"|",
         "deliver_time_interval":30
     } 
 }

Add an MRS dump task. The dump file format is Text.

POST https://{endpoint}/v2/{project_id}/streams/{stream_name}/transfer-tasks
         
 { 
  "destination_type": "MRS",
   "mrs_destination_descriptor":{
        "task_name":"task_8CWn",
        "consumer_strategy":"TRIM_HORIZON",
        "agency_name":"dis-global",
        "destination_file_type":"text",
        "mrs_cluster_id":"f8123fa6-99f1-4ed9-83f4-c827c7277d41",
        "mrs_cluster_name":"mrs_jxZP",
        "mrs_hdfs_path":"/user",
        "obs_bucket_path":"111111111",
        "file_prefix":"test",
        "hdfs_prefix_folder":"test",
        "deliver_time_interval":30,
        "retry_duration":1800
    }
}

Add an MRS dump task. The dump file format is Parquet.

POST https://{endpoint}/v2/{project_id}/streams/{stream_name}/transfer-tasks  
           
 { 
  "destination_type": "MRS",
   "mrs_destination_descriptor":{
        "task_name":"task_JvzV",
        "consumer_strategy":"LATEST",
        "agency_name":"dis-global",
        "destination_file_type":"parquet",
        "mrs_cluster_id":"13cfefa4-a410-4717-91ba-1b694ec92da7",
        "mrs_cluster_name":"mrs_gkFP_172",
        "mrs_hdfs_path":"/user",
        "obs_bucket_path":"111111111",
        "file_prefix":"test",
        "hdfs_prefix_folder":"test",
        "deliver_time_interval":30,
        "retry_duration":1800
    }
}

Add an MRS dump task. The dump file format is CarbonData.

POST https://{endpoint}/v2/{project_id}/streams/{stream_name}/transfer-tasks 
          
 { 
  "destination_type": "MRS",
   "mrs_destination_descriptor":{
        "task_name":"task_w6WG",
        "consumer_strategy":"TRIM_HORIZON",
        "agency_name":"dis-global",
"destination_file_type":"CarbonData",
        "carbon_properties":null,
        "mrs_cluster_id":"13cfefa4-a410-4717-91ba-1b694ec92da7",
        "mrs_cluster_name":"mrs_gkFP_172",
        "mrs_hdfs_path":"/user",
        "obs_bucket_path":"111111111",
        "file_prefix":"test",
        "hdfs_prefix_folder":"test",
        "deliver_time_interval":30,
        "retry_duration":1800
    }
}

Add a DLI dump task.

POST https://{endpoint}/v2/{project_id}/streams/{stream_name}/transfer-tasks
{
	"destination_type": "DLI",
	"dli_destination_descriptor": {
		"task_name": "task",
		"consumer_strategy": "LATEST",
		"agency_name": "dli-agency",
		"dli_database_name": "db",
		"dli_table_name": "table",
		"obs_bucket_path": "1111",
		"file_prefix": "2018",
		"deliver_time_interval": 300,
		"retry_duration ": 300
	}
}

Add a CloudTable HBase dump task.

POST https://{endpoint}/v2/{project_id}/streams/{stream_name}/transfer-tasks
{
	"destination_type": "CLOUDTABLE",
	"cloudtable_destination_descriptor": {
		"task_name": "task",
		"consumer_strategy": "TRIM_HORIZON",
		"cloudtable_cluster_name": "cloudtable_cluster",
		"cloudtable_cluster_id": "b8c095e2-db5f-4732-8a1d-eacd662e35dc",
		"cloudtable_table_name": "cloudtable_table",
		"cloudtable_row_key_delimiter": "|",
		"retry_duration": 1800,
		"obs_backup_bucket_path": "obs-test-hz",
		"backup_file_prefix": "2018",
		"cloudtable_schema": {
			"row_key": [{
				"value": "dataId",
				"type": "String"
			}],
			"columns": [{
				"column_family_name": "cfname1",
				"column_name": "ID",
				"value": "dataId",
				"type": "String"
			}, {
				"column_family_name": "cfname2",
				"column_name": "VALUE",
				"value": "dataValue",
				"type": "String"
			}]
		}
	}
}

Add a CloudTable OpenTSDB dump task.

{
	"destination_type": "CLOUDTABLE",
	"cloudtable_destination_descriptor": {
		"task_name": "task",
		"consumer_strategy": "LATEST",
		"cloudtable_cluster_name": "cloudtable_cluster",
		"cloudtable_cluster_id": "b8c095e2-db5f-4732-8a1d-eacd662e35dc",
		"retry_duration": 1800,
		"obs_backup_bucket_path": "obs-test-hz",
		"backup_file_prefix": "2018",
		"opentsdb_schema": [{
			"metric": [{
				"type": "Constant",
				"value": "age"
			}],
			"timestamp": {
				"value": "date",
				"type": "String",
				"format": "yyyy/MM/dd HH:mm:ss"
			},
			"value": {
				"value": "value",
				"type": "Bigint"
			},
			"tags": [{
				"name": "name",
				"value": "name",
				"type": "Bigint"
			}]
		}]
	}
}

Add a DWS dump task.

{
	"destination_type": "DWS",
	　　"dws_destination_descriptor": 　　　　{
		"task_name": "task",
		"consumer_strategy": "LATEST",
		　　　　　　"agency_name": "dis-global",
		　　　　　　"dws_cluster_name": "dws-Shang",
		　　　　　　"dws_cluster_id": "ea4a0a58-7b02-4d56-a3ec-ed67b498f8b9",
		　　　　　　"dws_database_name": "postgres",
		　　　　　　"dws_schema": "dbadmin",
		　　　　　　"dws_table_name": "dws_table",
		　　　　　　"dws_delimiter": "|",
		　　　　　　"user_name": "dbadmin",
		　　　　　　"user_password": "password",
		　　　　　　"kms_user_key_name": "KMS-key",
		　　　　　　"kms_user_key_id": "2260a755-c256-4211-aace-3c4358099b08",
		　　　　　　"obs_bucket_path": "dws-test",
		　　　　　　"file_prefix": "a",
		　　　　　　"deliver_time_interval": 300,
		　　　　　　"retry_duration": 1800,
		"dws_table_columns": "dev,id,online,module,appId,logTime,evvvv",
		"options": {
			"fill_missing_fields": "false",
			"ignore_extra_data": "false",
			"compatible_illegal_chars": "false"
		}
	}　　
}

**Table 1** Parameter description
Parameter	Mandatory	Type	Description
destination_type	No	String	Dump destination. Possible values: OBS MRS DLI CLOUDTABLE DWS
obs_destination_descriptor	No	Object	Parameter list of the OBS to which data in the DIS stream will be dumped. Data in a DIS stream cannot be dumped to multiple destinations. This parameter is left unspecified by default. If this parameter is left unspecified, data is not dumped to OBS.
mrs_destination_descriptor	No	Object	Parameter list of the MRS to which data in the DIS stream will be dumped. Data in a DIS stream cannot be dumped to multiple destinations. This parameter is left unspecified by default. If this parameter is left unspecified, data is not dumped to MRS.
dli_destination_descriptor	No	Object	Parameter list of the DLI to which data in the DIS stream will be dumped. Data in a DIS stream cannot be dumped to multiple destinations. This parameter is left unspecified by default. If this parameter is left unspecified, data is not dumped to DLI.
cloudtable_destination_descriptor	No	Object	Parameter list of the CloudTable to which data in the DIS stream will be dumped. Data in a DIS stream cannot be dumped to multiple destinations. This parameter is left unspecified by default. If this parameter is left unspecified, data is not dumped to CloudTable HBase or OpenTSDB.
dws_destination_descriptor	No	Object	Parameter list of the DWS to which data in the DIS stream will be dumped. Data in a DIS stream cannot be dumped to multiple destinations. This parameter is left unspecified by default. If this parameter is left unspecified, data is not dumped to DWS.

**Table 2** **obs_destination_descriptor** parameter description
Parameter	Mandatory	Type	Description
task_name	Yes	String	Name of the dump task. A task name is 1 to 64 characters long. Only letters, digits, hyphens (-), and underscores (_) are allowed.
destination_file_type	No	String	Dump file format. Possible values: Text Parquet CarbonData NOTE: The Parquet or CarbonData format can be selected only when Source Data Type is set to JSON and Dump Destination is set to OBS. Default value: text
agency_name	Yes	String	Name of the agency created in IAM. DIS uses an agency to access your specified resources. Agency parameter settings: Agency Type: Cloud service Cloud Service: DIS Validity Period: Permanent Set Policy to Tenant Administrator on the OBS project in the Global service region. This parameter cannot be left unspecified and the parameter value cannot exceed 64 characters.
obs_bucket_path	Yes	String	Name of the OBS bucket used to store data from the DIS stream.
file_prefix	No	String	Directory to store files that will be dumped to OBS. Different directory levels are separated by slashes (/) and cannot start with slashes. NOTE: This parameter is optional if streaming data is dumped to OBS. The value cannot exceed 50 characters. Only letters, digits, underscores (_), and slashes (/) are allowed. This parameter is left unspecified by default.
consumer_strategy	No	String	Specifies the offset. Latest: Maximum offset, indicating that the latest data will be extracted. TRIM_HORIZON: Minimum offset, indicating that the earliest data will be extracted. Default value: Latest
deliver_time_interval	Yes	Int	User-defined interval at which data is imported from the current DIS stream into OBS. If no data is pushed to the DIS stream during the current interval, no dump file package will be generated. NOTE: This parameter is mandatory if streaming data is dumped to OBS. Value range: 30s to 900s Default value: 300 Unit: second
partition_format	No	String	Directory structure of the object file written into OBS. The directory structure is in the format of yyyy/MM/dd/HH/mm (time at which the dump task was created). N/A: If this parameter is left unspecified, the time directory format will not be used. yyyy: year. yyyy/MM: year and month. yyyy/MM/dd: year, month, and day. yyyy/MM/dd/HH: year, month, day, and hour. yyyy/MM/dd/HH/mm: year, month, day, hour, and minute. For example, if the dump task was created at 14:49:00 on November 10, 2017, then the directory structure is 2017/11/10/14/49. 2017 is the level-1 directory. This parameter is left unspecified by default. NOTE: After the data is dumped successfully, the directory structure is obs_bucket_path/file_prefix/partition_format.
processing_schema	No	Object	Directory structure of the object file written into OBS. The directory structure is in the format of yyyy/MM/dd/HH/mm and generated based on the source data timestamp and the configured partition_format. NOTE: This parameter is mandatory if you want to customize the OBS directory to which the parquet file will be dumped.
record_delimiter	No	String	Delimiter for the dump file, which is used to separate the user data that is written into the dump file. Possible values: Comma (,) Semicolon (;) Vertical bar (\|) Newline (\n) NULL Default value: \n

**Table 3** **processing_schema** parameter description
Parameter	Mandatory	Type	Description
timestamp_name	Yes	String	Attribute name of the source data timestamp.
timestamp_type	Yes	String	Type of the source data timestamp. String Timestamp: 13-bit timestamp of the long type
timestamp_format	No	String	OBS directory generated based on the timestamp format. This parameter is mandatory when the timestamp type of the source data is String. Possible values: yyyy/MM/dd HH:mm:ss MM/dd/yyyy HH:mm:ss dd/MM/yyyy HH:mm:ss yyyy-MM-dd HH:mm:ss MM-dd-yyyy HH:mm:ss dd-MM-yyyy HH:mm:ss

The data dumped to OBS is the raw data uploaded by users, that is, the data decoded using Base64.

**Table 4** **mrs_destination_descriptor** parameter description
Parameter	Mandatory	Type	Description
agency_name	Yes	String	Name of the agency created in IAM. DIS uses an agency to access your specified resources. Agency parameter settings: Agency Type: Cloud service Cloud Service: DIS Validity Period: Permanent Policy NOTE: Both of the following permissions need to be configured. Set Policy to Tenant Administrator on the OBS project in the Global service region. Set Policy to MRS Administrator, Server Administrator, and Tenant Guest on the project in the region to which the chosen MRS cluster belongs. This parameter cannot be left unspecified and the parameter value cannot exceed 64 characters.
mrs_cluster_name	Yes	String	Name of the MRS cluster to which data in the DIS stream will be dumped. NOTE: Only the MRS cluster authenticated not by Kerberos is supported.
mrs_cluster_id	Yes	String	ID of the MRS cluster to which data in the DIS stream will be dumped.
mrs_hdfs_path	Yes	String	Hadoop Distributed File System (HDFS) path of the MRS cluster to which data in the DIS stream will be dumped.
obs_bucket_path	Yes	String	Name of the OBS bucket to which data in the DIS stream will be temporarily saved.
file_prefix	No	String	Self-defined directory created in the OBS bucket and used to temporarily store data in the DIS stream. Directory levels are separated by slashes (/) and cannot start with slashes. The value cannot exceed 50 characters. Only letters, digits, underscores (_), and slashes (/) are allowed. This parameter is left unspecified by default.
hdfs_prefix_folder	Yes	string	Directory to store files that will be dumped to HDFS of the chosen MRS cluster. Different directory levels are separated by slashes (/). This directory name is 0 to 50 characters long. This parameter is left unspecified by default.
consumer_strategy	No	String	Specifies the offset. Latest: Maximum offset, indicating that the latest data will be extracted. TRIM_HORIZON: Minimum offset, indicating that the earliest data will be extracted. Default value: Latest
deliver_time_interval	Yes	Int	User-defined interval at which data is imported from the DIS stream into HDFS of the chosen MRS cluster. If no data is pushed to the DIS stream during the current interval, no dump file package will be generated. Value range: 30s to 900s Unit: second Default value: 300
retry_duration	No	Int	Time duration for DIS to retry if data fails to be dumped. If the duration is exceeded but the dump still fails, the data will be backed up to OBS bucket name/file_prefix/mrs_error. Value range: 0s to 7200s Unit: second Default value: 1800 If the value is set to 0, no retry is allowed.

**Table 5** **dli_destination_descriptor** parameter description
Parameter	Mandatory	Type	Description
agency_name	Yes	String	Name of the agency created in IAM. DIS uses an agency to access your specified resources. Agency parameter settings: Agency Type: Cloud service Cloud Service: DIS Validity Period: Permanent Policy NOTE: Both of the following permissions need to be configured. Set Policy to Tenant Administrator on the OBS project in the Global service region. Set Policy to the DLI Service User permission on the project in the region to which the chosen DLI table belongs. This parameter cannot be left unspecified and the parameter value cannot exceed 64 characters.
dli_database_name	Yes	String	Name of the DLI database to which data in the DIS stream will be dumped.
dli_table_name	Yes	String	Name of the DLI table to which data in the DIS stream will be dumped. NOTE: Currently, only DLI tables are supported. Before selecting a DLI table, ensure that you have the permission to insert a DLI table.
consumer_strategy	No	String	Specifies the offset. Latest: Maximum offset, indicating that the latest data will be extracted. TRIM_HORIZON: Minimum offset, indicating that the earliest data will be extracted. Default value: Latest
obs_bucket_path	Yes	String	Name of the OBS bucket to which data in the DIS stream will be temporarily saved.
file_prefix	No	String	Self-defined directory created in the OBS bucket and used to temporarily store data in the DIS stream. Directory levels are separated by slashes (/) and cannot start with slashes. The value cannot exceed 50 characters. Only letters, digits, underscores (_), and slashes (/) are allowed. This parameter is left unspecified by default.
deliver_time_interval	Yes	Int	User-defined interval at which data is imported from the DIS stream into DLI. If no data is pushed to the DIS stream during the current interval, no dump file package will be generated. Value range: 30s to 900s Unit: second Default value: 300
retry_duration	No	Int	Time duration for DIS to retry if data fails to be dumped to DLI. If the duration is exceeded but the dump still fails, the data will be backed up to OBS bucket name/file_prefix/dli_error. Value range: 0s to 7200s Unit: second Default value: 1800 If the value is set to 0, no retry is allowed.

**Table 6** **cloudtable_destination_descriptor** parameter description
Parameter	Mandatory	Type	Description
consumer_strategy	No	String	Specifies the offset. Latest: Maximum offset, indicating that the latest data will be extracted. TRIM_HORIZON: Minimum offset, indicating that the earliest data will be extracted. Default value: Latest
cloudtable_cluster_name	Yes	String	Name of the CloudTable cluster to which data will be dumped. If data will be dumped to OpenTSDB, OpenTSDB must be enabled before the dump.
cloudtable_cluster_id	Yes	String	ID of the CloudTable cluster to which data will be dumped. If data will be dumped to OpenTSDB, OpenTSDB must be enabled before the dump.
cloudtable_table_name	No	String	HBase table name of the CloudTable cluster to which data will be dumped. The parameter is mandatory when data is dumped to the CloudTable HBase.
cloudtable_schema	No	Object	Schema configuration of the CloudTable HBase data. You can set either this parameter or opentsdb_schema, but this parameter is mandatory when data will be dumped to HBase. After this parameter is set, the JSON data in the stream can be converted to another format and then be imported to the CloudTable HBase.
cloudtable_row_key_delimiter	No	String	Delimiter used to separate the user data that generates HBase row keys. Possible values: , . \| ; \ - _ ~ Default value: .
opentsdb_schema	No	List<Object>	Schema configuration of the CloudTable OpenTSDB data. You can set either this parameter or cloudtable_schema, but this parameter is mandatory when data will be dumped to OpenTSDB. After this parameter is set, the JSON data in the stream can be converted to another format and then be imported to the CloudTable OpenTSDB.
obs_backup_bucket_path	No	String	Name of the OBS bucket used to back up data that failed to be dumped to CloudTable.
backup_file_prefix	No	String	Self-defined directory created in the OBS bucket and used to back up data that failed to be dumped to CloudTable. Directory levels are separated by slashes (/) and cannot start with slashes. A directory name cannot exceed 50 characters. Only letters, digits, and underscores (_) are allowed. This parameter is left unspecified by default.
retry_duration	No	Int	Time duration for DIS to retry if data fails to be dumped to CloudTable. If the duration is exceeded but the dump still fails, the data will be backed up to OBS bucket name/backup_file_prefix/cloudtable_error or OBS bucket name/backup_file_prefix/opentsdb_error. Value range: 0s to 7200s Unit: second Default value: 1800

**Table 7** **cloudtable_schema** parameter description
Parameter	Mandatory	Type	Description
row_key	Yes	List<Object>	HBase rowkey schema used by the CloudTable cluster to convert JSON data into HBase rowkeys. Value range: 1 to 64
columns	Yes	List<Object>	HBase column schema used by the CloudTable cluster to convert JSON data into HBase columns. Value range: 1 to 4096

**Table 8** **row_key** parameter description
Parameter	Mandatory	Type	Description
value	Yes	String	JSON attribute name, which is used to generate HBase rowkeys for JSON data in the DIS stream.
type	Yes	String	JSON attribute type of JSON data in the DIS stream. Possible values: Bigint Double Boolean Timestamp String Decimal

**Table 9** **columns** parameter description
Parameter	Mandatory	Type	Description
column_family_name	Yes	String	Name of the HBase column family to which data will be dumped.
column_name	Yes	String	Name of the HBase column to which data will be dumped. This value is 1 to 32 characters long. Only letters, digits, and underscores (_) are allowed.
value	Yes	String	JSON attribute name, which is used to generate HBase column values for JSON data in the DIS stream.
type	Yes	String	JSON attribute type of JSON data in the DIS stream. Possible values: Bigint Double Boolean Timestamp String Decimal

**Table 10** **opentsdb_schema** parameter description
Parameter	Mandatory	Type	Description
metric	Yes	List<Object>	Schema configuration of the OpenTSDB data metric in the CloudTable cluster. After this parameter is set, the JSON data in the stream can be converted to the metric of the OpenTSDB data. Value: 1
timestamp	Yes	Object	Schema configuration of the OpenTSDB data timestamp in the CloudTable cluster. After this parameter is set, the JSON data in the stream can be converted to the timestamp of the OpenTSDB data.
value	Yes	Object	Schema configuration of the OpenTSDB data value in the CloudTable cluster. After this parameter is set, the JSON data in the stream can be converted to the value of the OpenTSDB data.
tags	Yes	List<Object>	Schema configuration of the OpenTSDB data tags in the CloudTable cluster. After this parameter is set, the JSON data in the stream can be converted to the tags of the OpenTSDB data. Value range: 1 to 8

**Table 11** **metric** parameter description
Parameter	Mandatory	Type	Description
type	Yes	String	When type is set to Constant, the value of metric is the value of value. When value is set to String, the value of metric is the value of the JSON attribute of the user data in the stream.
value	Yes	String	Constant value or JSON attribute name of the user data in the stream. This value is 1 to 32 characters long. Only letters, digits, and periods (.) are allowed.

**Table 12** **timestamp** parameter description
Parameter	Mandatory	Type	Description
type	Yes	String	When type is set to Timestamp, the value type of the JSON attribute of the user data in the stream is Timestamp, and the timestamp of OpenTSDB can be generated without converting the data format. When type is set to String, the value type of the JSON attribute of the user data in the stream is Date, and the timestamp of OpenTSDB can be generated only after the data format is converted.
value	Yes	String	JSON attribute name of the user data in the stream. This value is 1 to 32 characters long. Only letters, digits, and underscores (_) are allowed.
format	No	String	This parameter is mandatory when type is set to String. When the value type of the JSON attribute of the user data in the stream is Date, format is required to convert the data format to generate the timestamp of OpenTSDB. Possible values: yyyy/MM/dd HH:mm:ss MM/dd/yyyy HH:mm:ss dd/MM/yyyy HH:mm:ss yyyy-MM-dd HH:mm:ss MM-dd-yyyy HH:mm:ss dd-MM-yyyy HH:mm:ss

**Table 13** **value** parameter description
Parameter	Mandatory	Type	Description
type	Yes	String	Type name of the JSON attribute of the user data in the stream. Possible values: Bigint Double Boolean Timestamp String Decimal
value	Yes	String	JSON attribute name of the user data in the stream. This value is 1 to 32 characters long. Only letters, digits, and periods (.) are allowed.

**Table 14** **tags** parameter description
Parameter	Mandatory	Type	Description
name	Yes	String	Tag name of the OpenTSDB data that stores the data in the stream. This value is 1 to 32 characters long. Only letters, digits, and underscores (_) are allowed.
type	Yes	String	Type name of the JSON attribute of the user data in the stream. Possible values: Bigint Double Boolean Timestamp String Decimal
value	Yes	String	Constant value or JSON attribute name of the user data in the stream. This value is 1 to 32 characters long. Only letters, digits, and underscores (_) are allowed.

**Table 15** **dws_destination_descriptor** parameter description
Parameter	Mandatory	Type	Description
consumer_strategy	No	String	Specifies the offset. Latest: Maximum offset, indicating that the latest data will be extracted. TRIM_HORIZON: Minimum offset, indicating that the earliest data will be extracted. Default value: Latest
agency_name	Yes	String	Name of the agency created in IAM. DIS uses an agency to access your specified resources. Agency parameter settings: Agency Type: Cloud service Cloud Service: DIS Validity Period: Permanent Policy NOTE: Both of the following permissions need to be configured. Set Policy to Tenant Administrator on the OBS project in the Global service region. Set Policy to DWS Administrator, Server Administrator, and Tenant Guest on the project and region to which the DWS cluster belongs. This parameter cannot be left unspecified and the parameter value cannot exceed 64 characters.
dws_cluster_name	Yes	String	Name of the DWS cluster to which data will be dumped.
dws_cluster_id	Yes	String	ID of the DWS cluster to which will be dumped.
dws_database_name	Yes	String	Name of the DWS database to which data in the DIS stream will be dumped.
dws_schema	Yes	String	Schema of the DWS database to which data will be dumped.
dws_table_name	Yes	String	Name of the table in the DWS database to which data will be dumped.
dws_delimiter	Yes	String	Delimiter used to separate the columns in the DWS tables. The delimiter can be a comma (,), semicolon (;), or vertical bar (\|).
user_name	Yes	String	Username of the DWS database to which data will be dumped.
user_password	Yes	String	Password of the DWS database to which data will be dumped.
kms_user_key_name	Yes	String	Key created in Key Management Service (KMS) and used to encrypt the password of the DWS database.
kms_user_key_id	Yes	String	ID of the key created in KMS and used to encrypt the password of the DWS database.
obs_bucket_path	Yes	String	Name of the OBS bucket to which data in the DIS stream will be temporarily saved.
file_prefix	No	String	Self-defined directory created in the OBS bucket and used to temporarily store data in the DIS stream. Directory levels are separated by slashes (/) and cannot start with slashes. The value cannot exceed 50 characters. Only letters, digits, underscores (_), and slashes (/) are allowed. This parameter is left unspecified by default.
deliver_time_interval	Yes	Int	Interval at which data is imported to the table of a DWS cluster. If no data is imported during the specific period of time, no dump file package is created. Value range: 30s to 900s Unit: second Default value: 300
retry_duration	No	Int	Time duration for DIS to retry if data failed to be dumped to the DWS cluster. If the duration is exceeded but the dump still fails, the data will be backed up to OBS bucket name/file_prefix/dws_error. Value range: 0s to 7200s Unit: second Default value: 1800
dws_table_columns	No	String	Column to be dumped to the DWS table. If the value is null or empty, all columns are dumped by default. The value is left blank by default.
options	No	Object	DWS fault tolerance option (used to specify various parameters of foreign table data).

**Table 16** **options** parameter description
Parameter	Mandatory	Type	Description
fill_missing_fields	No	String	Specifies whether to set the field to Null or enable an error message to be displayed in the error table when the last field in a row of the data source file is missing during database import. Value: true/on or false/off Default value: false/off
ignore_extra_data	No	String	Specifies whether to ignore the extra columns when the number of fields in the data source file is greater than the number of columns defined in the foreign table. This parameter is used only during data import. Value: true/on or false/off Default value: false/off
compatible_illegal_chars	No	String	Specifies whether to convert invalid characters based on the conversion rule and save the converted characters to the database, or whether to report an error and stop the import. Value: true/on or false/off Default value: false/off
reject_limit	No	String	Specifies the maximum number of data format errors allowed during the data import. If the number of data format errors does not reach the maximum, the data import is successful. Value range: integer or unlimited (no limit) Default value: 0, indicating that an error message is returned immediately a data format error occurs.

Response

If the operation is successfully performed, status code 201 is returned and the response message body is empty.
If the operation failed to be performed, identify the failure cause according to the response body and the instructions in Error Codes.

Response Code

Normal
201 Created
Failed
For more information, see Error Codes.

Products

Compute

Application

Dedicated Cloud

Storage

Management & Deployment

Migration

Network

Enterprise Intelligence

Video

Database

Edge Cloud Services

DevCloud

Security

Cloud Communications

Internet of Things

Solutions

Industry-Specific Solutions

General-Purpose Solutions

Security

DevOps

Enterprise Intelligence

Essential Platform

Big Data

Visual Cognition

Speech and Semantics

Support

Help Center

Customer Services

Developers

Console

语言 - Language

中国站 - 简体中文

中国站 - English

International - 简体中文

International - English

Help Center

Adding a Dump Task

Function

URI

Request

Response

Response Code