Help Center> Distributed Message Service for Kafka> API Reference> APIs V2 (Recommended)> Dumping Management> Creating a Dumping Task
View PDF

Creating a Dumping Task

Function

This API is used to create a dumping task.

Debugging

You can use API Explorer to debug this API.

URI

POST /v2/{project_id}/connectors/{connector_id}/sink-tasks

Table 1 Path parameters

Parameter

Mandatory

Type

Description

project_id

Yes

String

Project ID.

connector_id

Yes

String

Instance dump ID. The value can be obtained from response of the API for querying an instance.

Request Parameters

Table 2 Request body parameters

Parameter

Mandatory

Type

Description

source_type

Yes

String

Source data type. Currently, only BLOB is supported.

task_name

Yes

String

Name of the dumping task.

destination_type

Yes

String

Storage type. Currently, only OBS is supported.

obs_destination_descriptor

Yes

ObsDestinationDescriptor object

Description of the dump.
Table 3 ObsDestinationDescriptor

Parameter

Mandatory

Type

Description

topics

Yes

String

Names of the topics to be dumped. Multiple topics can be separated by commas (,). Regular expressions are supported. Example value: topic1,topic2.

topics_regex

No

String

Regular expression of the topics to be dumped. Set either the topics_regex parameter or topics parameter. Do not set both of them or set it to ".*".

consumer_strategy

Yes

String

Dump start offset.

  • latest: Messages are consumed from the end of the topic.

  • earliest: Messages are consumed from the start of the topic.

The default value is latest.

destination_file_type

Yes

String

Dump file format. Currently, only TXT files are supported.

access_key

Yes

String

Access key ID (AK).

secret_key

Yes

String

Secret access key (SK).

obs_bucket_name

Yes

String

Name of the OBS bucket used to store the data.

obs_path

No

String

OBS path. This parameter can be left blank by default. The value can contain a maximum of 50 characters, including letters, digits, underscores (_), and slashes (/). This parameter is left empty by default.

partition_format

No

String

Directory structure of the object file written into OBS. The directory structure is in the format of yyyy/MM/dd/HH/mm (time at which the dump task was created).

  • N/A: Leave this parameter empty, indicating that the date and time directory is not used.

  • yyyy: year.

  • yyyy/MM: year and month.

  • yyyy/MM/dd: year, month, and day.

  • yyyy/MM/dd/HH: year, month, day, and hour.

  • yyyy/MM/dd/HH/mm: year, month, day, hour, and minute. For example, 2017/11/10/14/49 means that the directory structure is 2017 > 11 > 10 > 14 > 49, where 2017 is the outermost folder.

Default value: empty.

After the data is dumped successfully, the storage directory structure is obs_bucket_path/file_prefix/partition_format. The default time zone is GMT+08:00.

record_delimiter

No

String

Delimiter for the dump file, which is used to separate the user data that is written into the dump file. The value can be:

  • Comma (,)

  • Semicolon (;)

  • Vertical bar (|)

  • Newline (\n)

  • NULL

Default value: newline (\n).

deliver_time_interval

Yes

Integer

User-defined interval at which data is imported into OBS. If no data is pushed during the current interval, no dump file package will be generated. Value range: 30 to 900 Unit: second.

NOTE:

This parameter is mandatory if streaming data is dumped to OBS.

Response Parameters

Status code: 200

Table 4 Response body parameters

Parameter

Type

Description

task_id

String

Task ID.

Example Requests

POST https://{endpoint}/v2/{project_id}/connectors/{connector_id}/sink-tasks

{
  "source_type" : "BLOB",
  "task_name" : "obsTransfer-1122976956",
  "destination_type" : "OBS",
  "obs_destination_descriptor" : {
    "consumer_strategy" : "earliest",
    "destination_file_type" : "TEXT",
    "access_key" : "XXXXXXXXXXXXXXXXXXXX",
    "secret_key" : "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
    "obs_bucket_name" : "6666",
    "obs_path" : "obsTransfer-1122976956",
    "partition_format" : "yyyy/MM/dd/HH/mm",
    "record_delimiter" : "\n",
    "deliver_time_interval" : 300,
    "topics" : "topic-test"
  }
}

Example Responses

Status code: 200

A dumping task is created successfully.

{
  "task_id" : "2962882a-386c-4c9d-bb59-3b4f55d82961"
}

Status Codes

Status Code

Description

200

A dumping task is created successfully.

Error Codes

See Error Codes.

Parent topic: Dumping Management