文档首页/ 数据接入服务 DIS/ API参考/ API说明/ 转储任务管理/ 添加OBS转储任务
更新时间：2024-10-21 GMT+08:00
在线调试
CLI示例
查看PDF
分享

添加OBS转储任务

功能介绍

本接口用于添加OBS转储任务。

调用方法

请参见如何调用API。

URI

POST /v2/{project_id}/streams/{stream_name}/transfer-tasks

表1 路径参数

参数

是否必选

参数类型

描述

project_id

是

String

项目ID。

stream_name

是

String

已创建的通道名称。

最大长度：60

请求参数

表2 请求Header参数

参数

是否必选

参数类型

描述

X-Auth-Token

是

String

用户Token。

通过调用IAM服务获取用户Token接口获取（响应消息头中X-Subject-Token的值）。
表3 请求Body参数

参数

是否必选

参数类型

描述

destination_type

是

String

转储任务类型。

  • OBS：转储到OBS

  • MRS：转储到MRS

  • DLI：转储到DLI

  • CLOUDTABLE：转储到CloudTable

  • DWS：转储到DWS

缺省值：NOWHERE

枚举值：

  • OBS

obs_destination_descriptor

否

OBSDestinationDescriptorRequest object

转储目的地为OBS的参数列表。
表4 OBSDestinationDescriptorRequest

参数

是否必选

参数类型

描述

task_name

是

String

转储任务的名称。 任务名称由英文字母、数字、中划线和下划线组成。长度为1～64个字符。

agency_name

是

String

在统一身份认证服务(IAM)中创建委托的名称，DIS需要获取IAM委托信息去访问您指定的资源。创建委托的参数设置如下：- 委托类型：云服务- 云服务：DIS- 持续时间：永久- “所属区域”为“全局服务”，“项目”为“对象存储服务”对应的“策略”包含“Tenant Administrator”。如果已经创建过委托，可以使用IAM服务提供的查询委托列表接口，获取有效可用的委托名称。取值范围：长度不超过64位，且不可配置为空。如果有在Console控制台使用转储任务，会提示自动创建委托，自动创建的委托名称为：dis_admin_agency

最大长度：64

deliver_time_interval

是

Integer

根据用户配置的时间，周期性的将数据导入OBS，若某个时间段内无数据，则此时间段不会生成打包文件。

单位：秒

最小值：30

最大值：900

缺省值：300

consumer_strategy

否

String

偏移量。

  • LATEST：最大偏移量，即获取最新的数据。

  • TRIM_HORIZON：最小偏移量，即读取最早的数据。

缺省值：LATEST

枚举值：

  • LATEST

  • TRIM_HORIZON

file_prefix

否

String

在OBS中存储通道文件的自定义目录，多级目录可用“/”进行分隔，不可以“/”开头。

取值范围：英文字母、数字、下划线和斜杠，最大长度为50个字符。

默认配置为空。

最大长度：50

partition_format

否

String

将转储文件的生成时间使用“yyyy/MM/dd/HH/mm”格式生成分区字符串，用来定义写到OBS的Object文件所在的目录层次结构。- N/A：置空，不使用日期时间目录。- yyyy：年- yyyy/MM：年/- yyyy/MM/dd：年/月/日- yyyy/MM/dd/HH：年/月/日/时- yyyy/MM/dd/HH/mm：年/月/日/时/分例如：2017/11/10/14/49，目录结构就是“2017 > 11 > 10 > 14 > 49”，“2017”表示最外层文件夹。默认值：空说明：数据转储成功后，存储的目录结构为“obs_bucket_path/file_prefix/partition_format”。

枚举值：

  • yyyy

  • yyyy/MM

  • yyyy/MM/dd

  • yyyy/MM/dd/HH

  • yyyy/MM/dd/HH/mm

obs_bucket_path

是

String

存储该通道数据的OBS桶名称。

destination_file_type

否

String

转储文件格式。

  • text：转储目标格式为TEXT，缺省值

  • parquet：转储目标格式为Parquet

  • carbon：转储目标格式为Carbon

说明：

“源数据类型”为“JSON”，“转储服务类型”为“OBS”时才可选择“parquet”或“carbon”格式。

缺省值：text

枚举值：

  • text

  • parquet

  • carbon

processing_schema

否

ProcessingSchema object

根据源数据的时间戳和已配置的"partition_format"生成对应的转储时间目录。将源数据的时间戳使用“yyyy/MM/dd/HH/mm”格式生成分区字符串，用来定义写到OBS的Object文件所在的目录层次结构。

record_delimiter

否

String

转储文件的记录分隔符，用于分隔写入转储文件的用户数据。

取值范围：

  • 逗号 ","，默认值

  • 分号 ";"

  • 竖线 "|"

  • 换行符 "\n"

缺省值：\n
表5 ProcessingSchema

参数

是否必选

参数类型

描述

timestamp_name

是

String

源数据时间戳的属性名称。

timestamp_type

是

String

源数据时间戳的类型。

  • String

  • Timestamp：Long类型的13位时间戳

timestamp_format

否

String

源数据时间戳的类型为String时必选，用于根据时间戳格式生成OBS的时间目录。

枚举值：

  • yyyy/MM/dd HH:mm:ss

  • MM/dd/yyyy HH:mm:ss

  • dd/MM/yyyy HH:mm:ss

  • yyyy-MM-dd HH:mm:ss

  • MM-dd-yyyy HH:mm:ss

  • dd-MM-yyyy HH:mm:ss

响应参数

无

请求示例

  • 添加OBS转储任务

    POST https://{Endpoint}/v2/{project_id}/streams/{stream_name}/transfer-tasks

{
  "destination_type" : "OBS",
  "obs_destination_descriptor" : {
    "task_name" : "newtask",
    "consumer_strategy" : "LATEST",
    "agency_name" : "dis_admin_agency",
    "destination_file_type" : "text",
    "obs_bucket_path" : "obsbucket",
    "file_prefix" : "",
    "partition_format" : "yyyy/MM/dd/HH/mm",
    "record_delimiter" : "|",
    "deliver_time_interval" : 30
  }
}

  • 添加OBS转储任务（转储文件格式是parquet）

    POST https://{Endpoint}/v2/{project_id}/streams/{stream_name}/transfer-tasks

{
  "destination_type" : "OBS",
  "obs_destination_descriptor" : {
    "task_name" : "newtask",
    "consumer_strategy" : "LATEST",
    "agency_name" : "dis_admin_agency",
    "destination_file_type" : "parquet",
    "obs_bucket_path" : "obsbucket",
    "file_prefix" : "",
    "partition_format" : "yyyy/MM/dd/HH/mm",
    "record_delimiter" : "|",
    "deliver_time_interval" : 30
  }
}

响应示例

无

SDK代码示例

SDK代码示例如下。

Java

  • 添加OBS转储任务

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
    		 
    package com.huaweicloud.sdk.test;

import com.huaweicloud.sdk.core.auth.ICredential;
import com.huaweicloud.sdk.core.auth.BasicCredentials;
import com.huaweicloud.sdk.core.exception.ConnectionException;
import com.huaweicloud.sdk.core.exception.RequestTimeoutException;
import com.huaweicloud.sdk.core.exception.ServiceResponseException;
import com.huaweicloud.sdk.dis.v2.region.DisRegion;
import com.huaweicloud.sdk.dis.v2.*;
import com.huaweicloud.sdk.dis.v2.model.*;


public class CreateObsTransferTaskSolution {

    public static void main(String[] args) {
        // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
        // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
        String ak = System.getenv("CLOUD_SDK_AK");
        String sk = System.getenv("CLOUD_SDK_SK");

        ICredential auth = new BasicCredentials()
                .withAk(ak)
                .withSk(sk);

        DisClient client = DisClient.newBuilder()
                .withCredential(auth)
                .withRegion(DisRegion.valueOf("<YOUR REGION>"))
                .build();
        CreateObsTransferTaskRequest request = new CreateObsTransferTaskRequest();
        CreateTransferTaskReq body = new CreateTransferTaskReq();
        OBSDestinationDescriptorRequest obsDestinationDescriptorbody = new OBSDestinationDescriptorRequest();
        obsDestinationDescriptorbody.withTaskName("newtask")
            .withObsBucketPath("obsbucket")
            .withAgencyName("dis_admin_agency")
            .withDestinationFileType(OBSDestinationDescriptorRequest.DestinationFileTypeEnum.fromValue("text"))
            .withRecordDelimiter("|")
            .withDeliverTimeInterval(30)
            .withFilePrefix("")
            .withPartitionFormat(OBSDestinationDescriptorRequest.PartitionFormatEnum.fromValue("yyyy/MM/dd/HH/mm"))
            .withConsumerStrategy(OBSDestinationDescriptorRequest.ConsumerStrategyEnum.fromValue("LATEST"));
        body.withObsDestinationDescriptor(obsDestinationDescriptorbody);
        body.withDestinationType(CreateTransferTaskReq.DestinationTypeEnum.fromValue("OBS"));
        request.withBody(body);
        try {
            CreateObsTransferTaskResponse response = client.createObsTransferTask(request);
            System.out.println(response.toString());
        } catch (ConnectionException e) {
            e.printStackTrace();
        } catch (RequestTimeoutException e) {
            e.printStackTrace();
        } catch (ServiceResponseException e) {
            e.printStackTrace();
            System.out.println(e.getHttpStatusCode());
            System.out.println(e.getRequestId());
            System.out.println(e.getErrorCode());
            System.out.println(e.getErrorMsg());
        }
    }
}

  • 添加OBS转储任务（转储文件格式是parquet）

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
    		 
    package com.huaweicloud.sdk.test;

import com.huaweicloud.sdk.core.auth.ICredential;
import com.huaweicloud.sdk.core.auth.BasicCredentials;
import com.huaweicloud.sdk.core.exception.ConnectionException;
import com.huaweicloud.sdk.core.exception.RequestTimeoutException;
import com.huaweicloud.sdk.core.exception.ServiceResponseException;
import com.huaweicloud.sdk.dis.v2.region.DisRegion;
import com.huaweicloud.sdk.dis.v2.*;
import com.huaweicloud.sdk.dis.v2.model.*;


public class CreateObsTransferTaskSolution {

    public static void main(String[] args) {
        // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
        // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
        String ak = System.getenv("CLOUD_SDK_AK");
        String sk = System.getenv("CLOUD_SDK_SK");

        ICredential auth = new BasicCredentials()
                .withAk(ak)
                .withSk(sk);

        DisClient client = DisClient.newBuilder()
                .withCredential(auth)
                .withRegion(DisRegion.valueOf("<YOUR REGION>"))
                .build();
        CreateObsTransferTaskRequest request = new CreateObsTransferTaskRequest();
        CreateTransferTaskReq body = new CreateTransferTaskReq();
        OBSDestinationDescriptorRequest obsDestinationDescriptorbody = new OBSDestinationDescriptorRequest();
        obsDestinationDescriptorbody.withTaskName("newtask")
            .withObsBucketPath("obsbucket")
            .withAgencyName("dis_admin_agency")
            .withDestinationFileType(OBSDestinationDescriptorRequest.DestinationFileTypeEnum.fromValue("parquet"))
            .withRecordDelimiter("|")
            .withDeliverTimeInterval(30)
            .withFilePrefix("")
            .withPartitionFormat(OBSDestinationDescriptorRequest.PartitionFormatEnum.fromValue("yyyy/MM/dd/HH/mm"))
            .withConsumerStrategy(OBSDestinationDescriptorRequest.ConsumerStrategyEnum.fromValue("LATEST"));
        body.withObsDestinationDescriptor(obsDestinationDescriptorbody);
        body.withDestinationType(CreateTransferTaskReq.DestinationTypeEnum.fromValue("OBS"));
        request.withBody(body);
        try {
            CreateObsTransferTaskResponse response = client.createObsTransferTask(request);
            System.out.println(response.toString());
        } catch (ConnectionException e) {
            e.printStackTrace();
        } catch (RequestTimeoutException e) {
            e.printStackTrace();
        } catch (ServiceResponseException e) {
            e.printStackTrace();
            System.out.println(e.getHttpStatusCode());
            System.out.println(e.getRequestId());
            System.out.println(e.getErrorCode());
            System.out.println(e.getErrorMsg());
        }
    }
}

Python

  • 添加OBS转储任务

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
    		 
    # coding: utf-8

import os
from huaweicloudsdkcore.auth.credentials import BasicCredentials
from huaweicloudsdkdis.v2.region.dis_region import DisRegion
from huaweicloudsdkcore.exceptions import exceptions
from huaweicloudsdkdis.v2 import *

if __name__ == "__main__":
    # The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    # In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak = os.environ["CLOUD_SDK_AK"]
    sk = os.environ["CLOUD_SDK_SK"]

    credentials = BasicCredentials(ak, sk)

    client = DisClient.new_builder() \
        .with_credentials(credentials) \
        .with_region(DisRegion.value_of("<YOUR REGION>")) \
        .build()

    try:
        request = CreateObsTransferTaskRequest()
        obsDestinationDescriptorbody = OBSDestinationDescriptorRequest(
            task_name="newtask",
            obs_bucket_path="obsbucket",
            agency_name="dis_admin_agency",
            destination_file_type="text",
            record_delimiter="|",
            deliver_time_interval=30,
            file_prefix="",
            partition_format="yyyy/MM/dd/HH/mm",
            consumer_strategy="LATEST"
        )
        request.body = CreateTransferTaskReq(
            obs_destination_descriptor=obsDestinationDescriptorbody,
            destination_type="OBS"
        )
        response = client.create_obs_transfer_task(request)
        print(response)
    except exceptions.ClientRequestException as e:
        print(e.status_code)
        print(e.request_id)
        print(e.error_code)
        print(e.error_msg)

  • 添加OBS转储任务（转储文件格式是parquet）

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
    		 
    # coding: utf-8

import os
from huaweicloudsdkcore.auth.credentials import BasicCredentials
from huaweicloudsdkdis.v2.region.dis_region import DisRegion
from huaweicloudsdkcore.exceptions import exceptions
from huaweicloudsdkdis.v2 import *

if __name__ == "__main__":
    # The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    # In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak = os.environ["CLOUD_SDK_AK"]
    sk = os.environ["CLOUD_SDK_SK"]

    credentials = BasicCredentials(ak, sk)

    client = DisClient.new_builder() \
        .with_credentials(credentials) \
        .with_region(DisRegion.value_of("<YOUR REGION>")) \
        .build()

    try:
        request = CreateObsTransferTaskRequest()
        obsDestinationDescriptorbody = OBSDestinationDescriptorRequest(
            task_name="newtask",
            obs_bucket_path="obsbucket",
            agency_name="dis_admin_agency",
            destination_file_type="parquet",
            record_delimiter="|",
            deliver_time_interval=30,
            file_prefix="",
            partition_format="yyyy/MM/dd/HH/mm",
            consumer_strategy="LATEST"
        )
        request.body = CreateTransferTaskReq(
            obs_destination_descriptor=obsDestinationDescriptorbody,
            destination_type="OBS"
        )
        response = client.create_obs_transfer_task(request)
        print(response)
    except exceptions.ClientRequestException as e:
        print(e.status_code)
        print(e.request_id)
        print(e.error_code)
        print(e.error_msg)

Go

  • 添加OBS转储任务

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
    		 
    package main

import (
	"fmt"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/core/auth/basic"
    dis "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/dis/v2"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/services/dis/v2/model"
    region "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/dis/v2/region"
)

func main() {
    // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak := os.Getenv("CLOUD_SDK_AK")
    sk := os.Getenv("CLOUD_SDK_SK")

    auth := basic.NewCredentialsBuilder().
        WithAk(ak).
        WithSk(sk).
        Build()

    client := dis.NewDisClient(
        dis.DisClientBuilder().
            WithRegion(region.ValueOf("<YOUR REGION>")).
            WithCredential(auth).
            Build())

    request := &model.CreateObsTransferTaskRequest{}
	destinationFileTypeObsDestinationDescriptor:= model.GetObsDestinationDescriptorRequestDestinationFileTypeEnum().TEXT
	recordDelimiterObsDestinationDescriptor:= "|"
	filePrefixObsDestinationDescriptor:= ""
	partitionFormatObsDestinationDescriptor:= model.GetObsDestinationDescriptorRequestPartitionFormatEnum().YYYY_MM_DD_HH_MM
	consumerStrategyObsDestinationDescriptor:= model.GetObsDestinationDescriptorRequestConsumerStrategyEnum().LATEST
	obsDestinationDescriptorbody := &model.ObsDestinationDescriptorRequest{
		TaskName: "newtask",
		ObsBucketPath: "obsbucket",
		AgencyName: "dis_admin_agency",
		DestinationFileType: &destinationFileTypeObsDestinationDescriptor,
		RecordDelimiter: &recordDelimiterObsDestinationDescriptor,
		DeliverTimeInterval: int32(30),
		FilePrefix: &filePrefixObsDestinationDescriptor,
		PartitionFormat: &partitionFormatObsDestinationDescriptor,
		ConsumerStrategy: &consumerStrategyObsDestinationDescriptor,
	}
	request.Body = &model.CreateTransferTaskReq{
		ObsDestinationDescriptor: obsDestinationDescriptorbody,
		DestinationType: model.GetCreateTransferTaskReqDestinationTypeEnum().OBS,
	}
	response, err := client.CreateObsTransferTask(request)
	if err == nil {
        fmt.Printf("%+v\n", response)
    } else {
        fmt.Println(err)
    }
}

  • 添加OBS转储任务（转储文件格式是parquet）

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
    		 
    package main

import (
	"fmt"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/core/auth/basic"
    dis "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/dis/v2"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/services/dis/v2/model"
    region "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/dis/v2/region"
)

func main() {
    // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak := os.Getenv("CLOUD_SDK_AK")
    sk := os.Getenv("CLOUD_SDK_SK")

    auth := basic.NewCredentialsBuilder().
        WithAk(ak).
        WithSk(sk).
        Build()

    client := dis.NewDisClient(
        dis.DisClientBuilder().
            WithRegion(region.ValueOf("<YOUR REGION>")).
            WithCredential(auth).
            Build())

    request := &model.CreateObsTransferTaskRequest{}
	destinationFileTypeObsDestinationDescriptor:= model.GetObsDestinationDescriptorRequestDestinationFileTypeEnum().PARQUET
	recordDelimiterObsDestinationDescriptor:= "|"
	filePrefixObsDestinationDescriptor:= ""
	partitionFormatObsDestinationDescriptor:= model.GetObsDestinationDescriptorRequestPartitionFormatEnum().YYYY_MM_DD_HH_MM
	consumerStrategyObsDestinationDescriptor:= model.GetObsDestinationDescriptorRequestConsumerStrategyEnum().LATEST
	obsDestinationDescriptorbody := &model.ObsDestinationDescriptorRequest{
		TaskName: "newtask",
		ObsBucketPath: "obsbucket",
		AgencyName: "dis_admin_agency",
		DestinationFileType: &destinationFileTypeObsDestinationDescriptor,
		RecordDelimiter: &recordDelimiterObsDestinationDescriptor,
		DeliverTimeInterval: int32(30),
		FilePrefix: &filePrefixObsDestinationDescriptor,
		PartitionFormat: &partitionFormatObsDestinationDescriptor,
		ConsumerStrategy: &consumerStrategyObsDestinationDescriptor,
	}
	request.Body = &model.CreateTransferTaskReq{
		ObsDestinationDescriptor: obsDestinationDescriptorbody,
		DestinationType: model.GetCreateTransferTaskReqDestinationTypeEnum().OBS,
	}
	response, err := client.CreateObsTransferTask(request)
	if err == nil {
        fmt.Printf("%+v\n", response)
    } else {
        fmt.Println(err)
    }
}

更多

更多编程语言的SDK代码示例，请参见API Explorer的代码示例页签，可生成自动对应的SDK代码示例。

状态码

状态码

描述

201

正常返回

错误码

请参见错误码。

父主题： 转储任务管理