文档首页/ 数据接入服务 DIS/ API参考/ API说明/ 转储任务管理/ 添加OBS转储任务

更新时间：2024-10-21 GMT+08:00

在线调试

CLI示例

查看PDF

添加OBS转储任务

功能介绍

本接口用于添加OBS转储任务。

调用方法

请参见如何调用API。

URI

POST /v2/{project_id}/streams/{stream_name}/transfer-tasks

表1 路径参数
参数	是否必选	参数类型	描述
project_id	是	String	项目ID。
stream_name	是	String	已创建的通道名称。最大长度：60

请求参数

表2 请求Header参数
参数	是否必选	参数类型	描述
X-Auth-Token	是	String	用户Token。通过调用IAM服务获取用户Token接口获取（响应消息头中X-Subject-Token的值）。

表3 请求Body参数
参数	是否必选	参数类型	描述
destination_type	是	String	转储任务类型。 OBS：转储到OBS MRS：转储到MRS DLI：转储到DLI CLOUDTABLE：转储到CloudTable DWS：转储到DWS 缺省值：NOWHERE 枚举值： OBS
obs_destination_descriptor	否	OBSDestinationDescriptorRequest object	转储目的地为OBS的参数列表。

表4 OBSDestinationDescriptorRequest
参数	是否必选	参数类型	描述
task_name	是	String	转储任务的名称。任务名称由英文字母、数字、中划线和下划线组成。长度为1～64个字符。
agency_name	是	String	在统一身份认证服务(IAM)中创建委托的名称，DIS需要获取IAM委托信息去访问您指定的资源。创建委托的参数设置如下：- 委托类型：云服务- 云服务：DIS- 持续时间：永久- “所属区域”为“全局服务”，“项目”为“对象存储服务”对应的“策略”包含“Tenant Administrator”。如果已经创建过委托，可以使用IAM服务提供的查询委托列表接口，获取有效可用的委托名称。取值范围：长度不超过64位，且不可配置为空。如果有在Console控制台使用转储任务，会提示自动创建委托，自动创建的委托名称为：dis_admin_agency 最大长度：64
deliver_time_interval	是	Integer	根据用户配置的时间，周期性的将数据导入OBS，若某个时间段内无数据，则此时间段不会生成打包文件。单位：秒最小值：30 最大值：900 缺省值：300
consumer_strategy	否	String	偏移量。 LATEST：最大偏移量，即获取最新的数据。 TRIM_HORIZON：最小偏移量，即读取最早的数据。缺省值：LATEST 枚举值： LATEST TRIM_HORIZON
file_prefix	否	String	在OBS中存储通道文件的自定义目录，多级目录可用“/”进行分隔，不可以“/”开头。取值范围：英文字母、数字、下划线和斜杠，最大长度为50个字符。默认配置为空。最大长度：50
partition_format	否	String	将转储文件的生成时间使用“yyyy/MM/dd/HH/mm”格式生成分区字符串，用来定义写到OBS的Object文件所在的目录层次结构。- N/A：置空，不使用日期时间目录。- yyyy：年- yyyy/MM：年/- yyyy/MM/dd：年/月/日- yyyy/MM/dd/HH：年/月/日/时- yyyy/MM/dd/HH/mm：年/月/日/时/分例如：2017/11/10/14/49，目录结构就是“2017 > 11 > 10 > 14 > 49”，“2017”表示最外层文件夹。默认值：空说明：数据转储成功后，存储的目录结构为“obs_bucket_path/file_prefix/partition_format”。枚举值： yyyy yyyy/MM yyyy/MM/dd yyyy/MM/dd/HH yyyy/MM/dd/HH/mm
obs_bucket_path	是	String	存储该通道数据的OBS桶名称。
destination_file_type	否	String	转储文件格式。 text：转储目标格式为TEXT，缺省值 parquet：转储目标格式为Parquet carbon：转储目标格式为Carbon 说明： “源数据类型”为“JSON”，“转储服务类型”为“OBS”时才可选择“parquet”或“carbon”格式。缺省值：text 枚举值： text parquet carbon
processing_schema	否	ProcessingSchema object	根据源数据的时间戳和已配置的"partition_format"生成对应的转储时间目录。将源数据的时间戳使用“yyyy/MM/dd/HH/mm”格式生成分区字符串，用来定义写到OBS的Object文件所在的目录层次结构。
record_delimiter	否	String	转储文件的记录分隔符，用于分隔写入转储文件的用户数据。取值范围：逗号 ","，默认值分号 ";" 竖线 "\|" 换行符 "\n" 缺省值：\n

表5 ProcessingSchema
参数	是否必选	参数类型	描述
timestamp_name	是	String	源数据时间戳的属性名称。
timestamp_type	是	String	源数据时间戳的类型。 String Timestamp：Long类型的13位时间戳
timestamp_format	否	String	源数据时间戳的类型为String时必选，用于根据时间戳格式生成OBS的时间目录。枚举值： yyyy/MM/dd HH:mm:ss MM/dd/yyyy HH:mm:ss dd/MM/yyyy HH:mm:ss yyyy-MM-dd HH:mm:ss MM-dd-yyyy HH:mm:ss dd-MM-yyyy HH:mm:ss

响应参数

无

请求示例

添加OBS转储任务

POST https://{Endpoint}/v2/{project_id}/streams/{stream_name}/transfer-tasks

{
  "destination_type" : "OBS",
  "obs_destination_descriptor" : {
    "task_name" : "newtask",
    "consumer_strategy" : "LATEST",
    "agency_name" : "dis_admin_agency",
    "destination_file_type" : "text",
    "obs_bucket_path" : "obsbucket",
    "file_prefix" : "",
    "partition_format" : "yyyy/MM/dd/HH/mm",
    "record_delimiter" : "|",
    "deliver_time_interval" : 30
  }
}

添加OBS转储任务（转储文件格式是parquet）

POST https://{Endpoint}/v2/{project_id}/streams/{stream_name}/transfer-tasks

{
  "destination_type" : "OBS",
  "obs_destination_descriptor" : {
    "task_name" : "newtask",
    "consumer_strategy" : "LATEST",
    "agency_name" : "dis_admin_agency",
    "destination_file_type" : "parquet",
    "obs_bucket_path" : "obsbucket",
    "file_prefix" : "",
    "partition_format" : "yyyy/MM/dd/HH/mm",
    "record_delimiter" : "|",
    "deliver_time_interval" : 30
  }
}

响应示例

无

SDK代码示例

SDK代码示例如下。

添加OBS转储任务

        
         
           
           
             package com.huaweicloud.sdk.test;

import com.huaweicloud.sdk.core.auth.ICredential;
import com.huaweicloud.sdk.core.auth.BasicCredentials;
import com.huaweicloud.sdk.core.exception.ConnectionException;
import com.huaweicloud.sdk.core.exception.RequestTimeoutException;
import com.huaweicloud.sdk.core.exception.ServiceResponseException;
import com.huaweicloud.sdk.dis.v2.region.DisRegion;
import com.huaweicloud.sdk.dis.v2.*;
import com.huaweicloud.sdk.dis.v2.model.*;


public class CreateObsTransferTaskSolution {

    public static void main(String[] args) {
        // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
        // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
        String ak = System.getenv("CLOUD_SDK_AK");
        String sk = System.getenv("CLOUD_SDK_SK");

        ICredential auth = new BasicCredentials()
                .withAk(ak)
                .withSk(sk);

        DisClient client = DisClient.newBuilder()
                .withCredential(auth)
                .withRegion(DisRegion.valueOf("<YOUR REGION>"))
                .build();
        CreateObsTransferTaskRequest request = new CreateObsTransferTaskRequest();
        CreateTransferTaskReq body = new CreateTransferTaskReq();
        OBSDestinationDescriptorRequest obsDestinationDescriptorbody = new OBSDestinationDescriptorRequest();
        obsDestinationDescriptorbody.withTaskName("newtask")
            .withObsBucketPath("obsbucket")
            .withAgencyName("dis_admin_agency")
            .withDestinationFileType(OBSDestinationDescriptorRequest.DestinationFileTypeEnum.fromValue("text"))
            .withRecordDelimiter("|")
            .withDeliverTimeInterval(30)
            .withFilePrefix("")
            .withPartitionFormat(OBSDestinationDescriptorRequest.PartitionFormatEnum.fromValue("yyyy/MM/dd/HH/mm"))
            .withConsumerStrategy(OBSDestinationDescriptorRequest.ConsumerStrategyEnum.fromValue("LATEST"));
        body.withObsDestinationDescriptor(obsDestinationDescriptorbody);
        body.withDestinationType(CreateTransferTaskReq.DestinationTypeEnum.fromValue("OBS"));
        request.withBody(body);
        try {
            CreateObsTransferTaskResponse response = client.createObsTransferTask(request);
            System.out.println(response.toString());
        } catch (ConnectionException e) {
            e.printStackTrace();
        } catch (RequestTimeoutException e) {
            e.printStackTrace();
        } catch (ServiceResponseException e) {
            e.printStackTrace();
            System.out.println(e.getHttpStatusCode());
            System.out.println(e.getRequestId());
            System.out.println(e.getErrorCode());
            System.out.println(e.getErrorMsg());
        }
    }
}

            

          

        
       

添加OBS转储任务（转储文件格式是parquet）

        
         
           
           
             package com.huaweicloud.sdk.test;

import com.huaweicloud.sdk.core.auth.ICredential;
import com.huaweicloud.sdk.core.auth.BasicCredentials;
import com.huaweicloud.sdk.core.exception.ConnectionException;
import com.huaweicloud.sdk.core.exception.RequestTimeoutException;
import com.huaweicloud.sdk.core.exception.ServiceResponseException;
import com.huaweicloud.sdk.dis.v2.region.DisRegion;
import com.huaweicloud.sdk.dis.v2.*;
import com.huaweicloud.sdk.dis.v2.model.*;


public class CreateObsTransferTaskSolution {

    public static void main(String[] args) {
        // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
        // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
        String ak = System.getenv("CLOUD_SDK_AK");
        String sk = System.getenv("CLOUD_SDK_SK");

        ICredential auth = new BasicCredentials()
                .withAk(ak)
                .withSk(sk);

        DisClient client = DisClient.newBuilder()
                .withCredential(auth)
                .withRegion(DisRegion.valueOf("<YOUR REGION>"))
                .build();
        CreateObsTransferTaskRequest request = new CreateObsTransferTaskRequest();
        CreateTransferTaskReq body = new CreateTransferTaskReq();
        OBSDestinationDescriptorRequest obsDestinationDescriptorbody = new OBSDestinationDescriptorRequest();
        obsDestinationDescriptorbody.withTaskName("newtask")
            .withObsBucketPath("obsbucket")
            .withAgencyName("dis_admin_agency")
            .withDestinationFileType(OBSDestinationDescriptorRequest.DestinationFileTypeEnum.fromValue("parquet"))
            .withRecordDelimiter("|")
            .withDeliverTimeInterval(30)
            .withFilePrefix("")
            .withPartitionFormat(OBSDestinationDescriptorRequest.PartitionFormatEnum.fromValue("yyyy/MM/dd/HH/mm"))
            .withConsumerStrategy(OBSDestinationDescriptorRequest.ConsumerStrategyEnum.fromValue("LATEST"));
        body.withObsDestinationDescriptor(obsDestinationDescriptorbody);
        body.withDestinationType(CreateTransferTaskReq.DestinationTypeEnum.fromValue("OBS"));
        request.withBody(body);
        try {
            CreateObsTransferTaskResponse response = client.createObsTransferTask(request);
            System.out.println(response.toString());
        } catch (ConnectionException e) {
            e.printStackTrace();
        } catch (RequestTimeoutException e) {
            e.printStackTrace();
        } catch (ServiceResponseException e) {
            e.printStackTrace();
            System.out.println(e.getHttpStatusCode());
            System.out.println(e.getRequestId());
            System.out.println(e.getErrorCode());
            System.out.println(e.getErrorMsg());
        }
    }
}

            

          

        
       

添加OBS转储任务

        
         
           
           
             # coding: utf-8

import os
from huaweicloudsdkcore.auth.credentials import BasicCredentials
from huaweicloudsdkdis.v2.region.dis_region import DisRegion
from huaweicloudsdkcore.exceptions import exceptions
from huaweicloudsdkdis.v2 import *

if __name__ == "__main__":
    # The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    # In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak = os.environ["CLOUD_SDK_AK"]
    sk = os.environ["CLOUD_SDK_SK"]

    credentials = BasicCredentials(ak, sk)

    client = DisClient.new_builder() \
        .with_credentials(credentials) \
        .with_region(DisRegion.value_of("<YOUR REGION>")) \
        .build()

    try:
        request = CreateObsTransferTaskRequest()
        obsDestinationDescriptorbody = OBSDestinationDescriptorRequest(
            task_name="newtask",
            obs_bucket_path="obsbucket",
            agency_name="dis_admin_agency",
            destination_file_type="text",
            record_delimiter="|",
            deliver_time_interval=30,
            file_prefix="",
            partition_format="yyyy/MM/dd/HH/mm",
            consumer_strategy="LATEST"
        )
        request.body = CreateTransferTaskReq(
            obs_destination_descriptor=obsDestinationDescriptorbody,
            destination_type="OBS"
        )
        response = client.create_obs_transfer_task(request)
        print(response)
    except exceptions.ClientRequestException as e:
        print(e.status_code)
        print(e.request_id)
        print(e.error_code)
        print(e.error_msg)

            

          

        
       

添加OBS转储任务（转储文件格式是parquet）

        
         
           
           
             # coding: utf-8

import os
from huaweicloudsdkcore.auth.credentials import BasicCredentials
from huaweicloudsdkdis.v2.region.dis_region import DisRegion
from huaweicloudsdkcore.exceptions import exceptions
from huaweicloudsdkdis.v2 import *

if __name__ == "__main__":
    # The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    # In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak = os.environ["CLOUD_SDK_AK"]
    sk = os.environ["CLOUD_SDK_SK"]

    credentials = BasicCredentials(ak, sk)

    client = DisClient.new_builder() \
        .with_credentials(credentials) \
        .with_region(DisRegion.value_of("<YOUR REGION>")) \
        .build()

    try:
        request = CreateObsTransferTaskRequest()
        obsDestinationDescriptorbody = OBSDestinationDescriptorRequest(
            task_name="newtask",
            obs_bucket_path="obsbucket",
            agency_name="dis_admin_agency",
            destination_file_type="parquet",
            record_delimiter="|",
            deliver_time_interval=30,
            file_prefix="",
            partition_format="yyyy/MM/dd/HH/mm",
            consumer_strategy="LATEST"
        )
        request.body = CreateTransferTaskReq(
            obs_destination_descriptor=obsDestinationDescriptorbody,
            destination_type="OBS"
        )
        response = client.create_obs_transfer_task(request)
        print(response)
    except exceptions.ClientRequestException as e:
        print(e.status_code)
        print(e.request_id)
        print(e.error_code)
        print(e.error_msg)

            

          

        
       

添加OBS转储任务

        
         
           
           
             package main

import (
	"fmt"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/core/auth/basic"
    dis "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/dis/v2"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/services/dis/v2/model"
    region "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/dis/v2/region"
)

func main() {
    // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak := os.Getenv("CLOUD_SDK_AK")
    sk := os.Getenv("CLOUD_SDK_SK")

    auth := basic.NewCredentialsBuilder().
        WithAk(ak).
        WithSk(sk).
        Build()

    client := dis.NewDisClient(
        dis.DisClientBuilder().
            WithRegion(region.ValueOf("<YOUR REGION>")).
            WithCredential(auth).
            Build())

    request := &model.CreateObsTransferTaskRequest{}
	destinationFileTypeObsDestinationDescriptor:= model.GetObsDestinationDescriptorRequestDestinationFileTypeEnum().TEXT
	recordDelimiterObsDestinationDescriptor:= "|"
	filePrefixObsDestinationDescriptor:= ""
	partitionFormatObsDestinationDescriptor:= model.GetObsDestinationDescriptorRequestPartitionFormatEnum().YYYY_MM_DD_HH_MM
	consumerStrategyObsDestinationDescriptor:= model.GetObsDestinationDescriptorRequestConsumerStrategyEnum().LATEST
	obsDestinationDescriptorbody := &model.ObsDestinationDescriptorRequest{
		TaskName: "newtask",
		ObsBucketPath: "obsbucket",
		AgencyName: "dis_admin_agency",
		DestinationFileType: &destinationFileTypeObsDestinationDescriptor,
		RecordDelimiter: &recordDelimiterObsDestinationDescriptor,
		DeliverTimeInterval: int32(30),
		FilePrefix: &filePrefixObsDestinationDescriptor,
		PartitionFormat: &partitionFormatObsDestinationDescriptor,
		ConsumerStrategy: &consumerStrategyObsDestinationDescriptor,
	}
	request.Body = &model.CreateTransferTaskReq{
		ObsDestinationDescriptor: obsDestinationDescriptorbody,
		DestinationType: model.GetCreateTransferTaskReqDestinationTypeEnum().OBS,
	}
	response, err := client.CreateObsTransferTask(request)
	if err == nil {
        fmt.Printf("%+v\n", response)
    } else {
        fmt.Println(err)
    }
}

            

          

        
       

添加OBS转储任务（转储文件格式是parquet）

        
         
           
           
             package main

import (
	"fmt"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/core/auth/basic"
    dis "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/dis/v2"
	"github.com/huaweicloud/huaweicloud-sdk-go-v3/services/dis/v2/model"
    region "github.com/huaweicloud/huaweicloud-sdk-go-v3/services/dis/v2/region"
)

func main() {
    // The AK and SK used for authentication are hard-coded or stored in plaintext, which has great security risks. It is recommended that the AK and SK be stored in ciphertext in configuration files or environment variables and decrypted during use to ensure security.
    // In this example, AK and SK are stored in environment variables for authentication. Before running this example, set environment variables CLOUD_SDK_AK and CLOUD_SDK_SK in the local environment
    ak := os.Getenv("CLOUD_SDK_AK")
    sk := os.Getenv("CLOUD_SDK_SK")

    auth := basic.NewCredentialsBuilder().
        WithAk(ak).
        WithSk(sk).
        Build()

    client := dis.NewDisClient(
        dis.DisClientBuilder().
            WithRegion(region.ValueOf("<YOUR REGION>")).
            WithCredential(auth).
            Build())

    request := &model.CreateObsTransferTaskRequest{}
	destinationFileTypeObsDestinationDescriptor:= model.GetObsDestinationDescriptorRequestDestinationFileTypeEnum().PARQUET
	recordDelimiterObsDestinationDescriptor:= "|"
	filePrefixObsDestinationDescriptor:= ""
	partitionFormatObsDestinationDescriptor:= model.GetObsDestinationDescriptorRequestPartitionFormatEnum().YYYY_MM_DD_HH_MM
	consumerStrategyObsDestinationDescriptor:= model.GetObsDestinationDescriptorRequestConsumerStrategyEnum().LATEST
	obsDestinationDescriptorbody := &model.ObsDestinationDescriptorRequest{
		TaskName: "newtask",
		ObsBucketPath: "obsbucket",
		AgencyName: "dis_admin_agency",
		DestinationFileType: &destinationFileTypeObsDestinationDescriptor,
		RecordDelimiter: &recordDelimiterObsDestinationDescriptor,
		DeliverTimeInterval: int32(30),
		FilePrefix: &filePrefixObsDestinationDescriptor,
		PartitionFormat: &partitionFormatObsDestinationDescriptor,
		ConsumerStrategy: &consumerStrategyObsDestinationDescriptor,
	}
	request.Body = &model.CreateTransferTaskReq{
		ObsDestinationDescriptor: obsDestinationDescriptorbody,
		DestinationType: model.GetCreateTransferTaskReqDestinationTypeEnum().OBS,
	}
	response, err := client.CreateObsTransferTask(request)
	if err == nil {
        fmt.Printf("%+v\n", response)
    } else {
        fmt.Println(err)
    }
}

            

          

        
       

更多编程语言的SDK代码示例，请参见API Explorer的代码示例页签，可生成自动对应的SDK代码示例。

状态码

状态码	描述
201	正常返回

错误码

请参见错误码。

父主题：转储任务管理

上一篇：转储任务管理

下一篇：查询转储任务列表

意见反馈

文档内容是否对您有帮助？

有帮助没帮助

提供反馈

提交成功！非常感谢您的反馈，我们会继续努力做到更好！您可在我的云声建议查看反馈及问题处理状态。

系统繁忙，请稍后重试

在使用文档中是否遇到以下问题

内容与产品页面不一致

内容不易理解

缺失示例代码

步骤不可操作

搜不到想要的内容

缺少最佳实践

意见反馈（选填）

0/500

请至少选择一项反馈信息并填写问题反馈

字符长度不能超过500

直接提交取消

如您有其它疑问，您也可以通过华为云社区问答频道来与我们联系探讨

智能客服提问云社区提问

添加OBS转储任务

功能介绍

调用方法

URI

请求参数

响应参数

请求示例

响应示例

SDK代码示例

状态码

错误码

相关文档

意见反馈

文档内容是否对您有帮助？

7*24

备案

专业服务

退订

建议反馈

售前咨询热线