Data Ingestion Service
Data Ingestion Service
All results for "
" in this service
All results for "
" in this service
What's New
Service Overview
What Is DIS?
Application Scenarios
Functions
Billing
Permissions Management
Related Services
Restrictions
Basic Concepts
Regions and AZs
Quota Description
Getting Started
General Procedure
Step 1: Creating a DIS Stream
Step 2: Preparing a DIS Application Development Environment
Step 3: Sending Data to DIS
Step 4: Obtaining Data from DIS
Obtaining Authentication Information
Interconnecting with OBS
Creating an IAM Agency
Practices
User Guide
IAM Permissions Management
Creating a User and Granting Permissions
Getting Started
General Procedure
Step 1: Creating a DIS Stream
Step 2: Preparing a DIS Application Development Environment
Step 3: Sending Data to DIS
Step 4: Obtaining Data from DIS
Obtaining Authentication Information
Interconnecting with OBS
Creating an IAM Agency
Managing DIS Streams
Listing DIS Streams
Viewing Stream Monitoring Metrics
Changing a Source Data Type
Managing a Source Data Schema
Managing Stream Tags
Managing Applications
Managing Authorization
Debugging Streams
Scaling Up/Down a Stream
Deleting a Stream
Using DIS
Checking and Configuring DNS Information
Uploading Data by Using Agent
DIS Agent Overview
Preparing for Installing DIS Agent
Installing DIS Agent
Configuring DIS Agent
Starting DIS Agent
Testing DIS Agent
Stopping DIS Agent
Using DIS Flume Plugin to Upload and Download Data
DIS Flume Plugin Overview
Preparing for Installing a DIS Flume Plugin
Installing a DIS Flume Plugin
Configuring the DIS Flume Plugin
Testing a DIS Flume Plugin
(Optional) Uninstalling a DIS Flume Plugin
Using a DIS Logstash Plugin to Upload and Download Data
DIS Logstash Plugin Overview
Preparing for Installing a DIS Logstash Plugin
Installing DIS Logstash Plugin Online
Installing DIS Logstash Plugin Offline
Configuring the DIS Logstash Plugin
Testing the DIS Logstash Plugin
(Optional) Uninstalling the DIS Logstash Plugin
Using Kafka Adapter to Upload and Download Data
Kafka Adapter Overview
Preparations
Uploading Data
Consumption Modes
Consumption Offset
Adaptation to the Native KafkaConsumer API
Using SparkStreaming SDK to Download Data
DIS Spark Streaming Overview
Preparing a DIS Spark Streaming Environment
Customizing a SparkStreaming Job
Using a DIS Flink Connector to Upload and Download Data
DIS Flink Connector Overview
Preparing a DIS Flink Connector Environment
Customizing a Flink Streaming Job
Managing a Dump Task
Creating a Dump Task
Dumping Data to OBS
Managing Enterprise Projects
Notifying Events
Event Notification Overview
Setting Notification
Viewing Events
Using Cloud Eye to Monitor DIS
Supported Metrics
Setting Alarm Rules
Querying Metrics
Best Practices
Collecting Incremental Log Data of Driving Behavior
API Reference
Before You Start
Overview
Concepts
API Overview
Calling APIs
Making an API Request
Authentication
Response
Application Example
API Description
Stream Management
Creating a Stream
Querying Streams
Deleting a Stream
Querying Details of a Stream
Changing Partition Quantity
Updating Stream Information
Adding Permission Policies
Querying Permission Policies
App Management
Creating a Consumption App
Querying Apps
Deleting an App
Querying App Details
Querying App Consumption Status
Checkpoint Management
Submitting Checkpoints
Querying Checkpoint Details
Deleting a Checkpoint
Data Management
Uploading Data
Downloading Data
Obtaining Data Cursors
Dump Task Management
Adding OBS Dump Tasks
Querying Dump Tasks
Deleting a Dump Task
Querying Dump Task Details
Starting Dump Tasks
Pausing Dump Tasks
Monitoring Management
Querying Stream Monitoring Data
Querying Partition Monitoring Data
Tag Management
Adding Tags to a Specified Stream
Querying Tags of a Specified Stream
Deleting Tags of a Specified Stream
Adding Resource Tags
Querying All Tags of a Specified Region
Using Tags to Filter Resources (Streams)
Deleting Resource Tags
Appendix
Error Codes
Status Codes
Obtaining a Project ID
SDK Reference
Overview
What Is DIS SDK
Content Navigation
Related Resources
SDK Download
Compatibility
How Do I Check Software Package Integrity?
Enabling DIS
Enabling DIS
Creating a DIS Stream
Obtaining Authentication Information
Getting Started with SDK
Using the Java SDK
Preparing the Environment
Configuring a Sample Project
Initializing a DIS SDK Client Instance
Creating a Stream
Creating a Dump Task
Updating a Dump Task
Deleting a Dump Task
Querying a Dump Task List
Querying Dump Details
Deleting a Stream
Querying a Stream List
Querying Stream Details
Downloading Streaming Data
Uploading Streaming Data
Obtaining the Data Cursor
Creating an Application
Deleting an Application
Adding a Checkpoint
Querying a Checkpoint
Changing Partition Quantity
Using Kafka Adapter to Upload and Download Data
Kafka Adapter Overview
Preparations
Uploading Data
Consumption Modes
Consumption Offset
Adaptation to the Native KafkaConsumer API
Using the Python SDK
Preparing the Installation Environment
Configuring a Sample Project
Initializing a DIS SDK Client Instance
Creating a Stream
Creating a Dump Task
Deleting a Stream
Deleting a Dump Task
Querying a Stream List
Querying a Dump Task List
Querying Stream Details
Querying Dump Details
Uploading Streaming Data in JSON Format
Uploading Streaming Data in Protobuf Format
Downloading Streaming Data
Creating an Application
Deleting an Application
Viewing Application Details
Querying an Application List
Adding a Checkpoint
Querying a Checkpoint
Changing Partition Quantity
Obtaining a Data Cursor
Error Codes
DIS Server-Side Error Codes
Change History
FAQs
General Questions
What Is DIS?
What Is a Partition?
What Can I Do with DIS?
What Advantages Does DIS Have?
Which Modules Do DIS Have?
How Do I Create a DIS Stream?
What Is the Difference Between Storing Data into DIS and Dumping Data Elsewhere?
How Do I Check Software Package Integrity?
How Do I Send and Retrieve Data Using DIS?
What Is Data Control?
How Do I Minimize the DIS Agency Permissions?
Dump Questions
How Does DIS Dump Data to a Specific Column of DWS?
How Does a Schema Support Default Fields or NULL Fields?
How Do I Access DIS Through a Direct Connect Connection?
How Do I Distinguish Different Types of Data When Accessing Data in Streams?
DIS Agent Questions
How Can I Configure Agent to Listen To Multiple Directories or Files?
How Can I Configure Recursive Listening for Directories on DIS Agent?
How Can I Configure DIS Agent?
How Do I Use Agent to Encrypt AK/SK?
General Reference
Glossary
Service Level Agreement
White Papers
Endpoints
Permissions