Data Forwarding

Overview

Data forwarding is seamlessly interconnected with other Huawei Cloud services to implement full-stack services for device data storage, computing, and analysis. IoTDA can forward data to the following services:

Data Ingestion Service (DIS): The platform works with DIS for efficient data collection, transmission, and distribution. You can download data by using the SDKs or APIs provided by DIS. You can also dump data to other cloud services Object Storage Service (OBS), MapReduce Service (MRS), Data Warehouse Service (DWS), and Data Lake Insight (DLI) for subsequent data processing like data storage and analysis.
Distributed Message Service (DMS) for Kafka: The platform works with DMS for Kafka to provide message queues for device data. Kafka is distributed messaging middleware that features high throughput, data persistence, horizontal scalability, and stream data processing. It adopts the publish-subscribe pattern and is widely used for log collection, data streaming, online/offline system analytics, and real-time monitoring. You can apply for instances as required and customize partitions and replicas for topics in the instances. The instances can be used right out of the box, taking off the deployment and O&M workload for you so that you can focus on developing your services.
OBS: The platform works with OBS for persistent device data storage. OBS is an object-based massive storage service that provides secure, reliable, and low-cost data storage. It can archive, back up, and store data reported by devices. OBS can work with Cloud Stream Service (CS) to analyze streaming data in real time. The analysis result can be used for data visualization in other cloud services or third-party applications.
Third-party application (HTTP push): An application calls the platform APIs Creating a Rule Triggering Condition, Creating a Rule Action, and Modifying a Rule Triggering Condition to configure and activate rules. The platform pushes the changed device service details and management details to the application with a specified URL. (Service details include device lifecycle management, device data reporting, device message status, and device status. Management details include software/firmware upgrade status and result.)
AMQP message queue: Data can be forwarded without interconnecting with other Huawei Cloud services. An application calls the platform APIs Creating a Rule Triggering Condition, Creating a Rule Action, and Modifying a Rule Triggering Condition to configure and activate rules. After a connection is established between the AMQP client and the platform, the platform pushes the changes to a specified AMQP message queue based on the type of data subscribed. For details, see AMQP Subscription/Push.
Log Tank Service (LTS): The platform works with LTS for log collection, analysis, query, storage, and visualization. You can use LTS for efficient device O&M, service trend analysis, security audits, and monitoring. An application calls the platform APIs Creating a Rule Triggering Condition, Creating a Rule Action, and Modifying a Rule Triggering Condition to configure and activate rules. The platform pushes service run logs of MQTT devices to the specified log group and log stream.
FunctionGraph: hosts event-driven functions in a serverless context while ensuring high availability, high scalability, and zero maintenance. All you need to do is write your code and set the execution conditions. An application calls the platform APIs Creating a Rule Triggering Condition, Creating a Rule Action, and Modifying a Rule Triggering Condition to configure and activate rules. You can process data reported by devices based on specified functions and forward the processed data to other Huawei Cloud services, applications, message middleware, or databases.
MRS: Kafka is a distributed, partitioned, replicated message publishing and subscription system. Its features are similar to those of the Java Message Service (JMS), but the design is completely different. Kafka provides features such as message persistence, high throughput, distribution, multi-client support and real-time processing. It applies to both online and offline message consumption, such as regular message collection, website activeness tracking, aggregation of statistical system operation data (monitoring data), and log collection. These scenarios engage large amounts of data collection for Internet services.
GaussDB(for Influx): reads and writes time series data in high concurrency scenarios and stores the data with compression algorithms. It allows you to query data using SQL-like statements and supports multi-dimension aggregate computing and visual analysis. It is widely used to monitor resources, services, IoT devices, and industrial production processes, evaluate production quality, and trace faults. GaussDB(for Influx) can achieve very high throughput and concurrency, so it can handle a large number of connections in a very short period of time, making it an excellent choice for IoT applications.
GaussDB(for MySQL): is a stable, reliable, and scalable cloud database service. An application calls the platform APIs Creating a Rule Triggering Condition, Creating a Rule Action, and Modifying a Rule Triggering Condition to configure and activate rules. The platform stores the forwarded data to the MySQL database based on the configured rule conditions and actions.

Figure 1 Data forwarding
Click to enlarge

To further explore data forwarding, see Forwarding Device Data to OBS for Long-Term Storage.

Procedure

Access the IoTDA service page and click Access Console.
In the navigation pane, choose Rules > Data Forwarding, and click Create Rule in the upper right corner.

Set forwarding data and click Create Rule.

Parameter	Description
Rule Name	Specify the name of a rule to create.
Description	Describe the rule.
Data Source	Device: Device information, such as device addition, deletion, and update, will be forwarded. When Data Source is set to Device, quick configuration is not supported. Device property: A property value reported by a device in a resource space will be forwarded. Click Quick Configuration on the right and select the product, property, and service data to forward. Device message: A message reported by a device in a resource space will be forwarded. Click Quick Configuration on the right and select data of a specified topic to forward. Select the product to which the topic belongs and enter the topic name. You can use a custom topic on the product details page or a preset topic. Device message status: The status of device messages exchanged between the device and platform will be forwarded. For details on the device message status, see Message Delivery Status. When Data Source is set to Device message status, quick configuration is not supported. Device status: The status change of a directly or an indirectly connected device in a resource space will be forwarded. Click Quick Configuration on the right to forward information about devices whose status is Online, Offline, or Abnormal to other services. For details on the status of devices directly connected to the IoT platform, see Device Status. Batch task: The batch task status will be forwarded. When Data Source is set to Batch Task, quick configuration is not supported. Product: Product information, such as product addition, deletion, and update, will be forwarded. When Data Source is set to Product, quick configuration is not supported. Asynchronous command status of the device: Status changes of asynchronous commands to devices using LwM2M over CoAP will be forwarded. For details on the asynchronous command status of devices, see Command Execution Status. When Data Source is set to Asynchronous command status of the device, quick configuration is not supported. Run log: Service run logs of MQTT devices will be forwarded. When Data Source is set to Run log, quick configuration is not supported.
Trigger	After you select a data source, the platform automatically matches trigger events.
Resource Space	You can select a single resource space or all resource spaces. If All resource spaces is selected, quick configuration is not supported.

Click the Set Forwarding Target tab, and then click Add to set a forwarding target.

Select Data Ingestion Service (DIS), Distributed Message Service (DMS) for Kafka, Object Storage Service (OBS), ROMA Connect, Third-party application (HTTP push), AMQP queue, FunctionGraph, Log Tank Service (LTS), GaussDB(for Influx), RDS for MySQL, or MapReduce (MRS).

Parameter	Value	Description
Forwarding Target	Data Ingestion Service (DIS)	Region: Select the region of the service to which data will be forwarded. If you are not authorized to access the service in this region, perform authorization as required. Stream Homing: You can select either of the following: In-house stream: Select a stream. If no DIS stream is available, create one on the DIS console. Delegated by others: You can use DIS streams authorized by other users. Obtain the stream ID from the DIS console.
	Distributed Message Service (DMS) for Kafka NOTE: Data can be forwarded only to Kafka premium instances. You need to enable automatic topic creation.	Region: Select the region of the service to which data will be forwarded. If you are not authorized to access the service in this region, perform authorization as required. Connection Address: Obtain the connection address by following the instructions provided in Accessing a Kafka Instance with SASL. Basic and standard instances support only access to Kafka premium instances over the Internet. Enterprise instances support access to Kafka premium instances over a private network. Topic: Customize a topic. Data Transmission Encryption: If data transmission encryption is enabled, enter the SASL username and password that you entered when buying a Kafka instance.
	Object Storage Service (OBS)	Region: Select the region of the service to which data will be forwarded. If you are not authorized to access the service in this region, perform authorization as required. OBS Bucket: Select a bucket as required. If no OBS bucket is available, create one on the OBS console. Custom Directory: Separate different directory levels are by slashes (/). The directory cannot start or end with a slash (/) or contain two or more consecutive slashes (/).
	ROMA Connect NOTE: Currently, only data of instances of Enterprise Edition can be forwarded to ROMA Connect.	Connection Address: Enter the connection address of the MQS. For details, see Preparations. Account Name/Password: Enter the username and password for logging in to the ROMA console. For details, see Preparations. Topic: Customize a topic.
	Third-party application (HTTP push)	You can use HTTP or HTTPS to push messages. For details on how to set parameters, see HTTP/HTTPS Subscription/Push.
	AMQP message queue	Message Queue: Select the queue to which messages are to be pushed. If no queue is available, create one. For details on the restrictions on message queue names, see Configuring AMQP Server Subscription.
	Distributed Message Service (DMS) for RocketMQ NOTE: Currently, only data of instances of Enterprise and Standard editions can be forwarded to DMS for RocketMQ.	Connection Address: Public access must be enabled for RocketMQ instances. Access the DMS for RocketMQ console to view the RocketMQ instance details. The value of Metadata Address (Public) is the connection address. SSL: If SSL is enabled for the RocketMQ instance, enable SSL here. Otherwise, disable SSL. Topic: Select a topic on the Topics page of the RocketMQ instance. Username and Password: Access the DMS for RocketMQ console to obtain the username and password on the Users page. NOTE: To use the user management function of the RocketMQ instance, ACL must be enabled. For details, see Step 1: Enable ACL. Ensure that the account has the permission to publish topics. For details, see Creating a User.
	FunctionGraph NOTE: Currently, only data of instances of Enterprise and Standard editions can be forwarded to FunctionGraph.	Function Name: Select the name (latest version) of the function to be called. Currently, cross-region function calling is not supported. If no function is available, create one.
	GaussDB(for Influx) NOTE: Currently, only data of instances of Enterprise and Standard editions can be forwarded to GaussDB(for Influx).	Database Instance Address: Enter the address for connecting to the InfluxDB instance. IoTDA Enterprise edition instances can connect to InfluxDB using private network IP addresses, while Standard edition instances support only public network connection. For details, see Connection Methods. Database Name: Enter a database name. If no database exists, go to the GaussDB(for Influx) console to create a database. Access Account and Access Password: Access the GaussDB(for Influx) console to obtain the account and password. For details, see Resetting the Administrator Password. Table: Enter the name of the target table (measurement). If the table does not exist, it will be automatically created. Field Mappings: Forwarding Field: Enter the attribute name of the data to be forwarded. Data to be forwarded is in JSON format. Separate multi-level attribute names with periods (.). For details about the format of data to be forwarded, see Data Transfer APIs. Target Field: Enter the column name of the database.
	RDS for MySQL NOTE: Currently, only data of instances of Enterprise and Standard editions can be forwarded to RDS for MySQL.	Database Instance Address: Enter the address for connecting to the MySQL instance. IoTDA Enterprise edition instances can connect to MySQL using private network IP addresses, while Standard edition instances support only public network connection. For details, see Connection Management. Database Name: Enter a database name. If no database exists, go to the RDS for MySQL console to create a database. Access Account and Access Password: Access the MySQL console to obtain the account and password. For details, see Account Management (Non-Administrator). SSL: Select whether to connect to the database in SSL mode. You are advised to use the SSL mode. If not, security risks may exist during data transmission. To use the SSL mode, configure an SSL connection in the database instance first. Table: Select the name of the table to which data is forwarded. Field Mappings: Forwarding Field: Enter the attribute name of the data to be forwarded. Data to be forwarded is in JSON format. Separate multi-level attribute names with periods (.). For details about the format of data to be forwarded, see Data Transfer APIs. Target Field: Enter the column name of the database.
	MRS Kafka NOTE: Currently, only data of instances of Enterprise Edition can be forwarded to MRS Kafka under the same Virtual Private Cloud (VPC).	Kerberos Authentication: Configure an authentication credential to enable this function. Connection Address: Enter the connection address of MRS Kafka. You can view the IP address of the node where Kafka is deployed in MRS Manager. Topic: Customize a topic. For details, see Managing Kafka Topics.