Integrating Message Data into Huawei Cloud Astro Zero Through the Data Access Function

Prerequisites

• You have created parking as the internal event into which received data is converted. For details, see Creating a Custom Event.
• The authentication information provided by Huawei Cloud Astro Zero has been configured on the third-party external data source side. For details, see the third-party document.
• If the ROMA Connect data source is connected, set ROMA Connect instance information in the platform by referring to Configuring the ROMA Connect Information.

Data Access

If any configuration data (including the configuration information of all diagram elements) of data access is modified in the development environment, you can configure whether to overwrite the data access configuration with the same name in other environments during packing.

1. Log in to the application designer by referring to Logging In to the Application Designer.
2. In the navigation pane, choose Integrations.
3. Click + next to Message Access. The page for creating a message access is displayed.
4. Click Create New. The page for creating a data access task is displayed.
5. Choose Elements > Source on the right, drag required diagram elements to the canvas on the left, configure basic information about the external data source, and click Save.
   • OneNET
     You can use OneNET to quickly configure HTTP messages and authentication settings, and receive messages from OneNet.

     Figure 1 OneNET configuration information page
     

     Figure 2 Defining metadata
     

     Table 1 Parameters for adding OneNET

     Area	Parameter	Description
     Configuration Information	Label	Label of OneNET, which is displayed on the page. Value: 1–64 characters.
     	Name	Unique ID of OneNET in the system, which cannot be changed after being created. The value must start with a letter and can contain only letters, digits, and underscores (_). It cannot end with an underscore (_).
     	Token	Data signature, that is, token, provided by OneNET.
     	Packet Encrypted	Indicates whether to encrypt the transmitted packets. If this option is selected, the packets are encrypted. If encryption is required, you need to set Aeskey.
     	Aeskey	Decryption key, which is provided by OneNET.
     Defining Metadata	MetaData	Defines OneNET metadata. If this parameter is set to empty {}, you can only transfer raw data but cannot use interceptors to process data. You can click Format Json to format the metadata through JSON. { "notifyType": "STRING", "requestId": "STRING", "timestamp": "STRING", "eventTime": "datetime", "deviceId": "STRING", "gatewayId": "STRING", "deviceService": { "name": "STRING" }, "service": { "serviceType": "STRING", "serviceId": "STRING", "body": "OBJECT", "data": { "battery_low": "number", "light": "STRING" } } } During data interconnection, the OneNET platform of China Mobile IoT performs authentication based on the interconnection standard provided by China Mobile IoT. The MD5 digest is obtained based on the configured token, random number in the packet, and packet content, and is compared with the digest in the packet, if they are the same, the packet is considered as a normal packet. Currently, China Mobile IoT supports only the MD5 algorithm to obtain data digests. The OneNET platform can encrypt the packet content before reporting it. Huawei Cloud Astro Zero must use the same encryption algorithm as the OneNET platform for decryption, and the obtained encryption vector must be the same as that of the OneNET platform. Therefore, the first 16 bits of the Aeskey (decryption key) are used as the encryption vector.

   • IoT Device Access (IoTDA)
     IoT Device Access allows you to quickly configure HTTP messages and authentication settings, and receive messages sent by IoTDA.

     Figure 3 IoTDA configuration information
     
     Figure 4 Defining metadata
     

     Table 2 Parameters

     Area	Parameter	Description
     Configuration Information	Label	Label of the data source device connected to IoTDA, which is displayed on the page. Value: 1–64 characters.
     	Name	Unique ID of a data source device connected to IoTDA. The value cannot be changed after being created. The value must start with a letter and can contain only letters, digits, and underscores (_). It cannot end with an underscore (_).
     	Format	Format of data reported by the device. The data transmission protocol can be HTTP or HTTPS, which is determined by IoTDA.
     	Authentication Mode	Set the authentication mode, which can be certificate authentication, password authentication, or none authentication.
     	Access Key	Click Generate AK/SK. The value is automatically generated. Save the generated access key to the local host. Developers need to use the access key to generate the authorization attribute on the device. This parameter is displayed only when Authentication Mode is set to Password.
     	Secret Key	Click Generate AK/SK. The value is automatically generated. Save the generated secret key to the local host. Developers need to use the secret key to generate the authorization attribute on the device. This parameter is displayed only when Authentication Mode is set to Password.
     Defining Metadata	MetaData	Defines IoTDA metadata. If this parameter is set to empty {}, you can only transfer raw data but cannot use interceptors to process data. You can click Format Json to format the metadata through JSON. { "notifyType": "STRING", "requestId": "STRING", "timestamp": "STRING", "eventTime": "datetime", "deviceId": "STRING", "gatewayId": "STRING", "deviceService": { "name": "STRING" }, "service": { "serviceType": "STRING", "serviceId": "STRING", "body": "OBJECT", "data": { "battery_low": "number", "light": "STRING" } } } The following is an example of valid data to be reported: { "notifyType": "deviceDataChanged", "requestId": "45678", "timestamp":"1900012929922992", "eventTime":"20151212T121212Z", "deviceId":"8b3979fc-b072-433b-b3f6-673072e1bc04", "gatewayId":"XXX", "deviceService":{"name":"buttery"}, "service": { "serviceType": "", "serviceId": "", "data":{ "battery_low":444, "light":"99", "authorization":"Bingo bDTKFSGi:ZwGnN+bb*****************************TjFRMA=" } } } authorization is generated based on the AK/SK, reported URL, and body data.

   • ROMA Connect
     MQS Source enables you to quickly configure MQS subscription parameters and receive messages published by MQS.

     Figure 5 ROMA Connect configuration information page
     
     Figure 6 Defining MQS metadata
     

     Table 3 Parameters for adding MQS

     Area	Parameter	Description
     Configuration Information	Label	Label of a data source, which is displayed on the page. Value: 1–64 characters.
     	Name	Unique ID of a data source in the system. The value cannot be changed after the data source is created. The value must start with a letter and can contain only letters, digits, and underscores (_). It cannot end with an underscore (_).
     	App ID	Application ID of the MQS platform. An application ID can reference a system parameter defined on the platform. The reference position is not limited. For example, !roma_app_prefix}other_roma_app_id.
     	App Secret	Request key of the MQS platform application.
     	Topic	MQS topic. A topic can reference a system parameter defined on the platform. The reference position is not limited. For example, {!roma_topic_prefix}other_roma_topic.
     	MQS Configuration > Tags	Tags of subscription messages, which can be used to specify the type of messages to be consumed. The default value * indicates that all types of messages are consumed. You can enter multiple tags and separate them with vertical bars (||), for example, tag1 || tag2 || tag3.
     	Channel Encrypted	Whether to enable data encryption. If this parameter is selected, the TLS protocol is used for MQS data transmission. This parameter is selected by default.
     	Consumer Position	If this parameter is selected, the data access task is stopped. When the task is restarted, all messages sent by ROMA Connect during the stop and restart period will be discarded. If the interval between stopping and restarting data access is less than 10 minutes, messages may not be lost. This parameter is deselected by default.
     Defining Metadata	MetaData	Defines MQS metadata. If this parameter is set to empty {}, you can only transfer raw data but cannot use interceptors to process data. You can click Format Json to format the metadata through JSON. { "notifyType": "STRING", "requestId": "STRING", "timestamp": "STRING", "eventTime": "datetime", "deviceId": "STRING", "gatewayId": "STRING", "deviceService": { "name": "STRING" }, "service": { "serviceType": "STRING", "serviceId": "STRING", "body": "OBJECT", "data": { "battery_low": "number", "light": "STRING" } } }

   • Kafka
     Data is collected from the internal or external Kafka of the platform. The data collected from the internal Kafka cannot be written back to the internal Kafka.

     Figure 7 Adding Kafka
     

     Table 4 Parameters

     Parameter	Description
     Label	Label of a data source, which is displayed on the page. Value: 1–64 characters.
     Name	Unique ID of a data source in the system. The value cannot be changed after the data source is created. The value must start with a letter and can contain only letters, digits, and underscores (_). It cannot end with an underscore (_).
     External Kafka	Astro Zero can collect data from the internal or external Kafka of the platform. The data collected from the internal Kafka cannot be exported to the internal Kafka. If this option is selected, data is collected from the external Kafka. Deselected: Data is collected from the internal Kafka of the platform.
     Broker List	This parameter is available only when External Kafka is selected. Address of the Kafka cluster for receiving events. Multiple node addresses can be separated by commas (,). The format is as follows: IP address of Kafka node 1:Port number of Kafka node 1,IP address of Kafka node 2:Port number of Kafka node 2, ... To obtain the IP address and port number of the Kafka, you can log in to each Kafka node and check the value of listeners in the server.properties file in ${KAFKA_HOME}/config. For example, 10.10.10.1:9091,10.10.10.2:9092.
     Topic	This parameter is available only when External Kafka is selected. Kafka topic to which the collected data belongs. A topic is a class name for storing and publishing messages. You can use topics to transfer and process events by category.
     Security Protocol	This parameter is available only when External Kafka is selected. If this parameter is set to SASL_SSL, the username and password are required for authentication when you access the external Kafka, and data is encrypted before transmission.
     Username	Username for accessing the external Kafka. This parameter is available only when External Kafka is selected.
     Password	Password for accessing the external Kafka. This parameter is available only when External Kafka is selected.
     Defining Metadata	This tab is available only when External Kafka is selected. Defines MQS metadata. If this parameter is set to empty {}, you can only transfer raw data but cannot use interceptors to process data. You can click Format Json to format the metadata through JSON. { "notifyType": "STRING", "requestId": "STRING", "timestamp": "STRING", "eventTime": "datetime", "deviceId": "STRING", "gatewayId": "STRING", "deviceService": { "name": "STRING" }, "service": { "serviceType": "STRING", "serviceId": "STRING", "body": "OBJECT", "data": { "battery_low": "number", "light": "STRING" } } }
     Event	Event associated with data collected from the Kafka of the platform. This parameter is available only when External Kafka is not selected.

6. Drag the required interceptor from Elements > Interceptor to the page on the left to intercept the source data, and click the save button.
   • Select: Data selector, which can select some data from the source data for sending.
     Figure 8 Configuration
     

     Table 5 Parameters

     Parameter	Description
     Label	Label name, which is displayed on the page. Value: 1–64 characters.
     Name	Unique ID of Select in the system, which cannot be changed after being created. The value must start with a letter and can contain only letters, digits, and underscores (_). It cannot end with an underscore (_).
     Field Selector	All metadata transferred from external data sources is displayed on the left. Select required data fields and click to move the selected data to the output fields on the right.

   • Assign: This element assigns the object data in the cache, transferred external data source data, constant data, or the value of the data expression to a parameter.
     Figure 9 Assigner configuration
     

     Table 6 Parameters

     Parameter	Description
     Label	Label name of the assigner, which is displayed on the page. Value: 1–64 characters.
     Name	Unique identifier of the assigner in the system. The value must start with a letter and can contain only letters, digits, and underscores (_). It cannot end with an underscore (_).
     Assignment	Click Add to assign a value. AssignTo: parameter to which a value is assigned. Source: Set the source data. Cache: object data in the cache. You need to click the button for adding cache in setting to add the data in advance. reference: metadata of the transferred external data source. formula: expression value. The format is XXX (data source parameter). constant: constant in the format of constant value.

   • Filter: This element filters external metadata based on configured filter criteria.
     Figure 10 Filter configuration
     

     Table 7 Parameters

     Parameter	Description
     Label	Label name of the filter, which is displayed on the page. Value: 1–64 characters.
     Name	Unique ID of Filter in the system, which cannot be changed after being created. The value must start with a letter and can contain only letters, digits, and underscores (_). It cannot end with an underscore (_).
     Condition Configuration	Set filter criteria. You can click Add to add multiple filter criteria.

7. In the Elements > Channel area, drag the required channel to the area under OneNet in the canvas on the left, configure basic information about the channel, and click the save button.
   Memory indicates a common data channel. When there is a large amount of data, you can select File.

   Figure 11 Memory channel configuration
   

   Table 8 Parameters for adding a memory channel

   Parameter	Description
   Label	Label name of a memory channel, which is displayed on the page. Value: 1–64 characters.
   Name	Unique ID of Memory in the system, which cannot be changed after being created. The value must start with a letter and can contain only letters, digits, and underscores (_). It cannot end with an underscore (_).
   Advanced Config	After this function is enabled, you can customize the channel capacity and maximum transmission volume. By default, this function is disabled and the default configuration is used. Capacity: maximum number of channel data records. Transaction Capacity: maximum number of data records sent to a single output source.

8. In the Elements > Sink area, drag the required element to the bottom of the canvas on the left, configure basic information about the element, and click the save button.
   • If the receiver is Kafka, the configuration is as follows:
     You can drag multiple Kafka elements to the canvas to set multiple event receivers.

     Figure 12 Data receiver configuration
     

     Table 9 Parameters

     Area	Parameter	Description
     Basic Information	Label	Label name of the data receiver Kafka, which is displayed on the page. Value: 1–64 characters.
     	Name	Unique ID of the data receiver Kafka in the system, which cannot be changed after being created. The value must start with a letter and can contain only letters, digits, and underscores (_). It cannot end with an underscore (_).
     	Event	Sets the output event.
     	Send to External	Whether to send data to the external Kafka. This function is disabled by default, indicating that data is exported to Kafka connected to Huawei Cloud Astro Zero. Broker List: address of the Kafka cluster for receiving events. Multiple node addresses can be separated by commas (,). The format is as follows: IP address of Kafka node 1:Port number of Kafka node 1,IP address of Kafka node 2:Port number of Kafka node 2, ... To obtain the IP address and port number of the Kafka, you can log in to each Kafka node and check the value of listeners in the server.properties file in ${KAFKA_HOME}/config. Example: 10.136.14.56:9092 The Kafka cluster that receives events does not need to be connected to Huawei Cloud Astro Zero. Topic: Kafka topic for receiving event data. For example, __BINGO_PROD_SYS_TOPIC. Whether to enable SASL_SSL: This parameter is displayed only when data is sent to an external Kafka. Whether to enable SASL_SSL authentication. If this function is enabled, the username and password are required for authentication when you access the external Kafka service, and data is encrypted before transmission.
     Configuration	Broker List	Enter the addresses of the Kafka cluster. Use commas (,) to separate multiple IP addresses, for example, ip1:port1,ip2:port2,ip3:port3. Do not enter the Kafka cluster address of the platform.
     	Topic	A topic is a class name for storing and publishing messages. You can use topics to transfer and process events by category. Do not enter the internal theme of the platform.
     	Security Protocol	If this parameter is set to SASL_SSL, the username and password are required for authentication when you access the external Kafka, and data is encrypted before transmission.
     	Kafka username	Username used by Kafka for SASL authentication.
     	Kafka password	Password used by Kafka for SASL authentication.
     Event Attribute	Partition	Whether to use parameter fields in events as Kafka message headers for sending events to different Kafka partitions during event distribution.
     	Attribute/Value	Configure the mapping between the transmitted data and the parameters defined in the event.

   • If the receiver is ROMA Connect, the configuration is as follows:
     Figure 13 Data receiver configuration
     

     Table 10 Parameters

     Description	Parameter
     Label	Label name of the recipient, which is displayed on the page. Value: 1–64 characters.
     Name	Unique ID of the recipient in the system. The ID cannot be changed after being created. The value must start with a letter and can contain only letters, digits, and underscores (_). It cannot end with an underscore (_).
     App ID	Application ID of the MQS platform. The application ID can reference a system parameter defined on the platform. The reference position is not limited, for example, !roma_app_prefix}other_roma_app_id.
     App Secret	Request key of the MQS platform application.
     Topic	MQS topic. A topic can reference a system parameter defined on the platform. The reference location is not limited, for example, {!roma_topic_prefix}other_roma_topic.
     Tags	Tags of subscription messages, which can be used to specify the type of messages to be consumed. The default value * indicates that all types of messages are consumed. You can enter multiple tags and separate them with vertical bars (||), for example, tag1 || tag2 || tag3.
     Encrypted	Whether data is encrypted for transmission.

9. Click in the upper part of the data access page, set the data access label and name, and click the save button.

   Table 11 Parameters

   Parameter	Description
   Label	Label name of the data access, which is displayed on the page.
   Name	Unique ID of the data access in the system, which cannot be changed after being created. The naming requirements are as follows: The value cannot exceed 64 characters, including the prefix namespace. To prevent duplicate data names among different tenants, each tenant must define a unique namespace when first creating an application. A tenant can create only one namespace. After being created, the namespace cannot be modified. Start with a letter and can contain only letters, digits, and underscores (_). Do not end with an underscore (_).
   Description	Enter the data access description as required. The description can contain a maximum of 255 characters.

10. Click in the upper part of the data access page to enable data access.
11. Click to start the data access.

    If the Kafka receives the data, the setting is successful. In the upper part of the data access page, click to stop the running data access service.

12. Copy the data push address in the execution success dialog box that is displayed and save the address.
13. Configure the data push address provided by Astro Zero on the third-party external data source side. For details, see the third-party document.
14. (Optional) Perform debugging or fault locating.

    In the upper part of the data access page, click to download logs and view the logs to locate the fault. If the data source or output source configuration is incorrect, you can download logs to locate the fault.

15. (Optional) Trace messages.

    On the right of the Message Access page, click the Message Tracing tab and click Start to start message tracing. The message tracing function is automatically disabled after a period (30 minutes by default).

    Figure 14 Starting message trace
    

    After message tracing is started, you can view the processing status of reported packets in real time.

    Figure 15 Checking the packet processing
    

Handling Exceptions

When the data source type is set to ROMA Connect, the MQS platform cannot receive messages after data access is configured. The EventTrigger configured by the service does not start the flow.

• Symptom: The ROMA Connect message track is gray. The possible cause is that the MQS service address is incorrectly configured or not configured.

  Solution: Check whether the roma-server parameter of the iotgateway container is configured. If no, set this parameter or contact O&M personnel to add the correct roma-server address (port number: 9776). Ensure that the network and MQS server are normal and compatible with the client version.

• Symptom: The ROMA Connect message track is red, which may be caused by the incorrect source definition of MQS.

  Solution: On the ROMA Connect platform, check whether the application ID is the same as the corresponding key and whether the application has subscribed to the topic. If the configuration is incorrect, the authentication fails.

• Symptom: The ROMA Connect message track is green, indicating that MQS successfully receives and forwards the message to Kafka. Huawei Cloud Astro Canvas also receives the message, but the flow may fail to be started.

  Solution: View the flume.log file of the iotgateway to check whether the MQS successfully receives the message. Check whether the message is successfully converted through kafka-console-cumer. Check whether the platform receives the event through the log tracing function of Huawei Cloud Astro Zero or the background logs of Huawei Cloud Astro Zero (the log keyword is OnEvent). If the corresponding flow or event log does not exist, the flow is not started. Contact Huawei technical support.