Creating an MQS-to-Database Integration Task

Overview

A data integration task is performed to convert the structure of device data in MQS and write the converted data into the database of the parking lot management system. The parking lot management system database uses MySQL as an example.

Procedure

1. Connect to the MQS data source.
   1. In the navigation pane on the left, choose Data Sources. In the upper right corner of the page, click Access Data Source.
   2. On the Default tab page, select MQS and click Next.
   3. Enter the configuration information about the MQS data source.

      Table 1 MQS data source configuration

      Parameter	Description
      Name	Enter the name of the MQS data source, which is user-defined.
      Integration Application	Select the integration application created in Registering a Device on ROMA Connect.
      Description	Enter a brief description of the data source.
      Connection Address	Select the MQS intranet address of the current ROMA Connect instance. You can view the MQS intranet address on the Instance Information page of the ROMA Connect console.
      Enable SSL	Set this parameter to Yes if MQS SASL_SSL is enabled and Intra-VPC Plaintext Access is disabled for the ROMA Connect instance. In other cases, set this parameter to No.
      SSL Username/Application Key	Mandatory for Enable SSL set to Yes. Set this parameter to the key of the integration application to which the topic created in Configuring MQS for Device Data Forwarding belongs.
      SSL Password/Application Secret	Mandatory for Enable SSL set to Yes. Set this parameter to the secret of the integration application to which the topic created in Configuring MQS for Device Data Forwarding belongs.

   4. After setting the parameters for the MQS data source, click Check Connectivity to test the data source connection.
      • If the test result is Data source connected successfully, click Create.
      • If the test result is Failed to connect to the data source, check and modify the connection parameters, and click Recheck until the connection is successful.
   5. Click Create.
2. Connect to the MySQL data source.
   1. On the Data Sources page, click Access Data Source in the upper right corner.
   2. On the Default tab page, select MySQL and click Next.
   3. Enter the configuration information about the MySQL data source.

      Table 2 MySQL data source configuration

      Parameter	Description
      Name	Enter the name of the data source, which is user-defined.
      Integration Application	Select the integration application created in Registering a Device on ROMA Connect.
      Description	Enter a brief description of the data source.
      Connection Mode	Select Default.
      Connection Address	Enter the IP address of the MySQL database.
      Port	Enter the port number for accessing the MySQL database. The default port number is 3306. Set this parameter based on the actual database configuration.
      Database Name	Enter the name of the database where the table resides.
      Encoding Format	Enter the encoding format used by the database.
      Timeout Interval (s)	Enter the timeout interval for connecting to the database, in seconds.
      Username	Enter the username used for accessing the database. The user must have the read and write permissions on the database.
      Password	Enter the password used for accessing the database.

   4. After setting the parameters for the MySQL data source, click Check Connectivity to test the data source connection.
      • If the test result is Data source connected successfully, go to the next step.
      • If the test result is Failed to connect to the data source, check and modify the connection parameters, and click Recheck until the connection is successful.
   5. Click Create.
3. Create a data integration task from MQS to MySQL.
   1. In the navigation pane on the left, choose Fast Data Integration > Task Management and click Create Common Task.
   2. On the Create Task page, set the parameters of the data integration task.
      1. Configure basic information about the task.

         Table 3 Basic task configuration

         Parameter	Description
         Task Name	Enter the task name, which is user-defined.
         Description	Enter a brief description of the task.
         Integration Mode	Select the mode of data integration. When MQS is used as the source data source, only real-time tasks are supported. In this case, select Real-Time.
         Tag	Add a tag to classify tasks for quick search. In this practice, you do not need to set this parameter.
         Enterprise Project	Select the enterprise project to which the task belongs. In this example, retain the default value default.

      2. Configure the MQS data source information at the source.

         Table 4 Source information configuration

         Parameter	Description
         Instance	Select the ROMA Connect instance that is being used.
         Integration Application	Select the integration application created in Registering a Device on ROMA Connect.
         Data Source Type	Select MQS.
         Data Source Name	Select the MQS data source that you configured in 1.
         Topic Name	Select the topic created in Configuring MQS for Device Data Forwarding.
         Parse	The data structure of the source MQS is different from that of the destination MySQL database. You need to parse and convert the data. In this example, select Yes.
         Data Root Field	This parameter specifies the path of the upper-layer common fields among all metadata in the data obtained from the source in JSON format. This parameter is not required in this practice.
         Data Type	Select the format of data obtained from the MQS data source. The data format must be the same as the actual data format stored in MQS. The device data stored in MQS is in JSON format. In this practice, select JSON.
         Offset	If you select Latest, the latest message data is integrated.
         Metadata	This parameter specifies the JSON data elements that are obtained from the source and need to be integrated into the destination. Take the data of parking slot sensors as an example. Integrate sensor and parking slot information into the destination database. Alias: deviceId. Type: String. Parsing Path: data.deviceId. Alias: status. Type: String. Parsing Path: data.status.
         Time Zone	Select the time zone used by the MQS data source so that ROMA Connect can identify the data timestamps.

      3. Configure the MySQL data source at the destination.

         Table 5 Destination information configuration

         Parameter	Description
         Instance	Select the ROMA Connect instance that is being used.
         Integration Application	Select the integration application created in Registering a Device on ROMA Connect.
         Data Source Type	Select MySQL.
         Data Source Name	Select the MySQL data source that you configured in 2.
         Table	Select the data table to which the integrated data in the MySQL database will be written. After selecting a data table, click Select Table Field to select the column fields for writing the parking space sensor information and parking status information into the database.
         Batch Number Field	This parameter is not required in this practice.

      4. Configure the data mapping rule from MQS to MySQL.

         Click Automatic Mapping. The mapping rules between the source and destination data fields are automatically created. If the fields in the data tables at the two ends are inconsistent, you need to select the corresponding source fields for the destination fields.

   3. Click Save.
   4. On the Task Management page, click Start on the right of the created data integration task to start the task.