Help Center/ ROMA Connect/ User Guide(new)/ Processors/ Data Mapper
Updated on 2024-11-25 GMT+08:00
View PDF
Share

Data Mapper

The data mapper processor is used to convert the data format. Subsequent nodes can reference the output data from a data mapper using ${payload}.

Configuring Parameters

Parameter

Description

Source Data Source

If the data source is a database, the field information is automatically displayed after a table is selected. In other cases, manually add fields or import a file. Only JSON, XML, and CSV files can be imported.

Destination Data Source

If the data source is a database, the field information is automatically displayed after a table is selected. In other cases, manually add fields or import a file. Only JSON, XML, and CSV files can be imported.

Subsequent nodes can reference the output data of a data mapper processor using ${node ID|payload.xxx}. The node ID is displayed after you click a data mapper node.

Destination - XML Data Example

If destination data source format is XML, Root Tag defaults to root. You can set this parameter to other values but cannot leave it empty.

Data in the demo.statistics table of the source MySQL database:

id

city

uuid1

suzhou

uuid2

shanghai

Mapping result:

<root>
    <custom>
        <key1>uuid1</key1>
        <key2>
            <level2>suzhou</level2>
        </key2>
    </custom>
    <custom>
        <key1>uuid2</key1>
        <key2>
            <level2>shanghai</level2>
        </key2>
    </custom>
</root>

Destination - CSV Data Example

Header: Whether to include the table header of the CSV file in the output. Options are true and false.

Delimiter: Data in each file column is separated by cells, which can be represented as tab characters, spaces, commas, or semicolons.

Scenario 1: Set the destination data source format to CSV, Header to true, and Delimiter to comma.

When a file is imported, if Header is set to true, the first line of the CSV file is the field names.

Data in the demo.statistics table of the source MySQL database:

id

city

uuid1

suzhou

uuid2

shanghai

Mapping result:

ManufacturerId, ProductName
uuid1, suzhou
uuid2, shanghai

Scenario 2: Set the destination data source format to CSV, Header to false, and Delimiter to comma.

When a file is imported, if Header is set to false, the header field of the imported file is not used. The filedN field is used by default.

Data in the demo.statistics table of the source MySQL database:

id

city

uuid1

suzhou

uuid2

shanghai

Mapping result:

uuid1, suzhou
uuid2, shanghai

Source - JSON Data Example

If the source data format is JSON, use the output of the source data source as the import file of the source, and select the encoding format (GB2312, UTF-8, or ASCII) of the file. The system automatically generates fields accordingly.

Set the destination data format to MySQL, and select a table name. The system automatically generates fields in the database table.

(Example) The following composite application calls he open API to list objects in a specified OBS bucket and map the bucket name and object names to corresponding fields in a specified MySQL table via data mapping, and then stores the bucket name and object names to MySQL. In this application, the OBS data serves as the source of data mapping, while MySQL is the destination. The open API calls the OBS connector to list bucket objects as the import file of the remote data source. Alternatively, you can also manually add fields based on the object list.

Content of the imported file:

{

"responseHeaders": {

"bucket-location": "cn-north-7",

"connection": "keep-alive",

"content-length": "504",

"content-type": "application/xml",

"date": "2024-04-24T07:07:43.000+00:00",

"id-2": "32AAAQAAEAABAAAQAAEAABAAAQAAEAABCSbOSbK ncLEgZnxffa8vFPuDgVIt7n9",

"request-id": "0000018F0EEE84734012ACF15A46395E",

"server": "OBS"

},

"originalHeaders": {

"connection": "keep-alive",

"content-length": "504",

"content-type": "application/xml",

"date": "Wed, 24 Apr 2024 07:07:43 GMT",

"server": "OBS",

"x-obs-bucket-location": "cn-north-7",

"x-obs-id-2": "32AAAQAAEAABAAAQAAEAABAAAQAAEAABCSbOSbK+ncLEgZnxffa8vFPuDgVIt7n9",

"x-obs-request-id": "0000018F0EEE84734012ACF15A46395E"

},

"statusCode": 200,

"objectSummaries": [

{

"bucketName": "test0223",

"objectKey": "test0424.txt",

"owner": {

"displayName": null,

"id": "0ab78f256080d2890feac0086234d7c0"

},

"metadata": {

"responseHeaders": {},

"originalHeaders": {},

"statusCode": 0,

"lastModified": "2024-04-24T06:51:46.566+00:00",

"contentLength": 19,

"contentType": null,

"contentEncoding": null,

"contentDisposition": null,

"cacheControl": null,

"contentLanguage": null,

"expires": null,

"etag": "\"0b26e313ed4a7ca6904b0e9369e5b957\"",

"contentMd5": null,

"crc64": null,

"storageClass": "STANDARD",

"webSiteRedirectLocation": null,

"nextPosition": -1,

"appendable": false,

"objectStorageClass": "STANDARD",

"allMetadata": {},

"metadata": {},

"requestId": ""

},

"objectContent": null

}

],

"commonPrefixes": [],

"extendCommonPrefixes": [],

"bucketName": "test0223",

"truncated": false,

"prefix": "",

"marker": "",

"maxKeys": 1000,

"delimiter": null,

"nextMarker": null,

"location": "cn-north-7",

"extenedCommonPrefixes": [],

"objects": [

{

"bucketName": "test0223",

"objectKey": "test0424.txt",

"owner": {

"displayName": null,

"id": "0ab78f256080d2890feac0086234d7c0"

},

"metadata": {

"responseHeaders": {},

"originalHeaders": {},

"statusCode": 0,

"lastModified": "2024-04-24T06:51:46.566+00:00",

"contentLength": 19,

"contentType": null,

"contentEncoding": null,

"contentDisposition": null,

"cacheControl": null,

"contentLanguage": null,

"expires": null,

"etag": "\"0b26e313ed4a7ca6904b0e9369e5b957\"",

"contentMd5": null,

"crc64": null,

"storageClass": "STANDARD",

"webSiteRedirectLocation": null,

"nextPosition": -1,

"appendable": false,

"objectStorageClass": "STANDARD",

"allMetadata": {},

"metadata": {},

"requestId": ""

},

"objectContent": null

}

],

"requestId": "0000018F0EEE84734012ACF15A46395E"

}

Data mapping result:

Parent topic: Processors