MongoDB Combine Function

Function Overview

The combine() function is used to process advanced fields in the MongoDB Reader plugin. This function migrates fields that are explicitly specified in field mapping and combines those that are not explicitly configured in MongoDB into a unified JSON string.

How It Works

When the combine() function is used in field mapping, DataArts Migration performs the following operations:

1. Migrate the fields that are explicitly specified in configuration.

2. Combine all the fields that are not explicitly specified in the document into a JSON object.

Example

Assume that the structure of the source MongoDB document is as follows:

{
  "_id": "user_001",
  "username": "john_doe",
  "email": "john@example.com",
  "age": 30,
  "address":
 {
    "city": "beijing",
    "district": "chaoyang"
  },
  "preferences":
 {
    "theme": "dark",
    "language": "zh-CN"
  },
  "last_login": "2024-01-15T10:30:00Z",
  "login_count": 142
}

The structure of the destination MySQL table is as follows:

CREATE TABLE `combine_test`
 (
  `id` varchar(100) DEFAULT NULL,
  `username` varchar(100) DEFAULT NULL,
  `email` varchar(100) DEFAULT NULL
,  `age` int(11) DEFAULT NULL,
  `address` varchar(100) DEFAULT NULL,
  `combine` text
) 
ENGINE=InnoDB DEFAULT CHARSET=utf8 

The _id, username, email, age, and address fields in the source document will be migrated independently through normal field mapping. All the other fields in the document will be combined into a JSON string using the combine() function and migrated to the combine field in the destination database.

To implement the preceding migration logic, you can configure the data migration job as follows:

Add a custom field to in field mapping, and set the key to combine and the value to the combine() function. In this way, all the fields that are not explicitly mapped are automatically aggregated into a JSON object and written to the combine column of the destination table.

Figure 1 Parameter settings

After the ETL task is configured and executed, data is migrated based on the preset mapping rule and written to the destination MySQL database.

Figure 2 Data written successfully