From a Common Relational Database
Common relational databases that can serve as the source include GaussDB(DWS), RDS for MySQL, RDS for PostgreSQL, RDS for SQL Server, Dameng, FusionInsight LibrA, PostgreSQL, Microsoft SQL Server, SAP HANA, and MyCAT.
To export data from the preceding databases, configure the source job parameters listed in Table 1.
Category |
Parameter |
Description |
Example Value |
---|---|---|---|
Basic parameters |
Use SQL Statement |
Whether you can use SQL statements to export data from a relational database |
No |
SQL Statement |
When Use SQL Statement is set to Yes, enter an SQL statement here. CDM exports data based on the SQL statement.
NOTE:
|
select id,name from sqoop.user; |
|
Schema/Tablespace |
Name of the schema or tablespace from which data will be extracted. This parameter is displayed when Use SQL Statement is set to No. Click the icon next to the text box to go to the page for selecting a schema or directly enter a schema or tablespace. If the desired schema or tablespace is not displayed, check whether the login account has the permissions required to query metadata.
NOTE:
The parameter value can contain wildcard characters (*), which is used to export all databases whose names start with a certain prefix or end with a certain suffix. The examples are as follows:
|
SCHEMA_E |
|
Table Name |
Name of the table from which data will be extracted. This parameter is displayed when Use SQL Statement is set to No. Click the icon next to the text box to go to the page for selecting the table or directly enter a table name. If the desired table is not displayed, confirm that the table exists or that the login account has the permissions required to query metadata. This parameter can be configured as a macro variable of date and time and a path name can contain multiple macro variables. When the macro variable of date and time works with a scheduled job, the incremental data can be synchronized periodically.
NOTE:
The table name can contain wildcard characters (*), which is used to export all tables whose names start with a certain prefix or end with a certain suffix. The number and types of fields in the tables must be the same. The examples are as follows:
|
table |
|
Advanced attributes |
Partition Column |
This parameter is displayed when Use SQL Statement is set to No, indicating that a field used to split data during data extraction. CDM splits a job into multiple tasks based on this field and executes the tasks concurrently. Fields with data distributed evenly are used, such as the sequential number field. Click the icon next to the text box to go to the page for selecting a field or directly enter a field.
NOTE:
|
id |
Where Clause |
WHERE clause used to specify the data extraction range. This parameter is displayed when Use SQL Statement is set to No. If this parameter is not set, the entire table is extracted. You can set a date macro variable to extract data generated on a specific date. |
DS='${dateformat(yyyy-MM-dd,-1,DAY)}' |
|
Null in Partition Column |
Whether the partition column can contain null values |
Yes |
|
Job Split Field |
Used to split a job into multiple subjobs for concurrent execution. |
- |
|
Minimum value of a split field |
Specifies the minimum value of Job Split Field during data extraction. |
- |
|
Maximum Split Field Value |
Specifies the maximum value of Job Split Field during data extraction. |
- |
|
Number of subjobs |
Specifies the number of subjobs split from a job based on the data range specified by the minimum and maximum values of Job Split Field. |
- |
|
Extract by Partition |
When data is exported from an MySQL database, data can be extracted from each partition in the partitioned table. If this function is enabled, you can configure Table Partition to specify specific MySQL table partitions from which data is extracted.
|
No |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot