Reference: CDM Performance Test Data
Background
The performance metrics provided in this document are for reference only. The performance at your site may be affected by factors such as the data source performance at the source or destination, network bandwidth, latency, and the data and service model. It is recommended that you test the speed with a small amount of data before migration.
Environment
- A xlarge CDM cluster of the 2.9.1 200 version
- A table which has 50 million rows and 100 columns, and three HDFS binary files which have 35.97 million rows and 100 columns, 66.67 million rows and 100 columns, and 100 million rows and 100 columns, respectively.
- Number of concurrent extraction jobs for determining the maximum extraction/write rate: 1, 10, 20, 30, and 50
Data Source Extraction and Write Performance Test Data
Table 1 and Table 2 provide the data extraction and write performance, respectively.
|
Data Source |
Specifications |
Version |
Extraction Rate for a Single Job (Lines per Second) |
Extraction Rate for Multiple Jobs (Lines per Second) |
|---|---|---|---|---|
|
RDS for MySQL |
8 vCPUs, 32 GB |
MySQL 5.7 |
42,052 |
195,313 (concurrency: 40) |
|
Oracle |
8 vCPUs, 16 GB |
19C |
18,539 |
18,706 (concurrency: 10) |
|
MRS HBase |
Master: 16 vCPUs, 64 GB x 3 Node: 8 vCPUs, 32 GB x 3 |
MRS 3.1.0 |
6,296 |
69,156 (concurrency: 30) |
|
MRS Hive |
Master: 16 vCPUs, 64 GB x 3 Node: 8 vCPUs, 32 GB x 3 |
MRS 3.1.0 |
22,321 |
170,068 (concurrency: 30) |
|
MRS HDFS (binary files) |
Master: 16 vCPUs, 64 GB x 3 Node: 8 vCPUs, 32 GB x 3 |
MRS 3.1.0 |
138,727 |
141,468 (concurrency: 20) |
|
125,556 |
126,990 (concurrency: 10) |
|||
|
120,919 |
120,919 (concurrency: 10) |
|||
|
DWS |
8 vCPUs, 16 GB |
8.1.1.300 |
13,434 |
/ |
|
DLI |
16 vCPUs |
SQL queue |
71,023 |
19,290 (concurrency: 20) |
|
Data Source |
Specifications |
Version |
Write Rate for a Single Job (Rows per Second) |
Optimal Write Rate (Rows per Second) |
|---|---|---|---|---|
|
RDS for MySQL |
8 vCPUs, 32 GB |
MySQL 5.7 |
2,658 |
/ |
|
Oracle |
8 vCPUs, 16 GB |
19C |
/ |
/ |
|
MRS Hbase |
Master: 16 vCPUs, 64 GB x 3 Node: 8 vCPUs, 32 GB x 3 |
MRS 3.1.0 |
3,959 |
4,120 (concurrency: 10) |
|
MRS Hive |
Master: 16 vCPUs, 64 GB x 3 Node: 8 vCPUs, 32 GB x 3 |
MRS 3.1.0 |
25,813 |
26,882 (concurrency: 10) |
|
MRS HDFS (binary files) |
Master: 16 vCPUs, 64 GB x 3 Node: 8 vCPUs, 32 GB x 3 |
MRS 3.1.0 |
65,075 |
90,155 (concurrency: 10) |
|
86,248 |
86,248 (concurrency: 1) |
|||
|
76,687 |
76,687 (concurrency: 1) |
|||
|
DWS |
8 vCPUs, 16 GB |
8.1.1.300 |
26,624 |
27,902 (concurrency: 10) |
|
DLI |
16 vCPUs |
SQL queue |
15,211 |
18,430 (concurrency: 10) |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.
