Help Center/ Cloud Data Migration/ Performance White Paper/ Reference: CDM Performance Test Data
Updated on 2024-03-29 GMT+08:00
View PDF
Share

Reference: CDM Performance Test Data

Background

The performance metrics provided in this document are for reference only. The performance at your site may be affected by factors such as the data source performance at the source or destination, network bandwidth, latency, and the data and service model. It is recommended that you test the speed with a small amount of data before migration.

Environment

  • An xlarge CDM cluster of the 2.9.1 200 version
  • A table which has 50 million rows and 100 columns, and three HDFS binary files which have 35.97 million rows and 100 columns, 66.67 million rows and 100 columns, and 100 million rows and 100 columns, respectively.
  • Number of concurrent extraction jobs for determining the maximum extraction/write rate: 1, 10, 20, 30, and 50

Data Source Extraction and Write Performance Test Data

Table 1 and Table 2 provide the data extraction and write performance, respectively.

Table 1 Data extraction performance

Data Source

Specifications

Version

Extraction Rate for a Single Job (Lines per Second)

Extraction Rate for Multiple Jobs (Lines per Second)

RDS for MySQL

8 vCPUs, 32 GB

MySQL 5.7

42,052

195,313 (concurrency: 40)

Oracle

8 vCPUs, 16 GB

19C

18,539

18,706 (concurrency: 10)

MRS HBase

Master: 16 vCPUs, 64 GB x 3

Node: 8 vCPUs, 32 GB x 3

MRS 3.1.0

6,296

69,156 (concurrency: 30)

MRS Hive

Master: 16 vCPUs, 64 GB x 3

Node: 8 vCPUs, 32 GB x 3

MRS 3.1.0

22,321

170,068 (concurrency: 30)

MRS HDFS (binary files)

Master: 16 vCPUs, 64 GB x 3

Node: 8 vCPUs, 32 GB x 3

MRS 3.1.0

138,727

141,468 (concurrency: 20)

125,556

126,990 (concurrency: 10)

120,919

120,919 (concurrency: 10)

DWS

8 vCPUs, 16 GB

8.1.1.300

13,434

/

DLI

16 vCPUs

SQL queue

71,023

19,290 (concurrency: 20)

MRS Hudi (MOR)

Master: 16 vCPUs, 64 GB x 3

Node: 8 vCPUs, 64 GB x 3

MRS 3.2.0

75187

467,289 (concurrency: 30)

MRS Hudi (COW)

Master: 16 vCPUs, 64 GB x 3

Node: 8 vCPUs, 64 GB x 3

MRS 3.2.0

84033

485,436 (concurrency: 30)

ClickHouse

Node: 8 vCPUs, 32 GB x 2

ClickHouse 22.3.2.2

187265

/

Elasticsearch

4 vCPUs, 8 GB x 6

Elasticsearch 7.10.2

28752

/

RDS for PostgreSQL

4 vCPUs, 32 GB (active/standby)

PostgreSQL 13.12

128865

1,351,351 (concurrency: 30)
Table 2 Data write performance

Data Source

Specifications

Version

Write Rate for a Single Job

(Rows per Second)

Write Rate for Multiple Jobs

(Rows per Second)

RDS for MySQL

8 vCPUs, 32 GB

MySQL 5.7

2,658

/

Oracle

8 vCPUs, 16 GB

19C

/

/

MRS HBase

Master: 16 vCPUs, 64 GB x 3

Node: 8 vCPUs, 32 GB x 3

MRS 3.1.0

3,959

4,120 (concurrency: 10)

MRS Hive

Master: 16 vCPUs, 64 GB x 3

Node: 8 vCPUs, 32 GB x 3

MRS 3.1.0

25,813

26,882 (concurrency: 10)

MRS HDFS (binary files)

Master: 16 vCPUs, 64 GB x 3

Node: 8 vCPUs, 32 GB x 3

MRS 3.1.0

65,075

90,155 (concurrency: 10)

86,248

86,248 (concurrency: 1)

76,687

76,687 (concurrency: 1)

DWS

8 vCPUs, 16 GB

8.1.1.300

26,624

27,902 (concurrency: 10)

DLI

16 vCPUs

SQL queue

15,211

18,430 (concurrency: 10)

MRS Hudi (MOR)

Master: 16 vCPUs, 64 GB x 3

Node: 8 vCPUs, 64 GB x 3

MRS 3.2.0

16345

183,150 (concurrency: 10)

MRS Hudi (COW)

Master: 16 vCPUs, 64 GB x 3

Node: 8 vCPUs, 64 GB x 3

MRS 3.2.0

21088

88,183 (concurrency: 20)

ClickHouse

Node: 8 vCPUs, 32 GB x 2

ClickHouse 22.3.2.2

93984

/

Elasticsearch

4 vCPUs, 8 GB x 6

Elasticsearch 7.10.2

22271

/

RDS for PostgreSQL

4 vCPUs, 32 GB (active/standby)

PostgreSQL 13.12

34746

153,374 (concurrency: 10)