Help Center/ MapReduce Service/ Best Practices/ Data Migration/ Information Collection Before Data Migrated to MRS
Updated on 2022-12-09 GMT+08:00
View PDF
Share

Information Collection Before Data Migrated to MRS

Offline big data migration is flexible. Before the migration, you need to understand the detailed information about the existing cluster to better make migration decisions.

Service Information Survey

  1. Architecture of the big data platform and services.
  2. The data flow diagram (including peak and average traffic) of the big data platform and services is used to identify the data access source of the platform. Data inflow mode of the big data platform (real-time data reporting and batch data extraction) and data flow direction of the analysis platform. Data flow direction between components on the platform. For example, which component is used to collect data, how the collected data flows to the next-layer component, which component is used to store data, and the workflow during data processing.
  3. Job Type: Hive SQL, Spark SQL, and Spark Python. For details about whether to use the third-party package of MRS, see the MRS Application Development Sample.
  4. Dispatching system: The dispatching system needs to interconnect with the MRS cluster.
  5. After the migration, the service interruption duration is allowed during service cutover to identify the platform service priority. Identify the services that cannot be interrupted during the migration, the services that can be interrupted for a short time, and the acceptable migration duration of the entire service migration to decide the migration sequence.
  6. The deployment requirements of a client.
  7. Service execution time segment and peak time segment.
  8. Number and function division of big data clusters, and service model of the analysis platform, including the services that each cluster or component is responsible for and types of data that each cluster or component processes. For example, the components used to process real-time and offline data, data format type, and compression algorithm.

Collecting Basic Cluster Information

Table 1 Basic cluster information

Parameter

Value

Description

Cluster name

-

-

Cluster version

-

Versions of clusters such as MRS, CDM, and FusionInsight.

Node quantity and specifications

-

Mandatory. Know the number of existing cluster nodes and node specifications.

If the cluster uses heterogeneous hardware, enter multiple specifications and the number of nodes. For details, see Table 2.

Example:

NameNode and ResourceManager are deployed on two 32 U 64 GB servers.

The HiveServer is deployed on two 32 U 64 GB servers.

DataNodes and NodeManagers are deployed on 20 16 U 32 GB servers.

Enable Kerberos authentication

-

Mandatory: Yes or No

Permission control and description

-

Mandatory. Investigate the components and configurations for which ACL permission control is enabled.

Involved components: Yarn, Hive, Impala and HBase.

Use Ranger, Sentry, or open-source component permission capabilities to control permissions.

Region/AZ

-

Cloud resource parameters

VPC

-

Cloud resource parameters

Subnet

-

Cloud resource parameters

Security group

-

Cloud resource parameters
Table 2 Hardware information survey table

Node Group

CPU and Memory

Disk and Network (by Node Group)

HDFS

Yarn

-

-

Disk information (data disk size, disk I/O, current disk usage, and I/O status)

Network (NIC bandwidth, network read/write speed, and peak value)

NameNode

DataNode

JouralNode

NodeManager

ResourceManager

master1

(16 U 64 GB)

-

-

1

-

1

-

1

master2

(16 U 64 GB)

-

-

1

-

1

-

1

master3

(16 U 64 GB)

-

-

-

-

1

-

-

Core-group1

(32 U 128 GB)*Quantity

-

-

-

1

-

1

-

Core-group1

(32 U 129 GB)

-

-

-

-

-

-

-

Core-group1

(32 U 130 GB)

-

-

-

-

-

-

-

Big data component information

Compare the big data component information with the version information of the planned new big data cluster to identify the possible impact of version differences on the migration process and the impact on service compatibility after the migration.

Table 3 Big data component information

Component

Source Cluster Version

Target Cluster Version

(Example: MRS 1.9.2)

Description

HDFS/OBS (or other file storage systems)

Hadoop 2.8.3

Hadoop 2.8.3

-

Hive

1.2.1

2.3.3

Database for storing metadata: MySQL

HBase

1.3.1

1.3.1

-

Spark

2.2.2

2.2.2

-

Kafka

1.1.0

1.1.0

-

Oozie

2.x

Self-built

-

mysql

5.7.1

RDS

-

Flink

1.7

1.7

-

...

...

...

-

Statistics on Inventory Data to Be Migrated and Data Volume

If HDFS is used as the file storage system, you can run the hadoop fs -du -h /user/test command to collect statistics on the file size in the path.
Table 4 Existing data volume statistics

Component

Path of the Data to Be Migrated

Data Volume

Number of Files or Tables

HDFS/OBS (or other file storage systems)

/user/helloworld

xx

Total: xxxx files

Number of files smaller than 2 MB: xxx

Hive

/user/hive/warehouse/

xx

Number of tables: xxx

HBase

/hbase

xx

Number of tables: xx

Number of regions: xx

Statistics on New Data Volume Every Day

Evaluate the data growth rate (by day or hour) based on the amount of new data generated every day. After the first full data migration, you can periodically migrate new data in the old cluster until the final service migration is complete.
Table 5 New data volume statistics

Component

Path of the Data to Be Migrated

New Data Volume

HDFS/OBS (or other file storage systems)

/user/helloworld

xx

Hive

/user/hive/warehouse/

xx

HBase

/hbase

xx

Network Egress Bandwidth Capability

  • Maximum network bandwidth and private line bandwidth that can be used for data migration (adjustable or not)
  • Period during which a data migration job can be executed every day

Collecting Streaming Kafka Cluster Information

Table 6 Streaming Kafka cluster information

Item

Description

Number and names of Kafka topics

-

Local data temporary storage duration of Kafka. If the configuration of each topic is different, collect the data by topic.

-

Number of copies and partitions of each topic. (The default value is 2. More copies indicate more reliable data and more disk space is consumed.) If the configuration of each topic is different, collect data by topic.

-

Kafka production and consumption traffic, which is refined to the topic level.

-

Configure acks in ACK of the Kafka client

-

Data Migration Model Example

  • In the customer service system diagram of an offline analysis platform, Spark Streaming consumes Kafka data and saves the data to HDFS. After small files are merged in HDFS, Hive Load loads the data to Hive tables. Operation personnel can use Presto to query Hive data.
    Figure 1 Service diagram of the source cluster
  • For offline big data platforms, including HDFS and Hive, service programs of Kafka, Spark Streaming, HDFS, Hive, and Presto must be deployed on the destination cluster.
    Figure 2 Migration diagram
Parent topic: Data Migration