On this page

Service Information Survey
Collecting Basic Cluster Information
Big data component information
Statistics on Inventory Data to Be Migrated and Data Volume
Statistics on New Data Volume Every Day
Network Egress Bandwidth Capability
Collecting Streaming Kafka Cluster Information
Data Migration Model Example

Show all

Help Center/ MapReduce Service/ Best Practices/ Data Migration/ Information Collection Before Data Migrated to MRS

Information Collection Before Data Migrated to MRS

Updated on 2022-12-09 GMT+08:00

View PDF

Offline big data migration is flexible. Before the migration, you need to understand the detailed information about the existing cluster to better make migration decisions.

Service Information Survey

Architecture of the big data platform and services.
The data flow diagram (including peak and average traffic) of the big data platform and services is used to identify the data access source of the platform. Data inflow mode of the big data platform (real-time data reporting and batch data extraction) and data flow direction of the analysis platform. Data flow direction between components on the platform. For example, which component is used to collect data, how the collected data flows to the next-layer component, which component is used to store data, and the workflow during data processing.
Job Type: Hive SQL, Spark SQL, and Spark Python. For details about whether to use the third-party package of MRS, see the MRS Application Development Sample.
Dispatching system: The dispatching system needs to interconnect with the MRS cluster.
After the migration, the service interruption duration is allowed during service cutover to identify the platform service priority. Identify the services that cannot be interrupted during the migration, the services that can be interrupted for a short time, and the acceptable migration duration of the entire service migration to decide the migration sequence.
The deployment requirements of a client.
Service execution time segment and peak time segment.
Number and function division of big data clusters, and service model of the analysis platform, including the services that each cluster or component is responsible for and types of data that each cluster or component processes. For example, the components used to process real-time and offline data, data format type, and compression algorithm.

Collecting Basic Cluster Information

**Table 1** Basic cluster information
Parameter	Value	Description
Cluster name	-	-
Cluster version	-	Versions of clusters such as MRS, CDM, and FusionInsight.
Node quantity and specifications	-	Mandatory. Know the number of existing cluster nodes and node specifications. If the cluster uses heterogeneous hardware, enter multiple specifications and the number of nodes. For details, see Table 2. Example: NameNode and ResourceManager are deployed on two 32 U 64 GB servers. The HiveServer is deployed on two 32 U 64 GB servers. DataNodes and NodeManagers are deployed on 20 16 U 32 GB servers.
Enable Kerberos authentication	-	Mandatory: Yes or No
Permission control and description	-	Mandatory. Investigate the components and configurations for which ACL permission control is enabled. Involved components: Yarn, Hive, Impala and HBase. Use Ranger, Sentry, or open-source component permission capabilities to control permissions.
Region/AZ	-	Cloud resource parameters
VPC	-	Cloud resource parameters
Subnet	-	Cloud resource parameters
Security group	-	Cloud resource parameters

**Table 2** Hardware information survey table
Node Group	CPU and Memory	Disk and Network (by Node Group)		HDFS			Yarn
-	-	Disk information (data disk size, disk I/O, current disk usage, and I/O status)	Network (NIC bandwidth, network read/write speed, and peak value)	NameNode	DataNode	JouralNode	NodeManager	ResourceManager
master1	(16 U 64 GB)	-	-	1	-	1	-	1
master2	(16 U 64 GB)	-	-	1	-	1	-	1
master3	(16 U 64 GB)	-	-	-	-	1	-	-
Core-group1	(32 U 128 GB)*Quantity	-	-	-	1	-	1	-
Core-group1	(32 U 129 GB)	-	-	-	-	-	-	-
Core-group1	(32 U 130 GB)	-	-	-	-	-	-	-

Big data component information

Compare the big data component information with the version information of the planned new big data cluster to identify the possible impact of version differences on the migration process and the impact on service compatibility after the migration.

**Table 3** Big data component information
Component	Source Cluster Version	Target Cluster Version (Example: MRS 1.9.2)	Description
HDFS/OBS (or other file storage systems)	Hadoop 2.8.3	Hadoop 2.8.3	-
Hive	1.2.1	2.3.3	Database for storing metadata: MySQL
HBase	1.3.1	1.3.1	-
Spark	2.2.2	2.2.2	-
Kafka	1.1.0	1.1.0	-
Oozie	2.x	Self-built	-
mysql	5.7.1	RDS	-
Flink	1.7	1.7	-
...	...	...	-

Statistics on Inventory Data to Be Migrated and Data Volume

If HDFS is used as the file storage system, you can run the hadoop fs -du -h /user/test command to collect statistics on the file size in the path.

**Table 4** Existing data volume statistics
Component	Path of the Data to Be Migrated	Data Volume	Number of Files or Tables
HDFS/OBS (or other file storage systems)	/user/helloworld	xx	Total: xxxx files Number of files smaller than 2 MB: xxx
Hive	/user/hive/warehouse/	xx	Number of tables: xxx
HBase	/hbase	xx	Number of tables: xx Number of regions: xx

Statistics on New Data Volume Every Day

Evaluate the data growth rate (by day or hour) based on the amount of new data generated every day. After the first full data migration, you can periodically migrate new data in the old cluster until the final service migration is complete.

**Table 5** New data volume statistics
Component	Path of the Data to Be Migrated	New Data Volume
HDFS/OBS (or other file storage systems)	/user/helloworld	xx
Hive	/user/hive/warehouse/	xx
HBase	/hbase	xx

Network Egress Bandwidth Capability

Maximum network bandwidth and private line bandwidth that can be used for data migration (adjustable or not)
Period during which a data migration job can be executed every day

Collecting Streaming Kafka Cluster Information

**Table 6** Streaming Kafka cluster information
Item	Description
Number and names of Kafka topics	-
Local data temporary storage duration of Kafka. If the configuration of each topic is different, collect the data by topic.	-
Number of copies and partitions of each topic. (The default value is 2. More copies indicate more reliable data and more disk space is consumed.) If the configuration of each topic is different, collect data by topic.	-
Kafka production and consumption traffic, which is refined to the topic level.	-
Configure acks in ACK of the Kafka client	-

Data Migration Model Example

In the customer service system diagram of an offline analysis platform, Spark Streaming consumes Kafka data and saves the data to HDFS. After small files are merged in HDFS, Hive Load loads the data to Hive tables. Operation personnel can use Presto to query Hive data.
Figure 1 Service diagram of the source cluster
For offline big data platforms, including HDFS and Hive, service programs of Kafka, Spark Streaming, HDFS, Hive, and Presto must be deployed on the destination cluster.
Figure 2 Migration diagram