Help Center/ DataArts Studio/ Best Practices/ Migration of IoV Big Data to the Lake Without Loss/ Using DES to Migrate Historical Data Generated One Month Ago
Updated on 2024-08-30 GMT+08:00
View PDF
Share

Using DES to Migrate Historical Data Generated One Month Ago

Migration Process

  1. Use a script to import the historical data generated one month ago to the DES Teleport box. For details about the operations related to DES Teleport boxes, see Data Express Service (DES).
  2. Use DES to deliver data to HUAWEI CLOUD data center.
  3. Use CDM to migrate data from DES to Object Storage Service (OBS).
  4. Use CDM to migrate data from OBS to MRS.

The operations on CDM are the same as those described in Using CDM to Migrate Data of the Last Month. File directories are transmitted in binary format and two clusters concurrently execute jobs.

Precautions

  • If the migration affects the source HDFS cluster, manually stop the job.
  • If a large number of jobs fail, perform the following operations:
    1. Check whether the DES Teleport box is fully written. If the Teleport box is fully written, clear the failed directories to ensure that the data written later is complete.
    2. Check the network connectivity.
    3. Check the source HDFS cluster. Check whether indicators are abnormal. If any indicator is abnormal, suspend the migration task.