Backing Up Doris Data

Data Backup Principles

Backup operations upload the data of a specified table or partition as Doris files to a remote store. After a user submits a backup request, the system performs the following operations:

1. Take snapshots and upload

   The system takes a snapshot of the specified table or partition data file. A snapshot only generates a hard link to the current data file, which is less time-consuming. After the snapshot, changes and imports to the table no longer affect the results of the backup. After the snapshot is completed, the snapshot files will be uploaded one by one. Snapshot uploads are done concurrently by each BE.

2. Prepare and upload metadata

   After the data file snapshot upload is complete, FE will first write the corresponding metadata to a local file, and then upload the local metadata file to the remote warehouse through the Broker, completing the final backup job.

• If a table to be backed up is a dynamic partitioned table, the dynamic partitioning function is automatically disabled after backup. Before restoring data, run the following command to manually enable dynamic partitioning for the table:

  ALTER TABLE tbl1 SET ("dynamic_partition.enable"="true")

• The data backup operation does not keep the colocate_with property of a table.

Prerequisite

• A cluster containing the Doris service has been created, and all services in the cluster are running properly.
• The node to be connected to the Doris database can communicate with the MRS cluster.
• The MySQL client has been installed. For details, see Installing a MySQL Client.
• Create a user with the Doris management permission.
  • Kerberos authentication is enabled for the cluster (the cluster is in security mode)

    On FusionInsight Manager, create a human-machine user, for example, dorisuser, create a role with Doris administrator permission, and bind the role to the user.

    Log in to FusionInsight Manager as the new user dorisuser and change the initial password.

  • Kerberos authentication is disabled for the cluster (the cluster is in normal mode)

    After connecting to Doris as user admin, create a role with administrator permissions, and bind the role to the user.

Backing Up Doris Data

1. Log in to the node where MySQL is installed and connect the Doris database.

   If Kerberos authentication is enabled for the cluster (the cluster is in security mode), run the following command to connect to the Doris database:

   export LIBMYSQL_ENABLE_CLEARTEXT_PLUGIN=1

   mysql -uDatabase login username -pDatabase login password -PConnection port for FE queries -hIP address of the Doris FE instance

   

   • To obtain the query connection port of the Doris FE instance, you can log in to FusionInsight Manager, choose Cluster > Services > Doris > Configurations, and query the value of query_port of the Doris service.
   • To obtain the IP address of the Doris FE instance, log in to FusionInsight Manager of the MRS cluster and choose Cluster > Services > Doris > Instances to view the IP address of any FE instance.
   • You can also use the MySQL connection software or Doris web UI to connect the database.

2. Run the following command to create a remote repository example_repo in HDFS:

   Kerberos authentication is enabled for the cluster (the cluster is in security mode)

   CREATE REPOSITORY `example_repo`

   WITH BROKER `hdfs_broker`

   ON LOCATION "hdfs://hadoop-name-node:25000/path/to/repo/"

   PROPERTIES

   (

   "hadoop.security.authentication"="kerberos",

   "kerberos_principal"="doris/hadoop.hadoop.com@HADOOP.COM",

   "kerberos_keytab"="${BIGDATA_HOME}/FusionInsight_Doris_8.3.1/install/FusionInsight-Doris-2.0.3/doris-fe/bin/doris.keytab"

   );

   Kerberos authentication is disabled for the cluster (the cluster is in normal mode)

   CREATE REPOSITORY `example_repo`

   WITH BROKER `hdfs_broker`

   ON LOCATION "hdfs://hadoop-name-node:25000/path/to/repo/"

   PROPERTIES

   (

   "username" = "hdfs",

   "password" = ""

   );

3. View the created repository.

   SHOW REPOSITORIES;

4. Back up data to example_repo. You can back up table data or partition data. The following are examples:
   • Fully back up data in the example_tbl table from example_db to example_repo.

     BACKUP SNAPSHOT example_db.snapshot_label1

     TO example_repo

     ON (example_tbl)

     PROPERTIES ("type" = "full");

   • Fully back up the example_tbl2 table and the p1 and p2 partitions of the example_tbl table in the example_db to example_repo.

     BACKUP SNAPSHOT example_db.snapshot_label2

     TO example_repo

     ON

     (

     example_tbl PARTITION (p1,p2),

     example_tbl2

     );

5. Run the following command to check the execution status of the backup job:

   show BACKUP;

6. Check whether the backup is successful in the remote repository.

   SHOW SNAPSHOT ON example_repo WHERE SNAPSHOT = "snapshot_label1";

   +-----------------+---------------------+--------+
   | Snapshot        | Timestamp           | Status |
   +-----------------+---------------------+--------+
   | snapshot_label1 | 2022-04-08-15-52-29 | OK     |
   +-----------------+---------------------+--------+
   1 row in set (0.15 sec)