Backing Up Doris Data

Updated on 2024-11-29 GMT+08:00

View PDF

Doris data can be backed up in form of files to a remote storage system through Broker. You can periodically back up and migrated data using snapshots.

NOTE:

Currently, only users with the ADMIN permission can perform backup and restoration operations.
There can be only one backup job that is being executed in a database.
Doris data can be backed up with smallest partition granularity. If a table contains a large amount of data, you can back up data by partition to reduce the retry cost upon failure.
The backup and restoration operations are performed on the actual data files. When a table has too many shards, or a shard has too many small versions, it may take a long time to back up or restore even if the total amount of data is small.
You can run the SHOW BACKUP or SHOW RESTORE command to view the job status. An error information may be displayed in the TaskErrMsg column. If the value in the State column is not CANCELLED, the job continues. Some tasks may be retried successfully. However, some tasks are in error, causing job failures.

Data Backup Principles

Backup operations upload the data of a specified table or partition as Doris files to a remote store. After a user submits a backup request, the system performs the following operations:

Take snapshots and upload
The system takes a snapshot of the specified table or partition data file. A snapshot only generates a hard link to the current data file, which is less time-consuming. After the snapshot, changes and imports to the table no longer affect the results of the backup. After the snapshot is completed, the snapshot files will be uploaded one by one. Snapshot uploads are done concurrently by each BE.
Prepare and upload metadata
After the data file snapshot upload is complete, FE will first write the corresponding metadata to a local file, and then upload the local metadata file to the remote warehouse through the Broker, completing the final backup job.

NOTE:

If a table to be backed up is a dynamic partitioned table, the dynamic partitioning function is automatically disabled after backup. Before restoring data, run the following command to manually enable dynamic partitioning for the table:
ALTER TABLE tbl1 SET ("dynamic_partition.enable"="true")
The data backup operation does not keep the colocate_with property of a table.

Prerequisite

A cluster containing the Doris service has been created, and all services in the cluster are running properly.
The node to be connected to the Doris database can communicate with the MRS cluster.
The MySQL client has been installed. For details, see Installing a MySQL Client.
Create a user with the Doris management permission.
- Kerberos authentication is enabled for the cluster (the cluster is in security mode)
  On FusionInsight Manager, create a human-machine user, for example, dorisuser, create a role with Doris administrator permission, and bind the role to the user.
  
  Log in to FusionInsight Manager as the new user dorisuser and change the initial password.
- Kerberos authentication is disabled for the cluster (the cluster is in normal mode)
  After connecting to Doris as user admin, create a role with administrator permissions, and bind the role to the user.

Backing Up Doris Data

Log in to the node where MySQL is installed and connect the Doris database.

If Kerberos authentication is enabled for the cluster (the cluster is in security mode), run the following command to connect to the Doris database:

export LIBMYSQL_ENABLE_CLEARTEXT_PLUGIN=1

mysql -uDatabase login username -pDatabase login password -PConnection port for FE queries -hIP address of the Doris FE instance
NOTE:
- To obtain the query connection port of the Doris FE instance, you can log in to FusionInsight Manager, choose Cluster > Services > Doris > Configurations, and query the value of query_port of the Doris service.
- To obtain the IP address of the Doris FE instance, log in to FusionInsight Manager of the MRS cluster and choose Cluster > Services > Doris > Instances to view the IP address of any FE instance.
- You can also use the MySQL connection software or Doris web UI to connect the database.

Run the following command to create a remote repository example_repo in HDFS:

Kerberos authentication is enabled for the cluster (the cluster is in security mode)

CREATE REPOSITORY `example_repo`

WITH BROKER `hdfs_broker`

ON LOCATION "hdfs://hadoop-name-node:25000/path/to/repo/"

PROPERTIES

(

"hadoop.security.authentication"="kerberos",

"kerberos_principal"="doris/hadoop.hadoop.com@HADOOP.COM",

"kerberos_keytab"="${BIGDATA_HOME}/FusionInsight_Doris_8.3.1/install/FusionInsight-Doris-2.0.3/doris-fe/bin/doris.keytab"

);

Kerberos authentication is disabled for the cluster (the cluster is in normal mode)

CREATE REPOSITORY `example_repo`

WITH BROKER `hdfs_broker`

ON LOCATION "hdfs://hadoop-name-node:25000/path/to/repo/"

PROPERTIES

(

"username" = "hdfs",

"password" = ""

);
View the created repository.

SHOW REPOSITORIES;

Back up data to example_repo. You can back up table data or partition data. The following are examples:
- Fully back up data in the example_tbl table from example_db to example_repo.
  BACKUP SNAPSHOT example_db.snapshot_label1
  
  TO example_repo
  
  ON (example_tbl)
  
  PROPERTIES ("type" = "full");
- Fully back up the example_tbl2 table and the p1 and p2 partitions of the example_tbl table in the example_db to example_repo.
  BACKUP SNAPSHOT example_db.snapshot_label2
  
  TO example_repo
  
  ON
  
  (
  
  example_tbl PARTITION (p1,p2),
  
  example_tbl2
  
  );
Run the following command to check the execution status of the backup job:

show BACKUP;

Check whether the backup is successful in the remote repository.

SHOW SNAPSHOT ON example_repo WHERE SNAPSHOT = "snapshot_label1";

+-----------------+---------------------+--------+
| Snapshot        | Timestamp           | Status |
+-----------------+---------------------+--------+
| snapshot_label1 | 2022-04-08-15-52-29 | OK     |
+-----------------+---------------------+--------+
1 row in set (0.15 sec)

Parent topic: Backing Up and Restoring Data

Previous topic: Backing Up and Restoring Data

Next topic: Restoring Doris Data

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot