Help Center/ MapReduce Service/ User Guide/ Managing Clusters/ Managing MRS Cluster Metadata/ MRS Cluster Metadata Storage in an External Data Source
Updated on 2025-12-10 GMT+08:00
View PDF

MRS Cluster Metadata Storage in an External Data Source

Metadata is data that describes other data, providing details such as its structure, storage location, and access permissions. In an MRS cluster, component metadata is stored by default within the local GaussDB database of the cluster. Deleting a cluster will also delete its metadata. To retain the metadata, you need to manually save it in advance.

MRS provides the data connection management function. This function allows metadata of components (such as Hive and Ranger) to be stored in external data sources, decoupling the data storage layer (such as HDFS) from compute engines (such as Spark and Flink).

For example, Hive metadata can be stored in an external relational database and will not be deleted when the current MRS cluster is deleted. In addition, multiple MRS clusters can share the same metadata.

Figure 1 MRS cluster metadata storage in an external data source

External Data Connections Supported by MRS

Table 1 MRS external data connections

Data Connection Type

Description

Applicable Version

Supported Engine

RDS PostgreSQL database

RDS for PostgreSQL is designed for enterprise online transactional processing (OLTP) scenarios requiring complex SQL processing. It supports NoSQL data types (such as JSON, XML, and HStore) and geographic information system (GIS) data types, and is renowned for its reliability and data integrity. It is suitable for internet websites, location-based applications, and complex data object processing.
  • MRS cluster version: clusters with the Hive component installed
  • PostgreSQL version: PostgreSQL 14

Hive

RDS MySQL database

RDS for MySQL is fully compatible with native MySQL, combining stability, reliability, and high performance. It features intelligent operations and maintenance, robust security, out-of-the-box usability, and automatic scaling.
  • MRS cluster version: clusters with the Hive or Ranger component installed
  • MySQL versions: MySQL 5.7.x and MySQL 8.0
  • Hive
  • Ranger

Notes and Constraints

  • When Hive metadata is switched between different clusters, MRS synchronizes only the permissions in the metadata database of the Hive component. The permission model on MRS is maintained on MRS Manager. Therefore, when Hive metadata is switched between clusters, the permissions of users or user groups cannot be automatically synchronized to MRS Manager of another cluster.
  • The VPC and subnet of the service for which an external data connection will be created must be the same as those of the MRS cluster to be interconnected.
  • The RDS database instance interconnected with the MRS cluster cannot be deleted. Otherwise, the cluster will be abnormal.
Parent Topic: Managing MRS Cluster Metadata