MRS Cluster Metadata Storage in an External Data Source
Metadata is data that describes other data, providing details such as its structure, storage location, and access permissions. By default, component metadata is stored in the MRS cluster's built-in local GaussDB database, which stores metadata and configuration details for each component. Because deleting an MRS cluster also deletes its metadata, you must manually back up the metadata in advance if you need to preserve it.
MRS provides the data connection management function. This function allows metadata of components (such as Hive and Ranger) to be stored in external data sources, decoupling the data storage layer (such as HDFS) from compute engines (such as Spark and Flink).
For example, Hive metadata can be stored in an external relational database and will not be deleted when the current MRS cluster is deleted. In addition, multiple MRS clusters can share the same metadata.
Typical applications include:
- Cross-cluster metadata sharing: Enables multiple MRS clusters to access the same Hive metadata, preventing data redundancy and inconsistency.
- Metadata retention after cluster deletion: Retains metadata when a cluster is released due to service adjustments, allowing subsequent clusters to reuse the metadata.
- Unified data lake metadata management: Uses LakeFormation to manage metadata across multiple data sources and compute engines in a unified manner.
The figure above illustrates the architecture for storing MRS cluster metadata in an external data source.
- MRS cluster components, such as Hive and Ranger, use data connections to connect to external data sources including RDS for PostgreSQL, RDS for MySQL, TaurusDB, and LakeFormation.
- Component metadata read and write requests are routed to these external data sources through data connection channels rather than the default local GaussDB database.
- Storage layers (such as HDFS) and compute engines (such as Spark and Flink) interact with metadata through standard component interfaces, making the underlying metadata storage location transparent to upper-layer services.
External Data Connections Supported by MRS
| Data Connection Type | Description | Applicable Version | Supported Engine |
|---|---|---|---|
| RDS PostgreSQL database | RDS for PostgreSQL is designed for enterprise online transactional processing (OLTP) scenarios requiring complex SQL processing. It supports NoSQL data types (such as JSON, XML, and HStore) and geographic information system (GIS) data types, and is renowned for its reliability and data integrity. It is suitable for internet websites, location-based applications, and complex data object processing. For more information, see What Is RDS for PostgreSQL? |
| Hive |
| RDS MySQL database | RDS for MySQL is fully compatible with native MySQL, combining stability, reliability, and high performance. It features intelligent operations and maintenance, robust security, out-of-the-box usability, and automatic scaling. For more information, see What Is RDS for MySQL? |
|
|
| TaurusDB | TaurusDB is a distributed relational database developed by Huawei. It supports distributed transactions and intra-city deployment across AZs for zero data loss, storage for petabytes of data, and scale-up to more than 1,000 nodes. For more information, see What Is TaurusDB? | MRS cluster versions: MRS 3.1.2-LTS.3, MRS 3.1.5, and MRS 3.3.0-LTS |
|
| LakeFormation | LakeFormation is a one-stop enterprise-class data lake and warehouse construction service. It provides APIs and a GUI for unified management of data lake metadata, and is compatible with Hive metadata and Ranger permission models. LakeFormation can connect to multiple compute engines and big data cloud services seamlessly to ensure quick building and easy operations of data lakes and unleash rich value of service data. LakeFormation is a serverless service that uses underlying resources to implement cross-AZ deployment, high reliability, auto scaling, unified metadata management, association between metadata and file directories, and interconnection with multiple compute engines. For more information, see What Is LakeFormation? | MRS cluster version: Kerberos-enabled clusters: MRS 3.3.0-LTS or later Kerberos-disabled clusters: MRS 3.6.0-LTS or later |
|
Using an external data source (RDS, TaurusDB, or LakeFormation) for metadata storage incurs additional charges. Billing models vary depending on the selected data source type. For example, RDS is billed by instance specification and duration, while LakeFormation is billed by usage duration. For details, see the billing guide for each respective product.
Notes and Constraints
- When you switch Hive metadata between clusters, MRS synchronizes only the database-level permissions in Hive metadata. The permission model on MRS is maintained on MRS Manager. Therefore, when Hive metadata is switched between clusters, the permissions of users or user groups cannot be automatically synchronized to MRS Manager of another cluster.
- The VPC, subnet, and security group of the service for which an external data connection will be created must be the same as those of the MRS cluster to be interconnected. If the security groups differ, configure security group rules to enable communication. For details, see Enabling Communication Between Instances in Different Security Groups.
- The RDS database instance interconnected with the MRS cluster cannot be deleted. Otherwise, the cluster will be abnormal.
Disassociating external metadata can lead to service failures. Perform this action during off-peak hours or only after confirming that it will not have any adverse impacts. After disassociation, component metadata is restored to the local cluster GaussDB database. Existing metadata in the external data source is not automatically synchronized to the local GaussDB database.
- MRS cannot connect to an external database that has SSL enabled. If SSL is enabled, the connection fails. To resolve this issue, disable SSL on the database.
- Disassociating external metadata can lead to service failures. Perform this action during off-peak hours or only after confirming that it will not have any adverse impacts.
- In MRS 3.6.0-LTS.1 and later, when multiple MRS clusters connect to the same external metadata database, sensitive information in table attributes cannot be encrypted for storage. Tables containing encrypted information can be used only in the local cluster and cannot be decrypted in other clusters. To disable this function, set metastore-ext.encrypt.parameters.enable to false.
FAQs
Q: Is metadata in the local GaussDB database automatically migrated to an external data source after a metadata switch?
A: No. Before switching metadata to an external data source, you must manually export the local metadata and import it into the external data source.
Q: How do I resolve permission conflicts when multiple clusters share the same external metadata?
A: MRS synchronizes only database permissions in Hive metadata. You must manually configure user and user group permissions on MRS Manager in each cluster to ensure permission consistency.
Q: How does external metadata affect cluster performance?
A: Deploy external data sources and MRS clusters in the same VPC and subnet to minimize network latency.
Helpful Links
- For details about creating an MRS cluster data connection and verifying external metadata storage, see Storing Ranger Metadata to RDS and Storing Hive Metadata to RDS.
- For details about viewing, modifying, and deleting data connections, see Managing MRS Cluster Data Connections.
What is your overall rating for this page?
Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot
