Metadata Sharing

Scenario

Multiple services and clusters use unified metadata to maximize data sharing, avoid unnecessary duplicate data, and maximize the value of service data.

Data sharing across clusters
LakeFormation supports data sharing across clusters. As shown in Figure 1, there are MRS cluster 1 and MRS cluster 2. The two clusters have decoupled storage and compute enabled, with data stored in OBS. A service user of cluster 1 creates data table T1 and writes data to T1. After LakeFormation is interconnected with cluster 1 and cluster 2 and related operation permissions are granted to the clusters, service users of cluster 2 can use the unified metadata management function of LakeFormation to query and analyze data in data table T1.
Figure 1 Data sharing across clusters
Data sharing across services and clusters
LakeFormation supports data sharing across services and clusters. As shown in Figure 2, there are MRS cluster 1 and GaussDB(DWS) cluster. The two clusters have decoupled storage and compute enabled, with data stored in OBS. Big data user A of MRS cluster 1 creates data table T1. After LakeFormation is interconnected with MRS cluster 1 and GaussDB(DWS) cluster and related operation permissions are granted to the clusters, data warehouse users of the GaussDB(DWS) cluster can use the unified metadata management function of LakeFormation to add partitions and write data to data table T1. Big data user B of MRS cluster 1 can read data from data table T1.

Figure 2 Data sharing across services and clusters

Advantages

Being compatible with the Hive metadata model, the SDK client supports easy and fast interconnection between compute engines and LakeFormation.
The API for querying permissions is compatible with the Ranger permission model.