What Is LakeFormation?

Updated on 2025-01-10 GMT+08:00

View PDF

DataArts Lake Formation (LakeFormation) is an enterprise-level one-stop data lake construction service. It adopts a storage-compute decoupling architecture and provides GUIs and APIs for unified lake metadata management. It is compatible with Hive metadata models and Ranger permission models and can interconnect with MapReduce Service (MRS),Data Lake Insight (DLI), ModelArts, DataArts Studio, and GaussDB(DWS). LakeFormation helps you to easily and efficiently build data lakes and operation services, accelerating the release of service data value.

LakeFormation is a serverless service that uses underlying resources to implement cross-AZ deployment, high reliability, auto scaling, unified metadata management, association between metadata and file directories, and interconnection with multiple compute engines.

LakeFormation Architecture

Figure 1 shows the LakeFormation architecture.

Figure 1 LakeFormation service architecture
Click to enlarge

LakeFormation provides metadata management, data permission management, console, and API functions.

The metadata management system of LakeFormation is built based on the Hive metadata model and supports metadata objects such as catalogs, databases, tables, and functions.
LakeFormation allows you to configure permission policies and the corresponding access permission control.
- Authorization entities include IAM users, user groups, and LakeFormation roles.
- You can grant permissions to metadata objects such as catalogs, databases, tables, columns, functions, and OBS parallel file system paths.
- Authorized operations include operations related to metadata objects and read and write operations on OBS paths.
On the LakeFormation console, you can manage instances, metadata, data permissions, data access, and tasks.
The API layer provides metadata APIs compatible with Hive and permission synchronization APIs compatible with Ranger to facilitate integration and interconnection with services such as MRS and GaussDB(DWS).

Advantages

Open ecosystem
LakeFormation complies with open source standards and supports seamless service evolution.
- Smooth interconnection with Hive/Spark/Flink/Trino using metadata APIs
- One-time authorization with Ranger
- Smooth metadata migration with effective migration tools
Data-AI convergence
LakeFormation eliminates the barriers in big data and implement data-AI convergence.
- Unified management of databases, tables, function models, and unstructured datasets
- Secure data sharing across services and clusters based on unified fine-grained data permission management
Large specifications (HA)
LakeFormation supports high reliability of ultra-large-scale big data services.
- Effective management for ultra-large-scale metadata
- Unified fine-grained permission management at scale
- Multi-AZ disaster recovery
Ease of use
LakeFormation provides metadata-based management capabilities.
- User-friendly serverless architecture
- Management capabilities such as data lake management and metadata statistics

Functions

Table 1 lists the common functions provided by LakeFormation.

Learning about the basic concepts of LakeFormation LakeFormation in advance helps you choose the optimal functions in actual situations.

**Table 1** LakeFormation LakeFormation Functions
Function	Description
Instance Type	LakeFormation provides different types of instances to meet customers' different requirements for performance and costs in different scenarios. For details, see Comparison Between Shared and Exclusive Instances.
Instance management	LakeFormation provides basic functions such as instance creation, overview, and deletion, helping you easily manage instances and accelerate the planning and deployment of services carried by the data lake.
Metadata management	LakeFormation allows you to create, modify, view, and delete data lake metadata catalogs, databases, and tables. It also supports metadata life cycle configuration. Easy management helps you initialize and operate the data lake with ease, manage all LakeFormation metadata in a unified manner, and quickly plan and deploy data lake services.
Data permission management	LakeFormation allows you to authorize, cancel, and view data resources such as catalogs, databases, and tables. It helps you implement convenient and unified data permission management for the data lake.
Task management	LakeFormation supports full or incremental migration of metadata and permissions from external services to the current LakeFormation instance for unified management.
Access management	LakeFormation provides unified access management capabilities. You can create an access client to establish a network connection channel for a specified client environment. In addition, you can view information such as the access IP address and access domain name in the client details for other cloud services to access LakeFormation instances.

Access Methods

You can access the LakeFormation service from the web-based console or using HTTPS APIs. In addition, LakeFormation provides an SDK client to facilitate interconnection and integration with compute engines.

Using APIs
If you want to integrate LakeFormation instances on the public cloud platform into a third-party system for secondary development, use the APIs to access LakeFormation instances. For details about the operations, see API Reference.
Web-based console
You can access LakeFormation on the console by selecting Analytics > LakeFormation from the service list if you have registered with the public cloud.
Using SDKs
- LakeFormation provides an SDK client compatible with Hive metadata models. If you need to connect compute engines such as Hive and Spark to LakeFormation for unified metadata management, you can use SDKs to access LakeFormation instances.
- LakeFormation provides REST APIs, allowing you to call APIs using HTTPS.