Help Center/ DataArts Lake Formation/ Service Overview/ What Is LakeFormation?
Updated on 2025-10-28 GMT+08:00
View PDF
Share

What Is LakeFormation?

DataArts Lake Formation (LakeFormation) is an enterprise-level one-stop data lake construction service. It adopts a storage-compute decoupling architecture and provides GUIs and APIs for unified lake metadata management. It is compatible with Hive metadata models and Ranger permission models and can interconnect with MapReduce Service (MRS), Data Warehouse Service (DWS), Data Lake Insight (DLI), ModelArts, DataArts Studio, and more compute engines and big data cloud services. LakeFormation helps you to easily and efficiently build data lakes and operation services, accelerating the release of service data value.

LakeFormation is a serverless service that uses underlying resources to implement cross-AZ deployment, high reliability, auto scaling, unified metadata management, association between metadata and file directories, and interconnection with multiple compute engines.

LakeFormation Architecture

Figure 1 shows the LakeFormation architecture.

Figure 1 LakeFormation service architecture

LakeFormation provides metadata management, data permission management, console, and API functions.

  • The metadata management system of LakeFormation is built based on the Hive metadata model and supports metadata objects such as catalogs, databases, tables, and functions.
  • LakeFormation allows you to configure permission policies and the corresponding access permission control.
    • Authorization entities include IAM users, user groups, and LakeFormation roles.
    • You can grant permissions to metadata objects such as catalogs, databases, tables, columns, functions, and OBS paths.
    • Authorized operations include operations related to metadata objects and read and write operations on OBS paths.
  • On the LakeFormation console, you can manage instances, metadata, data permissions, data access, and tasks.
  • The API layer provides metadata APIs compatible with Hive and permission synchronization APIs compatible with Ranger to facilitate integration and interconnection with services such as MRS and GaussDB(DWS).

Advantages

  • Open ecosystem

    LakeFormation complies with open source standards and supports seamless service evolution.

    • Smooth interconnection with Hive/Spark/Flink/Trino using metadata APIs
    • One-time authorization with Ranger
    • Smooth metadata migration with effective migration tools
  • Data-AI convergence

    LakeFormation eliminates the barriers in big data and implements data-AI convergence.

    • Unified management of databases, tables, function models, and unstructured datasets
    • Secure data sharing across services and clusters based on unified fine-grained data permission management
  • Large specifications (HA)

    LakeFormation supports high reliability of ultra-large-scale big data services.

    • Effective management for ultra-large-scale metadata
    • Unified fine-grained permission management at scale
    • Multi-AZ disaster recovery
  • Ease of use

    LakeFormation provides metadata-based management capabilities.

    • User-friendly serverless architecture
    • Management capabilities such as data lake management and metadata statistics

Access Methods

You can access the LakeFormation service from the web-based console or using HTTPS APIs. In addition, LakeFormation provides an SDK client to facilitate interconnection and integration with compute engines.

  • Using APIs

    If you want to integrate LakeFormation instances on the public cloud platform into a third-party system for secondary development, use the APIs to access LakeFormation instances. For details about the operations, see API Reference.

  • Web-based console

    You can access LakeFormation on the console by selecting Analytics > LakeFormation from the service list if you have registered with the public cloud.

  • Using SDKs
    • LakeFormation provides an SDK client compatible with Hive metadata models. If you need to connect compute engines such as Hive and Spark to LakeFormation for unified metadata management, you can use SDKs to access LakeFormation instances.
    • LakeFormation provides REST APIs, allowing you to call APIs using HTTPS.