Help Center/ DataArts Lake Formation/ Getting Started/ Constructing a Data Lake
Updated on 2024-11-21 GMT+08:00
View PDF
Share

Constructing a Data Lake

MRS can only connect to the hive catalog and the default database in LakeFormation.

After a LakeFormation instance is created, you can create catalogs and metadata such as databases and tables as required.

For example, perform the following steps to create a catalog named hive (fixed name, which cannot be customized) and create a database named default (fixed name, which cannot be customized) in the catalog. Assign permissions for the two data tables table_A and table_B contained in the database.

Creating an OBS Path for Storing Metadata

  1. Log in to the LakeFormation console.
  2. Click in the upper left corner of the page and choose Storage > Object Storage Service to access the Object Storage Service console.
  3. Click Parallel File Systems and click Create Parallel File System. On the displayed page, set the parameters, and click Create Now.

    • File System Name: Set the name of the parallel file system as required, for example, lakeformation-test.
    • Set other parameters based on the site requirements.

  4. On the Parallel File Systems page, click the name of the created file system, that is lakeformation-test.
  5. Click Files in the navigation pane, click Create Folder, enter a folder name, and click OK. Click the folder name and click Create Folder to create a subfolder.

    Repeat this step to create paths for storing metadata in sequence. The following paths are examples:

    • Catalog storage path: lakeformation-test/catalog1
    • Database storage path: lakeformation-test/catalog1/database1
    • Table storage path: lakeformation-test/catalog1/database1/table1 and lakeformation-test/catalog1/database1/table2
    • Function storage path: lakeformation-test/catalog1/database1/udf1

Creating a Catalog

  1. Log in to the LakeFormation console.
  2. In the upper left corner, click and choose Analytics > LakeFormation to access the LakeFormation console.
  3. Select the created LakeFormation instance (for example, lakeformation-test). For how to create an instance, see the operations described in Creating a LakeFormation Instance and a Client Connection. Choose Metadata > Catalog in the navigation pane.

    You only need one hive catalog per instance. If you already have one in the current instance, you can skip creating another one.

  4. Click Create in the upper right corner. On the displayed page, set the following parameters, and click Submit.

    • Catalog Name: hive

      The name is fixed and cannot be customized.

    • Select Location: Click , select a storage location, for example, obs://lakeformation-test/catalog1, and click OK.
    • Catalog Type: DEFAULT
    • Retain the default values for other parameters.

  5. After the catalog is created, you can view the catalog information on the Catalog page.

Creating a Database

  1. Log in to the LakeFormation console.
  2. In the upper left corner, click and choose Analytics > LakeFormation to access the LakeFormation console.
  3. Select the created LakeFormation instance (for example, lakeformation-test). For how to create an instance, see the operations described in Creating a LakeFormation Instance and a Client Connection. Choose Metadata > Database in the navigation pane.
  4. Select hive from the Catalog drop-down list box in the upper right corner.

    You only need one default database per instance. If you already have one in the current instance, you can skip creating another one.

  5. Click Create, set related parameters, and click Submit.

    • Database Name: default

      The name is fixed and cannot be customized.

    • Catalog: hive
    • Select Location: Click , select a storage location, for example, obs://lakeformation-test/catalog1/database1, and click OK.
    • Retain the default values for other parameters.

  6. After the catalog is created, you can view the database information on the Database page.

Creating a Table

  1. Log in to the LakeFormation console.
  2. In the upper left corner, click and choose Analytics > LakeFormation to access the LakeFormation console.
  3. Select the created LakeFormation instance (for example, lakeformation-test) from the drop-down list box on the left. For how to create an instance, see Creating a LakeFormation Instance and a Client Connection. Choose Metadata > Table in the navigation pane and select hive and default from the Catalog and Database drop-down lists in the upper right corner.
  4. Click Create, set related parameters, and click Submit.

    • Table Name: Enter a table name, for example, table_A.
    • Data Storage Location: Click , select a location for storing the table in the OBS parallel file system, for example, obs://lakeformation-test/catalog1/database1/table1, and click OK.
    • Set other parameters based on the site requirements.

  5. Create another table named table_B and set the storage path to lakeformation-test/catalog1/database1/table2 by referring to 4.
  6. After these two tables are created, you can check these tables on the Table page.

Authorizing Metadata

  1. Log in to the LakeFormation console.
  2. In the upper left corner, click and choose Analytics > LakeFormation to access the LakeFormation console.
  3. Select the target LakeFormation instance from the drop-down list box on the left and choose Data Permissions > Data Authorization.
  4. Click Authorize, authorize the hive catalog and the default database as required, and click OK.

    For details about authorization configurations and requirements, see Granting permissions.