更新时间:2026-11-13 GMT+08:00
分享

FlinkJob场景的LakeFormation授权

提交FlinkJob作业前,需通过委托完成LakeFormation的IAM细粒度授权和SQL资源授权(包括元数据、数据库、表、列和函数),以确保作业在Flink集群后台运行时能够正常访问所需数据。本节介绍FlinkJob场景的LakeFormation授权的具体操作。

授权操作

在使用LakeFormation资源前需要分别完成LakeFormation API的授权和LakeFormation SQL资源授权。各SQL语句的权限请参考Paimon、Iceberg和Hive。

  1. 使用委托,授权AI DataLake使用LakeFormation API。

    使用FlinkJob执行作业时,需要您先在IAM页面创建相关委托,并在配置作业时添加新建的委托信息。委托权限示例请参考创建自定义委托。

    而使用委托的场景,当操作主体变成AI DataLake云服务本身,则必须换取临时凭证以“代表用户”去访问资源。

    相关依赖:

    • 临时凭证使用DEW创建和保存,临时凭证的创建方法:具体操作请参考创建通用凭据。
    • 访问DEW同样要求AI DataLake具备访问DEW的委托授权。详细的权限说明参考表1。
      表1 访问DEW的委托授权

      权限描述

      权限

      DEW中的查询凭据的版本与凭据值ShowSecretVersion接口权限

      csms:secretVersion:get

      DEW中的查询凭据的版本列表ListSecretVersions接口权限

      csms:secretVersion:list

      DEW解密凭据的权限

      kms:dek:decrypt

    关于获取临时凭证的方法:

    • (推荐)获取临时凭证

      临时安全凭证包括临时AK/SK和SecurityToken,以“动态生成、短期有效、自动失效、无需手动保管”为核心特性,委托搭配临时凭证,完成服务访问,凭证到期后自动失效,从根源上降低了密钥泄露的风险,同时省去了维护密钥的繁琐成本,成为批量自动化处理、高安全等级业务的优选方案。了解更多临时安全凭证。

      相关操作指导:获取Flink作业委托临时凭证用于访问其他云服务

    • 获取访问DEW的临时凭证+获取永久AK/SK

      该方案适用于仅支持细粒度授权(Version=1.1)鉴权协议的服务,以及对凭证稳定性要求极高、无需频繁刷新的业务需求,提供了可靠的鉴权支撑。

      该方案通过委托+获取访问DEW的临时凭证,再通过DEW获取永久AK/SK,最终实现对其他云服务的安全访问。

      相关操作指导:Flink SQL使用DEW管理访问凭据、Flink Jar使用DEW获取访问凭证读写OBS

  2. LakeFormation SQL资源授权:授权使用LakeFormation具体资源。

    LakeFormation资源授权是指允许用户对特定资源(元数据、数据库、表、列和函数等)的访问权限,以此来控制对LakeFormation的数据和元数据的访问。

    在LakeFormation管理控制台,对资源授权。FlinkJob支持角色和委托两种主体类型。

    了解LakeFormation SQL资源权限请参考配置LakeFormation元数据权限。

SQL鉴权

  • 除了表格内的权限,IAM侧还需要配置lakeformation:policy:describe。
  • 由于Flink默认会校验default数据库,所以在执行任何SQL前,需要在LakeFormation控制台上先为目标Catalog下的default数据库授予DESCRIBE权限,授权操作请参考配置LakeFormation元数据权限。
  • 建议不要给lakeformation:policy:create权限,给了会被视为管理员,跳过SQL鉴权。
表2 Paimon鉴权

类型

SQL

LakeFormation SQL资源鉴权权限

IAM权限

SQL DDL

CREATE DATABASE

database:DESCRIBE

catalog:CREATE_DATABASE

catalog:DESCRIBE

"lakeformation:database:describe",

"lakeformation:database:create"

"lakeformation:catalog:describe"

CREATE TABLE my_table

database:DESCRIBE

database:CREATE_TABLE

table:DESCRIBE

catalog:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe",

"lakeformation:table:create"

"lakeformation:catalog:describe"

Create Table As Select

table:DESCRIBE

table:INSERT

table:SELECT

table:ALTER

database:CREATE_TABLE

database:DESCRIBE

catalog:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe",

"lakeformation:table:create"

"lakeformation:table:alter"

"lakeformation:catalog:describe"

Create Table Like

table:DESCRIBE

table:INSERT

database:CREATE_TABLE

database:DESCRIBE

catalog:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe",

"lakeformation:table:create"

"lakeformation:catalog:describe"

ALTER TABLE my_table SET (key1=val1);

database:DESCRIBE

table:DESCRIBE

table:ALTER

catalog:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe",

"lakeformation:table:alter"

"lakeformation:catalog:describe"

ALTER TABLE my_table RESET (key1=val1);

database:DESCRIBE

table:DESCRIBE

table:ALTER

catalog:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe",

"lakeformation:table:alter"

"lakeformation:catalog:describe"

ALTER TABLE my_table RENAME TO my_table_new;

database:DESCRIBE

table:DESCRIBE

table:ALTER

catalog:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe",

"lakeformation:table:alter"

"lakeformation:catalog:describe"

ALTER TABLE my_table ADD (c1 INT, c2 STRING);

database:DESCRIBE

table:DESCRIBE

table:ALTER

catalog:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe",

"lakeformation:table:alter"

"lakeformation:catalog:describe"

ALTER TABLE my_table RENAME c0 TO c1;

database:DESCRIBE

table:DESCRIBE

table:ALTER

catalog:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe",

"lakeformation:table:alter"

"lakeformation:catalog:describe"

ALTER TABLE my_table DROP (c1, c2);

database:DESCRIBE

table:DESCRIBE

table:ALTER

catalog:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe",

"lakeformation:table:alter"

"lakeformation:catalog:describe"

ALTER TABLE my_table MODIFY buy_count BIGINT COMMENT '';

database:DESCRIBE

table:DESCRIBE

table:ALTER

catalog:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe",

"lakeformation:table:alter"

"lakeformation:catalog:describe"

ALTER TABLE my_table ADD c INT AFTER b;

database:DESCRIBE

table:DESCRIBE

table:ALTER

catalog:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe",

"lakeformation:table:alter"

"lakeformation:catalog:describe"

ALTER TABLE my_table MODIFY col_a DOUBLE AFTER col_b;

database:DESCRIBE

table:DESCRIBE

table:ALTER

catalog:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe",

"lakeformation:table:alter"

"lakeformation:catalog:describe"

ALTER TABLE my_table MODIFY col_a DOUBLE;

database:DESCRIBE

table:DESCRIBE

table:ALTER

catalog:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe",

"lakeformation:table:alter"

"lakeformation:catalog:describe"

ALTER DATABASE [catalog_name.]db_name SET (key1=val1, key2=val2, ...);

database:DESCRIBE

database:ALTER

catalog:DESCRIBE

"lakeformation:database:describe",

"lakeformation:database:alter"

"lakeformation:catalog:describe"

SQL DML

INSERT INTO my_table SELECT ...

table:DESCRIBE

table:ALTER

table:SELECT

database:DESCRIBE

catalog:DESCRIBE

"lakeformation:table:describe",

"lakeformation:table:alter",

"lakeformation:database:describe"

"lakeformation:catalog:describe"

Auxiliary语句

DESCRIBE TABLE

database:DESCRIBE

table:DESCRIBE

catalog:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe"

"lakeformation:catalog:describe"

SHOW DATABASES

catalog:LIST_DATABASE

database:DESCRIBE

catalog:DESCRIBE

"lakeformation:database:describe"

"lakeformation:catalog:describe"

SHOW CURRENT DATABASE

catalog:LIST_DATABASE

database:DESCRIBE

catalog:DESCRIBE

"lakeformation:database:describe"

"lakeformation:catalog:describe"

SHOW CREATE TABLE

database:DESCRIBE

table:DESCRIBE

catalog:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe"

"lakeformation:catalog:describe"

SHOW COLUMNS

database:DESCRIBE

table:DESCRIBE

catalog:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe"

"lakeformation:catalog:describe"

SHOW PARTITIONS

database:DESCRIBE

table:DESCRIBE

catalog:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe"

"lakeformation:catalog:describe"

表3 Iceberg鉴权

类型

SQL

LakeFormation SQL资源鉴权权限

IAM权限

SQL DDL

CREATE DATABASE

database:DESCRIBE

catalog:CREATE_DATABASE

"lakeformation:database:describe",

"lakeformation:database:create"

CREATE TABLE my_table

database:DESCRIBE

database:CREATE_TABLE

table:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe",

"lakeformation:table:create"

Create Table Like

table:DESCRIBE

table:INSERT

database:CREATE_TABLE

database:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe",

"lakeformation:table:create"

DROP DATABASE

database:DESCRIBE

database:DROP

"lakeformation:database:describe",

"lakeformation:database:drop"

DROP TABLE

database:DESCRIBE

table:DESCRIBE

table:DROP

"lakeformation:database:describe",

"lakeformation:table:describe",

"lakeformation:table:drop"

ALTER TABLE

database:DESCRIBE

table:DESCRIBE

table:ALTER

"lakeformation:database:describe",

"lakeformation:table:describe",

"lakeformation:table:alter"

ALTER TABLE .. RENAME TO

database:DESCRIBE

database:CREATE_TABLE

table:DESCRIBE

table:ALTER

"lakeformation:database:describe",

"lakeformation:table:describe",

"lakeformation:table:alter"

"lakeformation:table:create"

SQL DML

INSERT INTO my_table SELECT ...

table:DESCRIBE

table:ALTER

table:SELECT

database:DESCRIBE

"lakeformation:table:describe",

"lakeformation:table:alter",

"lakeformation:database:describe"

INSERT OVERWRITE my_table SELECT ...

table:DESCRIBE

table:ALTER

table:UPDATE

database:DESCRIBE

"lakeformation:table:describe",

"lakeformation:table:alter",

"lakeformation:database:describe"

Auxiliary语句

DESCRIBE TABLE

database:DESCRIBE

table:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe"

SHOW DATABASES

catalog:LIST_DATABASE

database:DESCRIBE

"lakeformation:database:describe"

SHOW CURRENT DATABASE

catalog:LIST_DATABASE

database:DESCRIBE

"lakeformation:database:describe"

SHOW CREATE TABLE

database:DESCRIBE

table:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe"

SHOW TABLES

database:LIST_TABLE

database:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe"

SHOW COLUMNS

database:DESCRIBE

table:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe"

SHOW PARTITIONS

database:DESCRIBE

table:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe"

类型

SQL

LakeFormation SQL资源鉴权权限

IAM权限

SQL DDL

CREATE DATABASE

database:DESCRIBE

catalog:CREATE_DATABASE

"lakeformation:database:describe",

"lakeformation:database:create"

CREATE TABLE my_table

database:DESCRIBE

database:CREATE_TABLE

table:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe",

"lakeformation:table:create"

Create Table As Select

table:DESCRIBE

table:INSERT

table:SELECT

table:ALTER

database:CREATE_TABLE

database:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe",

"lakeformation:table:create",

"lakeformation:table:alter"

Create Table Like

table:DESCRIBE

table:INSERT

database:CREATE_TABLE

database:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe",

"lakeformation:table:create"

CREATE FUNCTION

database:DESCRIBE

database:CREATE_FUNC

"lakeformation:database:describe",

"lakeformation:function:describe",

"lakeformation:function:create"

DROP DATABASE

database:DESCRIBE

database:DROP

"lakeformation:database:describe",

"lakeformation:database:drop"

DROP TABLE

database:DESCRIBE

table:DESCRIBE

table:DROP

"lakeformation:database:describe",

"lakeformation:table:describe",

"lakeformation:table:drop"

DROP FUNCTION

database:DESCRIBE

function:DESCRIBE

function:DROP

"lakeformation:database:describe",

"lakeformation:function:describe",

"lakeformation:function:drop"

ALTER TABLE

database:DESCRIBE

database:create_table

table:DESCRIBE

table:ALTER

"lakeformation:database:describe",

"lakeformation:table:describe",

"lakeformation:table:alter"

ALTER DATABASE

database:DESCRIBE

database:ALTER

"lakeformation:database:describe",

"lakeformation:database:alter"

ALTER FUNCTION

database:DESCRIBE

function:DESCRIBE

function:ALTER

"lakeformation:database:describe",

"lakeformation:function:describe",

"lakeformation:function:alter"

SQL DML

INSERT INTO my_table SELECT ...

table:DESCRIBE

table:ALTER

table:SELECT

database:DESCRIBE

"lakeformation:table:describe",

"lakeformation:table:alter",

"lakeformation:database:describe"

INSERT OVERWRITE my_table SELECT ...

table:DESCRIBE

table:ALTER

table:UPDATE

database:DESCRIBE

"lakeformation:table:describe",

"lakeformation:table:alter",

"lakeformation:database:describe"

TRUNCATE TABLE my_table;

database:DESCRIBE

table:DESCRIBE

table:SELECT

table:UPDATE

"lakeformation:database:describe",

"lakeformation:table:describe",

"lakeformation:table:alter"

UPDATE table_identifier SET column1 = value1, column2 = value2, ... WHERE condition;

table:DESCRIBE

table:ALTER

table:SELECT

database:DESCRIBE

"lakeformation:table:describe",

"lakeformation:table:alter",

"lakeformation:database:describe"

DELETE FROM table_identifier WHERE conditions;

database:DESCRIBE

database:list_table

table:DESCRIBE

table:ALTER

table:UPDATE

table:SELECT

"lakeformation:database:describe",

"lakeformation:table:describe",

"lakeformation:table:alter"

Auxiliary语句

DESCRIBE TABLE

database:DESCRIBE

table:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe"

SHOW DATABASES

catalog:LIST_DATABASE

database:DESCRIBE

"lakeformation:database:describe"

SHOW CURRENT DATABASE

catalog:LIST_DATABASE

database:DESCRIBE

"lakeformation:database:describe"

SHOW CREATE TABLE

database:DESCRIBE

table:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe"

SHOW TABLES

database:LIST_TABLE

database:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe"

SHOW COLUMNS

database:DESCRIBE

table:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe"

SHOW PARTITIONS

database:DESCRIBE

table:DESCRIBE

"lakeformation:database:describe",

"lakeformation:table:describe"

SHOW FUNCTIONS

database:DESCRIBE

database:LIST_FUNC

"lakeformation:database:describe",

"lakeformation:function:describe"

相关文档