On this page

Show all

Help Center/ Data Lake Insight/ Developer Guide/ Flink Jobs/ Flink Job Agencies/ Flink OpenSource SQL Jobs Using DEW to Manage Access Credentials

Flink OpenSource SQL Jobs Using DEW to Manage Access Credentials

Updated on 2024-04-07 GMT+08:00

View PDF

Scenario

When DLI writes the output data of Flink jobs to MySQL or GaussDB(DWS), you need to set attributes such as the username and password in the connector. However, information such as usernames and passwords is highly sensitive and needs to be encrypted to ensure user data privacy.

Data Encryption Workshop (DEW) and Cloud Secret Management Service (CSMS) joint form a secure, reliable, and easy-to-use privacy data encryption and decryption solution.

Users or applications can use CSMS to create, retrieve, update, and delete credentials in a unified manner throughout the secret lifecycle. CSMS can help you eliminate risks incurred by hardcoding, plaintext configuration, and permission abuse.

This section walks you through on how to use DEW to manage access credentials for Flink OpenSource SQL jobs.

Prerequisites

A shared secret has been created on the DEW console and the secret value has been stored. For details, see Creating a Shared Secret.
An agency has been created and authorized for DLI to access DEW. The agency must have been granted the following permissions:
- Permission of the ShowSecretVersion interface for querying secret versions and secret values in DEW: csms:secretVersion:get.
- Permission of the ListSecretVersions interface for listing secret versions in DEW: csms:secretVersion:list.
- Permission to decrypt DEW secrets: kms:dek:decrypt
For details about agency permission examples, see Customizing DLI Agency Permissions and Agency Permission Policies in Common Scenarios.
DEW can be used to manage access credentials only in Flink 1.15. When creating a Flink job, select version 1.15 and configure the information of the agency that allows DLI to access DEW for the job.
On the DLI management console, create an enhanced datasource connection and configure the network connection between DLI and the data source.

For details, see Enhanced Datasource Connections.

Syntax

create table tableName(
  attr_name attr_type 
  (',' attr_name attr_type)* 
  (',' WATERMARK FOR rowtime_column_name AS watermark-strategy_expression)
)
with (
   ...
  'dew.endpoint'='',
  'dew.csms.secretName'='',
  'dew.csms.decrypt.fields'='',
  'dew.projectId'='',
  'dew.csms.version'=''
  
);

Parameter Description

**Table 1** Parameters
Parameter	Mandatory	Default Value	Data Type	Description
dew.endpoint	Yes	None	String	Endpoint of the DEW service to be used. See Regions and Endpoints. Configuration example: 'dew.endpoint'='kms.cn-xxxx.myhuaweicloud.com'
dew.projectId	No	Yes	String	ID of the project DEW belongs to. The default value is the ID of the project where the Flink job is. See Obtaining a Project ID.
dew.csms.secretName	Yes	None	String	Name of the shared secret in DEW's secret management. Configuration example: 'dew.csms.secretName'='secretInfo'
dew.csms.decrypt.fields	Yes	None	String	Specify which fields in the connector with attribute need to be decrypted using DEW's CSMS. Separate the field attributes with commas, for example, 'dew.csms.decrypt.fields'='field1,field2,field3'
dew.csms.version	No	Latest version	String	Version number (certificate version identifier) of the shared secret in DEW's secret management. If not specified, the latest version of the shared secret is obtained by default. Configuration example: 'dew.csms.version'='v1'

Example

This example demonstrates how to configure Flink OpenSource SQL to manage access credentials using DEW by generating random data through a DataGen table and outputting it to a MySQL result table.

Create an enhanced datasource connection between DLI and MySQL.
Create an agency for DLI to access DEW and complete authorization. For details, see Customizing DLI Agency Permissions.
Create a shared secret in DEW. For details, see Creating a Shared Secret.
1. Log in to the DEW management console.
2. In the navigation pane on the left, choose Cloud Secret Management Service > Secrets.
3. On the displayed page, click Create Secret. Set basic secret information.
4. In this example, the MySQL credential value is configured as follows:
  - "MySQLUsername":"demo"
  - "MySQLPassword":"*******", where ******* indicates the password for accessing the MySQL database.

Enter a SQL statement in the editing pane. The following is an example:

create table dataGenSource(
  user_id string,
  amount int
) with (
  'connector' = 'datagen',
  'rows-per-second' = '1', --Generate a piece of data per second.
  'fields.user_id.kind' = 'random', --Specify a random generator for the user_id field.
  'fields.user_id.length' = '3' --Limit the length of user_id to 3.
);

CREATE TABLE jdbcSink ( 
  user_id string,
  amount int
 ) 
WITH (  
 'connector' = 'jdbc',  
  'url? = 'jdbc:mysql://MySQLAddress:MySQLPort/flink',--flink is the MySQL database where the orders table locates.
 'table-name' = 'orders',  
 'username' = 'MySQLUsername',  -- Shared secret in DEW whose name is secretInfo and version is v1. The key MySQLUsername defines the secret value. The value is the user's sensitive information.
 'password' = 'MySQLPassword',  -- Shared secret in DEW whose name is secretInfo and version is v1. The key MySQLPassword defines the secret value. The value is the user's sensitive information.
 'sink.buffer-flush.max-rows' = '1',
 'dew.endpoint'='kms.cn-xxxx.myhuaweicloud.com', --Endpoint information for the DEW service being used
 'dew.csms.secretName'='secretInfo', --Name of the DEW shared secret
 'dew.csms.decrypt.fields'='username,password', --The password field value must be decrypted and replaced using DEW secret management.
 'dew.csms.version'='v1'
);

insert into jdbcSink select * from dataGenSOurce;

Parent topic: Flink Job Agencies

Previous topic: Flink Job Agencies

Next topic: Flink Jar Jobs Using DEW to Acquire Access Credentials for Reading and Writing Data from and to OBS

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot