Spark Jar Jobs Using DEW to Acquire Access Credentials for Reading and Writing Data from and to OBS

Scenario

To write the output data of a Spark Jar job to OBS, AK/SK is required for accessing OBS. To ensure the security of AK/SK data, you can use Data Encryption Workshop (DEW) and Cloud Secret Management Service (CSMS) for unified management of AK/SK, effectively avoiding sensitive information leakage and business risks caused by hard-coded or plaintext configuration of programs.

This section walks you through on how a Spark Jar job acquires an AK/SK to read and write data from and to OBS.

Prerequisites

• A shared secret has been created on the DEW console and the secret value has been stored. For details, see Creating a Shared Secret.
• An agency has been created and authorized for DLI to access DEW. The agency must have been granted the following permissions:
  • Permission of the ShowSecretVersion interface for querying secret versions and secret values in DEW: csms:secretVersion:get.
  • Permission of the ListSecretVersions interface for listing secret versions in DEW: csms:secretVersion:list.
  • Permission to decrypt DEW secrets: kms:dek:decrypt

    For details about agency permission examples, see Customizing DLI Agency Permissions and Agency Permission Policies in Common Scenarios.

• DEW can be used to manage access credentials only in Spark 3.3.1 (Spark general queue scenario) or later. When creating a Spark job, select version 3.3.1 and configure the information of the agency that allows DLI to access DEW for the job.

  For details about how to create a custom agency and configure it, see Customizing DLI Agency Permissions.

• To use this function, you need to configure AK/SK for all OBS buckets.

Syntax

On the Spark Jar job editing page, set Runtime Configuration as needed. The configuration information is as follows:

Different OBS buckets use different AK/SK authentication information. You can use the following configuration method to specify the AK/SK information based on the bucket. For details about the parameters, see Table 1.

spark.hadoop.fs.obs.bucket.USER_BUCKET_NAME.dew.access.key= USER_AK_CSMS_KEY
spark.hadoop.fs.obs.bucket.USER_BUCKET_NAME.dew.secret.key= USER_SK_CSMS_KEY
spark.hadoop.fs.obs.security.provider = com.dli.provider.UserObsBasicCredentialProvider
spark.hadoop.fs.dew.csms.secretName= CredentialName
spark.hadoop.fs.dew.endpoint=ENDPOINT
spark.hadoop.fs.dew.csms.version=VERSION_ID
spark.hadoop.fs.dew.csms.cache.time.second =CACHE_TIME
spark.dli.job.agency.name=USER_AGENCY_NAME

Parameter Description

Sample Code

This section describes how to write processed DataGen data to OBS. You need to modify the parameters in the sample Java code based on site requirements.

1. Create an agency for DLI to access DEW and complete authorization.

   For details, see Customizing DLI Agency Permissions.

2. Create a shared secret in DEW. For details, see Creating a Shared Secret.
   1. Log in to the DEW management console.
   2. In the navigation pane on the left, choose Cloud Secret Management Service > Secrets.
   3. On the displayed page, click Create Secret. Set basic secret information.
3. Set job parameters on the DLI Spark Jar job editing page.
   Spark Arguments

   spark.hadoop.fs.obs.bucket.USER_BUCKET_NAME.dew.access.key= USER_AK_CSMS_KEY
   spark.hadoop.fs.obs.bucket.USER_BUCKET_NAME.dew.secret.key= USER_SK_CSMS_KEY
   spark.hadoop.fs.obs.security.provider=com.dli.provider.UserObsBasicCredentialProvider
   spark.hadoop.fs.dew.csms.secretName=obsAkSk
   spark.hadoop.fs.dew.endpoint=kmsendpoint
   spark.hadoop.fs.dew.csms.version=v3
   spark.dli.job.agency.name=agency

4. Sample Code

   For details about the sample code, see Using Spark Jar Jobs to Read and Query OBS Data.