Help Center> Data Lake Insight> FAQs> Problems Related to Spark Jobs> Job Development> How Do I Set the AK/SK for a Queue to Operate an OBS Table?
Updated on 2024-05-16 GMT+08:00
View PDF

How Do I Set the AK/SK for a Queue to Operate an OBS Table?

(Recommended) Solution 1: Using a Temporary AK/SK

The temporary AK/SK is recommended. For details, see Obtaining a Temporary Access Key and Security Token in Identity and Access Management API Reference.

Hard-coded or plaintext AK and SK pose significant security risks. To ensure security, encrypt your AK and SK, store them in configuration files or environment variables, and decrypt them when needed.

Table 1 Obtaining access credentials

Type

Instruction

Description

Flink job

Flink OpenSource SQL Jobs Using DEW to Manage Access Credentials

Guideline for using DEW to manage and access credentials for Flink OpenSource SQL jobs. When writing the output data of Flink jobs to MySQL or GaussDB(DWS), set attributes such as the username and password in the connector.

Flink Jar Jobs Using DEW to Acquire Access Credentials for Reading and Writing Data from and to OBS

Guideline for Flink Jar jobs to acquire an AK/SK to read and write data from and to OBS.

Obtaining Temporary Credentials for Flink Job Agencies

DLI provides a common interface to obtain temporary credentials for Flink job agencies set by users during job launch. The interface encapsulates the obtained temporary credentials for the job agency in the com.huaweicloud.sdk.core.auth.BasicCredentials class.

Guideline for obtaining a temporary credential for a Flink job agency.

Spark job

Spark Jar Jobs Using DEW to Acquire Access Credentials for Reading and Writing Data from and to OBS

Guideline for Spark Jar jobs to acquire an AK/SK to read and write data from and to OBS.

Obtaining Temporary Credentials for Spark Job Agencies

Guideline for obtaining a temporary credential for a Spark Jar job agency.

Solution 2: Setting Up a Spark Jar Job to Obtain the AK/SK

  • To obtain the AK/SK, set the parameters as follows:
    • Create a SparkContext using code.
      val sc: SparkContext = new SparkContext()
sc.hadoopConfiguration.set("fs.obs.access.key", ak)
sc.hadoopConfiguration.set("fs.obs.secret.key", sk)
    • Create a SparkSession using code.
      val sparkSession: SparkSession = SparkSession
      .builder()
      .config("spark.hadoop.fs.obs.access.key", ak)
      .config("spark.hadoop.fs.obs.secret.key", sk)
      .enableHiveSupport()
      .getOrCreate()
  • To obtain the AK/SK and security token and use them together for authentication, set the parameters as follows:
    • Create a SparkContext using code.
      val sc: SparkContext = new SparkContext()
sc.hadoopConfiguration.set("fs.obs.access.key", ak)
sc.hadoopConfiguration.set("fs.obs.secret.key", sk)
sc.hadoopConfiguration.set("fs.obs.session.token", sts)
    • Create a SparkSession using code.
      val sparkSession: SparkSession = SparkSession
      .builder()
      .config("spark.hadoop.fs.obs.access.key", ak)
      .config("spark.hadoop.fs.obs.secret.key", sk)
      .config("spark.hadoop.fs.obs.session.token", sts)
      .enableHiveSupport()
      .getOrCreate()
Parent topic: Job Development

Job Development FAQs

more