On this page

Show all

Help Center/ Data Lake Insight/ FAQs/ Problems Related to Spark Jobs/ Job Development/ How Do I Set the AK/SK for a Queue to Operate an OBS Table?

How Do I Set the AK/SK for a Queue to Operate an OBS Table?

Updated on 2023-05-19 GMT+08:00

The temporary AK/SK is recommended. For details, see Obtaining a Temporary Access Key and Security Token in Identity and Access Management API Reference.

  • If the AK and SK are obtained, set the parameters as follows:
    • Create SparkContext using code
      val sc: SparkContext = new SparkContext()
      sc.hadoopConfiguration.set("fs.obs.access.key", ak)
      sc.hadoopConfiguration.set("fs.obs.secret.key", sk)
    • Create SparkSession using code
      val sparkSession: SparkSession = SparkSession
            .builder()
            .config("spark.hadoop.fs.obs.access.key", ak)
            .config("spark.hadoop.fs.obs.secret.key", sk)
            .enableHiveSupport()
            .getOrCreate()
  • If ak, sk, and securitytoken are obtained, the temporary AK/SK and security token must be used at the same time during authentication. The setting is as follows:
    • Create SparkContext using code
      val sc: SparkContext = new SparkContext()
      sc.hadoopConfiguration.set("fs.obs.access.key", ak)
      sc.hadoopConfiguration.set("fs.obs.secret.key", sk)
      sc.hadoopConfiguration.set("fs.obs.session.token", sts)
    • Create SparkSession using code
      val sparkSession: SparkSession = SparkSession
            .builder()
            .config("spark.hadoop.fs.obs.access.key", ak)
            .config("spark.hadoop.fs.obs.secret.key", sk)
            .config("spark.hadoop.fs.obs.session.token", sts)
            .enableHiveSupport()
            .getOrCreate()
NOTE:

For security purposes, you are advised not to include the AK and SK information in the OBS path. In addition, if a table is created in the OBS directory, the OBS path specified by the Path field cannot contain the AK and SK information.

Feedback

Feedback

Feedback

0/500

Selected Content

Submit selected content with the feedback