How Do I Set Up AK/SK So That a General Queue Can Access Tables Stored in OBS?

(Recommended) Solution 1: Using a Temporary AK/SK

The temporary AK/SK is recommended. For details, see Obtaining a Temporary Access Key and Security Token in Identity and Access Management API Reference.

Hard-coded or plaintext AK and SK pose significant security risks. To ensure security, encrypt your AK and SK, store them in configuration files or environment variables, and decrypt them when needed.

Solution 2: Setting Up a Spark Jar Job to Obtain the AK/SK

• To obtain the AK/SK, set the parameters as follows:
  • Create a SparkContext using code.

    val sc: SparkContext = new SparkContext()
    sc.hadoopConfiguration.set("fs.obs.access.key", ak)
    sc.hadoopConfiguration.set("fs.obs.secret.key", sk)

  • Create a SparkSession using code.

    val sparkSession: SparkSession = SparkSession
          .builder()
          .config("spark.hadoop.fs.obs.access.key", ak)
          .config("spark.hadoop.fs.obs.secret.key", sk)
          .enableHiveSupport()
          .getOrCreate()

• To obtain the AK/SK and security token and use them together for authentication, set the parameters as follows:
  • Create a SparkContext using code.

    val sc: SparkContext = new SparkContext()
    sc.hadoopConfiguration.set("fs.obs.access.key", ak)
    sc.hadoopConfiguration.set("fs.obs.secret.key", sk)
    sc.hadoopConfiguration.set("fs.obs.session.token", sts)

  • Create a SparkSession using code.

    val sparkSession: SparkSession = SparkSession
          .builder()
          .config("spark.hadoop.fs.obs.access.key", ak)
          .config("spark.hadoop.fs.obs.secret.key", sk)
          .config("spark.hadoop.fs.obs.session.token", sts)
          .enableHiveSupport()
          .getOrCreate()