Configuring a Storage-Compute Decoupled Cluster (AK/SK)

Using Hadoop to Access OBS

• Add the following content to the core-site.xml file in the Client installation directory/HDFS/hadoop/etc/hadoop directory on the HDFS client.

  <property>
      <name>fs.obs.access.key</name>
      <value>ak</value>
  </property>
  <property>
      <name>fs.obs.secret.key</name>
      <value>sk</value>
  </property>
  <property>
      <name>fs.obs.endpoint</name>
      <value>obs endpoint</value>
  </property>

  If you use commands that need to submit jobs to Yarn, such as distcp, you need to add the preceding content to the core-site.xml file in the Yarn directory ($client_home/Yarn/config) on the MRS client.

  

  AK and SK will be displayed as plaintext in the configuration file. Exercise caution when setting AK and SK in the file.

  After the configuration is added, you can directly access data on OBS without manually adding the AK/SK and endpoint. For example, run the following command to view the file list of the test_obs_orc directory in the obs-test file system:

  hadoop fs -ls "obs://obs-test/test_obs_orc"

• Add AK/SK and endpoint to the command line to access data on OBS.

  hadoop fs -Dfs.obs.endpoint=xxx -Dfs.obs.access.key=xx -Dfs.obs.secret.key=xx -ls "obs://obs-test/ test_obs_orc"

Using Hive to Access OBS

1. Log in to the service configuration page.
   • For versions earlier than MRS 3.x, log in to the cluster details page and choose Components > Hive > Service Configuration.
   • For MRS 3.x or later, log in to FusionInsight Manager. For details, see Accessing FusionInsight Manager (MRS 3.x or Later). Choose Cluster > Services > Hive > Configurations.

2. In the configuration type drop-down box, switch Basic Configurations to All Configurations.
3. Configure the AK and SK of OBS.
   • For versions earlier than MRS 3.x, click Hive, select Customization, and add the following two configurations to core.site.customized.configs: Name: fs.obs.access.key; Value: AK for accessing OBS; Name: fs.obs.secret.key; Value: SK for accessing OBS.
     Figure 1 Setting the AK/SK for accessing OBS
     
   • For MRS 3.x or later, click Hive(Service), select Customization, and add the following two configurations to core.site.customized.configs: Name: fs.obs.access.key; Value: AK for accessing OBS; Name: fs.obs.secret.key; Value: SK for accessing OBS.
     Figure 2 Configuring the AK/SK for accessing OBS
     

4. Save the configurations and restart Hive.
5. Access the OBS directory in Beeline. For example, run the following command to create a Hive table and specify that data is stored in the test_obs directory in the test-bucket file system:

   create table test_obs(a int, b string) row format delimited fields terminated by "," stored as textfile location "obs://test-bucket/test_obs";

Using Spark to Access OBS

• SparkSQL depends on Hive. Therefore, when configuring OBS on Spark, you need to modify the OBS configuration used in Using Hive to Access OBS.
• In MRS 3.3.0-LTS and later versions, the Spark2x component is renamed Spark, and the role names in the component are also changed. For example, JobHistory2x is changed to JobHistory. Refer to the descriptions and operations related to the component name and role names in the document based on your MRS version.

• spark-beeline and spark-sql

  You can use spark-beeline or spark-sql to log in to the Spark client and run the following commands to configure AK and SK information for accessing OBS:

  set fs.obs.access.key=AK

  set fs.obs.secret.key=SK

  set fs.obs.endpoint=OBS Endpoint

• spark-beeline
  The spark-beeline can access OBS by configuring service parameters on Manager. The procedure is as follows:

  1. Log in to the service configuration page.
     • For versions earlier than MRS 3.x, log in to the cluster details page and choose Components > Spark > Service Configuration.
     • For MRS 3.x or later, log in to FusionInsight Manager. For details, see Accessing FusionInsight Manager (MRS 3.x or Later). Choose Cluster > Services > Spark2x > Configurations.
  2. In the configuration type drop-down box, switch Basic Configurations to All Configurations.
  3. Choose JDBCServer > OBS, and set values for fs.obs.access.key and fs.obs.secret.key.

     If the preceding two parameters cannot be found in the current cluster, choose JDBCServer > Customization in the navigation tree on the left and add the two parameters to the customized parameter spark.core-site.customized.configs.

     Figure 3 Parameters for adding an OBS
     
  4. Save the configurations and restart Spark.
  5. Access OBS in spark-beeline. For example, access the obs://obs-demo-input/table/ directory.

     create table test(id int) location 'obs://obs-demo-input/table/';

• spark-sql and spark-submit

  Both spark-sql and spark-submit can access OBS if you add the following content to the core-site.xml configuration file in the Client installation directory/Spark/spark/conf directory:

  <property>
      <name>fs.obs.access.key</name>
      <value>ak</value>
  </property>
  <property>
      <name>fs.obs.secret.key</name>
      <value>sk</value>
  </property>
  <property>
      <name>fs.obs.endpoint</name>
      <value>obs endpoint</value>
  </property>

Using Presto to Access OBS

1. Go to the cluster details page and choose Components > Presto > Service Configuration.
2. In the configuration type drop-down box, switch Basic Configurations to All Configurations.
3. Search for and configure the following parameters:
   • Set fs.obs.access.key to AK.
   • Set fs.obs.secret.key to SK.

   If the preceding two parameters cannot be found in the current cluster, choose Presto > Hive in the navigation tree on the left and add the two parameters to the customized parameter core.site.customized.configs.

4. Save the configurations and restart Presto.
5. Choose Components > Hive > Service Configuration.
6. In the configuration type drop-down box, switch Basic Configurations to All Configurations.
7. Search for and configure the following parameters:
   • Set fs.obs.access.key to AK.
   • Set fs.obs.secret.key to SK.

8. Save the configurations and restart Hive.
9. On the Presto client, run the following statement to create a schema and set location to an OBS path:

   presto_cli.sh

   CREATE SCHEMA hive.demo WITH (location = 'obs://obs-demo/presto-demo/');

10. Create a table in the schema. The table data is stored in the OBS file system. The following is an example.

    USE hive.demo;

    CREATE TABLE Table name (id int);

    INSERT INTO Table name VALUES (2); In this command, 2 is only used an example. Replace it with the real value.

    CREATE TABLE hive.demo.demo_table WITH (format = 'ORC') AS SELECT * FROM tpch.sf1.customer;

Using Flink to Access OBS

Add the following configuration to the Flink configuration file of the MRS client in Client installation path/Flink/flink/conf/flink-conf.yaml:

fs.obs.access.key: ak  
fs.obs.secret.key: sk  
fs.obs.endpoint: OBS Endpoint

AK and SK will be displayed as plaintext in the configuration file. Exercise caution when setting AK and SK in the file.

After the configuration is added, you can directly access data on OBS without manually adding the AK/SK and endpoint.