How Do I Connect an MRS Cluster Client to OBS Using an AK/SK Pair?

Obtaining an AK/SK Pair and Endpoint

Obtain an AK, SK, and endpoint before interconnecting cluster components with OBS.

• Obtaining an AK/SK pair
  1. Log in to the Huawei Cloud management console. Hover over the username in the upper right corner and select My Credentials from the drop-down list.
  2. Click Access Keys. You can obtain the AK from the access key list and SK from the downloaded CSV file.
     If there is no AK available, you can generate one by clicking Add Access Key and and download it by entering the verification code or password and clicking OK.

     

     • You can only download the file before the page is closed. But, if you are unable to obtain the AK, you can recreate it.
     • To ensure security, keep your AK secure and update it regularly by deleting the old one and creating a new one.
• Obtaining an endpoint

  For how to obtain an endpoint, refer to Regions and Endpoints.

Using Hadoop to Access OBS

There are two ways to connect Hadoop to OBS. The first way is to add the AK/SK pair and endpoint to the core-site.xml file in the HDFS client installation directory. The second way is to add the AK/SK pair and endpoint when running Hadoop commands.

• Add the following content to the core-site.xml file in the Client installation directory/HDFS/hadoop/etc/hadoop directory on the HDFS client. For how to install the MRS client, see Installing the Client.

  <property>
      <name>fs.obs.access.key</name>
      <value>AK prepared in Obtaining an AK/SK Pair and Endpoint</value>
  </property>
  <property>
      <name>fs.obs.secret.key</name>
      <value>SK prepared in Obtaining an AK/SK Pair and Endpoint</value>
  </property>
  <property>
      <name>fs.obs.endpoint</name>
      <value>OBS endpoint prepared in Obtaining an AK/SK Pair and Endpoint</value>
  </property>

  If you use commands that need to submit jobs to YARN, such as distcp, add the preceding content to the core-site.xml file in the YARN directory (Client installation directory/Yarn/config) on the MRS client.

  

  AK and SK will be displayed as plaintext in the configuration file. Exercise caution when setting AK and SK in the file.

  Once the configuration is added, you can access data on OBS without the need to manually add the AK/SK and endpoint. For example, run the following command to view the file list of the test_obs_orc directory in the obs-test file system:

  cd Client installation directory

  source bigdata_env

  kinit Component operation user (Skip this step if Kerberos authentication is disabled for the cluster.)

  hadoop fs -ls "obs://obs-test/test_obs_orc"

• Add the AK/SK pair and endpoint to the command line to access data on OBS.

  cd Client installation directory

  source bigdata_env

  kinit Component operation user (Skip this step if Kerberos authentication is disabled for the cluster.)

  hadoop fs -Dfs.obs.endpoint=Endpoint prepared in Obtaining an AK/SK Pair and Endpoint -Dfs.obs.access.key=AK prepared in Obtaining an AK/SK Pair and Endpoint -Dfs.obs.secret.key=SK prepared in Obtaining an AK/SK Pair and Endpoint -ls "obs://obs-test/ test_obs_orc"

Using Hive to Access OBS

1. Log in to the service configuration page.
   • For versions earlier than MRS 3.x, log in to the cluster details page and choose Components > Hive > Service Configuration.
   • For MRS 3.x or later, log in to FusionInsight Manager. For details, see Accessing MRS Manager. Choose Cluster > Services > Hive > Configurations.

2. In the configuration type drop-down box, switch Basic Configurations to All Configurations.
3. Configure the AK and SK of OBS.
   • For versions earlier than MRS 3.x, click Hive, select Customization, and add the following configurations to core.site.customized.configs: Name: fs.obs.access.key; Value: AK prepared in Obtaining an AK/SK Pair and Endpoint; Name: fs.obs.secret.key; Value: SK prepared in Obtaining an AK/SK Pair and Endpoint.
     Figure 1 Setting the AK/SK for accessing OBS
     
   • For MRS 3.x or later, click Hive(Service), select Customization, and add the following configurations to core.site.customized.configs: Name: fs.obs.access.key; Value: AK prepared in Obtaining an AK/SK Pair and Endpoint; Name: fs.obs.secret.key; Value: SK prepared in Obtaining an AK/SK Pair and Endpoint.
     Figure 2 Configuring the AK/SK for accessing OBS
     

4. Save the configurations and restart Hive.
5. Access the OBS directory in Beeline. For example, run the following command to create a Hive table and specify that data is stored in the test_obs directory in the test-bucket file system:

   cd Client installation directory

   source bigdata_env

   kinit Component operation user (Skip this step if Kerberos authentication is disabled for the cluster.)

   create table test_obs(a int, b string) row format delimited fields terminated by "," stored as textfile location "obs://test-bucket/test_obs";

Using Spark to Access OBS

• SparkSQL depends on Hive. Therefore, when configuring OBS on Spark, you need to modify the OBS configuration used in Using Hive to Access OBS.
• In MRS 3.3.0-LTS and later versions, the Spark2x component is renamed Spark, and the role names in the component are also changed. For example, JobHistory2x is changed to JobHistory. Refer to the descriptions and operations related to the component name and role names in the document based on your MRS version.

• spark-beeline and spark-sql

  You can use spark-beeline or spark-sql to log in to the Spark client and run the following commands to configure AK and SK information for accessing OBS:

  set fs.obs.access.key=AK prepared in Obtaining an AK/SK Pair and Endpoint;

  set fs.obs.secret.key=SK prepared in Obtaining an AK/SK Pair and Endpoint;

  set fs.obs.endpoint=Endpoint prepared in Obtaining an AK/SK Pair and Endpoint;

• spark-beeline
  The spark-beeline can access OBS by configuring service parameters on Manager. The procedure is as follows:

  1. Log in to the service configuration page.
     • For versions earlier than MRS 3.x, log in to the cluster details page and choose Components > Spark > Service Configuration.
     • For MRS 3.x or later, log in to FusionInsight Manager. For details, see Accessing MRS Manager. Choose Cluster > Services > Spark2x > Configurations.
  2. In the configuration type drop-down box, switch Basic Configurations to All Configurations.
  3. Choose JDBCServer > OBS and set fs.obs.access.key and fs.obs.secret.key.

     If the preceding two parameters do not exist in the current cluster, choose JDBCServer > Customization in the navigation pane on the left and add the AK and SK prepared in Obtaining an AK/SK Pair and Endpoint to spark.core-site.customized.configs.

     Figure 3 Adding parameters for accessing OBS
     
  4. Save the configurations and restart Spark.
  5. Access OBS in spark-beeline. For example, access the obs://obs-demo-input/table/ directory.

     create table test(id int) location 'obs://obs-demo-input/table/';

• spark-sql and spark-submit

  Both spark-sql and spark-submit can access OBS if you add the following content to the core-site.xml configuration file in the Client installation directory/Spark/spark/conf directory:

  <property>
      <name>fs.obs.access.key</name>
      <value>AK prepared in Obtaining an AK/SK Pair and Endpoint</value>
  </property>
  <property>
      <name>fs.obs.secret.key</name>
      <value>SK prepared in Obtaining an AK/SK Pair and Endpoint</value>
  </property>
  <property>
      <name>fs.obs.endpoint</name>
      <value>Endpoint prepared in Obtaining an AK/SK Pair and Endpoint</value>
  </property>

Using Presto to Access OBS

1. Go to the cluster details page and choose Components > Presto > Service Configuration.
2. In the configuration type drop-down box, switch Basic Configurations to All Configurations.
3. Search for and configure the following parameters:
   • Set fs.obs.access.key to the AK prepared in Obtaining an AK/SK Pair and Endpoint.
   • Set fs.obs.secret.key to the SK prepared in Obtaining an AK/SK Pair and Endpoint.

   If the preceding two parameters cannot be found in the current cluster, choose Presto > Hive in the navigation tree on the left and add the two parameters to the customized parameter core.site.customized.configs.

4. Save the configurations and restart Presto.
5. Choose Components > Hive > Service Configuration.
6. In the configuration type drop-down box, switch Basic Configurations to All Configurations.
7. Search for and configure the following parameters:
   • Set fs.obs.access.key to the AK prepared in Obtaining an AK/SK Pair and Endpoint.
   • Set fs.obs.secret.key to the SK prepared in Obtaining an AK/SK Pair and Endpoint.

8. Save the configurations and restart Hive.
9. On the Presto client, run the following statement to create a schema and set location to an OBS path:

   presto_cli.sh

   CREATE SCHEMA hive.demo WITH (location = 'obs://obs-demo/presto-demo/');

10. Create a table in the schema. The table data is stored in the OBS file system. The following is an example:

    USE hive.demo;

    CREATE TABLE Table name (id int);

    INSERT INTO Table name VALUES (2); In this command, 2 is only used an example. Replace it with the real value.

    CREATE TABLE hive.demo.demo_table WITH (format = 'ORC') AS SELECT * FROM tpch.sf1.customer;

Using Flink to Access OBS

Add the following configuration to the Flink configuration file of the MRS client in Client installation path/Flink/flink/conf/flink-conf.yaml:

fs.obs.access.key: AK prepared in Obtaining an AK/SK Pair and Endpoint  
fs.obs.secret.key: SK prepared in Obtaining an AK/SK Pair and Endpoint  
fs.obs.endpoint: Endpoint prepared in Obtaining an AK/SK Pair and Endpoint

AK and SK will be displayed as plaintext in the configuration file. Exercise caution when setting AK and SK in the file.

Once the configuration is added, you can access data on OBS without the need to manually add the AK/SK and endpoint.