Help Center/ MapReduce Service/ User Guide (Ankara Region)/ Configuring a Cluster with Decoupled Storage and Compute/ Interconnecting with OBS Using the Cluster Agency Mechanism/ Configuring a Storage-Compute Decoupled Cluster (AK/SK)
Updated on 2024-11-29 GMT+08:00
View PDF
Share

Configuring a Storage-Compute Decoupled Cluster (AK/SK)

In or later, OBS can be interconnected with MRS using obs://. Currently, Hadoop, Hive, Spark, Presto, and Flink are supported. HBase cannot use obs:// to interconnect with OBS.

MRS provides the following configuration modes for accessing OBS. You can select one of them. The agency mode is recommended.

  • Bind an agency of the ECS type to an MRS cluster to access OBS, preventing the AK/SK from being exposed in the configuration file. For details, see Configuring a Storage-Compute Decoupled Cluster (Agency).
  • Configure the AK/SK in an MRS cluster. The AK/SK will be exposed in the configuration file in plaintext. Exercise caution when performing this operation. For details, see the following part in this section.
  • To improve data write performance, log in to the Manager and choose Cluster > Services > Name of the service to be modified > Configurations. Change the value of fs.obs.buffer.dir to the data disk directory.
  • In the storage-compute decoupled scenario, the OBS parallel file system must be used to configure a cluster. For details, see . Using common object buckets will greatly affect the cluster performance.
  • In MRS 3.2.0-LTS.1 and later versions, components prevent mis-deletion by default. That is, file data deleted by component users is not directly deleted but stored in the recycle bin directory in the OBS file system.

    To save OBS space, you need to enable periodical deletion of file data from the OBS recycle bin by referring to Configuring the Policy for Clearing Component Data in the Recycle Bin.

  • Configuration files containing authentication passwords pose security risks. Delete such files after configuration or store them securely.
  • Commands carrying authentication passwords pose security risks. Disable historical command recording before running such commands to prevent information leakage.

Using Hadoop to Access OBS

  • Add the following content to the core-site.xml file in the Client installation directory/HDFS/hadoop/etc/hadoop directory on the HDFS client. 
    <property>
    <name>fs.obs.access.key</name>
    <value>ak</value>
</property>
<property>
    <name>fs.obs.secret.key</name>
    <value>sk</value>
</property>
<property>
    <name>fs.obs.endpoint</name>
    <value>obs endpoint</value>
</property>

    If you use commands that need to submit jobs to Yarn, such as distcp, you need to add the preceding content to the core-site.xml file in the Yarn directory ($client_home/Yarn/config) on the MRS client.

    AK and SK will be displayed as plaintext in the configuration file. Exercise caution when setting AK and SK in the file.

    After the configuration is added, you can directly access data on OBS without manually adding the AK/SK and endpoint. For example, run the following command to view the file list of the test_obs_orc directory in the obs-test file system:

    hadoop fs -ls "obs://obs-test/test_obs_orc"

  • Add AK/SK and endpoint to the command line to access data on OBS.

    hadoop fs -Dfs.obs.endpoint=xxx -Dfs.obs.access.key=xx -Dfs.obs.secret.key=xx -ls "obs://obs-test/ test_obs_orc"

Using Hive to Access OBS

  1. Log in to the service configuration page.

    Log in to FusionInsight Manager. For details, see Accessing FusionInsight Manager. Choose Cluster > Services > Hive > Configurations.

  2. In the configuration type drop-down box, switch Basic Configurations to All Configurations.
  3. Search for fs.obs.access.key and fs.obs.secret.key and set them to the AK and SK of OBS respectively.

    If the preceding two parameters cannot be found in the current cluster, choose Hive > Customization in the navigation tree on the left and add the two parameters to the customized parameter core.site.customized.configs.

  4. Save the configurations and restart Hive.
  5. Access the OBS directory in Beeline. For example, run the following command to create a Hive table and specify that data is stored in the test_obs directory in the test-bucket file system:

    create table test_obs(a int, b string) row format delimited fields terminated by "," stored as textfile location "obs://test-bucket/test_obs";

Using Spark to Access OBS

  • SparkSQL depends on Hive. Therefore, when configuring OBS on Spark, you need to modify the OBS configuration used in Using Hive to Access OBS.
  • In MRS 3.3.0-LTS and later versions, the Spark2x component is renamed Spark, and the role names in the component are also changed. For example, JobHistory2x is changed to JobHistory. Refer to the descriptions and operations related to the component name and role names in the document based on your MRS version.
  • spark-beeline and spark-sql

    You can use spark-beeline or spark-sql to log in to the Spark client and run the following commands to configure AK and SK information for accessing OBS:

    set fs.obs.access.key=AK

    set fs.obs.secret.key=SK

    set fs.obs.endpoint=OBS Endpoint

  • spark-beeline
    The spark-beeline can access OBS by configuring service parameters on Manager. The procedure is as follows:
    1. Log in to the service configuration page.

      Log in to FusionInsight Manager. For details, see Accessing FusionInsight Manager. Choose Cluster > Services > Spark2x > Configurations.

    2. In the configuration type drop-down box, switch Basic Configurations to All Configurations.
    3. Choose JDBCServer > OBS, and set values for fs.obs.access.key and fs.obs.secret.key.

      If the preceding two parameters cannot be found in the current cluster, choose JDBCServer > Customization in the navigation tree on the left and add the two parameters to the customized parameter spark.core-site.customized.configs.

      Figure 1 Parameters for adding an OBS
    4. Save the configurations and restart Spark.
    5. Access OBS in spark-beeline. For example, access the obs://obs-demo-input/table/ directory.

      create table test(id int) location 'obs://obs-demo-input/table/';

  • spark-sql and spark-submit

    Both spark-sql and spark-submit can access OBS if you add the following content to the core-site.xml configuration file in the Client installation directory/Spark/spark/conf directory:

    <property>
    <name>fs.obs.access.key</name>
    <value>ak</value>
</property>
<property>
    <name>fs.obs.secret.key</name>
    <value>sk</value>
</property>
<property>
    <name>fs.obs.endpoint</name>
    <value>obs endpoint</value>
</property>

Using Flink to Access OBS

Add the following configuration to the Flink configuration file of the MRS client in Client installation path/Flink/flink/conf/flink-conf.yaml: 
fs.obs.access.key: ak  
fs.obs.secret.key: sk  
fs.obs.endpoint: OBS Endpoint

AK and SK will be displayed as plaintext in the configuration file. Exercise caution when setting AK and SK in the file.

After the configuration is added, you can directly access data on OBS without manually adding the AK/SK and endpoint.