Help Center/ MapReduce Service/ User Guide/ Configuring a Cluster with Decoupled Storage and Compute/ MRS Storage-Compute Decoupling
Updated on 2024-04-11 GMT+08:00
View PDF

MRS Storage-Compute Decoupling

In scenarios that require large storage capacity and elastic compute resources, MRS enables you to store data in OBS and use an MRS cluster for data computing only. In this way, storage and compute are separated.

In the big data storage-compute decoupling scenario, the OBS parallel file system must be used to configure a cluster. Using common object buckets will greatly affect the cluster performance.

Perform the following steps to use the storage-compute decoupling function:

  1. Configure a cluster with decoupled storage and compute.
    Select one of the following configurations (Using an agency is recommended.):
    • Bind an agency of the ECS type to an MRS cluster to access OBS, preventing the AK/SK from being exposed in the configuration file. For details, see Configuring a Storage-Compute Decoupled Cluster (Agency).
    • Configure the AK/SK in an MRS cluster. The AK/SK will be exposed in the configuration file in plaintext. Exercise caution when performing this operation. For details, see Configuring a Storage-Compute Decoupled Cluster (AK/SK).
    • MRS uses the Guardian component to connect to OBS, providing other components with the capabilities of obtaining temporary authentication credentials and fine-grained permission control for accessing OBS. For details, see Interconnecting the Guardian Service with OBS.
      • Currently, using Guardian components to connect to OBS is supported only in MRS 3.3.0-LTS or later versions. For details about configurations in clusters of other versions, see Interconnecting MRS with OBS Using an Agency.
      • Job submission based on the Guardian storage and compute decoupling management plane depends on JobGateway instead of Executor.
  2. Use the cluster.

    After the required permissions for accessing OBS are obtained, components in the MRS cluster can access the corresponding files through the client.

    For details about how to configure components to access OBS, see the following content: