MRS Storage-Compute Decoupling

Updated on 2024-11-29 GMT+08:00

View PDF

In scenarios that require large storage capacity and elastic compute resources, MRS enables you to store data in OBS and use an MRS cluster for data computing only. In this way, storage and compute are separated.

NOTE:

In the big data storage-compute decoupling scenario, the OBS parallel file system must be used to configure a cluster. Using common object buckets will greatly affect the cluster performance.

Perform the following steps to use the storage-compute decoupling function:

Configure a cluster with decoupled storage and compute.
Select one of the following configurations (Using an agency is recommended.):
- Bind an agency of the ECS type to an MRS cluster to access OBS, preventing the AK/SK from being exposed in the configuration file. For details, see Configuring a Storage-Compute Decoupled Cluster (Agency).
- Configure the AK/SK in an MRS cluster. The AK/SK will be exposed in the configuration file in plaintext. Exercise caution when performing this operation. For details, see Configuring a Storage-Compute Decoupled Cluster (AK/SK).
- MRS uses the Guardian component to connect to OBS, providing other components with the capabilities of obtaining temporary authentication credentials and fine-grained permission control for accessing OBS. For details, see Interconnecting the Guardian Service with OBS.
  NOTE:
  Currently, using Guardian components to connect to OBS is supported only in MRS 3.3.0-LTS or later versions. For details about configurations in clusters of other versions, see Interconnecting MRS with OBS Using an Agency.
  
  Job submission based on the Guardian storage and compute decoupling management plane depends on JobGateway instead of Executor.
Use the cluster.
After the required permissions for accessing OBS are obtained, components in the MRS cluster can access the corresponding files through the client.

For details about how to configure components to access OBS, see the following content:
- Interconnecting MRS with OBS Using an Agency
- Interconnecting Components with OBS Using Guardian

Parent topic: Configuring a Cluster with Decoupled Storage and Compute

Previous topic: Configuring a Cluster with Decoupled Storage and Compute

Next topic: Interconnecting with OBS Using the Cluster Agency Mechanism

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot