On this page

Show all

Help Center/ MapReduce Service/ User Guide (Kuala Lumpur Region)/ Managing Clusters/ Job Management/ Using Encrypted OBS Data for Job Running

Using Encrypted OBS Data for Job Running

Updated on 2022-08-12 GMT+08:00

View PDF

In encrypted data in OBS file systems can be used to run jobs, and the encrypted job running results can be stored in OBS file systems. Currently, data can be accessed only through an OBS protocol.

OBS supports data encryption and decryption using KMS keys. All encryption and decryption operations are performed on OBS, and keys are managed by DEW.

To use the OBS encryption function in MRS, you must have the KMS Administrator permissions and configure the following settings for the corresponding component:

NOTE:

If the OBS permission control function is enabled in a cluster, the default agency MRS_ECS_DEFAULT_AGENCY configured on the ECS or the AK/SK of the custom agency is used for accessing OBS. OBS uses the received AK/SK to access DEW to obtain the KMS key status. Therefore, you need to bind the KMS Administrator policy to the used agency. Otherwise, OBS returns the "403 Forbidden" error when processing encrypted data. Currently, the KMS Administrator policy is bound to the agency MRS_ECS_DEFAULT_AGENCY by default. If you use a custom agency, you need to manually bind the policy to your custom agency.

Prerequisites

You have configured the function of accessing OBS from MRS first to use the OBS encryption function. For details, see Configuring a Storage-Compute Decoupled Cluster (Agency).

Hive Configuration

Log in to the MRS management console. In the navigation tree on the left, choose Clusters > Active Clusters and click the cluster name.
Choose Components > Hive > Service Configuration.

Switch Basic to All, and search for and set the following parameters:

**Table 1** Data encryption parameters
Parameter	Value	Description
fs.obs.server-side-encryption-type	SSE-KMS	SSE-KMS: KMS keys are used for encryption and decryption NONE: The encryption function is disabled.
fs.obs.server-side-encryption-key	-	(Optional) This parameter indicates an ID of the KMS key used for encryption. If fs.obs.server-side-encryption-type is set to SSE-KMS and this parameter is not set, OBS uses the default KMS key for encryption.
fs.obs.connection.ssl.enabled	true	Whether to establish a secure connection with OBS. true: The secure connection is enabled. To use OBS encryption and decryption, this parameter must be set to true. false: The secure connection is disabled.

Click Save Configuration and select Restart the affected services or instances. Click OK.

Hadoop Configuration

Method 1: Configuration on the GUI

Log in to the MRS management console. In the navigation tree on the left, choose Clusters > Active Clusters and click the cluster name.
Choose Components > HDFS > Service Configuration.

Switch Basic to All, and search for and set the following parameters:

**Table 2** Data encryption parameters
Parameter	Value	Description
fs.obs.server-side-encryption-type	SSE-KMS	SSE-KMS: KMS keys are used for encryption and decryption NONE: The encryption function is disabled.
fs.obs.server-side-encryption-key	-	ID of the KMS key used for encryption. This parameter is optional. If fs.obs.server-side-encryption-type is set to SSE-KMS and this parameter is not set, OBS uses the default KMS key for encryption.
fs.obs.connection.ssl.enabled	true	Whether to establish a secure connection with OBS. true: The secure connection is enabled. To use OBS encryption and decryption, this parameter must be set to true. false: The secure connection is disabled.

Click Save Configuration and select Restart the affected services or instances. Click OK.
Log in to the Master node as user root. The password is the password of user root you set when you create the cluster. If the cluster has multiple Master nodes, log in to each Master node and repeat 5 to 7.
Run the following command to switch to the client directory, for example, /opt/Bigdata/client:

cd /opt/Bigdata/client
Run the following command to update client configurations, and enter the username and password. The username is admin, and the password is the password of user admin you set when you create the cluster.

./ autoRefreshConfig.sh

Method 2: Configuration Through the Client Configuration File

Add the following parameter settings to the client configuration file, for example, /opt/Bigdata/client/HDFS/hadoop/etc/hadoop/core-site.xml, on the Master node. If the cluster has multiple Master nodes, log in to each Master node and perform this operation.

**Table 3** Data encryption parameters
Parameter	Value	Description
fs.obs.server-side-encryption-type	SSE-KMS	SSE-KMS: KMS keys are used for encryption and decryption NONE: The encryption function is disabled.
fs.obs.server-side-encryption-key	-	ID of the KMS key used for encryption. This parameter is optional. If fs.obs.server-side-encryption-type is set to SSE-KMS and this parameter is not set, OBS uses the default KMS key for encryption.
fs.obs.connection.ssl.enabled	true	Whether to establish a secure connection with OBS. true: The secure connection is enabled. To use OBS encryption and decryption, this parameter must be set to true. false: The secure connection is disabled.

HBase Configuration

Method 1: Configuration on the GUI

Log in to the MRS management console. In the navigation tree on the left, choose Clusters > Active Clusters and click the cluster name.
Choose Components > HBase > Service Configuration.

Switch Basic to All, and search for and set the following parameters:

**Table 4** Data encryption parameters
Parameter	Value	Description
fs.obs.server-side-encryption-type	SSE-KMS	SSE-KMS: KMS keys are used for encryption and decryption NONE: The encryption function is disabled.
fs.obs.server-side-encryption-key	-	ID of the KMS key used for encryption. This parameter is optional. If fs.obs.server-side-encryption-type is set to SSE-KMS and this parameter is not set, OBS uses the default KMS key for encryption.
fs.obs.connection.ssl.enabled	true	Whether to establish a secure connection with OBS. true: The secure connection is enabled. To use OBS encryption and decryption, this parameter must be set to true. false: The secure connection is disabled.

Click Save Configuration and select Restart the affected services or instances. Click OK.
Log in to the Master node as user root. The password is the password of user root you set when you create the cluster. If the cluster has multiple Master nodes, log in to each Master node and repeat 5 to 7.
Run the following command to switch to the client directory, for example, /opt/Bigdata/client:

cd /opt/Bigdata/client
Run the following command to update client configurations, and enter the username and password. The username is admin, and the password is the password of user admin you set when you create the cluster.

./ autoRefreshConfig.sh

Method 2: Configuration Through the Client Configuration File

Add the following parameter settings to the client configuration file, for example, /opt/Bigdata/client/HBase/hbase/conf/core-site.xml, on the Master node. If the cluster has multiple Master nodes, log in to each Master node and perform this operation.

**Table 5** Data encryption parameters
Parameter	Value	Description
fs.obs.server-side-encryption-type	SSE-KMS	SSE-KMS: KMS keys are used for encryption and decryption NONE: The encryption function is disabled.
fs.obs.server-side-encryption-key	-	ID of the KMS key used for encryption. This parameter is optional. If fs.obs.server-side-encryption-type is set to SSE-KMS and this parameter is not set, OBS uses the default KMS key for encryption.
fs.obs.connection.ssl.enabled	true	Whether to establish a secure connection with OBS. true: The secure connection is enabled. To use OBS encryption and decryption, this parameter must be set to true. false: The secure connection is disabled.

Spark Configuration

Method 1: Configuration on the GUI

Log in to the MRS management console. In the navigation tree on the left, choose Clusters > Active Clusters and click the cluster name.
Choose Components > Spark > Service Configuration.

Switch Basic to All, and search for and set the following parameters:

**Table 6** Data encryption parameters
Parameter	Value	Description
fs.obs.server-side-encryption-type	SSE-KMS	SSE-KMS: KMS keys are used for encryption and decryption NONE: The encryption function is disabled.
fs.obs.server-side-encryption-key	-	ID of the KMS key used for encryption. This parameter is optional. If fs.obs.server-side-encryption-type is set to SSE-KMS and this parameter is not set, OBS uses the default KMS key for encryption.
fs.obs.connection.ssl.enabled	true	Whether to establish a secure connection with OBS. true: The secure connection is enabled. To use OBS encryption and decryption, this parameter must be set to true. false: The secure connection is disabled.

Click Save Configuration and select Restart the affected services or instances. Click OK.
Log in to the Master node as user root. The password is the password of user root you set when you create the cluster. If the cluster has multiple Master nodes, log in to each Master node and repeat 5 to 7.
Run the following command to switch to the client directory, for example, /opt/Bigdata/client:

cd /opt/Bigdata/client
Run the following command to update client configurations, and enter the username and password. The username is admin, and the password is the password of user admin you set when you create the cluster.

./autoRefreshConfig.sh

Method 2: Configuration Through the Client Configuration File

Add the following parameter settings to the client configuration file, for example, /opt/Bigdata/client/Spark/spark/conf/core-site.xml, on the Master node. If the cluster has multiple Master nodes, log in to each Master node and perform this operation.

**Table 7** Data encryption parameters
Parameter	Value	Description
fs.obs.server-side-encryption-type	SSE-KMS	SSE-KMS: KMS keys are used for encryption and decryption NONE: The encryption function is disabled.
fs.obs.server-side-encryption-key	-	ID of the KMS key used for encryption. This parameter is optional. If fs.obs.server-side-encryption-type is set to SSE-KMS and this parameter is not set, OBS uses the default KMS key for encryption.
fs.obs.connection.ssl.enabled	true	Whether to establish a secure connection with OBS. true: The secure connection is enabled. To use OBS encryption and decryption, this parameter must be set to true. false: The secure connection is disabled.

Presto Configuration

Log in to the MRS management console. In the navigation tree on the left, choose Clusters > Active Clusters and click the cluster name.
Choose Components > Presto > Service Configuration.

Switch Basic to All, and search for and set the following parameters:

**Table 8** Data encryption parameters
Parameter	Value	Description
fs.obs.server-side-encryption-type	SSE-KMS	SSE-KMS: KMS keys are used for encryption and decryption NONE: The encryption function is disabled.
fs.obs.server-side-encryption-key	-	ID of the KMS key used for encryption. This parameter is optional. If fs.obs.server-side-encryption-type is set to SSE-KMS and this parameter is not set, OBS uses the default KMS key for encryption.
fs.obs.connection.ssl.enabled	true	Whether to establish a secure connection with OBS. true: The secure connection is enabled. To use OBS encryption and decryption, this parameter must be set to true. false: The secure connection is disabled.

Click Save Configuration and select Restart the affected services or instances. Click OK.

Parent topic: Job Management

Previous topic: Deleting a Job

Next topic: Configuring Job Notification Rules

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot