Using Encrypted OBS Data for Job Running

In MRS 1.9.x encrypted data in OBS file systems can be used to run jobs, and the encrypted job running results can be stored in OBS file systems. Currently, data can be accessed only through an OBS protocol.

OBS supports data encryption and decryption using KMS keys. All encryption and decryption operations are performed on OBS, and keys are managed by DEW.

To use the OBS encryption function in MRS, you must have the KMS Administrator permissions and configure the following settings for the corresponding component:

If the OBS permission control function is enabled in a cluster, the default agency MRS_ECS_DEFAULT_AGENCY configured on the ECS or the AK/SK of the custom agency is used for accessing OBS. OBS uses the received AK/SK to access DEW to obtain the KMS key status. Therefore, you need to bind the KMS Administrator policy to the used agency. Otherwise, OBS returns the "403 Forbidden" error when processing encrypted data. Currently, the KMS Administrator policy is bound to the agency MRS_ECS_DEFAULT_AGENCY by default. If you use a custom agency, you need to manually bind the policy to your custom agency.

Prerequisites

You have configured the function of accessing OBS from MRS first to use the OBS encryption function. For details, see Configuring a Storage-Compute Decoupled Cluster (Agency).

Hive Configuration

Log in to the MRS management console. In the navigation tree on the left, choose Clusters > Active Clusters and click the cluster name.
Choose Components > Hive > Service Configuration.

Switch Basic to All, and search for and set the following parameters:

Click to enlarge

**Table 1** Data encryption parameters
Parameter	Value	Description
fs.obs.server-side-encryption-type	SSE-KMS	SSE-KMS: KMS keys are used for encryption and decryption NONE: The encryption function is disabled.
fs.obs.server-side-encryption-key	-	(Optional) This parameter indicates an ID of the KMS key used for encryption. If fs.obs.server-side-encryption-type is set to SSE-KMS and this parameter is not set, OBS uses the default KMS key for encryption.
fs.obs.connection.ssl.enabled	true	Whether to establish a secure connection with OBS. true: The secure connection is enabled. To use OBS encryption and decryption, this parameter must be set to true. false: The secure connection is disabled.

Click Save Configuration and save the modified parameters as prompted.

Hadoop Configuration

Method 1: Configuration on the GUI

Log in to the MRS management console. In the navigation tree on the left, choose Clusters > Active Clusters and click the cluster name.
Choose Components > HDFS > Service Configuration.

Switch Basic to All, and search for and set the following parameters:

**Table 2** Data encryption parameters
Parameter	Value	Description
fs.obs.server-side-encryption-type	SSE-KMS	SSE-KMS: KMS keys are used for encryption and decryption NONE: The encryption function is disabled.
fs.obs.server-side-encryption-key	-	ID of the KMS key used for encryption. This parameter is optional. If fs.obs.server-side-encryption-type is set to SSE-KMS and this parameter is not set, OBS uses the default KMS key for encryption.
fs.obs.connection.ssl.enabled	true	Whether to establish a secure connection with OBS. true: The secure connection is enabled. To use OBS encryption and decryption, this parameter must be set to true. false: The secure connection is disabled.

Click Save Configuration and operate as prompted.
Log in to the Master node as user root. The password is the password of user root you set when you create the cluster. If the cluster has multiple Master nodes, log in to each Master node and repeat 5 to 7.
Run the following command to switch to the client directory, for example, /opt/Bigdata/client:

cd /opt/Bigdata/client
Run the following command to update client configurations, and enter the username and password. The username is admin, and the password is the password of user admin you set when you create the cluster.

./ autoRefreshConfig.sh

Method 2: Configuration Through the Client Configuration File

Add the following parameter settings to the client configuration file, for example, /opt/Bigdata/client/HDFS/hadoop/etc/hadoop/core-site.xml, on the Master node. If the cluster has multiple Master nodes, log in to each Master node and perform this operation.

**Table 3** Data encryption parameters
Parameter	Value	Description
fs.obs.server-side-encryption-type	SSE-KMS	SSE-KMS: KMS keys are used for encryption and decryption NONE: The encryption function is disabled.
fs.obs.server-side-encryption-key	-	ID of the KMS key used for encryption. This parameter is optional. If fs.obs.server-side-encryption-type is set to SSE-KMS and this parameter is not set, OBS uses the default KMS key for encryption.
fs.obs.connection.ssl.enabled	true	Whether to establish a secure connection with OBS. true: The secure connection is enabled. To use OBS encryption and decryption, this parameter must be set to true. false: The secure connection is disabled.

HBase Configuration

Method 1: Configuration on the GUI

Log in to the MRS management console. In the navigation tree on the left, choose Clusters > Active Clusters and click the cluster name.
Choose Components > HBase > Service Configuration.

Switch Basic to All, and search for and set the following parameters:

**Table 4** Data encryption parameters
Parameter	Value	Description
fs.obs.server-side-encryption-type	SSE-KMS	SSE-KMS: KMS keys are used for encryption and decryption NONE: The encryption function is disabled.
fs.obs.server-side-encryption-key	-	ID of the KMS key used for encryption. This parameter is optional. If fs.obs.server-side-encryption-type is set to SSE-KMS and this parameter is not set, OBS uses the default KMS key for encryption.
fs.obs.connection.ssl.enabled	true	Whether to establish a secure connection with OBS. true: The secure connection is enabled. To use OBS encryption and decryption, this parameter must be set to true. false: The secure connection is disabled.

Click Save Configuration and operate as prompted.
Log in to the Master node as user root. The password is the password of user root you set when you create the cluster. If the cluster has multiple Master nodes, log in to each Master node and repeat 5 to 7.
Run the following command to switch to the client directory, for example, /opt/Bigdata/client:

cd /opt/Bigdata/client
Run the following command to update client configurations, and enter the username and password. The username is admin, and the password is the password of user admin you set when you create the cluster.

./ autoRefreshConfig.sh

Method 2: Configuration Through the Client Configuration File

Add the following parameter settings to the client configuration file, for example, /opt/Bigdata/client/HBase/hbase/conf/core-site.xml, on the Master node. If the cluster has multiple Master nodes, log in to each Master node and perform this operation.

**Table 5** Data encryption parameters
Parameter	Value	Description
fs.obs.server-side-encryption-type	SSE-KMS	SSE-KMS: KMS keys are used for encryption and decryption NONE: The encryption function is disabled.
fs.obs.server-side-encryption-key	-	ID of the KMS key used for encryption. This parameter is optional. If fs.obs.server-side-encryption-type is set to SSE-KMS and this parameter is not set, OBS uses the default KMS key for encryption.
fs.obs.connection.ssl.enabled	true	Whether to establish a secure connection with OBS. true: The secure connection is enabled. To use OBS encryption and decryption, this parameter must be set to true. false: The secure connection is disabled.

Spark Configuration

Method 1: Configuration on the GUI

Log in to the MRS management console. In the navigation tree on the left, choose Clusters > Active Clusters and click the cluster name.
Choose Components > Spark > Service Configuration.

Switch Basic to All, and search for and set the following parameters:

**Table 6** Data encryption parameters
Parameter	Value	Description
fs.obs.server-side-encryption-type	SSE-KMS	SSE-KMS: KMS keys are used for encryption and decryption NONE: The encryption function is disabled.
fs.obs.server-side-encryption-key	-	ID of the KMS key used for encryption. This parameter is optional. If fs.obs.server-side-encryption-type is set to SSE-KMS and this parameter is not set, OBS uses the default KMS key for encryption.
fs.obs.connection.ssl.enabled	true	Whether to establish a secure connection with OBS. true: The secure connection is enabled. To use OBS encryption and decryption, this parameter must be set to true. false: The secure connection is disabled.

Click Save Configuration and operate as prompted.
Log in to the Master node as user root. The password is the password of user root you set when you create the cluster. If the cluster has multiple Master nodes, log in to each Master node and repeat 5 to 7.
Run the following command to switch to the client directory, for example, /opt/Bigdata/client:

cd /opt/Bigdata/client
Run the following command to update client configurations, and enter the username and password. The username is admin, and the password is the password of user admin you set when you create the cluster.

./autoRefreshConfig.sh

Method 2: Configuration Through the Client Configuration File

Add the following parameter settings to the client configuration file, for example, /opt/Bigdata/client/Spark/spark/conf/core-site.xml, on the Master node. If the cluster has multiple Master nodes, log in to each Master node and perform this operation.

**Table 7** Data encryption parameters
Parameter	Value	Description
fs.obs.server-side-encryption-type	SSE-KMS	SSE-KMS: KMS keys are used for encryption and decryption NONE: The encryption function is disabled.
fs.obs.server-side-encryption-key	-	ID of the KMS key used for encryption. This parameter is optional. If fs.obs.server-side-encryption-type is set to SSE-KMS and this parameter is not set, OBS uses the default KMS key for encryption.
fs.obs.connection.ssl.enabled	true	Whether to establish a secure connection with OBS. true: The secure connection is enabled. To use OBS encryption and decryption, this parameter must be set to true. false: The secure connection is disabled.

Presto Configuration

Log in to the MRS management console. In the navigation tree on the left, choose Clusters > Active Clusters and click the cluster name.
Choose Components > Presto > Service Configuration.

Switch Basic to All, and search for and set the following parameters:

**Table 8** Data encryption parameters
Parameter	Value	Description
fs.obs.server-side-encryption-type	SSE-KMS	SSE-KMS: KMS keys are used for encryption and decryption NONE: The encryption function is disabled.
fs.obs.server-side-encryption-key	-	ID of the KMS key used for encryption. This parameter is optional. If fs.obs.server-side-encryption-type is set to SSE-KMS and this parameter is not set, OBS uses the default KMS key for encryption.
fs.obs.connection.ssl.enabled	true	Whether to establish a secure connection with OBS. true: The secure connection is enabled. To use OBS encryption and decryption, this parameter must be set to true. false: The secure connection is disabled.