Help Center > > User Guide> Managing Active Clusters> Managing Jobs> Using Encrypted OBS Data for Job Running

Using Encrypted OBS Data for Job Running

Updated at: Apr 28, 2020 GMT+08:00

In MRS 1.9.0, MRS 2.1.0, or later versions, encrypted data in OBS buckets can be used to run jobs, and the encrypted job running results can be stored in OBS buckets. Currently, data can be accessed only through an OBS protocol.

OBS supports data encryption and decryption using KMS keys. All encryption and decryption operations are performed on OBS, and keys are managed by DEW.

To use the OBS encryption function in MRS, you must have the KMS Administrator permissions and configure the following settings for the corresponding component:

If the OBS permission control function is enabled in a cluster, the default agency MRS_ECS_DEFAULT_AGENCY configured on the ECS or the AK/SK of the custom agency is used for accessing OBS. OBS uses the received AK/SK to access DEW to obtain the KMS key status. Therefore, you need to bind the KMS Administrator policy to the used agency. Otherwise, OBS returns the "403 Forbidden" error when processing encrypted data. Currently, the KMS Administrator policy is bound to the agency MRS_ECS_DEFAULT_AGENCY by default. If you use a custom agency, you need to manually bind the policy to your custom agency.

Prerequisites

Before using the OBS encryption function, some parameters must be configured for MRS to access OBS. For details, see Accessing OBS Using obs.

Hive Configuration

  1. Log in to the MRS management console. In the navigation tree on the left, choose Clusters > Active Clusters and click the cluster name.
  2. Choose Components > Hive > Service Configuration.
  3. Set Type to All, and search for the following parameters and configure them.

    Table 1 Data encryption parameters

    Parameter

    Value

    Description

    fs.obs.server-side-encryption-type

    SSE-KMS

    • SSE-KMS: KMS keys are used for encryption and decryption
    • NONE: The encryption function is disabled.

    fs.obs.server-side-encryption-key

    -

    ID of the KMS key used for encryption. This parameter is optional.

    If fs.obs.server-side-encryption-type is set to SSE-KMS and this parameter is not set, OBS uses the default KMS key for encryption.

    fs.obs.connection.ssl.enabled

    true

    Whether to establish a secure connection with OBS.

    • true: The secure connection is enabled. To use OBS encryption and decryption, this parameter must be set to true.
    • false: The secure connection is disabled.

  4. Click Save Configuration and select Restart the affected services or instances. Click OK.

Hadoop Configuration

Method 1: Configuration on the GUI

  1. Log in to the MRS management console. In the navigation tree on the left, choose Clusters > Active Clusters and click the cluster name.
  2. Choose Components > HDFS > Service Configuration.
  3. Set Type to All, and search for the following parameters and configure them.

    Table 2 Data encryption parameters

    Parameter

    Value

    Description

    fs.obs.server-side-encryption-type

    SSE-KMS

    • SSE-KMS: KMS keys are used for encryption and decryption
    • NONE: The encryption function is disabled.

    fs.obs.server-side-encryption-key

    -

    ID of the KMS key used for encryption. This parameter is optional.

    If fs.obs.server-side-encryption-type is set to SSE-KMS and this parameter is not set, OBS uses the default KMS key for encryption.

    fs.obs.connection.ssl.enabled

    true

    Whether to establish a secure connection with OBS.

    • true: The secure connection is enabled. To use OBS encryption and decryption, this parameter must be set to true.
    • false: The secure connection is disabled.

  4. Click Save Configuration and select Restart the affected services or instances. Click OK.
  5. Log in to the Master node as user root. The password is the password of user root you set when you create the cluster. If the cluster has multiple Master nodes, log in to each Master node and repeat 5 to 7.
  6. Run the following command to switch to the client directory, for example, /opt/client:

    cd /opt/client

  7. Run the following command to update client configurations, and enter the username and password. The username is admin, and the password is the password of user admin you set when you create the cluster.

    ./ autoRefreshConfig.sh

Method 2: Configuration Through the Client Configuration File

Add the following parameter settings to the client configuration file /opt/client/HDFS/hadoop/etc/hadoop/core-site.xml on the Master node. If the cluster has multiple Master nodes, log in to each Master node and perform this operation.

Table 3 Data encryption parameters

Parameter

Value

Description

fs.obs.server-side-encryption-type

SSE-KMS

  • SSE-KMS: KMS keys are used for encryption and decryption
  • NONE: The encryption function is disabled.

fs.obs.server-side-encryption-key

-

ID of the KMS key used for encryption. This parameter is optional.

If fs.obs.server-side-encryption-type is set to SSE-KMS and this parameter is not set, OBS uses the default KMS key for encryption.

fs.obs.connection.ssl.enabled

true

Whether to establish a secure connection with OBS.

  • true: The secure connection is enabled. To use OBS encryption and decryption, this parameter must be set to true.
  • false: The secure connection is disabled.

HBase Configuration

Method 1: Configuration on the GUI

  1. Log in to the MRS management console. In the navigation tree on the left, choose Clusters > Active Clusters and click the cluster name.
  2. Choose Components > HBase > Service Configuration.
  3. Set Type to All, and search for the following parameters and configure them.

    Table 4 Data encryption parameters

    Parameter

    Value

    Description

    fs.obs.server-side-encryption-type

    SSE-KMS

    • SSE-KMS: KMS keys are used for encryption and decryption
    • NONE: The encryption function is disabled.

    fs.obs.server-side-encryption-key

    -

    ID of the KMS key used for encryption. This parameter is optional.

    If fs.obs.server-side-encryption-type is set to SSE-KMS and this parameter is not set, OBS uses the default KMS key for encryption.

    fs.obs.connection.ssl.enabled

    true

    Whether to establish a secure connection with OBS.

    • true: The secure connection is enabled. To use OBS encryption and decryption, this parameter must be set to true.
    • false: The secure connection is disabled.

  4. Click Save Configuration and select Restart the affected services or instances. Click OK.
  5. Log in to the Master node as user root. The password is the password of user root you set when you create the cluster. If the cluster has multiple Master nodes, log in to each Master node and repeat 5 to 7.
  6. Run the following command to switch to the client directory, for example, /opt/client:

    cd /opt/client

  7. Run the following command to update client configurations, and enter the username and password. The username is admin, and the password is the password of user admin you set when you create the cluster.

    ./ autoRefreshConfig.sh

Method 2: Configuration Through the Client Configuration File

Add the following parameter settings to the client configuration file /opt/client/HBase/hbase/conf/core-site.xml on the Master node. If the cluster has multiple Master nodes, log in to each Master node and perform this operation.

Table 5 Data encryption parameters

Parameter

Value

Description

fs.obs.server-side-encryption-type

SSE-KMS

  • SSE-KMS: KMS keys are used for encryption and decryption
  • NONE: The encryption function is disabled.

fs.obs.server-side-encryption-key

-

ID of the KMS key used for encryption. This parameter is optional.

If fs.obs.server-side-encryption-type is set to SSE-KMS and this parameter is not set, OBS uses the default KMS key for encryption.

fs.obs.connection.ssl.enabled

true

Whether to establish a secure connection with OBS.

  • true: The secure connection is enabled. To use OBS encryption and decryption, this parameter must be set to true.
  • false: The secure connection is disabled.

Spark Configuration

Method 1: Configuration on the GUI

  1. Log in to the MRS management console. In the navigation tree on the left, choose Clusters > Active Clusters and click the cluster name.
  2. Choose Components > Spark > Service Configuration.
  3. Set Type to All, and search for the following parameters and configure them.

    Table 6 Data encryption parameters

    Parameter

    Value

    Description

    fs.obs.server-side-encryption-type

    SSE-KMS

    • SSE-KMS: KMS keys are used for encryption and decryption
    • NONE: The encryption function is disabled.

    fs.obs.server-side-encryption-key

    -

    ID of the KMS key used for encryption. This parameter is optional.

    If fs.obs.server-side-encryption-type is set to SSE-KMS and this parameter is not set, OBS uses the default KMS key for encryption.

    fs.obs.connection.ssl.enabled

    true

    Whether to establish a secure connection with OBS.

    • true: The secure connection is enabled. To use OBS encryption and decryption, this parameter must be set to true.
    • false: The secure connection is disabled.

  4. Click Save Configuration and select Restart the affected services or instances. Click OK.
  5. Log in to the Master node as user root. The password is the password of user root you set when you create the cluster. If the cluster has multiple Master nodes, log in to each Master node and repeat 5 to 7.
  6. Run the following command to switch to the client directory, for example, /opt/client:

    cd /opt/client

  7. Run the following command to update client configurations, and enter the username and password. The username is admin, and the password is the password of user admin you set when you create the cluster.

    ./ autoRefreshConfig.sh

Method 2: Configuration Through the Client Configuration File

Add the following parameter settings to the client configuration file /opt/client/Spark/spark/conf/core-site.xml on the Master node. If the cluster has multiple Master nodes, log in to each Master node and perform this operation.

Table 7 Data encryption parameters

Parameter

Value

Description

fs.obs.server-side-encryption-type

SSE-KMS

  • SSE-KMS: KMS keys are used for encryption and decryption
  • NONE: The encryption function is disabled.

fs.obs.server-side-encryption-key

-

ID of the KMS key used for encryption. This parameter is optional.

If fs.obs.server-side-encryption-type is set to SSE-KMS and this parameter is not set, OBS uses the default KMS key for encryption.

fs.obs.connection.ssl.enabled

true

Whether to establish a secure connection with OBS.

  • true: The secure connection is enabled. To use OBS encryption and decryption, this parameter must be set to true.
  • false: The secure connection is disabled.

Presto Configuration

  1. Log in to the MRS management console. In the navigation tree on the left, choose Clusters > Active Clusters and click the cluster name.
  2. Choose Components > Presto > Service Configuration.
  3. Set Type to All, and search for the following parameters and configure them.

    Table 8 Data encryption parameters

    Parameter

    Value

    Description

    fs.obs.server-side-encryption-type

    SSE-KMS

    • SSE-KMS: KMS keys are used for encryption and decryption
    • NONE: The encryption function is disabled.

    fs.obs.server-side-encryption-key

    -

    ID of the KMS key used for encryption. This parameter is optional.

    If fs.obs.server-side-encryption-type is set to SSE-KMS and this parameter is not set, OBS uses the default KMS key for encryption.

    fs.obs.connection.ssl.enabled

    true

    Whether to establish a secure connection with OBS.

    • true: The secure connection is enabled. To use OBS encryption and decryption, this parameter must be set to true.
    • false: The secure connection is disabled.

  4. Click Save Configuration and select Restart the affected services or instances. Click OK.

Did you find this page helpful?

Submit successfully!

Thank you for your feedback. Your feedback helps make our documentation better.

Failed to submit the feedback. Please try again later.

Which of the following issues have you encountered?







Please complete at least one feedback item.

Content most length 200 character

Content is empty.

OK Cancel