Help Center/ MapReduce Service/ User Guide (ME-Abu Dhabi Region)/ FusionInsight Manager Operation Guide (Applicable to 3.x)/ Security Management/ Security Hardening/ HFile and WAL Encryption
Updated on 2022-12-08 GMT+08:00
View PDF
Share

HFile and WAL Encryption

HFile and WAL Encryption

  • Setting the HFile and WAL encryption mode to SMS4 or AES has a great impact on the system and will cause data loss in case of any misoperation. Therefore, this operation is not recommended.
  • Batch data import using Bulkload does not support data encryption.

HFile and Write ahead log (WAL) in HBase are not encrypted by default. To encrypt them, perform the following operations.

  1. On any HBase node, run the following commands to create a key file as user omm:

    sh ${BIGDATA_HOME}/FusionInsight_HD_8.0.2.1/install/FusionInsight-HBase-2.2.3/hbase/bin/hbase-encrypt.sh <path>/hbase.jks <type> <length> <alias>

    • <path>/hbase.jks indicates the path of the generated jks file.
    • <type> indicates the encryption type, which can be SMS 4 or AES.
    • <length> indicates the key length. SMS 4 supports 16-bit and AES supports 128-bit.
    • <alias> indicates the alias name of key file. When you create the key file for the first time, retain the default value omm.

    For example, to generate an SMS4 encryption key, run the following command:

    sh ${BIGDATA_HOME}/FusionInsight_HD_ 8.0.2.1/install/FusionInsight-HBase-2.2.3/hbase/bin/hbase-encrypt.sh /home/hbase/conf/hbase.jks SMS4 16 omm

    To generate an AES encryption key, run the following command:

    sh ${BIGDATA_HOME}/FusionInsight_HD_8.0.2.1/install/FusionInsight-HBase-2.2.3/hbase/bin/hbase-encrypt.sh /home/hbase/conf/hbase.jks AES 128 omm

    • The cluster operation user must have the rw permission of the <path>/hbase.jks directory. The directory requires already exists.
    • After running the command, enter the same <password> four times. The password encrypted in 3 is the same as the password in this step.

  2. Distribute the generated key files to the same directory on all nodes in the cluster and assign read and write permission to user omm.

    • You need to select a safe procedure to distribute keys based on the enterprise security requirements.
    • If the key files of some nodes are lost, repeat the step to copy the key files from other nodes.

  3. On FusionInsight Manager, set hbase.crypto.keyprovider.parameters.encryptedtext to the encrypted password. Set hbase.crypto.keyprovider.parameters.uri to the path and name of the key file.

    • Format of hbase.crypto.keyprovider.parameters.uri: jceks://<key_Path_Name>.

      <key_Path_Name> indicates the path of the key file. For example, if the path of the key file is /home/hbase/conf/hbase.jks, set this parameter to jceks:///home/hbase/conf/hbase.jks.

    • Format of hbase.crypto.keyprovider.parameters.encryptedtext: <encrypted_password>.

      <encrypted_password> indicates the encrypted password generated during the key file creation. The parameter value is displayed in ciphertext. Run the following command as user omm to obtain the related encrypted password on the nodes where HBase service is installed:

      sh ${BIGDATA_HOME}/FusionInsight_HD_8.0.2.1/install/FusionInsight-HBase-2.2.3/hbase/bin/hbase-encrypt.sh

      After running the command, enter the <password>. The password is the same as that in 1.

  4. On FusionInsight Manager, set hbase.crypto.key.algorithm to SMS4 or AES to use SMS4 or AES for HFile encryption.
  5. On FusionInsight Manager, set hbase.crypto.wal.algorithm to SMS4 or AES to use SMS4 or AES for WAL encryption.
  6. On FusionInsight Manager, set hbase.regionserver.wal.encryption to true.
  7. Save the settings and restart the HBase service for the settings to take effect.
  8. Set the encryption mode when creating an HBase table. <type> indicates the encryption type.

    • When creating a table using commands, directly set the encryption mode to SMS4 or AES.

      create '<table name>', {NAME => 'd', ENCRYPTION => '<type>'}

    • When creating a table using code, set the encryption mode to SMS4 or AES by adding the following information to the code: 
      public void testCreateTable() { String tableName = "user"; Configuration conf = getConfiguration(); HTableDescriptor htd = new HTableDescriptor(TableName.valueOf(tableName)); 
     
    HColumnDescriptor hcd = new HColumnDescriptor("info"); //Set the encryption mode to SMS4 or AES. hcd.setEncryptionType("<type>"); htd.addFamily(hcd); 
 
    HBaseAdmin admin = null; try { admin = new HBaseAdmin(conf); 
         
        if(!admin.tableExists(tableName)) { admin.createTable(htd); } } catch (IOException e) { e.printStackTrace(); } finally { if(admin != null) { try { admin.close(); } catch (IOException e) { e.printStackTrace(); } } } }

  9. If you have configured SMS4 or AES encryption by performing 1 to 7 but do not set the related encryption parameter when creating the table in 8, the inserted data is not encrypted.

    In this case, you can perform the following steps to encrypt inserted data:

    1. Run the flush command for the table to import the data in the memory to the HFile.

      flush'<table_name>'

    2. Run the following commands to modify the table properties:

      disable'<table_name>'

      alter'<table_name>',NAME=>'<column_name>',ENCRYPTION => '<type>'

      enable'<table_name>'

    3. Insert a new data record and flush the table.

      A new data record must be inserted so that the HFile will generate a new HFile and the unencrypted data inserted previously will be rewritten and encrypted.

      put'<table_name>','id2','f1:c1','value222222222222222222222222222222222'

      flush'<table_name>'

    4. Perform the following step to rewrite the HFile:
      major_compact'<table_name>'

      During this step, the HBase table is disabled and cannot provide services. Exercise caution when you perform this step.

Modifying a Key File

Modifying a key file has a great impact on the system and will cause data loss in case of any misoperation. Therefore, this operation is not recommended.

During the HFile and WAL Encryption operation, the related key file must be generated and its password must be set to ensure system security. After a period of running, you can replace the key file with a new one to encrypt HFile and WAL.

  1. Run the following command to generate a new key file as user omm:

    sh ${BIGDATA_HOME}/FusionInsight_HD_8.0.2.1/install/FusionInsight-HBase-2.2.3/hbase/bin/hbase-encrypt.sh <path>/hbase.jks <type> <length> <alias-new>

    • <path>/hbase.jks: indicates the path of the generated hbase.jks file. The path and file name must be consistent with those of the key file generated in HFile and WAL Encryption.
    • <alias-new>: indicates the alias of the key file. The alias must be different with that of the old key file.
    • <type> indicates the encryption type, which can be SMS 4 or AES.
    • <length> indicates the key length. SMS 4 supports 16-bit and AES supports 128-bit.

    For example, to generate an SMS4 encryption key, run the following command:

    sh ${BIGDATA_HOME}/FusionInsight_HD_8.0.2.1/install/FusionInsight-HBase-2.2.3/hbase/bin/hbase-encrypt.sh /home/hbase/conf/hbase.jks SMS4 16 omm_new

    To generate an AES encryption key, run the following command:

    sh ${BIGDATA_HOME}/FusionInsight_HD_8.0.2.1/install/FusionInsight-HBase-2.2.3/hbase/bin/hbase-encrypt.sh /home/hbase/conf/hbase.jks AES 128 omm_new

    • The cluster operation user must have the rw permission of the <path>/hbase.jks directory. The directory requires already exists.
    • After running the command, enter the same <password> three times. The password indicates the password of key files. The password of the old key file can be used, which does not cause any security risk.

  2. Distribute the generated key files to the same directory on all nodes in the cluster and assign read and write permission to user omm.

    You need to select a safe procedure to distribute keys based on the enterprise security requirements.

  3. On the HBase service configuration page of FusionInsight Manager, add custom configuration items, set hbase.crypto.master.key.name to omm_new, set hbase.crypto.master.alternate.key.name to omm, and save the settings.

  4. Restart the HBase service for the configuration to take effect.
  5. In HBase shell, run the major compact command to generate the HFile file based on the new encryption algorithm.

    major_compact '<table_name>'

  6. You can view the major compact progress from the HMaster web page.

  7. When all items in Compaction Progress reach 100% and those in Remaining KVs are 0, run the following command as user omm to destroy the old key file:

    sh ${BIGDATA_HOME}/FusionInsight_HD_8.0.2.1/install/FusionInsight-HBase-2.2.3/hbase/bin/hbase-encrypt.sh <path>/hbase.jks <alias-old>

    • <path>/hbase.jks: indicates the path of the generated hbase.jks file. The path and file name must be consistent with those of the key file generated.
    • <alias-old>: indicates the alias of the old key file to be deleted.

    For example:

    sh ${BIGDATA_HOME}/FusionInsight_HD_8.0.2.1/install/FusionInsight-HBase-2.2.3/hbase/bin/hbase-encrypt.sh /home/hbase/conf/hbase.jks omm

    The cluster operation user must have the rw permission for the <path>/hbase.jks directory. The directory requires already exists.

  8. Repeat 2 and distribute the updated key files again.
  9. Delete the HBase self-defined configuration item hbase.crypto.master.alternate.key.name added in 3 from FusionInsight Manager.
  10. Repeat 4 for the configuration to take effect.
Parent topic: Security Hardening