Help Center/ MapReduce Service/ Component Operation Guide (LTS) (Ankara Region)/ Using Kafka/ Kafka Encryption and Decryption
Updated on 2024-11-29 GMT+08:00
View PDF
Share

Kafka Encryption and Decryption

Scenario

After a RangerKMS instance is installed for Ranger in an MRS cluster, the encryption and decryption feature is available for Kafka. You can create encrypted Kafka topics. The client automatically encrypts data and transmits it to the Kafka service. The encrypted data is stored on a local disk. The client application reads the encrypted data from the server and automatically decrypts the data on the client.

Prerequisites

  • The Ranger service and RangerKMS instance have been installed in the cluster.
  • To use this function, the Kafka client JAR package of MRS is required. The open-source client is not available for this function.

Procedure

  1. Assign key permissions to users of different roles by referring to Using the RangerKMS Native UI to Manage Permissions and Keys. The following table lists the required permission.

    Table 1 Key permission

    Task

    Required Key Permission

    Creating a topic
    • Get Metadata: Get the metadata of a key.
    • Generate EEK: Generate an EEK.

    Producing data
    • Decrypt EEK: Decrypt an EEK.

    Consuming data
    • Decrypt EEK: Decrypt an EEK.

  2. Log in to the node where the Kafka client is installed as the client installation user and run the following commands to configure environment variables and pass user authentication:

    cd Kafka client installation path

    source bigdata_env

    kinit Component service user

  3. Add a configuration item to the Kafka client configuration file to enable the encryption and decryption feature.

    cd Kafka/kafka/config

    Add the following parameters to the client.properties, producer.properties, and consumer.properties files respectively and save the changes: 
    encryption.keyprovider.class = org.apache.kafka.clients.encryption.RangerKeyProvider
encryption.keymanager.class = org.apache.kafka.clients.encryption.RangerKeyManager
encryption.keyprovider.rangerkms = https://IP address of the RangerKMS instance:Port number,https://IP address of the RangerKMS instance:Port number
    • With the preceding configurations, the producer writes encrypted data to the encrypted topic by default.
    • Without the preceding configurations, the producer writes plaintext data to the encrypted topic by default.
    • With the preceding configurations, the consumer decrypts the encrypted data in the encrypted topic by default.
    • If the preceding configurations are added for the consumer and a plaintext topic is subscribed, the plaintext data will be decrypted into garbled characters.
    • Without the preceding configurations, the consumer does not decrypt the encrypted data in the encrypted topic.
    • To obtain the IP address of the RangerKMS instance, log in to FusionInsight Manager, choose Cluster > Services > Ranger > Instances, and view and record the service IP address.
    • To obtain the port, log in to FusionInsight Manager, choose Cluster > Services > Ranger, and click Configurations > All Configurations. Search for ranger.service.https.port, and view and record the value for the RangerKMS instance.

  4. (Optional) Set the following parameters on the client as you need.

    Parameter

    Description

    Default Value

    kms.request.retries

    Number of retries allowed to access the RangerKMS instance

    2

    encryption.keyprovider.rangerkms.hostname

    IP address and host name of the RangerKMS instance, for example, ip|hostname,ip2|hostanme2

    null

    ranger.eek.cache.size

    Number of tables for caching keys

    1000

    ranger.eek.max.age.second

    Expiration time of cached keys, in seconds

    300

  5. Run the following command to switch to the client directory, for example, /opt/client/Kafka/kafka/bin.

    cd Kafka client installation directory/Kafka/kafka/bin

  6. Create a topic.

    When you run the client script to create a topic, use --bootstrap-server. The --zookeeper command is not supported.

    kafka-topics.sh --create --topic Topic name --partitions 1 --replication-factor 3 --bootstrap-server Broker IP address:port--command-config ../config/client.properties --config encryption.keyname=Key name

    • To obtain the IP address of the Broker, log in to FusionInsight Manager, choose Cluster > Services > Kafka > Instances, and view and record the service IP address.
    • The port number of the Kafka cluster is defaulted to 21007 in security mode and 21005 in normal mode.
    • Key name: name of the key used in 1

  7. Write data to the encrypted topic.

    Run the client script command to write data to the encrypted topic:

    sh kafka-console-producer.sh --broker-list Broker IP address:Port --topic Topic name--producer.config ../config/producer.properties

  8. Read encrypted topic data.

    sh kafka-console-consumer.sh --topic Topic name--bootstrap-server Broker IP address:Port --consumer.config ../config/consumer.properties

Parent topic: Using Kafka