Kafka Encryption and Decryption

Updated on 2024-11-29 GMT+08:00

View PDF

Scenario

After a RangerKMS instance is installed for Ranger in an MRS cluster, the encryption and decryption feature is available for Kafka. You can create encrypted Kafka topics. The client automatically encrypts data and transmits it to the Kafka service. The encrypted data is stored on a local disk. The client application reads the encrypted data from the server and automatically decrypts the data on the client.

Prerequisites

The Ranger service and RangerKMS instance have been installed in the cluster.
To use this function, the Kafka client JAR package of MRS is required. The open-source client is not available for this function.

Procedure

Assign key permissions to users of different roles by referring to Using the RangerKMS Native UI to Manage Permissions and Keys. The following table lists the required permission.

**Table 1** Key permission
Task	Required Key Permission
Creating a topic	Get Metadata: Get the metadata of a key. Generate EEK: Generate an EEK.
Producing data	Decrypt EEK: Decrypt an EEK.
Consuming data	Decrypt EEK: Decrypt an EEK.

Log in to the node where the Kafka client is installed as the client installation user and run the following commands to configure environment variables and pass user authentication:

cd Kafka client installation path

source bigdata_env

kinit Component service user
Add a configuration item to the Kafka client configuration file to enable the encryption and decryption feature.

cd Kafka/kafka/config
Add the following parameters to the client.properties, producer.properties, and consumer.properties files respectively and save the changes:
```
encryption.keyprovider.class = org.apache.kafka.clients.encryption.RangerKeyProvider
encryption.keymanager.class = org.apache.kafka.clients.encryption.RangerKeyManager
encryption.keyprovider.rangerkms = https://IP address of the RangerKMS instance:Port number,https://IP address of the RangerKMS instance:Port number
```
NOTICE:
- With the preceding configurations, the producer writes encrypted data to the encrypted topic by default.
- Without the preceding configurations, the producer writes plaintext data to the encrypted topic by default.
- With the preceding configurations, the consumer decrypts the encrypted data in the encrypted topic by default.
- If the preceding configurations are added for the consumer and a plaintext topic is subscribed, the plaintext data will be decrypted into garbled characters.
- Without the preceding configurations, the consumer does not decrypt the encrypted data in the encrypted topic.
- To obtain the IP address of the RangerKMS instance, log in to FusionInsight Manager, choose Cluster > Services > Ranger > Instances, and view and record the service IP address.
- To obtain the port, log in to FusionInsight Manager, choose Cluster > Services > Ranger, and click Configurations > All Configurations. Search for ranger.service.https.port, and view and record the value for the RangerKMS instance.

(Optional) Set the following parameters on the client as you need.

Parameter	Description	Default Value
kms.request.retries	Number of retries allowed to access the RangerKMS instance	2
encryption.keyprovider.rangerkms.hostname	IP address and host name of the RangerKMS instance, for example, ip\|hostname,ip2\|hostanme2	null
ranger.eek.cache.size	Number of tables for caching keys	1000
ranger.eek.max.age.second	Expiration time of cached keys, in seconds	300

Run the following command to switch to the client directory, for example, /opt/client/Kafka/kafka/bin.

cd Kafka client installation directory/Kafka/kafka/bin
Create a topic.

When you run the client script to create a topic, use --bootstrap-server. The --zookeeper command is not supported.

kafka-topics.sh --create --topic Topic name --partitions 1 --replication-factor 3 --bootstrap-server Broker IP address:port--command-config ../config/client.properties --config encryption.keyname=Key name
NOTE:
- To obtain the IP address of the Broker, log in to FusionInsight Manager, choose Cluster > Services > Kafka > Instances, and view and record the service IP address.
- The port number of the Kafka cluster is defaulted to 21007 in security mode and 21005 in normal mode.
- Key name: name of the key used in 1
Write data to the encrypted topic.

Run the client script command to write data to the encrypted topic:

sh kafka-console-producer.sh --broker-list Broker IP address:Port --topic Topic name--producer.config ../config/producer.properties
Read encrypted topic data.

sh kafka-console-consumer.sh --topic Topic name--bootstrap-server Broker IP address:Port --consumer.config ../config/consumer.properties

Parent topic: Using Kafka

Previous topic: Kafka Token Authentication Mechanism Tool Usage

Next topic: Using Kafka UI

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot