Configuring the Encrypted Transmission

Scenario

This topic describes how to configure the server and client parameters of the Flume service (including the Flume role) after the cluster is installed to ensure proper service running.

This section applies to MRS 3.x or later.

Prerequisites

The cluster and Flume service have been installed.

Procedure

Generate the certificate trust lists of the server and client of the Flume role respectively.
1. Remotely log in to the node using ECM where the Flume server is to be installed as user omm. Go to the ${BIGDATA_HOME}/FusionInsight_Porter_8.1.0.1/install/FusionInsight-Flume-1.9.0/flume/bin directory.
  cd ${BIGDATA_HOME}/FusionInsight_Porter_8.1.0.1/install/FusionInsight-Flume-1.9.0/flume/bin
  
  The version 8.1.0.1 is used as an example. Replace it with the actual version number.
2. Run the following command to generate and export the server and client certificates of the Flume role:
  sh geneJKS.sh -f xxx -g xxx
  The generated certificate is saved in the ${BIGDATA_HOME}/FusionInsight_Porter_8.1.0.1/install/FusionInsight-Flume-1.9.0/flume/conf path .
  - flume_sChat.jks is the certificate library of the Flume role server. flume_sChat.crt is the exported file of the flume_sChat.jks certificate. -f indicates the password of the certificate and certificate library.
  - flume_cChat.jks is the certificate library of the Flume role client. flume_cChat.crt is the exported file of the flume_cChat.jks certificate. -g indicates the password of the certificate and certificate library.
  - flume_sChatt.jks and flume_cChatt.jks are the SSL certificate trust lists of the Flume server and client, respectively.
  All user-defined passwords involved in this section must meet the following strength requirements:
  
  The password must contain at least four types of uppercase letters, lowercase letters, digits, and special characters.
  
  The password must contain 8 to 64 characters.
  
  It is recommended that the user-defined passwords be changed periodically (for example, every three months), and certificates and trust lists be generated again to ensure security.

Configure the server parameters of the Flume role and upload the configuration file to the cluster.

Remotely log in to any node where the Flume role is located as user omm using ECM. Run the following command to go to the ${BIGDATA_HOME}/FusionInsight_Porter_8.1.0.1/install/FusionInsight-Flume-1.9.0/flume/bin directory:
cd ${BIGDATA_HOME}/FusionInsight_Porter_8.1.0.1/install/FusionInsight-Flume-1.9.0/flume/bin
Run the following command to generate and obtain Flume server keystore password, trust list password, and keystore-password encrypted private key information. Enter the password twice and confirm the password. It is the password of the flume_sChat.jks certificate library.
./genPwFile.sh

cat password.property

Use the Flume configuration tool on the FusionInsight Manager portal to configure the server parameters and generate the configuration file.

Log in to FusionInsight Manager. Choose Services > Flume > Configuration Tool.
Figure 1 Choosing Configuration Tool
Set Agent Name to server. Select the source, channel, and sink to be used, drag them to the GUI on the right, and connect them.
For example, use Avro Source, File Channel, and HDFS Sink, as shown in Figure 2.

Figure 2 Example for the Flume configuration tool
Double-click the source, channel, and sink. Set corresponding configuration parameters by seeing Table 1 based on the actual environment.
- If the server parameters of the Flume role have been configured, you can choose Services > Flume > Instance on FusionInsight Manager. Then select the corresponding Flume role instance and click the Download button behind the flume.config.file parameter on the Instance Configurations page to obtain the existing server parameter configuration file. Choose Services > Flume > Import to change the relevant configuration items of encrypted transmission after the file is imported.
- It is recommended that the numbers of Sources, Channels, and Sinks do not exceed 40 during configuration file import. Otherwise, the response time may be very long.

Click Export to save the properties.properties configuration file to the local. There can be security risks if a configuration file contains the authentication password. You are advised to delete the configuration file or use other secure methods to keep the password.

**Table 1** Parameters to be modified of the Flume role server
Parameter	Description	Example Value
ssl	Specifies whether to enable the SSL authentication. (You are advised to enable this function to ensure security.) true indicates that the function is enabled. false indicates that the client authentication function is not enabled.	true
keystore	Indicates the server certificate.	${BIGDATA_HOME}/FusionInsight_Porter_8.1.0.1/install/FusionInsight-Flume-1.9.0/flume/conf/flume_sChat.jks
keystore-password	Specifies the password of the key library, which is the password required to obtain the keystore information. Enter the value of password obtained in 2.b.	-
truststore	Indicates the SSL certificate trust list of the server.	${BIGDATA_HOME}/FusionInsight_Porter_8.1.0.1/install/FusionInsight-Flume-1.9.0/flume/conf/flume_sChatt.jks
truststore-password	Specifies the trust list password, which is the password required to obtain the truststore information. Enter the value of password obtained in 2.b.	-

Log in to FusionInsight Manager and choose Cluster > Name of the desired cluster > Services > Flume. On the displayed page, click the Flume role under Role.
Figure 3 Clicking the Flume role
Select the Flume role of the node where the configuration file is to be uploaded, choose Instance Configurations > Import beside the flume.config.file, and select the properties.properties file.
Figure 4 Uploading a file
- An independent server configuration file can be uploaded to each Flume instance.
- This step is required for updating the configuration file. Modifying the configuration file on the background is an improper operation because the modification will be overwritten after configuration synchronization.
Click Save, and then click OK. Click Finish.

Set the client parameters of the Flume role.

Run the following commands to copy the generated client certificate (flume_cChat.jks) and client trust list (flume_cChatt.jks) to the client directory, for example, /opt/flume-client/fusionInsight-flume-1.9.0/conf/. (The Flume client must have been installed.) 10.196.26.1 is the service plane IP address of the node where the client resides.
scp ${BIGDATA_HOME}/FusionInsight_Porter_8.1.0.1/install/FusionInsight-Flume-1.9.0/flume/conf/flume_cChat.jks user@10.196.26.1:/opt/flume-client/fusionInsight-flume-1.9.0/conf/

scp ${BIGDATA_HOME}/FusionInsight_Porter_8.1.0.1/install/FusionInsight-Flume-1.9.0/flume/conf/flume_cChatt.jks user@10.196.26.1:/opt/flume-client/fusionInsight-flume-1.9.0/conf/

When copying the client certificate, you need to enter the password of user user of the host (for example, 10.196.26.1) where the client resides.
Log in to the node where the Flume client is decompressed as user user. Run the following command to go to the client directory opt/flume-client/fusionInsight-flume-1.9.0/bin.
cd opt/flume-client/fusionInsight-flume-1.9.0/bin
Run the following command to generate and obtain Flume client keystore password, trust list password, and keystore-password encrypted private key information. Enter the password twice and confirm the password. The password is the same as the password of the certificate whose alias is flumechatclient and the password of the flume_cChat.jks certificate library.
./genPwFile.sh

cat password.property
If the following error message is displayed, run the export JAVA_HOME=JDK path command.
```
JAVA_HOME is null in current user,please install the JDK and set the JAVA_HOME
```
Run the echo $SCC_PROFILE_DIR command to check whether the SCC_PROFILE_DIR environment variable is empty.
- If yes, run the source .sccfile command.
- If no, go to 3.e.

Use the Flume configuration tool on FusionInsight Manager to configure the Flume role client parameters and generate a configuration file.

Log in to FusionInsight Manager and choose Cluster > Name of the desired cluster > Services > Flume > Configuration Tool.
Figure 5 Choosing Configuration Tool
Set Agent Name to client. Select the source, channel, and sink to be used, drag them to the GUI on the right, and connect them.
For example, use SpoolDir Source, File Channel, and Avro Sink, as shown in Figure 6.

Figure 6 Example for the Flume configuration tool
Double-click the source, channel, and sink. Set corresponding configuration parameters by seeing Table 2 based on the actual environment.
- If the client parameters of the Flume role have been configured, you can obtain the existing client parameter configuration file from client installation directory/fusioninsight-flume-1.9.0/conf/properties.properties to ensure that the configuration is in concordance with the previous. Log in to FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Flume > Configuration Tool > Import, import the file, and modify the configuration items related to encrypted transmission.
- It is recommended that the numbers of Sources, Channels, and Sinks do not exceed 40 during configuration file import. Otherwise, the response time may be very long.
- A unique checkpoint directory needs to be configured for each File Channel.

Click Export to save the properties.properties configuration file to the local.

**Table 2** Parameters to be modified of the Flume role client
Parameter	Description	Example Value
ssl	Indicates whether to enable the SSL authentication. (You are advised to enable this function to ensure security.) true indicates that the function is enabled. false indicates that the client authentication function is not enabled.	true
keystore	Specified the client certificate.	/opt/flume-client/fusionInsight-flume-1.9.0/conf/flume_cChat.jks
keystore-password	Specifies the password of the key library, which is the password required to obtain the keystore information. Enter the value of password obtained in 3.c.	-
truststore	Indicates the SSL certificate trust list of the client.	/opt/flume-client/fusionInsight-flume-1.9.0/conf/flume_cChatt.jks
truststore-password	Specifies the trust list password, which is the password required to obtain the truststore information. Enter the value of password obtained in 3.c.	-