Generating Configuration Files for the Flume Server and Client

Scenario

This section describes how to configure Flume server and client parameters after the cluster and the Flume service are installed to ensure proper running of the service.

This section applies to MRS 3.x or later.

By default, the cluster network environment is secure and the SSL authentication is not enabled during the data transmission process. For details about how to use the encryption mode, see Configuring an Encrypted Flume Data Collection Task.

Prerequisites

• The Flume client has been installed.
• The cluster and Flume service have been installed.
• The network environment of the cluster is secure.

Procedure

1. Configure the client parameters of the Flume role.
   1. Use the Flume configuration tool on FusionInsight Manager to configure the Flume role client parameters and generate a configuration file.
      1. Log in to FusionInsight Manager. Choose Cluster > Services > Flume > Configuration Tool.
         Figure 1 Clicking Configuration Tool
         
      2. Set Agent Name to client. Select and drag the source, channel, and sink to be used to the GUI on the right, and connect them.
         For example, use SpoolDir Source, File Channel, and Avro Sink, as shown in Figure 2.

         Figure 2 Example for the Flume configuration tool
         
      3. Double-click the source, channel, and sink. Set corresponding configuration parameters by referring to Table 1 based on the actual environment.
         

         • If the client parameters of the Flume role have been configured, you can obtain the existing client parameter configuration file from client installation directory/fusioninsight-flume-1.9.0/conf/properties.properties to ensure that the configuration is in concordance with the previous. Log in to FusionInsight Manager, choose Cluster > Services > Flume > Configuration > Import, import the file, and modify the configuration items related to non-encrypted transmission.
         • It is recommended that the numbers of Sources, Channels, and Sinks do not exceed 40 during configuration file import. Otherwise, the response time may be very long.

         Table 1 Parameters to be modified for the Flume role client

         Parameter	Description	Example Value
         ssl	Specifies whether to enable the SSL authentication. (You are advised to enable this function to ensure security.) Only Sources of the Avro type have this configuration item. true indicates that the function is enabled. false indicates that the function is not enabled.	false

      4. Click Export to save the properties.properties configuration file to the local server.
   2. Upload the properties.properties file to flume/conf/ under the installation directory of the Flume client.

2. Configure the server parameters of the Flume role and upload the configuration file to the cluster.
   1. Use the Flume configuration tool on the FusionInsight Manager portal to configure the server parameters and generate the configuration file.
      1. Log in to FusionInsight Manager. Choose Cluster > Services > Flume > Configuration Tool.
         Figure 3 Clicking Configuration Tool
         
      2. Set Agent Name to server. Select and drag the source, channel, and sink to be used to the GUI on the right, and connect them.

         For example, use Avro Source, File Channel, and HDFS Sink, as shown in Figure 4.

         Figure 4 Example for the Flume configuration tool
         
      3. Double-click the source, channel, and sink. Set corresponding configuration parameters by referring to Table 2 based on the actual environment.
         

         • If the server parameters of the Flume role have been configured, you can choose Cluster > Services > Flume > Instance on FusionInsight Manager. Then select the corresponding Flume role instance and click the Download button behind the flume.config.file parameter on the Instance Configurations page to obtain the existing server parameter configuration file. Choose Cluster > Service > Flume > Configurations > Import, import the file, and modify the configuration items related to non-encrypted transmission.
         • It is recommended that the numbers of Sources, Channels, and Sinks do not exceed 40 during configuration file import. Otherwise, the response time may be very long.
         • A unique checkpoint directory needs to be configured for each File Channel.

         Table 2 Parameters to be modified for the Flume role server

         Parameter	Description	Example Value
         ssl	Specifies whether to enable the SSL authentication. (You are advised to enable this function to ensure security.) Only Sources of the Avro type have this configuration item. true indicates that the function is enabled. false indicates that the function is not enabled.	false

      4. Click Export to save the properties.properties configuration file to the local server.
   2. Log in to FusionInsight Manager and choose Cluster > Services > Flume. On the Instances tab page, click Flume.
      Figure 5 Clicking the Flume role
      
   3. Select the Flume role of the node where the configuration file is to be uploaded, choose Instance Configurations > Import beside the flume.config.file, and select the properties.properties file.
      Figure 6 Uploading a file
      
      

      • An independent server configuration file can be uploaded to each Flume instance.
      • This step is required for updating the configuration file. Modifying the configuration file on the background is an improper operation because the modification will be overwritten after configuration synchronization.
   4. Click Save, and then click OK.
   5. Click Finish.