Typical Scenario: Collecting Local Static Logs and Uploading Them to HBase

Scenario

This section describes how to use Flume client to collect static logs from a local computer and upload them to the flume_test table of HBase.

By default, the cluster network environment is secure and the SSL authentication is not enabled during the data transmission process. For details about how to use the encryption mode, see Configuring the Encrypted Transmission. The configuration can apply to scenarios where only the client or the server is configured, for example, Client/Server:Spooldir Source+File Channel+HBase Sink.

Prerequisites

• The cluster, HBase, and Flume service have been installed.
• The Flume client has been installed. For details about how to install the client, see Installing the Flume Client on Clusters.
• The network environment of the cluster is secure.
• An HBase table has been created by running the create 'flume_test', 'cf' command.
• MRS cluster administrator has understood service requirements and prepared HBase administrator flume_hbase.

Procedure

1. On FusionInsight Manager, choose System > User and choose More > Download Authentication Credential to download the Kerberos certificate file of user flume_hbase and save it to the local host.
   Figure 1 Downloading the authentication credential
   

2. Configure the client parameters of the Flume role.
   1. Use the Flume configuration tool on FusionInsight Manager to configure the Flume role client parameters and generate a configuration file.
      1. Log in to FusionInsight Manager and choose Cluster > Name of the desired cluster > Services > Flume > Configuration Tool.
         Figure 2 Choosing Configuration Tool
         
      2. Set Agent Name to client. Select the source, channel, and sink to be used, drag them to the GUI on the right, and connect them.

         Use SpoolDir Source, File Channel, and Avro Sink, as shown in Figure 3.

         Figure 3 Example for the Flume configuration tool
         
      3. Double-click the source, channel, and sink. Set corresponding configuration parameters by seeing Table 1 based on the actual environment.
         

         • If the client parameters of the Flume role have been configured, you can obtain the existing client parameter configuration file from client installation directory/fusioninsight-flume-1.9.0/conf/properties.properties to ensure that the configuration is in concordance with the previous. Log in to FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Flume > Configuration Tool > Import, import the file, and modify the configuration items related to non-encrypted transmission.
         • It is recommended that the numbers of Sources, Channels, and Sinks do not exceed 40 during configuration file import. Otherwise, the response time may be very long.
      4. Click Export to save the properties.properties configuration file to the local.

         Table 1 Parameters to be modified of the Flume role client

         Parameter	Description	Example Value
         Name	The value must be unique and cannot be left blank.	test
         spoolDir	Specifies the directory where the file to be collected resides. This parameter cannot be left blank. The directory needs to exist and have the write, read, and execute permissions on the flume running user.	/srv/BigData/hadoop/data1/zb
         trackerDir	Specifies the path for storing the metadata of files collected by Flume.	/srv/BigData/hadoop/data1/tracker
         batchSize	Specifies the number of events that Flume sends in a batch (number of data pieces). A larger value indicates higher performance and lower timeliness.	61200
         dataDirs	Specifies the directory for storing buffer data. The run directory is used by default. Configuring multiple directories on disks can improve transmission efficiency. Use commas (,) to separate multiple directories. If the directory is inside the cluster, the /srv/BigData/hadoop/dataX/flume/data directory can be used. dataX ranges from data1 to dataN. If the directory is outside the cluster, it needs to be independently planned.	/srv/BigData/hadoop/data1/flume/data
         checkpointDir	Specifies the directory for storing the checkpoint information, which is under the run directory by default. If the directory is inside the cluster, the /srv/BigData/hadoop/dataX/flume/checkpoint directory can be used. dataX ranges from data1 to dataN. If the directory is outside the cluster, it needs to be independently planned.	/srv/BigData/hadoop/data1/flume/checkpoint
         transactionCapacity	Specifies the transaction size, that is, the number of events in a transaction that can be processed by the current Channel. The size cannot be smaller than the batchSize of Source. Setting the same size as batchSize is recommended.	61200
         hostname	Specifies the name or IP address of the host whose data is to be sent. This parameter cannot be left blank. Name or IP address must be configured to be the name or IP address that the Avro source associated with it.	192.168.108.11
         port	Specifies the port that sends the data. This parameter cannot be left blank. It must be consistent with the port that is monitored by the connected Avro Source.	21154
         ssl	Specifies whether to enable the SSL authentication. (You are advised to enable this function to ensure security.) Only Sources of the Avro type have this configuration item. true indicates that the function is enabled. false indicates that the client authentication function is not enabled.	false

   2. Upload the properties.properties file to flume/conf/ under the installation directory of the Flume client.

3. Configure the server parameters of the Flume role and upload the configuration file to the cluster.
   1. Use the Flume configuration tool on the FusionInsight Manager portal to configure the server parameters and generate the configuration file.
      1. Log in to FusionInsight Manager and choose Cluster > Name of the desired cluster > Services > Flume > Configuration Tool.
         Figure 4 Choosing Configuration Tool
         
      2. Set Agent Name to server. Select the source, channel, and sink to be used, drag them to the GUI on the right, and connect them.

         For example, use Avro Source, File Channel, and HBase Sink, as shown in Figure 5.

         Figure 5 Example for the Flume configuration tool
         
      3. Double-click the source, channel, and sink. Set corresponding configuration parameters by seeing Table 2 based on the actual environment.
         

         • If the server parameters of the Flume role have been configured, you can choose Cluster > Name of the desired cluster > Services > Flume > Instance on FusionInsight Manager. Then select the corresponding Flume role instance and click the Download button behind the flume.config.file parameter on the Instance Configurations page to obtain the existing server parameter configuration file. Choose Cluster > Name of the desired cluster > Services > Flume > Configuration Tool > Import, import the file, and modify the configuration items related to non-encrypted transmission.
         • It is recommended that the numbers of Sources, Channels, and Sinks do not exceed 40 during configuration file import. Otherwise, the response time may be very long.
         • A unique checkpoint directory needs to be configured for each File Channel.
      4. Click Export to save the properties.properties configuration file to the local.

         Table 2 Parameters to be modified of the Flume role server

         Parameter	Description	Example Value
         Name	The value must be unique and cannot be left blank.	test
         bind	Specifies the IP address to which Avro Source is bound. This parameter cannot be left blank. It must be configured as the IP address that the server configuration file will upload.	192.168.108.11
         port	Specifies the ID of the port that the Avro Source monitors. This parameter cannot be left blank. It must be configured as an unused port.	21154
         ssl	Specifies whether to enable the SSL authentication. (You are advised to enable this function to ensure security.) Only Sources of the Avro type have this configuration item. true indicates that the function is enabled. false indicates that the client authentication function is not enabled.	false
         dataDirs	Specifies the directory for storing buffer data. The run directory is used by default. Configuring multiple directories on disks can improve transmission efficiency. Use commas (,) to separate multiple directories. If the directory is inside the cluster, the /srv/BigData/hadoop/dataX/flume/data directory can be used. dataX ranges from data1 to dataN. If the directory is outside the cluster, it needs to be independently planned.	/srv/BigData/hadoop/data1/flumeserver/data
         checkpointDir	Specifies the directory for storing the checkpoint information, which is under the run directory by default. If the directory is inside the cluster, the /srv/BigData/hadoop/dataX/flume/checkpoint directory can be used. dataX ranges from data1 to dataN. If the directory is outside the cluster, it needs to be independently planned.	/srv/BigData/hadoop/data1/flumeserver/checkpoint
         transactionCapacity	Specifies the transaction size, that is, the number of events in a transaction that can be processed by the current Channel. The size cannot be smaller than the batchSize of Source. Setting the same size as batchSize is recommended.	61200
         table	Specifies the HBase table name. This parameter cannot be left blank.	flume_test
         columnFamily	Specifies the HBase column family name. This parameter cannot be left blank.	cf
         batchSize	Specifies the maximum number of events written to HBase by Flume in a batch.	61200
         kerberosPrincipal	Specifies the Kerberos authentication user, which is mandatory in security versions. This configuration is required only in security clusters.	flume_hbase
         kerberosKeytab	Specifies the file path for Kerberos authentication, which is mandatory in security versions. This configuration is required only in security clusters.	/opt/test/conf/user.keytab NOTE: Obtain the user.keytab file from the Kerberos certificate file of the user flume_hbase. In addition, ensure that the user who installs and runs the Flume client has the read and write permissions on the user.keytab file.

   2. Log in to FusionInsight Manager and choose Cluster > Name of the desired cluster > Services > Flume. On the displayed page, click the Flume role on the Instance tab page.
      Figure 6 Clicking the Flume role
      
   3. Select the Flume role of the node where the configuration file is to be uploaded, choose Instance Configurations > Import beside the flume.config.file, and select the properties.properties file.
      Figure 7 Uploading a file
      
      

      • An independent server configuration file can be uploaded to each Flume instance.
      • This step is required for updating the configuration file. Modifying the configuration file on the background is an improper operation because the modification will be overwritten after configuration synchronization.
   4. Click Save, and then click OK.
   5. Click Finish.

4. Verify log transmission.
   1. Go to the directory where the HBase client is installed.

      cd /Client installation directory/ HBase/hbase

      kinit flume_hbase (Enter the password.)

   2. Run the hbase shell command to access the HBase client.
   3. Run the scan 'flume_test' statement. Logs are written in the HBase column family by line.

      hbase(main):001:0> scan 'flume_test'
      ROW                                                          COLUMN+CELL                                                                                                                                                                    
      2017-09-18 16:05:36,394 INFO  [hconnection-0x415a3f6a-shared--pool2-t1] ipc.AbstractRpcClient: RPC Server Kerberos principal name for service=ClientService is hbase/hadoop.<system domain name>@<system domain name>
       default4021ff4a-9339-4151-a4d0-00f20807e76d                 column=cf:pCol, timestamp=1505721909388, value=Welcome to flume                                                                                                                
       incRow                                                      column=cf:iCol, timestamp=1505721909461, value=\x00\x00\x00\x00\x00\x00\x00\x01                                                                                                
      2 row(s) in 0.3660 seconds