Help Center/ MapReduce Service/ Component Operation Guide (LTS)/ Using Flume/ Non-Encrypted Transmission/ Typical Scenario: Collecting Local Static Logs and Uploading Them to HDFS
Updated on 2022-11-18 GMT+08:00
View PDF

Typical Scenario: Collecting Local Static Logs and Uploading Them to HDFS

Scenario

This section describes how to use Flume client to collect static logs from a local PC and save them to the /flume/test directory on HDFS.

By default, the cluster network environment is secure and the SSL authentication is not enabled during the data transmission process. For details about how to use the encryption mode, see Configuring the Encrypted Transmission. The configuration applies to scenarios where only the Flume is configured, for example, Spooldir Source+Memory Channel+HDFS Sink.

Prerequisites

  • The cluster, HDFS, and Flume service have been installed.
  • The Flume client has been installed. For details about how to install the client, see Installing the Flume Client on Clusters.
  • The network environment of the cluster is secure.
  • User flume_hdfs has been created, and the HDFS directory and data used for log verification have been authorized to the user.

Procedure

  1. On FusionInsight Manager, choose System > Permission > User, select user flume_hdfs, and choose More > Download Authentication Credential to download the Kerberos certificate file of user flume_hdfs and save it to the local host.

    Figure 1 Downloading the authentication credential

  2. Set Flume parameters.

    Use Flume on FusionInsight Manager to configure the Flume role client parameters and generate a configuration file.
    1. Log in to FusionInsight Manager. Choose Cluster > Services > Flume > Configuration Tool.
      Figure 2 Choosing Configuration Tool
    2. Set Agent Name to client. Select the source, channel, and sink to be used, drag them to the GUI on the right, and connect them.

      Use SpoolDir Source, Memory Channel, and HDFS Sink.

      Figure 3 Example for the Flume configuration tool
    3. Double-click the source, channel, and sink. Set corresponding configuration parameters by referring to Table 1 based on the actual environment.
      • If you want to continue using the properties.propretites file by modifying it, log in to FusionInsight Manager, choose Cluster > Name of the desired cluster > Services. On the page that is displayed, choose Flume. On the displayed page, click the Configuration Tool tab, click Import, import the file, and modify the configuration items related to non-encrypted transmission.
      • It is recommended that the numbers of Sources, Channels, and Sinks do not exceed 40 during configuration file import. Otherwise, the response time may be very long.
    4. Click Export to save the properties.properties configuration file to the local.
      Table 1 Parameters to be modified of the Flume role client

      Parameter

      Description

      Example Value

      Name

      The value must be unique and cannot be left blank.

      test

      spoolDir

      Specifies the directory where the file to be collected resides. This parameter cannot be left blank. The directory needs to exist and have the write, read, and execute permissions on the flume running user.

      /srv/BigData/hadoop/data1/zb

      trackerDir

      Specifies the path for storing the metadata of files collected by Flume.

      /srv/BigData/hadoop/data1/tracker

      batch-size

      Specifies the number of events that Flume sends in a batch.

      61200

      hdfs.path

      Specifies the HDFS data write directory. This parameter cannot be left blank.

      hdfs://hacluster/flume/test

      hdfs.inUsePrefix

      Specifies the prefix of the file that is being written to HDFS.

      TMP_

      hdfs.batchSize

      Specifies the maximum number of events that can be written to HDFS once.

      61200

      hdfs.kerberosPrincipal

      Specifies the Kerberos authentication user, which is mandatory in security versions. This configuration is required only in security clusters.

      flume_hdfs

      hdfs.kerberosKeytab

      Specifies the keytab file path for Kerberos authentication, which is mandatory in security versions. This configuration is required only in security clusters.

      /opt/test/conf/user.keytab

      NOTE:

      Obtain the user.keytab file from the Kerberos certificate file of the user flume_hdfs. In addition, ensure that the user who installs and runs the Flume client has the read and write permissions on the user.keytab file.

      hdfs.useLocalTimeStamp

      Specifies whether to use the local time. Possible values are true and false.

      true

  3. Upload the configuration file.

    Upload the file exported in 2.d to the Flume client installation directory/fusioninsight-flume-Flume component version number/conf directory of the cluster

  1. Verify log transmission.

    1. Log in to FusionInsight Manager as a user who has the management permission on HDFS. For details, see Accessing FusionInsight Manager. Choose Cluster > Services > HDFS. On the page that is displayed, click the NameNode(Node name,Active) link next to NameNode WebUI to go to the HDFS web UI. On the displayed page, choose Utilities > Browse the file system.
      Figure 4 Accessing the HDFS WebUI
    2. Check whether the data is generated in the /flume/test directory on the HDFS.
      Figure 5 Checking HDFS directories and files