Help Center/ MapReduce Service/ User Guide (Kuala Lumpur Region)/ MRS Cluster Component Operation Gudie/ Using Flume/ Non-Encrypted Transmission/ Typical Scenario: Collecting Logs from Kafka and Uploading Them to HDFS Through the Flume Client
Updated on 2022-12-14 GMT+08:00

Typical Scenario: Collecting Logs from Kafka and Uploading Them to HDFS Through the Flume Client

Scenario

This section describes how to use Flume to collect logs from the Topic list (test1) of Kafka client and save them to the /flume/test directory on HDFS.

This section applies to MRS 3.x or later.

By default, the cluster network environment is secure and the SSL authentication is not enabled during the data transmission process. For details about how to use the encryption mode, see Configuring the Encrypted Transmission.

Prerequisites

  • The cluster, HDFS, Kafka, and Flume service have been installed.
  • You have created user flume_hdfs and authorized the HDFS directory and data to be operated during log verification.
  • The network environment of the cluster is secure.

Procedure

  1. On FusionInsight Manager, choose System > User and choose More > Download Authentication Credential to download the Kerberos certificate file of user flume_hdfs and save it to the local host.
  2. Configure the client parameters of the Flume role.

    1. Use the Flume configuration tool on FusionInsight Manager to configure the Flume role client parameters and generate a configuration file.
      1. Log in to FusionInsight Manager and choose Cluster > Name of the desired cluster > Services > Flume > Configuration Tool.
      2. Set Agent Name to client. Select the source, channel, and sink to be used, drag them to the GUI on the right, and connect them.

        For example, use Kafka Source, File Channel, and HDFS Sink.

      3. Double-click the source, channel, and sink. Set corresponding configuration parameters by seeing Table 1 based on the actual environment.
        • If the client parameters of the Flume role have been configured, you can obtain the existing client parameter configuration file from client installation directory/fusioninsight-flume-1.9.0/conf/properties.properties to ensure that the configuration is in concordance with the previous. Log in to FusionInsight Manager, choose Cluster > Name of the desired cluster > Services > Flume > Configuration Tool > Import, import the file, and modify the configuration items related to non-encrypted transmission.
        • It is recommended that the numbers of Sources, Channels, and Sinks do not exceed 40 during configuration file import. Otherwise, the response time may be very long.
      4. Click Export to save the properties.properties configuration file to the local.
        Table 1 Parameters to be modified of the Flume role client

        Parameter

        Description

        Example Value

        Name

        The value must be unique and cannot be left blank.

        test

        kafka.topics

        Specifies the subscribed Kafka topic list, in which topics are separated by commas (,). This parameter cannot be left blank.

        test1

        kafka.consumer.group.id

        Specifies the data group ID obtained from Kafka. This parameter cannot be left blank.

        flume

        kafka.bootstrap.servers

        Specifies the bootstrap IP address and port list of Kafka. The default value is all Kafka lists in a Kafka cluster. If Kafka has been installed in the cluster and its configurations have been synchronized, this parameter can be left blank.

        192.168.101.10:21007

        batchSize

        Specifies the number of events that Flume sends in a batch (number of data pieces).

        61200

        dataDirs

        Specifies the directory for storing buffer data. The run directory is used by default. Configuring multiple directories on disks can improve transmission efficiency. Use commas (,) to separate multiple directories. If the directory is inside the cluster, the /srv/BigData/hadoop/dataX/flume/data directory can be used. dataX ranges from data1 to dataN. If the directory is outside the cluster, it needs to be independently planned.

        /srv/BigData/hadoop/data1/flume/data

        checkpointDir

        Specifies the directory for storing the checkpoint information, which is under the run directory by default. If the directory is inside the cluster, the /srv/BigData/hadoop/dataX/flume/checkpoint directory can be used. dataX ranges from data1 to dataN. If the directory is outside the cluster, it needs to be independently planned.

        /srv/BigData/hadoop/data1/flume/checkpoint

        transactionCapacity

        Specifies the transaction size, that is, the number of events in a transaction that can be processed by the current Channel. The size cannot be smaller than the batchSize of Source. Setting the same size as batchSize is recommended.

        61200

        hdfs.path

        Specifies the HDFS data write directory. This parameter cannot be left blank.

        hdfs://hacluster/flume/test

        hdfs.inUsePrefix

        Specifies the prefix of the file that is being written to HDFS.

        TMP_

        hdfs.batchSize

        Specifies the maximum number of events that can be written to HDFS once.

        61200

        hdfs.kerberosPrincipal

        Specifies the Kerberos authentication user, which is mandatory in security versions. This configuration is required only in security clusters.

        flume_hdfs

        hdfs.kerberosKeytab

        Specifies the keytab file path for Kerberos authentication, which is mandatory in security versions. This configuration is required only in security clusters.

        /opt/test/conf/user.keytab

        NOTE:

        Obtain the user.keytab file from the Kerberos certificate file of the user flume_hdfs. In addition, ensure that the user who installs and runs the Flume client has the read and write permissions on the user.keytab file.

        hdfs.useLocalTimeStamp

        Specifies whether to use the local time. Possible values are true and false.

        true

    2. Upload the properties.properties file to flume/conf/ under the installation directory of the Flume client.
    3. To connect the Flume client to the HDFS, you need to add the following configuration:
      1. Download the Kerberos certificate of account flume_hdfs and obtain the krb5.conf configuration file. Upload the configuration file to the fusioninsight-flume-1.9.0/conf/ directory on the node where the client is installed.
      2. In fusioninsight-flume-1.9.0/conf/, create the jaas.conf configuration file.

        vi jaas.conf

        KafkaClient {
        com.sun.security.auth.module.Krb5LoginModule required
        useKeyTab=true
        keyTab="/opt/test/conf/user.keytab"
        principal="flume_hdfs@<System domain name>"
        useTicketCache=false
        storeKey=true
        debug=true;
        };

        Values of keyTab and principal vary depending on the actual situation.

      3. Obtain configuration files core-site.xml and hdfs-site.xml from /opt/FusionInsight_Cluster_<Cluster ID>_Flume_ClientConfig/Flume/config and upload them to fusioninsight-flume-1.9.0/conf/.
    4. Restart the Flume service.

  1. Verify log transmission.

    1. Log in to FusionInsight Manager as a user who has the management permission on HDFS. For details, see Accessing FusionInsight Manager (MRS 3.x or Later). Choose Cluster > Name of the desired cluster > Services > HDFS, click the HDFS WebUI link of NameNode (Node name, Active) to go to the HDFS WebUI, and choose Utilities > Browse the file system.
    2. Check whether the data is generated in the /flume/test directory on the HDFS.