Help Center> MapReduce Service> Component Operation Guide (ME-Abu Dhabi Region)> Using Flume> Non-Encrypted Transmission> Typical Scenario: Collecting Local Static Logs and Uploading Them to Kafka
Updated on 2022-12-08 GMT+08:00
View PDF

Typical Scenario: Collecting Local Static Logs and Uploading Them to Kafka

Scenario

This section describes how to use Flume to collect static logs from a local host (service IP address: 192.168.108.11) and save them to the topic list (test1) of Kafka.

This section applies to MRS 3.x and later versions.

By default, the cluster network environment is secure and the SSL authentication is not enabled during the data transmission process. For details about how to use the encryption mode, see Configuring the Encrypted Transmission. The configuration can apply to scenarios where only the server is configured, for example, Server:Spooldir Source+File Channel+Kafka Sink.

Prerequisites

  • The cluster, Kafka, and Flume service have been installed.
  • The network environment of the cluster is secure.
  • You have understood service requirements and prepared Kafka administrator flume_kafka.

Procedure

  1. Configure the client parameters of the Flume role.

    1. Use the Flume configuration tool on Manager to configure the Flume role client parameters and generate a configuration file.
      1. Log in to FusionInsight Manager. Choose Cluster > Services > Flume > Configuration Tool.
      2. Set Agent Name to client. Select and drag the source, channel, and sink to be used to the GUI on the right, and connect them.

        Use SpoolDir Source, File Channel, and Avro Sink.

      3. Double-click the source, channel, and sink. Set corresponding configuration parameters by referring to Table 1 based on the actual environment.
        • If the client parameters of the Flume role have been configured, you can obtain the existing client parameter configuration file from client installation directory/fusioninsight-flume-1.9.0/conf/properties.properties to ensure that the configuration is in concordance with the previous. Log in to Manager, choose Cluster > Services > Flume > Configurations > Import, import the file, and modify the configuration items related to non-encrypted transmission.
        • It is recommended that the numbers of Sources, Channels, and Sinks do not exceed 40 during configuration file import. Otherwise, the response time may be very long.
      4. Click Export to save the properties.properties configuration file to the local server.
        Table 1 Parameters to be modified of the Flume role client

        Parameter

        Description

        Example Value

        Name

        The value must be unique and cannot be left blank.

        test

        spoolDir

        Specifies the directory where the file to be collected resides. This parameter cannot be left blank. The directory needs to exist and have the write, read, and execute permissions on the flume running user.

        /srv/BigData/hadoop/data1/zb

        trackerDir

        Specifies the path for storing the metadata of files collected by Flume.

        /srv/BigData/hadoop/data1/tracker

        batchSize

        Specifies the number of events that Flume sends in a batch (number of data pieces). A larger value indicates higher performance and lower timeliness.

        61200

        dataDirs

        Specifies the directory for storing buffer data. The run directory is used by default. Configuring multiple directories on disks can improve transmission efficiency. Use commas (,) to separate multiple directories. If the directory is inside the cluster, the /srv/BigData/hadoop/dataX/flume/data directory can be used. dataX ranges from data1 to dataN. If the directory is outside the cluster, it needs to be independently planned.

        /srv/BigData/hadoop/data1/flume/data

        checkpointDir

        Specifies the directory for storing the checkpoint information, which is under the run directory by default. If the directory is inside the cluster, the /srv/BigData/hadoop/dataX/flume/checkpoint directory can be used. dataX ranges from data1 to dataN. If the directory is outside the cluster, it needs to be independently planned.

        /srv/BigData/hadoop/data1/flume/checkpoint

        transactionCapacity

        Specifies the transaction size, that is, the number of events in a transaction that can be processed by the current channel. The size cannot be smaller than the batchSize of Source. Setting the same size as batchSize is recommended.

        61200

        hostname

        Specifies the name or IP address of the host whose data is to be sent. This parameter cannot be left blank. The parameter must be configured to be the name or IP address of the host where the connected Avro Source resides.

        192.168.108.11

        port

        Specifies the port that sends the data. This parameter cannot be left blank. It must be configured to be the port that is listened to by the connected Avro Source.

        21154

        ssl

        Specifies whether to enable the SSL authentication. (You are advised to enable this function to ensure security.)

        Only Sources of the Avro type have this configuration item.

        • true indicates that the function is enabled.
        • false indicates that the function is not enabled.

        false
    2. Upload the properties.properties file to flume/conf/ under the installation directory of the Flume client.

  2. Configure the server parameters of the Flume role and upload the configuration file to the cluster.

    1. Use the Flume configuration tool on the FusionInsight Manager portal to configure the server parameters and generate the configuration file.
      1. Log in to FusionInsight Manager. Choose Cluster > Services > Flume > Configuration Tool.
      2. Set Agent Name to server. Select and drag the source, channel, and sink to be used to the GUI on the right, and connect them.

        Avro Source, File Channel, and Kafka Sink are used.

      3. Double-click the source, channel, and sink. Set corresponding configuration parameters by referring to Table 2 based on the actual environment.
        • If the server parameters of the Flume role have been configured, you can choose Cluster > Services > Flume > Instance on Manager. Then select the corresponding Flume role instance and click the Download button behind the flume.config.file parameter on the Instance Configurations page to obtain the existing server parameter configuration file. Choose Cluster > Services > Flume > Configurations > Import, import the file, and modify the configuration items related to non-encrypted transmission.
        • It is recommended that the numbers of Sources, Channels, and Sinks do not exceed 40 during configuration file import. Otherwise, the response time may be very long.
        • A unique checkpoint directory needs to be configured for each File Channel.
      4. Click Export to save the properties.properties configuration file to the local server.
        Table 2 Parameters to be modified of the Flume role server

        Parameter

        Description

        Example Value

        Name

        The value must be unique and cannot be left blank.

        test

        bind

        Specifies the IP address to which Avro Source is bound. This parameter cannot be left blank. It must be configured as the IP address that the server configuration file will upload.

        192.168.108.11

        port

        Specifies the port that Avro Source listens to. This parameter cannot be left blank. It must be configured as an unused port.

        21154

        ssl

        Specifies whether to enable the SSL authentication. (You are advised to enable this function to ensure security.)

        Only Sources of the Avro type have this configuration item.

        • true indicates that the function is enabled.
        • false indicates that the function is not enabled.

        false

        dataDirs

        Specifies the directory for storing buffer data. The run directory is used by default. Configuring multiple directories on disks can improve transmission efficiency. Use commas (,) to separate multiple directories. If the directory is inside the cluster, the /srv/BigData/hadoop/dataX/flume/data directory can be used. dataX ranges from data1 to dataN. If the directory is outside the cluster, it needs to be independently planned.

        /srv/BigData/hadoop/data1/flumeserver/data

        checkpointDir

        Specifies the directory for storing the checkpoint information, which is under the run directory by default. If the directory is inside the cluster, the /srv/BigData/hadoop/dataX/flume/checkpoint directory can be used. dataX ranges from data1 to dataN. If the directory is outside the cluster, it needs to be independently planned.

        /srv/BigData/hadoop/data1/flumeserver/checkpoint

        transactionCapacity

        Specifies the transaction size, that is, the number of events in a transaction that can be processed by the current channel. The size cannot be smaller than the batchSize of Source. Setting the same size as batchSize is recommended.

        61200

        kafka.topics

        Specifies the list of subscribed Kafka topics, which are separated by commas (,). This parameter cannot be left blank.

        test1

        kafka.bootstrap.servers

        Specifies the bootstrap IP address and port list of Kafka. The default value is all Kafka lists in a Kafka cluster. If Kafka has been installed in the cluster and its configurations have been synchronized, this parameter can be left blank.

        192.168.101.10:21007
    2. Log in to FusionInsight Manager and choose Cluster > Services > Flume. On the Instance tab page, click the Flume role.
    3. Select the Flume role of the node where the configuration file is to be uploaded, choose Instance Configurations > Import beside the flume.config.file, and select the properties.properties file.
      • An independent server configuration file can be uploaded to each Flume instance.
      • This step is required for updating the configuration file. Modifying the configuration file on the background is an improper operation because the modification will be overwritten after configuration synchronization.
    4. Click Save, and then click OK.
    5. Click Finish.

  1. Verify log transmission.

    1. Log in to the Kafka client.

      cd /Client installation directory/Kafka/kafka

      kinit flume_kafka (Enter the password.)

    2. Read data from a Kafka topic.

      bin/kafka-console-consumer.sh --topic topic name --bootstrap-server Kafka service IP address of the node where the role instance is located: 21007 --consumer.config config/consumer.properties --from-beginning

      The system displays the contents of the file to be collected.

      [root@host1 kafka]# bin/kafka-console-consumer.sh --topic test1 --bootstrap-server 192.168.101.10:21007 --consumer.config config/consumer.properties --from-beginning
Welcome to flume