Using Flume Server to Collect Logs from Kafka to HDFS

Updated on 2024-10-08 GMT+08:00

View PDF

Scenario

This section describes how to use the Flume server to collect logs from the topic list (test1) of Kafka and save them to the /flume/test directory on HDFS.

This section applies to MRS 3.x or later.

NOTE:

By default, the cluster network environment is secure and the SSL authentication is not enabled during the data transmission process. For details about how to use the encryption mode, see Configuring an Encrypted Flume Data Collection Task. The configuration applies to scenarios where only the Flume is configured, for example, Kafka Source+Memory Channel+HDFS Sink.

Prerequisites

The cluster has been installed, including the HDFS, Kafka, and Flume services.
The network environment of the cluster is secure.
You have created user flume_hdfs and authorized the HDFS directory and data to be operated during log verification.

Procedure

On FusionInsight Manager, choose System > User and choose More > Download Authentication Credential to download the Kerberos certificate file of user flume_hdfs and save it to the local host.

Figure 1 Downloading the authentication credential

Configure the server parameters of the Flume role.

Use the Flume configuration tool on FusionInsight Manager to configure the Flume role server parameters and generate a configuration file.

Log in to FusionInsight Manager and choose Cluster > Services. On the page that is displayed, choose Flume. On the displayed page, click the Configuration Tool tab.
Figure 2 Choosing Configuration Tool
Set Agent Name to server. Select the source, channel, and sink to be used, drag them to the GUI on the right, and connect them.
For example, use Kafka Source, Memory Channel, and HDFS Sink.

Figure 3 Example for the Flume configuration tool

Double-click the source, channel, and sink. Set corresponding configuration parameters by referring to Table 1 based on the actual environment.

NOTE:

If you want to continue using the properties.propretites file by modifying it, log in to FusionInsight Manager, choose Cluster > Name of the desired cluster > Services. On the page that is displayed, choose Flume. On the displayed page, click the Configuration Tool tab, click Import, import the file, and modify the configuration items related to non-encrypted transmission.
It is recommended that the numbers of Sources, Channels, and Sinks do not exceed 40 during configuration file import. Otherwise, the response time may be very long.

**Table 1** Parameters to be modified for the Flume role server
Parameter	Description	Example Value
Name	The value must be unique and cannot be left blank.	test
kafka.topics	Specifies the subscribed Kafka topic list, in which topics are separated by commas (,). This parameter cannot be left blank.	test1
kafka.consumer.group.id	Specifies the data group ID obtained from Kafka. This parameter cannot be left blank.	flume
kafka.bootstrap.servers	Specifies the bootstrap IP address and port list of Kafka. The default value is all Kafka lists in a Kafka cluster. If Kafka has been installed in the cluster and its configurations have been synchronized, this parameter can be left blank.	192.168.101.10:9092
batchSize	Specifies the number of events that Flume sends in a batch (number of data pieces).	61200
hdfs.path	Specifies the HDFS data write directory. This parameter cannot be left blank.	hdfs://hacluster/flume/test
hdfs.filePrefix	Specifies the file name prefix after data is written to HDFS.	TMP_
hdfs.batchSize	Specifies the maximum number of events that can be written to HDFS once.	61200
hdfs.kerberosPrincipal	Specifies the Kerberos authentication user, which is mandatory in security versions. This configuration is required only in security clusters.	flume_hdfs
hdfs.kerberosKeytab	Specifies the keytab file path for Kerberos authentication, which is mandatory in security versions. This configuration is required only in security clusters.	/opt/test/conf/user.keytab NOTE: Obtain the user.keytab file from the Kerberos certificate file of the user flume_hdfs. In addition, ensure that the user who installs and runs the Flume client has the read and write permissions on the user.keytab file.
hdfs.useLocalTimeStamp	Specifies whether to use the local time. Possible values are true and false.	true

Click Export to save the properties.properties configuration file to the local.

Upload the configuration file.

Log in to FusionInsight Manager and choose Cluster > Services > Flume. In the Instances tab, click the Flume role whose configuration file is to be uploaded. On the Instance Configurations page, click Upload File on the right of flume.config.file, select the properties.properties file exported in 2.d.

Verify log transmission.
1. Log in to FusionInsight Manager as a user who has the management permission on HDFS. For details, see Accessing FusionInsight Manager (MRS 3.x or Later). Choose Cluster > Services > HDFS. On the page that is displayed, click the NameNode(Node name,Active) link next to NameNode WebUI to go to the HDFS web UI. On the displayed page, choose Utilities > Browse the file system.
2. Check whether the data is generated in the /flume/test directory on the HDFS.
  Figure 4 Checking HDFS directories and files

Parent topic: Configuring a Non-Encrypted Flume Data Collection Task

Previous topic: Using Flume Server to Collect Dynamic Logs from Local Host to HDFS

Next topic: Using Flume Client to Collect Logs from Kafka to HDFS

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot