Installing the Flume Client

Updated on 2024-10-25 GMT+08:00

View PDF

To use Flume to collect logs, you must install the Flume client on a log host. You can create an ECS and install the Flume client on it.

Prerequisites

A cluster with the Flume component has been created.
The log host is in the same VPC and subnet with the MRS cluster.
You have obtained the username and password for logging in to the log host.
The installation directory is automatically created if it does not exist. If it exists, the directory must be left blank. The directory path cannot contain any space.

Installing the Flume Client

Obtain the software package.

Log in to the FusionInsight Manager. Choose Cluster > Name of the target cluster > Services > Flume. On the Flume service page that is displayed, choose More > Download Client in the upper right corner and set Select Client Type to Complete Client to download the Flume service client file.

The file name of the client is FusionInsight_Cluster_<Cluster ID>_Flume_Client.tar. This section takes the client file FusionInsight_Cluster_1_Flume_Client.tar as an example.
Upload the software package.

Upload the software package to a directory, for example, /opt/client, on the node where the Flume client is to be installed as user user.

NOTE:

user is the user who installs and runs the Flume client.
Decompress the software package.

Log in to the node where the Flume service client is to be installed as user user. Go to the directory where the installation package is installed, for example, /opt/client, and run the following command to decompress the installation package to the current directory:

cd /opt/client

tar -xvf FusionInsight_Cluster_1_Flume_Client.tar
Verify the software package.

Run the sha256sum -c command to verify the decompressed file. If OK is returned, the verification is successful. Example:

sha256sum -c FusionInsight_Cluster_1_Flume_ClientConfig.tar.sha256
```
FusionInsight_Cluster_1_Flume_ClientConfig.tar: OK
```
Decompress the package.

tar -xvf FusionInsight_Cluster_1_Flume_ClientConfig.tar
To install the Flume client on a node outside the cluster, perform the following steps to configure the installation environment. Skip this step if you install Flume client on a node in the cluster.
1. Run the following command to install the client running environment to a new directory, for example, /opt/Flumeenv. A directory is automatically generated during the client installation.
  sh /opt/client/FusionInsight_Cluster_1_Flume_ClientConfig/install.sh /opt/Flumeenv
  
  If the following information is displayed, the client running environment is successfully installed:
```
Components client installation is complete.
```
2. Run the following command to set environment variables:
  source /opt/Flumeenv/bigdata_env
Check whether the number of clients is 1.
- If yes, use the independent installation mode and go to 8. The installation is complete.
- If no, use the batch installation mode and go to 9.
Run the following command in the Flume client installation directory to install the client to a specified directory (for example, opt/FlumeClient): After the client is installed successfully, the installation is complete.

cd /opt/client/FusionInsight_Cluster_1_Flume_ClientConfig/Flume/FlumeClient

./install.sh -d /opt/FlumeClient -f MonitorServerService IP address or host name of the role -c User service configuration filePath for storing properties.properties -s CPU threshold -l /var/log/Bigdata -e FlumeServer service IP address or host name -n Flume
NOTE:
- -d: Flume client installation path
- (Optional) -f: IP addresses or host names of two MonitorServer roles. The IP addresses or host names are separated by commas (,). If this parameter is not configured, the Flume client does not send alarm information to MonitorServer and information about the client cannot be viewed on the FusionInsight Manager GUI.
- (Optional) -c: Service configuration file, which needs to be generated on the configuration tool page of the Flume server based on your service requirements by referring to Flume Service Configuration Guide. Upload the file to any directory on the node where the client is to be installed. If this parameter is not specified during the installation, you can upload the generated service configuration file properties.properties to the /opt/FlumeClient/fusioninsight-flume-1.9.0/conf directory after the installation.
- (Optional) -s: cgroup threshold. The value is an integer ranging from 1 to 100 x N. N indicates the number of CPU cores. The default threshold is -1, indicating that the processes added to the cgroup are not restricted by the CPU usage.
- (Optional) -l: Log path. The default value is /var/log/Bigdata. The user user must have the write permission on the directory. When the client is installed for the first time, a subdirectory named flume-client is generated. After the installation, subdirectories named flume-client-n will be generated in sequence. The letter n indicates a sequence number, which starts from 1 in ascending order. In the /conf/ directory of the Flume client installation directory, modify the ENV_VARS file and search for the FLUME_LOG_DIR attribute to view the client log path.
- (Optional) -e: Service IP address or host name of FlumeServer, which is used to receive statistics for the monitoring indicator reported by the client.
- (Optional) -n: Name of the Flume client. You can choose Cluster > Name of the desired cluster > Service > Flume > Flume Management on FusionInsight Manager to view the client name on the corresponding node.
- If the following error message is displayed, run the export JAVA_HOME=JDK path command. You can run the echo $JAVA_HOME command to query the JDK path.
  JAVA_HOME is null in current user,please install the JDK and set the JAVA_HOME
- When installing a cross-platform client in a cluster, go to the /opt/client/FusionInsight_Cluster_1_Flume_ClientConfig/Flume/FusionInsight-Flume-1.9.0.tar.gz directory to install the Flume client.
Go to the directory for installing clients in batches.

cd /opt/client/FusionInsight_Cluster_1_Flume_ClientConfig/Flume/FlumeClient/batch_install

NOTE:

When installing a cross-platform client in a cluster, go to the /opt/client/FusionInsight_Cluster_1_Flume_ClientConfig/Flume/FusionInsight-Flume-1.9.0.tar.gz directory to install the Flume client.
Configure the host_info.cfg file. The format of the configuration file is as follows:
host_ip="",user="",password="",install_path="",flume_config_file="",monitor_server_ip="",log_path="",flume_server_ip="",cgroup_threshold="",client_name=""
NOTE:
- (Mandatory) host_ip: IP address of the node where the Flume client is to be installed.
- (Mandatory) user: User name for logging in to the node where the Flume client is to be installed remotely. The user must have the crontab access permission. You can use echo Username > /etc/cron.allow to grant the user the permission.
- (Mandatory) password: Password for logging in to the Flume client to be installed remotely. There can be security risks if a configuration file contains the authentication password. You are advised to delete the configuration file or use other secure methods to keep the password.
- (Mandatory) install_path: Installation path of the Flume client.
- (Optional) flume_config_file: Configuration file for Flume running. You are advised to specify this configuration file during Flume installation. If you do not set this parameter, retain the value "" and do not delete the parameter.
- (Optional) monitor_server_ip: Service IP address of the Flume MonitorServer in the cluster. You can check the IP address on FusionInsight Manager. You can select either of the two IP addresses. If the IP address is not configured, the client does not send alarm information to the cluster in the scenario where a process is faulty.
- (Optional) log_path: Path for storing Flume run logs. If this parameter is not set, logs are recorded in /var/log/Bigdata/flume-client-Index by default. Index value: If there is only one client in this path, the value is 1. If there are multiple clients, the index value is incremented by 1.
- (Optional) flume_server_ip: Service IP address of the Flume server. The indicator information of the client is reported to the cluster from this node. The indicator information about the client can be displayed on the web client. If the indicator information is not configured, the client does not display the indicator information.
- (Optional) cgroup_threshold: cgroup threshold. The value is an integer ranging from 1 to 100 x N. N indicates the number of CPU cores. The default threshold is -1, indicating that the processes added to the cgroup are not restricted by the CPU usage.
- (Optional) client_name: Client name. The client name is displayed on the client monitoring page. If the client name is not configured, the client name is empty.
- If multiple nodes need to be added, the format is as follows:
  host_ip="ip1",user="user1",password="******",install_path="/tmp/flumeclient",flume_config_file="",monitor_server_ip="xxx.xxx.xxx.xxx",log_path="",flume_server_ip="xxx.xxx.xxx.xxx",cgroup_threshold="",client_name="ip1-client-1"
  
  host_ip="ip2",user="user1",password="******",install_path="/tmp/flumeclient",flume_config_file="",monitor_server_ip="xxx.xxx.xxx.xxx",log_path="",flume_server_ip="xxx.xxx.xxx.xxx",cgroup_threshold="",client_name="ip2-client-1"
  
  host_ip="ip3",user="user1",password="******",install_path="/tmp/flumeclient",flume_config_file="",monitor_server_ip="xxx.xxx.xxx.xxx",log_path="",flume_server_ip="xxx.xxx.xxx.xxx",cgroup_threshold="",client_name="ip3-client-1"
Run the following command to install the Flume client in batches. The installation is complete, as shown in the following figure.

./batch_install.sh -p /opt/client/FusionInsight_Cluster_1_Flume_Client.tar
Delete the password information from the host_info.cfg file.

After the batch installation is complete, delete the password information from the host_info.cfg file immediately. Otherwise, the password may be disclosed.

Flume Client Cgroup Usage Guide

This section describes how to join and log out of a cgroup, query the cgroup status, and change the cgroup CPU threshold.

Join Cgroup
Assume that the Flume client installation path is /opt/FlumeClient, and the cgroup CPU threshold is 50%. Run the following command to join a cgroup:

cd /opt/FlumeClient/fusioninsight-flume-1.9.0/bin

./flume-manage.sh cgroup join 50
NOTE:
- This command can be used to join a cgroup and change the cgroup CPU threshold.
- The value of the CPU threshold of a cgroup ranges from 1 to 100 x N. N indicates the number of CPU cores.
Check Cgroup status
Assume that the Flume client installation path is /opt/FlumeClient. Run the following commands to query the cgroup status:

cd /opt/FlumeClient/fusioninsight-flume-1.9.0/bin

./flume-manage.sh cgroup status
Exit Cgroup
Assume that the Flume client installation path is /opt/FlumeClient. Run the following commands to exit cgroup:

cd /opt/FlumeClient/fusioninsight-flume-1.9.0/bin

./flume-manage.sh cgroup exit
NOTE:
- After the client is installed, the default cgroup is automatically created. If the -s parameter is not configured during client installation, the default value -1 is used. The default value indicates that the agent process is not restricted by the CPU usage.
- Joining or exiting a cgroup does not affect the agent process. Even if the agent process is not started, the joining or exiting operation can be performed successfully, and the operation will take effect after the next startup of the agent process.
- After the client is uninstalled, the cgroups created during the client installation are automatically deleted.

Parent topic: Using Flume

Previous topic: Flume Service Model Configuration

Next topic: Quickly Using Flume to Collect Node Logs

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot