Installing the Flume Client
To use Flume to collect logs, you must install the Flume client on the log host. You can create an ECS to install the client.
- A streaming cluster with the Flume component has been created.
- The log host is in the same VPC and subnet with the cluster. For details, see Using an MRS Client on Nodes Outside a Cluster.
- You have obtained the username and password for logging in to the log host.
- Create an ECS that meets the requirements in the prerequisites.
- On the MRS cluster details page, click Components.
- If the Components tab is not displayed on the cluster details page, perform IAM user synchronization first. (On the Dashboard tab page of the cluster details page, click on the right side of IAM User Sync to synchronize IAM users.)
- For MRS 2.0.1 or earlier, log in to MRS Manager. For details, see Accessing MRS Manager. Then, choose Services.
- In Client Type, select All client files.
- In Save Path, select Remote host.
- Set Host IP Address to the IP address of the ECS, set Host Port to 22, and set Save Path to /tmp.
- If the default port 22 for logging in to an ECS using SSH has been changed, set Host Port to the new port.
- Save Path contains a maximum of 256 characters.
- Set Login User to root.
If other users are used, ensure that they have read, write, and execute permission on the save path.
- Select Password or SSH Private Key in Login Mode.
- Password: Enter the password of user root set during cluster creation.
- SSH Private Key: Select and upload the key file used for creating the cluster.
- Click OK to start downloading the client to the ECS.
If the following information is displayed, the client package is successfully saved.
Client files downloaded to the remote host successfully.
If the following information is displayed, check the username, password, and security group configurations of the remote host. Ensure that the username and password are correct and an inbound rule of the SSH(22) port has been added to the security group of the remote host. And then, go to 3 to download the client again.
Failed to connect to the server. Please check the network connection or parameter settings.
- Choose Flume > Instance. Query the Business IP Address of any Flume instance and any two MonitorServer instances.
- Log in to the ECS using VNC. See "Logging In to a Linux ECS Using VNC" in the Elastic Cloud Server User Guide (Getting Started > Logging In to an ECS > Logging In to a Linux ECS Using VNC).
All images support Cloud-Init. The preset username for Cloud-init is root and the password is the one you set during cluster creation. It is recommended that you change the password upon the first login.
- On the ECS, switch to user root and copy the installation package to the /opt directory.
sudo su - root
cp /tmp/MRS_Flume_Client.tar /opt
- Run the following command in the /opt directory to decompress the package and obtain the verification file and the configuration package of the client:
tar -xvf MRS_Flume_Client.tar
- Run the following command to verify the configuration package of the client:
sha256sum -c MRS_Flume_ClientConfig.tar.sha256
The command output is as follows:
- Run the following command to decompress MRS_Flume_ClientConfig.tar:
tar -xvf MRS_Flume_ClientConfig.tar
- Run the following command to install the client running environment to a new directory, for example, /opt/Flumeenv. The directory is automatically generated during installation.
sh /opt/MRS_Flume_ClientConfig/install.sh /opt/Flumeenv
If the following information is displayed, the client running environment is successfully installed:
Components client installation is complete.
- Run the following command to configure the environment variable:
- Run the following commands to decompress the Flume client package:
tar -xvf FusionInsight-Flume-1.6.0.tar.gz
- Run the following command to check whether the password of the current user has expired:
chage -l root
If the value of Password expires is earlier than the current time, the password has expired. Run the chage -M -1 root command to validate the password.
- Run the following command to install the Flume client to a new directory, for example, /opt/FlumeClient. The directory is automatically generated during installation.
sh /opt/MRS_Flume_ClientConfig/Flume/install.sh -d /opt/FlumeClient -f Service IP addresses of the MonitorServer instances -c Path of the Flume configuration file -l /var/log/ -e Service IP address of Flume -n Name of the Flume client
The parameters are described as follows:
- -d: indicates the installation path of the Flume client.
- -f: (optional) indicates the service IP addresses of the two MonitorServer instances, separated by a comma. If the IP addresses are not configured, the Flume client will not send alarm information to MonitorServer, and the client information will not be displayed on MRS Manager.
- -c: (optional) indicates the properties.properties configuration file that the Flume client loads after installation. If this parameter is not specified, the fusioninsight-flume-1.6.0/conf/properties.properties file in the client installation directory is used by default. The configuration file of the client is empty. You can modify properties.properties as required and the Flume client will load it automatically.
- -l: (optional) indicates the log directory. The default value is /var/log/Bigdata.
- -e: (optional) indicates the service IP address of the Flume instance. It is used to receive the monitoring indicators reported by the client.
- -n: (optional) indicates the name of the Flume client.
- IBM JDK does not support -Xloggc. You must change -Xloggc to -Xverbosegclog in flume/conf/flume-env.sh. For 32-bit JDK, the value of -Xmx must not exceed 3.25 GB.
- In flume/conf/flume-env.sh, the default value of -Xmx is 4 GB. If the client memory is too small, you can change it to 512 MB or even 1 GB.
For example, run sh install.sh -d /opt/FlumeClient.
If the following information is displayed, the client is successfully installed:
install flume client successfully.