Installing the Flume Client on Clusters of Versions Earlier Than MRS 3.x

Scenario

To use Flume to collect logs, you must install the Flume client on a log host. You can create an ECS and install the Flume client on it.

This section applies to MRS 3.x or earlier clusters.

Prerequisites

• A streaming cluster with the Flume component has been created.
• The log host is in the same VPC and subnet with the MRS cluster.
• You have obtained the username and password for logging in to the log host.

Procedure

1. Create an ECS that meets the requirements.
2. Go to the cluster details page and choose Components.
   

   If the Components tab is unavailable, complete IAM user synchronization first. (On the Dashboard page, click Synchronize on the right side of IAM User Sync to synchronize IAM users.)

3. Click Download Client.
   1. In Client Type, select All client files.
   2. In Download to, select Remote host.
   3. Set Host IP Address to the IP address of the ECS, Host Port to 22, and Save Path to /tmp.
      • If the default port 22 for logging in to an ECS through SSH has been changed, set Host Port to a new port.
      • The value of Save Path contains a maximum of 256 characters.
   4. Set Login User to root.

      If another user is used, ensure that the user has permissions to read, write, and execute the save path.

   5. Select Password or SSH Private Key for Login Mode.
      • Password: Enter the password of user root set during cluster creation.
      • SSH Private Key: Select and upload the key file used for creating the cluster.
   6. Click OK to generate a client file.

      If the following information is displayed, the client package is saved.

      Client files downloaded to the remote host successfully.

      If the following information is displayed, check the username, password, and security group configurations of the remote host. Ensure that the username and password are correct and an inbound rule of the SSH (22) port has been added to the security group of the remote host. And then, go to 3 to download the client again.

      Failed to connect to the server. Please check the network connection or parameter settings.

      Figure 1 Downloading a client
      

4. Choose Flume > Instance. Query the Business IP Address of any Flume instance and any two MonitorServer instances.

   

5. Log in to the ECS using VNC. For details, see "User Guide" > "Login Using VNC" in Elastic Cloud Server. ("Instances" > "Logging In to a Linux ECS").

   All images support Cloud-Init. The preset username for Cloud-Init is root and the password is the one you set during cluster creation. You are advised to change the password upon the first login.

6. On the ECS, switch to user root and copy the installation package to the /opt directory.

   sudo su - root

   cp /tmp/MRS_Flume_Client.tar /opt

7. Run the following command in the /opt directory to decompress the package and obtain the verification file and the configuration package of the client:

   tar -xvf MRS_Flume_Client.tar

8. Run the following command to verify the configuration package of the client:

   sha256sum -c MRS_Flume_ClientConfig.tar.sha256

   If the following information is displayed, the file package is successfully verified:

   MRS_Flume_ClientConfig.tar: OK

9. Run the following command to decompress MRS_Flume_ClientConfig.tar:

   tar -xvf MRS_Flume_ClientConfig.tar

10. Run the following command to install the client running environment to a new directory, for example, /opt/Flumeenv. A directory is automatically generated during the client installation.

    sh /opt/MRS_Flume_ClientConfig/install.sh /opt/Flumeenv

    If the following information is displayed, the client running environment is successfully installed:

    Components client installation is complete.

11. Run the following command to configure environment variables:

    source /opt/Flumeenv/bigdata_env

12. Run the following commands to decompress the Flume client package:

    cd /opt/MRS_Flume_ClientConfig/Flume

    tar -xvf FusionInsight-Flume-1.6.0.tar.gz

13. Run the following command to check whether the password of the current user has expired:

    chage -l root

    If the value of Password expires is earlier than the current time, the password has expired. Run the chage -M -1 root command to validate the password.

14. Run the following command to install the Flume client to a new directory, for example, /opt/FlumeClient. A directory is automatically generated during the client installation.

    sh /opt/MRS_Flume_ClientConfig/Flume/install.sh -d /opt/FlumeClient -f service IP address of the MonitorServer instance -c path of the Flume configuration file -l /var/log/ -e service IP address of Flume -n name of the Flume client

    The parameters are described as follows:

    • -d: indicates the installation path of the Flume client.
    • (Optional) -f: indicates the service IP addresses of the two MonitorServer instances, separated by a comma (,). If the IP addresses are not configured, the Flume client will not send alarm information to MonitorServer, and the client information will not be displayed on MRS Manager.
    • (Optional) -c: indicates the properties.properties configuration file that the Flume client loads after installation. If this parameter is not specified, the fusioninsight-flume-1.6.0/conf/properties.properties file in the client installation directory is used by default. The configuration file of the client is empty. You can modify the configuration file as required and the Flume client will load it automatically.
    • (Optional) -l: indicates the log directory. The default value is /var/log/Bigdata.
    • (Optional) -e: indicates the service IP address of the Flume instance. It is used to receive the monitoring indicators reported by the client.
    • (Optional) -n: indicates the name of the Flume client.
    • IBM JDK does not support -Xloggc. You must change -Xloggc to -Xverbosegclog in flume/conf/flume-env.sh. For 32-bit JDK, the value of -Xmx must not exceed 3.25 GB.
    • In flume/conf/flume-env.sh, the default value of -Xmx is 4 GB. If the client memory is too small, you can change it to 512 MB or even 1 GB.

    For example, run sh install.sh -d /opt/FlumeClient.

    If the following information is displayed, the client is successfully installed:

    install flume client successfully.