Help Center/ MapReduce Service/ Component Operation Guide (Normal)/ Using Flume/ Common Issues About Flume/ How Do I Configure a Custom Flume Script
Updated on 2024-10-24 GMT+08:00
View PDF
Share

How Do I Configure a Custom Flume Script

Flume supports customized scripts to be run before or after transmission for making preparations.

This section applies to MRS 3.x or later.

The Flume client is not installed

  1. Obtain the software package.

    Log in to the FusionInsight Manager. Choose Cluster > Name of the target cluster > Services > Flume. On the Flume service page that is displayed, choose More > Download Client in the upper right corner and set Client Type to Complete Client to download the Flume service client file.

    The file name of the client is FusionInsight_Cluster_<Cluster ID>_Flume_Client.tar. This section takes the client file FusionInsight_Cluster_1_Flume_Client.tar as an example.

  2. Upload the software package. Upload the software package to a directory, for example, /opt/client, on the node where the Flume client is to be installed as user user.

    user is the user who installs and runs the Flume client.

  3. Decompress the software package.

    Log in to the node where the Flume service client is to be installed as user user. Go to the directory where the installation package is installed, for example, /opt/client, and run the following command to decompress the installation package to the current directory:

    cd /opt/client

    tar -xvf FusionInsight_Cluster_1_Flume_Client.tar

  4. Verify the software package.

    Run the sha256sum -c command to verify the decompressed file. If OK is returned, the verification is successful. Example:

    sha256sum -c FusionInsight_Cluster_1_Flume_ClientConfig.tar.sha256

    FusionInsight_Cluster_1_Flume_ClientConfig.tar: OK

  5. Decompress the package.

    tar -xvf FusionInsight_Cluster_1_Flume_ClientConfig.tar

  6. Configure client.per-check.shell in the /opt/client/FusionInsight_Cluster_1_Flume_ClientConfig/Flume/FlumeClient/flume/conf/flume-check.properties file on the client to point to the absolute path of plugin.sh.

    The configuration is as follows:

    client.per-check.shell=/opt/client/FusionInsight_Cluster_1_Flume_ClientConfig/Flume/FlumeClient/flume/plugins.s/plugin.sh

    plugins = com.huawei.flume.services.FlumePreTransmitService

    flume.check.default.interval = 15

  7. Configure the /opt/client/FusionInsight_Cluster_1_Flume_ClientConfig/Flume/FlumeClient/flume/conf/plugin.conf file and define the scripts to be invoked and related parameters.

    The configuration is as follows:

    RUN_PLUGIN="PLUGIN_LIST_1"

    LOG_TO_HDFS_PATH="/yxs"

    LOG_TO_HDFS_ENCODE_PATH="${LOG_TO_HDFS_PATH}/Flume_Encoded/"

    PLUGIN_LINK_DIR="/tmp/yxs1"

    PLUGIN_MV_TARGET_DIR="/tmp/yxs2"

    PLUGIN_SUFFIX="COMPLETED"

    PLUGIN_LIST_1="mv_complete.sh --linkdir ${PLUGIN_LINK_DIR} --mvtargetdir ${PLUGIN_MV_TARGET_DIR} --suffix ${PLUGIN_SUFFIX}"

  8. Install and start the Flume client. For details about how to install the client, see Installing the Flume Client.

The Flume client has been installed

  1. Configure client.per-check.shell in the flume-check.properties file on the client to point to the absolute path of plugin.sh.

    For example, if the Flume client installation path is /opt/FlumeClient, the flume-check.properties file is stored in the /opt/FlumeClient/fusioninsight-flume-1.9.0/conf directory,

    The configuration is as follows:

    client.per-check.shell=/opt/FlumeClient/fusioninsight-flume-1.9.0/plugins.s/plugin.sh

    plugins = com.huawei.flume.services.FlumePreTransmitService

    flume.check.default.interval = 15

  2. Configure plugin.conf to define the script to be invoked and related parameters.

    For example, if the Flume client installation path is /opt/FlumeClient, the plugin.conf file is stored in the /opt/FlumeClient/fusioninsight-flume-1.9.0/conf directory,

    The configuration is as follows:

    RUN_PLUGIN="PLUGIN_LIST_1"

    LOG_TO_HDFS_PATH="/yxs"

    LOG_TO_HDFS_ENCODE_PATH="${LOG_TO_HDFS_PATH}/Flume_Encoded/"

    PLUGIN_LINK_DIR="/tmp/yxs1"

    PLUGIN_MV_TARGET_DIR="/tmp/yxs2"

    PLUGIN_SUFFIX="COMPLETED"

    PLUGIN_LIST_1="mv_complete.sh --linkdir ${PLUGIN_LINK_DIR} --mvtargetdir ${PLUGIN_MV_TARGET_DIR} --suffix ${PLUGIN_SUFFIX}"

  3. Run the following command in the bin directory, for example, /opt/FlumeClient/fusioninsight-flume-1.9.0/bin, of the client installation path to restart the Flume client:

    ./flume-manage.sh restart