Help Center> MapReduce Service> Component Operation Guide> Using Flume> Using Environment Variables in the properties.properties File
View PDF

Using Environment Variables in the properties.properties File

Scenario

This section describes how to use environment variables in the properties.properties configuration file.

This section applies to MRS 3.x and later versions.

Prerequisites

You have installed Flume service or client.

Procedure

  1. Add variables to the Flume client installation directory/fusioninsight-flume-Flume Version/conf/flume-env.sh file.

    Add variables:

    export Variable name = Variable value

    Example:

    JAVA_OPTS="-Xms2G -Xmx4G -XX:CMSFullGCsBeforeCompaction=1 -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -DpropertiesImplementation=org.apache.flume.node.EnvVarResolverProperties"

    export TAILDIR_PATH=/tmp/flumetest/201907/20190703/1/.*log.*

  2. Restart the Flume instance process.

    1. Log in to FusionInsight Manager.
    2. Choose Cluster > Name of the target cluster > Services > Flume > Instance, select all Flume instances, choose More > Restart Instance, enter the password, and click OK.
    • Do not restart the Flume process by restarting the Flume service on the Manager page, after flume-env.sh takes effect in the flume server. Otherwise, user-defined environment variables are lost.
    • Ensure that flume-env.sh takes effect before configuring the properties.properties as instructed in 3 and uploading the file on the GUI. If the operation sequence is not standard, user-defined environment variables may be lost.

  3. In the properties.properties configuration file, use the ${variable name} format to reference the variable, taking the client as an example.

    Example:

    client.sources.s1.type = TAILDIR
client.sources.s1.filegroups = f1
client.sources.s1.filegroups.f1 = ${TAILDIR_PATH}
client.sources.s1.positionFile = /tmp/flumetest/201907/20190703/1/taildir_position.json
client.sources.s1.channels = c1

Parent topic: Using Flume