Testing a DIS Flume Plugin

bin/flume-ng agent --conf-file conf/flume-conf.properties.template --name agent --conf conf/ -Dflume.root.logger=INFO,console

In the preceding information, bin/flume-ng agent indicates that Flume Agent will be started; conf-file indicates the path of the configuration file written by the user; name indicates the agent name in the configuration file; conf indicates the conf/ path of Flume.

View the log file. If the log file contains information similar to "source disSource started", DIS Source starts normally. In the preceding information, disSource indicates the DIS Source name configured by the user.

Upload data to a stream to which DIS Source points. If Flume does not report an error and DIS Sink can obtain data, the download is successful.

If the sample configuration in Step 2 is used, the data obtained from DIS is output to the console in byte array format.

You can modify the configuration file based on the flume-conf.properties.template provided by Flume. The following provides a file sample:

agent.sources = exec
agent.channels = memoryChannel
agent.sinks = dissink
#Define EXEC Source (which is used to monitor the dis.txt file in the tmp directory).
agent.sources.exec.type = exec
agent.sources.exec.command = tail -F /tmp/dis.txt
agent.sources.exec.shell = /bin/bash -c
agent.sources.exec.channels = memoryChannel
#Define a stream.
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.capacity = 10000
# Define DIS Sink (which is used to output data to the DIS stream).
agent.sinks.dissink.channel = memoryChannel
agent.sinks.dissink.type = com.cloud.dis.adapter.flume.sink.DISSink
agent.sinks.dissink.streamName = YOU_DIS_STREAM_NAME
agent.sinks.dissink.ak = YOU_ACCESS_KEY_ID
agent.sinks.dissink.sk = YOU_SECRET_KEY_ID
agent.sinks.dissink.region = YOU_Region
agent.sinks.dissink.projectId = YOU_PROJECT_ID
agent.sinks.dissink.endpoint = https://dis.${region}.myhuaweicloud.com
agent.sinks.dissink.resultLogLevel = INFO

To start Flume from the Flume installation directory, run the following sample command:

bin/flume-ng agent --conf-file conf/flume-conf.properties.template --name agent --conf conf/ -Dflume.root.logger=INFO,console

In the preceding information, bin/flume-ng agent indicates that Flume Agent will be started; conf-file indicates the path of the configuration file written by the user; name indicates the agent name in the configuration file; conf indicates the conf/ path of Flume.

View the log file. If the log file contains information similar to "Dis flume sink [dissink] start", it indicates that DIS Sink starts normally. The value of dissink is the DIS Sink name configured by the user.

Ingest data into the Flume source. Set the resultLogLevel of the DIS Sink to a value that is not OFF and higher than the log level of log4j. If logs similar to the following are displayed, the DIS Flume Sink has successfully uploaded data to the DIS.

CurrentPut 5 events[success 5 / failed 0] spend 131 ms.

If the sample configuration described in Step 2 is used, you can create a dis.txt file in the tmp directory and append content to the file. After Flume is started, each row of the appended content is read by Flume and sent to the DIS stream through DIS Sink.