Testing a DIS Flume Plugin

Updated on 2023-06-21 GMT+08:00

View PDF

Testing DIS Source

Start PuTTY and log in to the server on which Flume is installed.

Ensure that the configuration file containing information about the DIS Source is ready.

You can modify the configuration file based on the flume-conf.properties.template provided by Flume. The following provides a file sample:

agent.sources = dissource
agent.channels = memoryChannel
agent.sinks = loggerSink
#Define DIS Source (which is used to obtain data from DIS).
agent.sources.dissource.channels = memoryChannel
agent.sources.dissource.type = com.cloud.dis.adapter.flume.source.DISSource 
agent.sources.dissource.streams = YOU_DIS_STREAM_NAME
agent.sources.dissource.ak = YOU_ACCESS_KEY_ID
agent.sources.dissource.sk = YOU_SECRET_KEY_ID
agent.sources.dissource.region = YOU_Region
agent.sources.dissource.projectId = YOU_PROJECT_ID
agent.sources.dissource.endpoint = https://dis.${region}.cloud.com
agent.sources.dissource.group.id = YOU_APP_NAME
#Define a stream.
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.capacity = 10000
#Define Logger Sink (which is used to output data to the console).
agent.sinks.loggerSink.type = logger
agent.sinks.loggerSink.channel = memoryChannel

Start Flume. For details about the startup command, see the instructions at the Apache Flume official website.
To start Flume from the Flume installation directory, run the following sample command:
```
bin/flume-ng agent --conf-file conf/flume-conf.properties.template --name agent --conf conf/ -Dflume.root.logger=INFO,console
```
In the preceding information, bin/flume-ng agent indicates that Flume Agent will be started; conf-file indicates the path of the configuration file written by the user; name indicates the agent name in the configuration file; conf indicates the conf/ path of Flume.

View the log file. If the log file contains information similar to "source disSource started", DIS Source starts normally. In the preceding information, disSource indicates the DIS Source name configured by the user.
Check that DIS Source can successfully download data from DIS.

Upload data to a stream to which DIS Source points. If Flume does not report an error and DIS Sink can obtain data, the download is successful.

NOTE:

If the sample configuration in Step 2 is used, the data obtained from DIS is output to the console in byte array format.
Log in to the DIS console. Two minutes later, check monitoring data of the DIS stream specified in Table 1. If data download (blue lines) is displayed, DIS Source is running successfully.

Testing DIS Sink

Start PuTTY and log in to the server on which Flume is installed.

Ensure that the configuration file containing information about DIS Sink is ready.

You can modify the configuration file based on the flume-conf.properties.template provided by Flume. The following provides a file sample:

agent.sources = exec
agent.channels = memoryChannel
agent.sinks = dissink
#Define EXEC Source (which is used to monitor the dis.txt file in the tmp directory).
agent.sources.exec.type = exec
agent.sources.exec.command = tail -F /tmp/dis.txt
agent.sources.exec.shell = /bin/bash -c
agent.sources.exec.channels = memoryChannel
#Define a stream.
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.capacity = 10000
# Define DIS Sink (which is used to output data to the DIS stream).
agent.sinks.dissink.channel = memoryChannel
agent.sinks.dissink.type = com.cloud.dis.adapter.flume.sink.DISSink
agent.sinks.dissink.streamName = YOU_DIS_STREAM_NAME
agent.sinks.dissink.ak = YOU_ACCESS_KEY_ID
agent.sinks.dissink.sk = YOU_SECRET_KEY_ID
agent.sinks.dissink.region = YOU_Region
agent.sinks.dissink.projectId = YOU_PROJECT_ID
agent.sinks.dissink.endpoint = https://dis.${region}.myhuaweicloud.com
agent.sinks.dissink.resultLogLevel = INFO

Start Flume. For details about the startup command, see the instructions at the Apache Flume official website.

To start Flume from the Flume installation directory, run the following sample command:
```
bin/flume-ng agent --conf-file conf/flume-conf.properties.template --name agent --conf conf/ -Dflume.root.logger=INFO,console
```
In the preceding information, bin/flume-ng agent indicates that Flume Agent will be started; conf-file indicates the path of the configuration file written by the user; name indicates the agent name in the configuration file; conf indicates the conf/ path of Flume.

View the log file. If the log file contains information similar to "Dis flume sink [dissink] start", it indicates that DIS Sink starts normally. The value of dissink is the DIS Sink name configured by the user.
Check that the DIS Sink can successfully upload data to DIS.

Ingest data into the Flume source. Set the resultLogLevel of the DIS Sink to a value that is not OFF and higher than the log level of log4j. If logs similar to the following are displayed, the DIS Flume Sink has successfully uploaded data to the DIS.
```
CurrentPut 5 events[success 5 / failed 0] spend 131 ms.
```
NOTE:

If the sample configuration described in Step 2 is used, you can create a dis.txt file in the tmp directory and append content to the file. After Flume is started, each row of the appended content is read by Flume and sent to the DIS stream through DIS Sink.
Log in to the DIS console. Two minutes later, check monitoring data of the DIS stream specified in Table 2. If data upload (indicated in green lines) is displayed, DIS Sink is running successfully.

Parent topic: Using DIS Flume Plugin to Upload and Download Data

Previous topic: Configuring the DIS Flume Plugin

Next topic: (Optional) Uninstalling a DIS Flume Plugin

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot