Commissioning a Kafka Application

Prerequisites

• The client can log in to the Elastic Cloud Server (ECS) of MRS. For details about how to log in to the ECS, see Getting Started > Logging In to an ECS Using SSH in the ECS User Guide.
• The sample project has been compiled using Maven.

Example: Packing the Maven Project to Run Examples on Linux

1. Run the mvn package command to generate a JAR file, for example, kafka-examples-1.6.0.jar, and obtain it from the target directory in the project directory.
2. Run the mvn dependency:copy-dependencies -DoutputDirectory=kafka-examples-lib -DincludeScope=compile command to export the JAR files that the Kafka sample project depends on, for example, to the kafka-examples-lib directory.
3. A JAR file and a lib directory are generated in the specified path.

   

4. Copy the generated dependency library folder (kafka-examples-lib) to any directory in a Linux environment of the MRS service, for example: /opt/example, and then copy the generated JAR file to the /opt/example/kafka-examples-lib directory.
5. Copy the conf directory of the sample project to the directory where the dependent library folder is located, that is, /opt/example, and create the logs directory for storing the run logs of the JAR file.
6. Switch to the root user and change the directories conf, kafka-examples-lib, and logs to omm, which is owned by the omm:wheel user group. Run the following command to switch to the user group:

   sudo su - root

   chown -R omm:wheel /opt/example/*

   

7. Switch to user omm and go to the /opt/example directory. Ensure that all files in the conf directory and the dependent library file directory can be read by the current user. Ensure that the JDK has been installed and Java environment variables have been set. Then run the java -cp .:/opt/example/conf:/opt/example/kafka-examples-lib/* com.huawei.bigdata.kafka.example.Producer command to run the sample project.

   su - omm

   chmod 750 /opt/example

   cd /opt/example

   java -cp .:/opt/example/conf:/opt/example/kafka-examples-lib/* com.huawei.bigdata.kafka.example.Producer

Observing the Running Result

The running result of the JAR file of the sample program project can be viewed in the client.log file in the logs directory. The log4j.properties file does not output the running status by default. To view the running information of the program, you need to configure the log4j.properties file as follows:

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

kafka.logs.dir=logs

log4j.rootLogger=INFO, stdout, kafkaAppender 

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n

log4j.logger.kafka=INFO, kafkaAppender

log4j.appender.kafkaAppender=org.apache.log4j.DailyRollingFileAppender
log4j.appender.kafkaAppender.DatePattern='.'yyyy-MM-dd-HH
log4j.appender.kafkaAppender.File=${kafka.logs.dir}/client.log
log4j.appender.kafkaAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.kafkaAppender.layout.ConversionPattern=[%d] %p %m (%c)%n

# Turn on all our debugging info
#log4j.logger.kafka.producer.async.DefaultEventHandler=DEBUG, kafkaAppender
#log4j.logger.kafka.client.ClientUtils=DEBUG, kafkaAppender
#log4j.logger.kafka.perf=DEBUG, kafkaAppender
#log4j.logger.kafka.perf.ProducerPerformance$ProducerThread=DEBUG, kafkaAppender
#log4j.logger.org.I0Itec.zkclient.ZkClient=DEBUG

Add kafkaAppender to rootLogger and change the log level to the level that you want to observe.