Sample Projects of MRS Components

Updated on 2024-08-12 GMT+08:00

View PDF

To obtain the sample projects from Obtaining the MRS Application Development Sample Project, switch the branch to the version that matches the MRS cluster, download the package to a local directory, and decompress the package to obtain the sample code project of each component.

The MRS sample code library provides sample projects of basic functions of each component. Table 1 lists the projects of the current version.

**Table 1** Sample projects of each component
Component	Sample Project Location		Description
ClickHouse	clickhouse-examples		Java program that creates and deletes ClickHouse data tables, and inserts and queries data in MRS clusters This program establishes server connections, creates databases and data tables, inserts data, queries data, and deletes data tables.
Flink	If Kerberos authentication is enabled for the cluster, the sample project directory is flink-examples/flink-examples-security. If Kerberos authentication is disabled for the cluster, the sample project directory is flink-examples/flink-examples-normal.	FlinkCheckpointJavaExample	Java/Scala program for Flink asynchronous checkpointing In this project, the program uses custom operators to continuously generate data. The generated data is a quadruple of long, string, string, and integer values. The program collects statistic results and displays them on the terminal. A checkpoint is triggered every other 6 seconds and the checkpoint result is stored in HDFS.
		FlinkCheckpointScalaExample
		FlinkKafkaJavaExample	Java/Scala program that uses a Flink job to produce and consume data from Kafka. In this project, assume that a Flink service receives one message per second. The Producer application sends data to Kafka, the Consumer application receives data from Kafka, and the program processes and prints the data.
		FlinkKafkaScalaExample
		FlinkPipelineJavaExample	Java/Scala sample program for Flink job pipeline. In this example, a publisher job generates 10,000 data records per second, and the other two jobs subscribe to the data, respectively. After receiving the data, the subscriber jobs convert data formats, sample the data, and output the samples.
		FlinkPipelineScalaExample
		FlinkSqlJavaExample	SQL job submission through Jar jobs on the client
		FlinkStreamJavaExample	Java/Scale program for Flink to construct DataStream This program analyzes user log data based on service requirements, reads text data, generates DataStreams, filters data that meets specified conditions, and obtains results.
		FlinkStreamScalaExample
		FlinkStreamSqlJoinExample	Flink SQL Join program This program calls APIs of the flink-connector-kafka module to produce and consume data. It generates Table1 and Table2, uses Flink SQL to perform joint query on the tables, and displays results.
HBase	hbase-examples	hbase-example	HBase data read and write This program calls HBase APIs to create user tables, import user data, add and query user information, and create secondary indexes for user tables.
		hbase-rest-example	A development example for using HBase REST interfaces. This program uses REST APIs to query HBase cluster information, obtain tables, use NameSpaces, and manipulate tables.
		hbase-thrift-example	A development example for accessing HBase ThriftServer. This program accesses ThriftServer to manipulate tables, and write data to and read data from tables.
		hbase-zk-example	A development example for HBase to access ZooKeeper. You can use the same client process to access MRS ZooKeeper and third-party ZooKeeper at the same time. The HBase client accesses MRS ZooKeeper, and the customer application accesses third-party ZooKeeper.
HDFS	If Kerberos authentication is enabled for the cluster, the sample project directory is hdfs-example-security. If Kerberos authentication is disabled for the cluster, the sample project directory is hdfs-example-normal.		Java program for HDFS file operations. This program creates HDFS folders, writs files, appends file content, reads files, and deletes files or folders.
HDFS	hdfs-c-example		A C language development example for using HDFS. This program connects the HDFS file system and implements file operation functions, such as creating, reading, writing, appending, and deleting files.
Hive	hive-jdbc-example		Java program for Hive JDBC to process data In this project, JDBC APIs are used to connect Hive and perform data operations. You can use JDBC APIs to create tables, load data, and query data. You can access FusionInsight ZooKeeper and third-party ZooKeeper in the same client process at the same time.
	hive-jdbc-example-multizk
	hcatalog-example		Java program for Hive HCatalog to process data HCatalog APIs are used to define and query MRS Hive metadata with Hive CLI.
	python3-examples		Python 3 program to connect Hive and execute SQL statements. This program uses Python 3 to connect Hive and submits data analysis tasks.
Kafka	kafka-examples		Java program for processing Kafka streaming data The program is developed based on Kafka Streams to count words in each message by reading messages in the input topic and to output the result in key-value pairs by consuming data in the output topic.
Manager	manager-examples		Program for calling FusionInsight Manager APIs This program calls Manager APIs to create, modify, and delete cluster users.
MapReduce	If Kerberos authentication is enabled for the cluster, the sample project directory is mapreduce-example-security. If Kerberos authentication is disabled for the cluster, the sample project directory is mapreduce-example-normal.		Java program for submitting MapReduce jobs This program runs a MapReduce statistics data job to analyze and process data and output data required by users. It illustrates how to write MapReduce jobs to access multiple service components in HDFS, HBase, and Hive, helping you to develop for key operations such as authentication and configuration loading.
Oozie	If Kerberos authentication is enabled for the cluster, the sample project directory is oozie-examples/ooziesecurity-examples. If Kerberos authentication is disabled for the cluster, the sample project directory is oozie-examples/oozienormal-examples.	OozieMapReduceExample	Program for submitting MapReduce jobs with Oozie. This program demonstrates how to use Java APIs to submit MapReduce jobs, query job status, and perform offline analysis on website log files.
		OozieSparkHBaseExample	Program for using Oozie to schedule Spark jobs to access HBase.
		OozieSparkHiveExample	Program for using Oozie to schedule Spark jobs to access Hive.
Spark	If Kerberos authentication is enabled for the cluster, the sample project directory is spark-examples/sparksecurity-examples. If Kerberos authentication is disabled for the cluster, the sample project directory is spark-examples/sparknormal-examples.	SparkHbasetoCarbonJavaExample	Java program for Spark to synchronize HBase data to CarbonData. In this project, the program writes data to HBase in real time for point queries. Data is synchronized to CarbonData tables in batches at a specified interval for analytical queries.
		SparkHbasetoHbaseJavaExample	Java/Scala/Python program that uses Spark to read data from and then write data to HBase The program uses Spark jobs to analyze and summarize data of two HBase tables.
		SparkHbasetoHbasePythonExample
		SparkHbasetoHbaseScalaExample
		SparkHivetoHbaseJavaExample	Java/Scala/Python program that uses Spark to read data from Hive and then write data to HBase The program uses Spark jobs to analyze and summarize data of a Hive table and write result to an HBase table.
		SparkHivetoHbasePythonExample
		SparkHivetoHbaseScalaExample
		SparkJavaExample	Java/Python/Scala/R program of Spark Core tasks The program reads text data from HDFS and then calculates and analyzes the data. SparkRExample is only available for clusters with Kerberos authentication enabled.
		SparkPythonExample
		SparkScalaExample
		SparkRExample
		SparkLauncherJavaExample	Java/Scala program that uses Spark Launcher to submit jobs The program uses the org.apache.spark.launcher.SparkLauncher class through Java/Scala commands to submit Spark jobs.
		SparkLauncherScalaExample
		SparkOnHbaseJavaExample	Java/Scala/Python program in the Spark on HBase scenario The program uses HBase as data sources. In this project, data is stored in HBase in Avro format. Data is read from HBase, and the read data is filtered.
		SparkOnHbasePythonExample
		SparkOnHbaseScalaExample
		SparkOnHudiJavaExample	Java/Scala/Python program in the Spark on Hudi scenario The program uses Spark jobs to perform operations such as insertion, query, update, incremental query, query at a specific time, and data deletion on Hudi.
		SparkOnHudiPythonExample
		SparkOnHudiScalaExample
		SparkOnMultiHbaseScalaExample	Scala program that uses Spark to access HBase in two clusters at the same time This program is only available for clusters with Kerberos authentication enabled.
		SparkSQLJavaExample	Java/Python/Scala program of Spark SQL tasks The program reads text data from HDFS and then calculates and analyzes the data.
		SparkSQLPythonExample
		SparkSQLScalaExample
		SparkStreamingKafka010JavaExample	Java/Scala program used by Spark Streaming to receive data from Kafka and perform statistical analysis The program accumulates and calculates the stream data in Kafka in real time and calculates the total number of records of each word.
		SparkStreamingKafka010ScalaExample
		SparkStreamingtoHbaseJavaExample010	Java/Scala/Python sample project used by Spark Streaming to read Kafka data and write the data into HBase The program starts a task every 5 seconds to read data from Kafka and updates the data to a specified HBase table.
		SparkStreamingtoHbasePythonExample010
		SparkStreamingtoHbaseScalaExample010
		SparkStructuredStreamingJavaExample	The program uses Structured Streaming in Spark jobs to call Kafka APIs to obtain word records. Word records are classified to obtain the number of records of each word.
		SparkStructuredStreamingPythonExample
		SparkStructuredStreamingScalaExample
		SparkThriftServerJavaExample	Java/Scala program for Spark SQL access through JDBC. In this sample, a custom JDBCServer client and JDBC connections are used to create, load data to, query, and delete tables.
		SparkThriftServerScalaExample
		StructuredStreamingADScalaExample	Structured Streaming is used to read advertisement request data, display data, and click data from Kafka, obtain effective display statistics and click statistics in real time, and write the statistics to Kafka.
		StructuredStreamingStateScalaExample	This Spark structured streaming program collects statistics on the number of events in each session and the start and end timestamp of the sessions in different batches, and outputs the sessions that the state is updated in this batch.

Previous topic: Obtaining the MRS Application Development Sample Project

Next topic: Using Open-source JAR File Conflict Lists

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot