Development Plan

Updated on 2024-08-10 GMT+08:00

View PDF

Overview

This section describes how to use Spark to perform operations such as data insertion, query, update, incremental query, query at a specific time point, and data deletion on Hudi.

For details, see the sample code.

Packaging the Project

Use the Maven tool provided by IDEA to pack the project and generate a JAR file. For details, see Commissioning a Spark Application in a Linux Environment.
NOTE:

The Python sample code does not need to be packaged using Maven.
Upload the generated JAR file to any directory (for example, /opt/example/) on the server where the Spark client is located.

Running the Task

Log in to the Spark client node and run the following commands:
source Client installation directory/bigdata_env

source Client installation directory/Hudi/component_env
After compiling and building the sample code, you can use the spark-submit command to perform the write, update, query, and delete operations in sequence.
- Run the Java sample project.
  spark-submit --class com.huawei.bigdata.hudi.examples.HoodieWriteClientExample /opt/example/hudi-java-examples-1.0.jar hdfs://hacluster/tmp/example/hoodie_java hoodie_java
  
  /opt/example/hudi-java-examples-1.0.jar indicates the JAR file path, hdfs://hacluster/tmp/example/hoodie_java indicates the storage path of the Hudi table, and hoodie_java indicates the name of the Hudi table.
- Run the Scala sample project.
  spark-submit --class com.huawei.bigdata.hudi.examples.HoodieDataSourceExample /opt/example/hudi-scala-examples-1.0.jar hdfs://hacluster/tmp/example/hoodie_scala hoodie_scala
  
  /opt/example/hudi-scala-examples-1.0.jar indicates the JAR file path, hdfs://hacluster/tmp/example/hoodie_scala indicates the storage path of the Hudi table, and hoodie_Scala indicates the name of the Hudi table.
- Run the Python sample project.
  spark-submit /opt/example/HudiPythonExample.py hdfs://hacluster/tmp/huditest/example/python hudi_trips_cow
  
  hdfs://hacluster/tmp/huditest/example/python indicates the storage path of the Hudi table, and hudi_trips_cow indicates the name of the Hudi table.

Parent topic: Using Spark to Execute the Hudi Sample Project

Previous topic: Using Spark to Execute the Hudi Sample Project

Next topic: Using Spark to Execute the Hudi Sample Project (Scala)

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot