Help Center>
Data Lake Insight>
FAQs>
Problems Related to Spark Jobs>
Job Development>
How Do I Read Uploaded Files for a Spark Jar Job?
Updated on 2024-01-23 GMT+08:00
How Do I Read Uploaded Files for a Spark Jar Job?
You can use SparkFiles to read the file submitted using –-file form a local path: SparkFiles.get("Name of the uploaded file").
![](https://support.huaweicloud.com/intl/en-us/dli_faq/public_sys-resources/note_3.0-en-us.png)
- The file path in the Driver is different from that obtained by the Executor. The path obtained by the Driver cannot be passed to the Executor.
- You still need to call SparkFiles.get("filename") in Executor to obtain the file path.
- The SparkFiles.get() method can be called only after Spark is initialized.
Figure 1 Adding other dependencies
![Click to enlarge](https://support.huaweicloud.com/intl/en-us/dli_faq/en-us_image_0000001442646205.png)
The java code is as follows:
package main.java import org.apache.spark.SparkFiles import org.apache.spark.sql.SparkSession import scala.io.Source object DliTest { def main(args:Array[String]): Unit = { val spark = SparkSession.builder .appName("SparkTest") .getOrCreate() // Driver: obtains the uploaded file. println(SparkFiles.get("test")) spark.sparkContext.parallelize(Array(1,2,3,4)) // Executor: obtains the uploaded file. .map(_ => println(SparkFiles.get("test"))) .map(_ => println(Source.fromFile(SparkFiles.get("test")).mkString)).collect() } }
Parent topic: Job Development
Job Development FAQs
- How Do I Use Spark to Write Data into a DLI Table?
- How Do I Set the AK/SK for a Queue to Operate an OBS Table?
- How Do I View the Resource Usage of DLI Spark Jobs?
- How Do I Use Python Scripts to Access the MySQL Database If the pymysql Module Is Missing from the Spark Job Results Stored in MySQL?
- How Do I Run a Complex PySpark Program in DLI?
- How Does a Spark Job Access a MySQL Database?
- How Do I Use JDBC to Set the spark.sql.shuffle.partitions Parameter to Improve the Task Concurrency?
- How Do I Read Uploaded Files for a Spark Jar Job?
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
The system is busy. Please try again later.
For any further questions, feel free to contact us through the chatbot.
Chatbotmore