Spark Troubleshooting

This section describes common faults that occur when Spark is used and troubleshooting methods.

• Common Spark Task Submission Faults
  • Spark Task Submission Failure
  • Spark Streaming Task Submission Issues
  • Incorrect Parameter Format Is Displayed When a Spark Task Is Submitted
  • Error "ClassNotFoundException" Is Reported When a Spark Task Is Submitted
  • Driver Displays a Message Indicating That the Running Memory Exceeds the Threshold When a Spark Task Is Submitted
  • Error "Can't get the Kerberos realm" Is Reported When a Spark Task Is Submitted in Yarn-Cluster Mode
  • ApplicationMaster Fails to Start Twice When a Spark Task Is Submitted in Yarn-client Mode
  • Job Status Is error After a Spark Job Is Submitted Through an API
  • Failed to Connect to ResourceManager When a Spark Task Is Submitted
  • Failed to Connect to the Driver When a Spark Job Is Submitted on a Node Outside the Cluster
• Common Spark Task Running Faults
  • Spark Task Execution Failure
  • An Error Is Reported When the Split Size Is Changed for a Running Spark Application
  • A Spark Job Fails to Run Due to Incorrect JAR File Import
  • Spark Job Suspended Due to Insufficient Memory or Lack of JAR Packages
  • DataArts Studio Failed to Schedule Spark Jobs
  • Large Number of Shuffle Results Are Lost During Spark Task Execution
  • Failed to View Spark Task Logs
  • Spark Streaming Task Failed but Status Shows as Successful in Yarn WebUI
  • Why JRE fatal error after running Spark application multiple times?
  • Spark Shuffle Exception Handling
• Common Faults of Spark Component Exceptions
  • Spark, Hive, and Yarn Are Unavailable Due to Insufficient Disk Capacity
  • Failed to Start spark-sql and spark-shell Due to JDK Version Mismatch
  • Failed to Create or Delete a Table in Spark Beeline
  • An Error Is Reported When spark-beeline Is Used to Query a Hive View
  • Why Do I Fail to Create a Table in the Specified Location on OBS After Logging to Spark Beeline?
  • Failed to Load Data to a Hive Table Across File Systems by Running SQL Statements Using Spark Shell
  • An Error Occurs When SparkSQL Reads the ORC Table
  • What Should I Do If Statistics of Hudi or Hive Tables Created Using Spark SQLs Are Empty Before Data Is Inserted?
  • Failed to Switch to the Log Page from stderr and stdout on the Native Spark Web UI
  • Why the Job Information Obtained from the restful Interface of an Ended Spark Application Is Incorrect?
  • Why Cannot I Switch from the Yarn Web UI to the Spark Web UI?
  • Apps Cannot Be Displayed on the JobHistory Page When an Empty Part File Is Loaded
  • Why Does Spark Fail to Export a Table with the Duplicate Field Names?
  • Why Does the Foreign Table Query Fail When Multiple Foreign Tables Are Created in the Same Directory?
  • Why Does the Cluster Port Fail to Connect When a Client Outside the Cluster Is Installed or Used?
  • How Do I Handle the Exception Occurred When I Query Datasource Avro Formats?
  • Failed to Query Table Statistics by Partition Using Non-Standard Time Format When the Partition Column in the Table Creation Statement is timestamp
  • How Do I Use Special Characters with TIMESTAMP and DATE?
• Common Spark Permission Authentication Faults
  • Authentication Fails When Spark Connects to Other Services
  • Authentication Fails When Spark Connects to Kafka
  • How Does Spark Access External Cluster Components?
• Common JDBCServer Faults
  • JDBCServer Connection Failure
• Common JobHistory Faults
  • ALM-43006 Is Repeatedly Reported for the MRS Cluster
  • What Can I Do If an Error Occurs when I Access the Application Page Because the Application Cached by HistoryServer Is Recycled?
  • Why Is the Native Page of an Application in Spark JobHistory Displayed Incorrectly?