What Should I Do If the Number of Read Rows Is the Same as That of Write Rows, Both Numbers No Longer Increase, and the Job Stays in Running State?
Possible Cause
CDM writes data to the Hive temporary table and then run a Spark SQL statement to write data to Hudi. The number of written rows is the number of rows written to the Hive temporary table. When the number of rows no longer increases, all the source data has been read and written to the Hive table. However, the job is executing a Spark SQL statement and the execution is complete only after the Spark SQL statement is executed.
Troubleshooting
Open the log, search for insert into, find the following log, and view Yarn task details on MRS Resource Manager based on Yarn ApplicationId in the log.
The speed of executing the Spark SQL statement is closely related to the resources of the tenant queue. Before executing Hudi tasks, ensure that the tenant queue has sufficient resources.
Hudi Destination Case Library FAQs
- What Should I Do If the Auto Creation of the Hudi Table Fails Due to a Schema Mismatch?
- What Should I Do If a Hudi Job Stays in Booting Status for a Long Time and then Fails and the "Read timed out" Error Is Contained in the Log?
- What Should I Do If the Number of Read Rows Is the Same as That of Write Rows, Both Numbers No Longer Increase, and the Job Stays in Running State?
- What Should I Do If a Job Stays in Running State but the Number of Written Rows Is 0?
- What Should I Do If Data Fails to Be Written to Hudi Using Spark SQL?
- What Should I Do If a Job Fails Due to Intermittent Disconnection, Timeout, or Connection Termination of the Source Link?
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbotmore
