What Should I Do If Error "java.lang.NumberFormatException: For input string" Occurs During Data Synchronization to Hudi?

Symptom

An exception occurs when data is written to a Hudi table during incremental synchronization. The error log of JobManager or TaskManager contains message "java.lang.NumberFormatException: For input string".

Figure 1 Error message

Possible Causes

The field type at the source has changed (for example, from int to varchar), but the field type on Hudi has not been updated and is still int. As a result, the source data type is incompatible with Hudi, and the job encounters an exception.

Solution

Real-time migration jobs with Hudi as the destination do not support DDL operations such as changing the field type. You need to stop the real-time migration jobs and use the schema evolution function of Spark SQL and Hudi to change the field type of Hudi. After the change, restart the real-time migration job.

For details about how to change the field type of a Hudi table, see Evolution Introduction in the MRS documentation. The following is an example SQL statement used to change the field type. For details about the SQL statements, see the MRS documentation.

# Spark SQLset hoodie.schema.evolution.enable=true;ALTER TABLE {my_table} ALTER COLUMN {my_column} TYPE string;