How Do I Handle the Exception Occurred When I Query Datasource Avro Formats?

Question

An error is reported when I query Datasource Avro formats, and the message "Caused by: org.apache.spark.sql.avro.IncompatibleSchemaException" is displayed.

Answer

The datasource Avro formats are not compatible with the current data formats.

1. For new Avro files, set spark.sql.forceConvertSchema.enabled to true before you create them. This forcibly converts Avro formats to the specified data types and changes the schema at a time.
2. For existing Avro files, set spark.sql.forceConvertSchema.enabled to true before the query. If the query fails, run the refresh table command to clear the cache and then set query parameters. The Avro formats are forcibly converted to the specified data types, and the schema is temporarily modified on the client.