How Do I Use Special Characters with TIMESTAMP and DATE?

Question

In versions later than Spark 3.2.0, TIMESTAMP(*) or DATE(*) is not supported. The asterisk (*) can be any of the following characters:

• epoch
• today
• yesterday
• tomorrow
• now

The default format supported is either the timestamp '*' or data '*' format. If the previous syntax is used to insert data into the data table, a NULL value will be returned.

Answer

To make the spark.sql.convert.special.datetime parameter compatible with the previous syntax, run the following command on the Spark client:

set spark.sql.convert.special.datetime=true;