Typical Delta Configurations
To set Delta parameters while submitting a DLI Spark SQL job, access the SQL Editor page and click Settings in the upper right corner. In the Parameter Settings area, set the parameters.
Parameter |
Description |
Default Value |
---|---|---|
spark.databricks.delta.retentionDurationCheck.enabled |
Whether the retention period is checked when vacuum clears files that are no longer referenced. |
true |
spark.databricks.delta.properties.defaults.deletedFileRetentionDuration or delta.deletedFileRetentionDuration |
Retention period of files that are no longer referenced by Delta. If spark.databricks.delta.retentionDurationCheck.enabled is set to true, an exception will be thrown when files that have not yet reahced their retention period are cleared. |
168 hours (1 week) |
spark.databricks.delta.properties.defaults.logRetentionDuration or delta.logRetentionDuration |
Expiration time of Delta log files. When a Delta log executes a checkpoint action, it scans for expired files that need deletion. If such files are found, they are removed to prevent the Delta log files from growing indefinitely. |
30 days |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.