Typical Delta Configurations
To set Delta parameters while submitting a DLI Spark SQL job, access the SQL Editor page and click Settings in the upper right corner. In the Parameter Settings area, set the parameters.
Parameter | Description | Default Value |
|---|---|---|
spark.databricks.delta.retentionDurationCheck.enabled | Whether the retention period is checked when vacuum clears files that are no longer referenced. | true |
spark.databricks.delta.properties.defaults.deletedFileRetentionDuration or delta.deletedFileRetentionDuration | Retention period of files that are no longer referenced by Delta. If spark.databricks.delta.retentionDurationCheck.enabled is set to true, an exception will be thrown when files that have not yet reahced their retention period are cleared. | 168 hours (1 week) |
spark.databricks.delta.properties.defaults.logRetentionDuration or delta.logRetentionDuration | Expiration time of Delta log files. When a Delta log executes a checkpoint action, it scans for expired files that need deletion. If such files are found, they are removed to prevent the Delta log files from growing indefinitely. | 30 days |
Feedback
Was this page helpful?
Provide feedbackThank you very much for your feedback. We will continue working to improve the documentation.
