What Should I If Only Redundant Metadata Fails to Be Deleted After a Metadata Migration Job Is Successfully Executed?

Symptom

Metadata Deletion Policy is set to Delete redundant metadata when configuring a metadata migration job on the LakeFormation management console. After the migration job is executed, a message is displayed, indicating that the job is partially successful. The logs in the following picture show that the migration job is successful, but redundant metadata fails to be deleted.

Fault Handling

Prepare a custom filtering policy file, modify the metadata migration job, and run the migration task using the custom filtering policy.

Procedure

1. Prepare a custom filtering policy file test.json. The following file content indicates that table tbl_1 in database db_1, function func_1 in database db_1, and database db_2 are to be deleted.

   {
       "metadataPattern": [
           {
               "databasePattern": "db_1",
               "tablePatterns": [
                   "tbl_1"
               ],
               "functionPatterns": [
                   "func_1"
               ],
               "action": "DELETE"
           },
           {
               "databasePattern": "db_2",
               "action": "DELETE"
           }
       ],
       "partitionEnable": true
   }

   

   • action is set to DELETE, indicating that a deletion job is executed.
   • To delete a database, specify only the database name.
   • To delete a table, specify the database name and table name. You can specify multiple table names in a database.
   • To delete a function, specify the database name and function name. You can specify multiple function names in a database.

   Note: The objects to be migrated and deleted are metadata.

2. Log in to the console.
3. Click in the upper left corner and choose Storage > Object Storage Service. The OBS page is displayed.

   Upload the test.json file prepared in Step 1 to OBS, for example, obs://lakeformation/test.

4. Click in the upper left corner and choose Analytics > LakeFormation. The LakeFormation console is displayed.
5. Select the LakeFormation instance to be operated from the drop-down list on the left and choose Jobs > Metadata Migration in the navigation pane.
6. Locate a job and click Modify in the Operation column. Set the following parameters and click Submit.
   • Metadata Filtering Policy: Select Custom rule.
   • Filtering Policy Storage Location: Select obs://lakeformation/test.
   • Filtering Policy File Name: Enter test.json.

7. Run the migration task again. The task is executed successfully.