Why Is An Error Reported When a Spark Jar Job Uses LakeFormation Metadata?

Symptom

When a Spark Jar job reads and writes LakeFormation metadata, the following error message is displayed when the job is started:

field 'location' must match '^(obs|har)://.+/.+$'

The job cannot continue, and the table or database fails to be created.

The detailed error information is as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

2025-07-25 09:47:55,782 | INFO  | [main] | Send request, http method: GET, url: 
https://172.16.xxx/v1/02f6e35xxxxx/instances/default/catalogs/lgl/databases/default, request Id: 75d0ee10-xxxx| com.huawei.cloud.dalf.lakecat.client.ApiClient.getResponse(ApiClient.java:392)
2025-07-25 09:47:56,259 | INFO  | [main] | Send request, http method: GET, url: 
https://172.16.xxx/v1/02f6e35xxxxx/instances/default/catalogs/lgl/databases/default, request Id: eef1c48f-xxx| com.huawei.cloud.dalf.lakecat.client.ApiClient.getResponse(ApiClient.java:392)
2025-07-25 09:47:56,433 | INFO  | [main] | Send request, http method: POST, url: 
https://172.16.xxx/v1/02f6e35xxxxx/instances/default/catalogs/lgl/databases, request Id: e6b4e588-xxx | com.huawei.cloud.dalf.lakecat.client.ApiClient.getResponse(ApiClient.java:392)
2025-07-25 09:47:56,495 | ERROR | [main] | field 'location' must match '^(obs|har)://.+/.+$' | com.huawei.cloud.dalf.lakecat.client.hiveclient.impl.HiveExceptionHandler.printLog(HiveExceptionHandler.java:51)
com.huawei.cloud.dalf.lakecat.client.exception.LakeFormationLakeCatClientException: 
400 Bad Request: "{"error_code":"common.00000400","error_msg":"field 'location' must match '^(obs|har)://.+/.+$'"}"
	at com.huawei.cloud.dalf.lakecat.client.exception.LakeFormationExceptionHandler.exceptionHandler(LakeFormationExceptionHandler.java:57) ~

Root Cause Analysis

This exception indicates that the location field of a database or table does not pass the regular expression verification of LakeFormation.

The default database is missing. An error message is displayed when Spark creates the default database.

• When Spark Catalog is started, it checks whether the default database exists in the current catalog.
  • If yes, Spark reads data from the default database.
  • If no, Spark automatically creates the database.

    However, the location generated by Spark by default does not meet the LakeFormation rule. As a result, an error is reported when the default database is referenced.

• The location specified in the statement for creating a database must start with obs://. Otherwise, the LakeFormation verification rules are not met.

Solution

Before using the new catalog, manually create the default database and specify a valid OBS path to prevent hidden risks caused by automatic Spark creation.

• An error is reported when an existing self-built database or table is checked.
  • Check and modify the location of the corresponding database or table to ensure that the location starts with obs://.
  • If the OBS path cannot be modified, delete it, create a new one, and specify the correct OBS path in the table or database creation statement.
• Manually create and initialize the default database.
  1. Log in to the LakeFormation console and choose Metadata > Database.
  2. Select the catalog to be used and click Create.
     Figure 1 Creating a database
     
  3. Set Database Name to default and Select Location to a valid OBS path, for example, obs://your-bucket/path/to/default/.
     Figure 2 Creating the default database and configuring the correct OBS path
     
  4. After the database is created, submit the Spark Jar job again.