Help Center/ DataArts Lake Formation/ API Reference/ API/ LakeCat/ Zone-based Management/ Adding Partitions in Batches

Updated on 2024-02-21 GMT+08:00

View PDF

Adding Partitions in Batches

Function

This API is used to add partitions in batches.

URI

POST /v1/{project_id}/instances/{instance_id}/catalogs/{catalog_name}/databases/{database_name}/tables/{table_name}/partitions/batch-create

**Table 1** Path Parameters
Parameter	Mandatory	Type	Description
project_id	Yes	String	Project ID. For how to obtain the project ID, see Obtaining a Project ID (lakeformation_04_0026.xml).
instance_id	Yes	String	LakeFormation instance ID. The value is automatically generated when the instance is created, for example, 2180518f-42b8-4947-b20b-adfc53981a25.
catalog_name	Yes	String	Catalog name. The value should contain 1 to 256 characters. Only letters, numbers, and underscores (_) are allowed.
database_name	Yes	String	Database name. The value should contain 1 to 128 characters. Only letters, numbers, hyphens (-), and underscores (_) are allowed.
table_name	Yes	String	Table name. The value should contain 1 to 256 characters. Only letters, numbers, hyphens (-), and underscores (_) are allowed.

Request Parameters

**Table 2** Request header parameters
Parameter	Mandatory	Type	Description
X-Auth-Token	Yes	Array of strings	Tenant token.

**Table 3** Request body parameters
Parameter	Mandatory	Type	Description
if_not_exist	Yes	Boolean	Whether to skip a normal partition. The default value is false.
partitions	Yes	Array of PartitionInput objects	Information about the added partition.

**Table 4** PartitionInput
Parameter	Mandatory	Type	Description
partition_values	Yes	Array of strings	Partition values.
create_time	Yes	String	Creation time.
last_access_time	Yes	String	Last access time.
parameters	Yes	Map<String,String>	Parameters.
storage_descriptor	Yes	StorageDescriptor object	Data storage.

**Table 5** StorageDescriptor
Parameter	Mandatory	Type	Description
columns	Yes	Array of Column objects	All fields except partition columns.
location	No	String	Path, for example, obs://location/uri/.
compressed	Yes	Boolean	Whether to enable data compression. The default value is false.
input_format	No	String	Input format.
output_format	No	String	Output format.
number_of_buckets	No	Integer	Number of buckets. The default value is 0.
bucket_columns	No	Array of strings	Bucket field.
sort_columns	No	Array of Order objects	A list that specifies the bucket sorting method.
serde_info	Yes	SerDeInfo object	Serialization/Deserialization information.
parameters	Yes	Map<String,String>	Descriptor storage parameter. key can contain 255 characters at most. value can contain a maximum of 4,000 characters.
skewed_info	No	SkewedInfo object	Information about values (skewed values) that often appear in a column.
stored_as_sub_directories	No	Boolean	Whether to store data in subdirectories.

**Table 6** Column
Parameter	Mandatory	Type	Description
column_type	Yes	String	Column type, including array, bigint, binary, boolean, char, date, decimal, double, float, int, interval, map, set, smallint, string, struct, timestamp, tinyint, union, and varchar.
column_name	Yes	String	Column name. The value can contain 1 to 767 characters. Only letters, digits, and special characters (_-+*(),) are allowed.
comment	No	String	Description of a column. Enter a maximum of 4,000 characters.

**Table 7** Order
Parameter	Mandatory	Type	Description
column	No	String	Column name. The value can contain 1 to 767 characters. Only letters, digits, and special characters (_-+*(),) are allowed.
sort_order	No	Integer	Sorting method. The values are ==1 (ascending order) and ==0 (descending order).

**Table 8** SerDeInfo
Parameter	Mandatory	Type	Description
name	No	String	Names.
serialization_library	No	String	Class that implements serialization/deserialization.
parameters	No	Map<String,String>	Parameters. The value of key ranges from 1 to 255. The maximum value of value is 4000.

**Table 9** SkewedInfo
Parameter	Mandatory	Type	Description
skewed_column_names	Yes	Array of strings	List of skewed columns.
skewed_column_value_location_maps	Yes	Map<String,String>	Mapping between skew values and addresses.
skewed_column_values	Yes	Array<Array<String>>	List of skewed values.

Response Parameters

Status code: 201

**Table 10** Response body parameters
Parameter	Type	Description
[items]	Array of Partition objects	List of newly added partitions.

**Table 11** Partition
Parameter	Type	Description
catalog_name	String	Catalog name.
catalog_id	String	catalogID.
database_name	String	Database name.
database_id	String	Database ID.
table_name	String	Table name.
table_id	String	Table ID.
partition_id	String	Partition ID.
partition_values	Array of strings	Partition values.
create_time	String	Creation time.
last_access_time	String	Last access time.
parameters	Map<String,String>	Parameter table.
storage_descriptor	StorageDescriptor object	Data storage.

**Table 12** StorageDescriptor
Parameter	Type	Description
columns	Array of Column objects	All fields except partition columns.
location	String	Path, for example, obs://location/uri/.
compressed	Boolean	Whether to enable data compression. The default value is false.
input_format	String	Input format.
output_format	String	Output format.
number_of_buckets	Integer	Number of buckets. The default value is 0.
bucket_columns	Array of strings	Bucket field.
sort_columns	Array of Order objects	A list that specifies the bucket sorting method.
serde_info	SerDeInfo object	Serialization/Deserialization information.
parameters	Map<String,String>	Descriptor storage parameter. key can contain 255 characters at most. value can contain a maximum of 4,000 characters.
skewed_info	SkewedInfo object	Information about values (skewed values) that often appear in a column.
stored_as_sub_directories	Boolean	Whether to store data in subdirectories.

**Table 13** Column
Parameter	Type	Description
column_type	String	Column type, including array, bigint, binary, boolean, char, date, decimal, double, float, int, interval, map, set, smallint, string, struct, timestamp, tinyint, union, and varchar.
column_name	String	Column name. The value can contain 1 to 767 characters. Only letters, digits, and special characters (_-+*(),) are allowed.
comment	String	Description of a column. Enter a maximum of 4,000 characters.

**Table 14** Order
Parameter	Type	Description
column	String	Column name. The value can contain 1 to 767 characters. Only letters, digits, and special characters (_-+*(),) are allowed.
sort_order	Integer	Sorting method. The values are ==1 (ascending order) and ==0 (descending order).

**Table 15** SerDeInfo
Parameter	Type	Description
name	String	Names.
serialization_library	String	Class that implements serialization/deserialization.
parameters	Map<String,String>	Parameters. The value of key ranges from 1 to 255. The maximum value of value is 4000.

**Table 16** SkewedInfo
Parameter	Type	Description
skewed_column_names	Array of strings	List of skewed columns.
skewed_column_value_location_maps	Map<String,String>	Mapping between skew values and addresses.
skewed_column_values	Array<Array<String>>	List of skewed values.

Status code: 400

**Table 17** Response body parameters
Parameter	Type	Description
error_code	String	Error code.
error_msg	String	Error message.
solution_msg	String	Solution.

Status code: 404

**Table 18** Response body parameters
Parameter	Type	Description
error_code	String	Error code.
error_msg	String	Error message.
solution_msg	String	Solution.

Status code: 500

**Table 19** Response body parameters
Parameter	Type	Description
error_code	String	Error code.
error_msg	String	Error message.
solution_msg	String	Solution.

Example Requests

POST https://{endpoint} /v1/{project_id}/instances/{instance_id}/catalogs/{catalog_name}/databases/{database_name}/tables/{table_name}/partitions/batch-create

{
  "if_not_exist" : "false",
  "partitions" : [ {
    "partition_values" : [ "string" ],
    "create_time" : "2023-05-26T08:03:42.267Z",
    "last_access_time" : "2023-05-26T08:03:42.267Z",
    "parameters" : {
      "additionalProp1" : "string",
      "additionalProp2" : "string",
      "additionalProp3" : "string"
    },
    "storage_descriptor" : {
      "columns" : [ {
        "column_type" : "string",
        "column_name" : "c1",
        "comment" : "string"
      } ],
      "location" : "string",
      "compressed" : false,
      "input_format" : "string",
      "output_format" : "string",
      "number_of_buckets" : 0,
      "bucket_columns" : [ "string" ],
      "sort_columns" : [ {
        "column" : "string",
        "sort_order" : 1
      } ],
      "serde_info" : {
        "name" : "string",
        "serialization_library" : "string",
        "parameters" : {
          "additionalProp1" : "string",
          "additionalProp2" : "string",
          "additionalProp3" : "string"
        }
      },
      "parameters" : {
        "additionalProp1" : "string",
        "additionalProp2" : "string",
        "additionalProp3" : "string"
      },
      "skewed_info" : {
        "skewed_column_names" : [ "string" ],
        "skewed_column_value_location_maps" : {
          "additionalProp1" : "string",
          "additionalProp2" : "string",
          "additionalProp3" : "string"
        },
        "skewed_column_values" : [ [ ] ]
      },
      "stored_as_sub_directories" : false
    }
  } ]
}

Example Responses

Status code: 201

List of newly added partitions.

[ {
  "catalog_name" : "catcd6a92bfc2c74a509dc8501b8124505c",
  "database_name" : "db4cfc99ac3cea421abc157c7bd49f6f16",
  "table_name" : "tblbb79f3733c0747adbb94981b412fe48f",
  "partition_values" : [ "48e1f86dc4394a218b832d94411805fe", "5a2ab07a663c49a98cc606e6ffa0b178" ],
  "create_time" : "2023-05-30T09:06:41.372+00:00",
  "last_access_time" : "2023-05-30T09:06:41.000+00:00",
  "parameters" : {
    "key1" : "value1",
    "transient_lastDdlTime" : "120",
    "classification" : "other"
  },
  "storage_descriptor" : {
    "columns" : [ {
      "column_type" : "string",
      "column_name" : "column_prefix0",
      "comment" : "83e99b4daff04e75b28263bf44495a3c"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix1",
      "comment" : "d947669d90884014b60264b2770849ae"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix2",
      "comment" : "21edd8e92f2448deb036f8e4682ec694"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix3",
      "comment" : "fc25f38e8b694aebb34a2f84ef4a62f7"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix4",
      "comment" : "3443079e616d447f8f35da2014754701"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix5",
      "comment" : "5f7dda02ac8e4813b3eb4858abb7c520"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix6",
      "comment" : "2a6ae2e5bc3f4add99133106f2cd526c"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix7",
      "comment" : "84d50424269a41fbb5528b350e9d0b13"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix8",
      "comment" : "e3f1d2fa80454554a22e8564ec9c4832"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix9",
      "comment" : "f11315794e6d40daa2cb8347c61e5a46"
    } ],
    "location" : "obs://location/test/database/b593fc09871e4c9886ed931a3e51f9e5",
    "compressed" : false,
    "input_format" : "8c470072d43344a3a3bfabd2c7f9d6ec",
    "output_format" : "74ee5be43fb548e7b988390df05134fa",
    "number_of_buckets" : 0,
    "bucket_columns" : [ ],
    "sort_columns" : [ ],
    "serde_info" : {
      "name" : "9b3ea84748854c2d86a71ae37e246293",
      "serialization_library" : "e9632ffbbfbe425d8eac91e765e046e1",
      "parameters" : {
        "e5ae426044054a5f8e362b9b5729dbab" : "33d8296203f94760b1e4b7db55edce0d"
      }
    },
    "parameters" : {
      "7e27b28f439e40bea098dd65bc857334" : "ab893cac08bb4374beb8ff00a02e9a73",
      "c698760d6de74d0c92eee65bea283215" : "8ba9a058a5cc4ebaac6f43c9771e167a"
    },
    "skewed_info" : {
      "skewed_column_names" : [ ],
      "skewed_column_value_location_maps" : { },
      "skewed_column_values" : [ ]
    },
    "stored_as_sub_directories" : false
  }
}, {
  "catalog_name" : "catcd6a92bfc2c74a509dc8501b8124505c",
  "database_name" : "db4cfc99ac3cea421abc157c7bd49f6f16",
  "table_name" : "tblbb79f3733c0747adbb94981b412fe48f",
  "partition_values" : [ "e4af64d55308403a9abf1999afbbe2be", "079f4cd2ee5b4de6922ef8f0c92c82ae" ],
  "create_time" : "2023-05-30T09:06:41.372+00:00",
  "last_access_time" : "2023-05-30T09:06:41.000+00:00",
  "parameters" : {
    "key1" : "value1",
    "transient_lastDdlTime" : "120",
    "classification" : "other"
  },
  "storage_descriptor" : {
    "columns" : [ {
      "column_type" : "string",
      "column_name" : "column_prefix0",
      "comment" : "b40e5a6b6b3c4a538c4b7daa8bc49300"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix1",
      "comment" : "d7de59704feb4218ad6c3899f16f72e3"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix2",
      "comment" : "215d3961dd0c4858b884e50157da1d4b"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix3",
      "comment" : "b8e4888680ca449492be2344a81aba1d"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix4",
      "comment" : "1ac15b288c354834830062e48b04e3c8"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix5",
      "comment" : "5e65189b3f28434c8b96a56fb19b3005"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix6",
      "comment" : "7944cf9158514a96905712d686bc7c18"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix7",
      "comment" : "c6b9396476e541e9ab34a8eb5fda9953"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix8",
      "comment" : "978fe9b8a1d1437391e83553ae361bfc"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix9",
      "comment" : "4f7ed315c7c44c5683500104464f4776"
    } ],
    "location" : "obs://location/test/database/3971910eb87a41ee9d7096d538232c33",
    "compressed" : false,
    "input_format" : "620befc9b9a447fb95ecfb0d33fb636c",
    "output_format" : "fc943b36eaec4d1da525734e56374d32",
    "number_of_buckets" : 0,
    "bucket_columns" : [ ],
    "sort_columns" : [ ],
    "serde_info" : {
      "name" : "9286f98baef44841883166345635faed",
      "serialization_library" : "ac9cdbe5004d468798bc57b4ed10f602",
      "parameters" : {
        "cef1b8d74bdd43459793b00b4deff6ae" : "e66acd0f816047ae9e00d01a4045c340"
      }
    },
    "parameters" : {
      "0546e622cf2042598ed4f31e0b3b3276" : "be46b5accfdc41ba93f50fbe16b782c2",
      "a5b63ea213ab48be8c3c361fe93f75d2" : "c0f1215c944040cfad2e69fbe7cb4b0b"
    },
    "skewed_info" : {
      "skewed_column_names" : [ ],
      "skewed_column_value_location_maps" : { },
      "skewed_column_values" : [ ]
    },
    "stored_as_sub_directories" : false
  }
}, {
  "catalog_name" : "catcd6a92bfc2c74a509dc8501b8124505c",
  "database_name" : "db4cfc99ac3cea421abc157c7bd49f6f16",
  "table_name" : "tblbb79f3733c0747adbb94981b412fe48f",
  "partition_values" : [ "e5ade865888a4bb0a22b71505dd0989d", "788dfdb29db342f488a6155a55875f5b" ],
  "create_time" : "2023-05-30T09:06:41.372+00:00",
  "last_access_time" : "2023-05-30T09:06:41.000+00:00",
  "parameters" : {
    "key1" : "value1",
    "transient_lastDdlTime" : "120",
    "classification" : "other"
  },
  "storage_descriptor" : {
    "columns" : [ {
      "column_type" : "string",
      "column_name" : "column_prefix0",
      "comment" : "a4398b0a8f404ad1a2d8d350efdcd812"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix1",
      "comment" : "c2d17faf55de41ddb9f098a684dd8473"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix2",
      "comment" : "5f2de67835ff49aba7b661712ef059a6"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix3",
      "comment" : "26ce9cc09f3545df80181cbed87d4da2"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix4",
      "comment" : "f5b65b8db66349b3a1f8eace0ea33334"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix5",
      "comment" : "3f9b100dbc2f45d58d6fbcfa63e0d806"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix6",
      "comment" : "22c5f279e9d04cada8bdee6ddbd38af7"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix7",
      "comment" : "7ac1468020094057bf2e1e41f3e035b8"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix8",
      "comment" : "8c8178e2fe694ace80e78d5c985f7165"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix9",
      "comment" : "3f8d23d533fe44b295067de73a8f1b68"
    } ],
    "location" : "obs://location/test/database/8c351a1aca6b426a936be4453da30c35",
    "compressed" : false,
    "input_format" : "9fb9f9d02b3044febe447a1ad4ca5297",
    "output_format" : "ba8acae75da24d858ebe99f4ef2fce84",
    "number_of_buckets" : 0,
    "bucket_columns" : [ ],
    "sort_columns" : [ ],
    "serde_info" : {
      "name" : "6e4b2479321842d191f1b540b348a4f1",
      "serialization_library" : "22209422df114f589b4ec28d19d86e52",
      "parameters" : {
        "a5c4f39d83d047748dcd1218764c7a1c" : "9e2256acd089474ba469d329d35ace13"
      }
    },
    "parameters" : {
      "997ebfa4c6e343789a92d20f7961c865" : "2a1899624e084fb4b26595469cd23f32",
      "86887611dd7345c88f3de16538cb3587" : "ac4aab973d50429d9803cb418c9d44e1"
    },
    "skewed_info" : {
      "skewed_column_names" : [ ],
      "skewed_column_value_location_maps" : { },
      "skewed_column_values" : [ ]
    },
    "stored_as_sub_directories" : false
  }
} ]

Status code: 400

Bad Request

{
  "error_code" : "common.01000001",
  "error_msg" : "failed to read http request, please check your input, code: 400, reason: Type mismatch., cause: TypeMismatchException"
}

Status code: 401

Unauthorized

{
  "error_code": 'APIG.1002',
  "error_msg": 'Incorrect token or token resolution failed'
}

Status code: 403

Forbidden

{
  "error" : {
    "code" : "403",
    "message" : "X-Auth-Token is invalid in the request",
    "error_code" : null,
    "error_msg" : null,
    "title" : "Forbidden"
  },
  "error_code" : "403",
  "error_msg" : "X-Auth-Token is invalid in the request",
  "title" : "Forbidden"
}

Status code: 404

Not Found

{
  "error_code" : "common.01000001",
  "error_msg" : "response status exception, code: 404"
}

Status code: 408

Request Timeout

{
  "error_code" : "common.00000408",
  "error_msg" : "timeout exception occurred"
}

Status code: 500

Internal Server Error

{
  "error_code" : "common.00000500",
  "error_msg" : "internal error"
}

Status Codes

Status Code	Description
201	List of newly added partitions.
400	Bad Request
401	Unauthorized
403	Forbidden
404	Not Found
408	Request Timeout
500	Internal Server Error

Error Codes

See Error Codes.

Parent topic: Zone-based Management

Previous topic: Listing Partitions

Next topic: Deleting Partitions in Batches

Feedback

Was this page helpful?

Helpful Not helpful

Provide feedback

Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.

The system is busy. Please try again later.

Which of the following issues have you encountered?

Content is inconsistent with the product UI

Unclear descriptions

Lack of examples or code

Incorrect steps

Can't find what I need

Lack of best practices

Feedback (optional)

0/500

Select at least one type of issue, and enter your comments or suggestions.

Enter a maximum of 500 characters.

Submit Cancel

For any further questions, feel free to contact us through the chatbot.

Chatbot