Updated on 2024-02-21 GMT+08:00

Adding Partitions in Batches

Function

This API is used to add partitions in batches.

URI

POST /v1/{project_id}/instances/{instance_id}/catalogs/{catalog_name}/databases/{database_name}/tables/{table_name}/partitions/batch-create

Table 1 Path Parameters

Parameter

Mandatory

Type

Description

project_id

Yes

String

Project ID. For how to obtain the project ID, see Obtaining a Project ID (lakeformation_04_0026.xml).

instance_id

Yes

String

LakeFormation instance ID. The value is automatically generated when the instance is created, for example, 2180518f-42b8-4947-b20b-adfc53981a25.

catalog_name

Yes

String

Catalog name. The value should contain 1 to 256 characters. Only letters, numbers, and underscores (_) are allowed.

database_name

Yes

String

Database name. The value should contain 1 to 128 characters. Only letters, numbers, hyphens (-), and underscores (_) are allowed.

table_name

Yes

String

Table name. The value should contain 1 to 256 characters. Only letters, numbers, hyphens (-), and underscores (_) are allowed.

Request Parameters

Table 2 Request header parameters

Parameter

Mandatory

Type

Description

X-Auth-Token

Yes

Array of strings

Tenant token.

Table 3 Request body parameters

Parameter

Mandatory

Type

Description

if_not_exist

Yes

Boolean

Whether to skip a normal partition. The default value is false.

partitions

Yes

Array of PartitionInput objects

Information about the added partition.

Table 4 PartitionInput

Parameter

Mandatory

Type

Description

partition_values

Yes

Array of strings

Partition values.

create_time

Yes

String

Creation time.

last_access_time

Yes

String

Last access time.

parameters

Yes

Map<String,String>

Parameters.

storage_descriptor

Yes

StorageDescriptor object

Data storage.

Table 5 StorageDescriptor

Parameter

Mandatory

Type

Description

columns

Yes

Array of Column objects

All fields except partition columns.

location

No

String

Path, for example, obs://location/uri/.

compressed

Yes

Boolean

Whether to enable data compression. The default value is false.

input_format

No

String

Input format.

output_format

No

String

Output format.

number_of_buckets

No

Integer

Number of buckets. The default value is 0.

bucket_columns

No

Array of strings

Bucket field.

sort_columns

No

Array of Order objects

A list that specifies the bucket sorting method.

serde_info

Yes

SerDeInfo object

Serialization/Deserialization information.

parameters

Yes

Map<String,String>

Descriptor storage parameter. key can contain 255 characters at most. value can contain a maximum of 4,000 characters.

skewed_info

No

SkewedInfo object

Information about values (skewed values) that often appear in a column.

stored_as_sub_directories

No

Boolean

Whether to store data in subdirectories.

Table 6 Column

Parameter

Mandatory

Type

Description

column_type

Yes

String

Column type, including array, bigint, binary, boolean, char, date, decimal, double, float, int, interval, map, set, smallint, string, struct, timestamp, tinyint, union, and varchar.

column_name

Yes

String

Column name. The value can contain 1 to 767 characters. Only letters, digits, and special characters (_-+*(),) are allowed.

comment

No

String

Description of a column. Enter a maximum of 4,000 characters.

Table 7 Order

Parameter

Mandatory

Type

Description

column

No

String

Column name. The value can contain 1 to 767 characters. Only letters, digits, and special characters (_-+*(),) are allowed.

sort_order

No

Integer

Sorting method. The values are ==1 (ascending order) and ==0 (descending order).

Table 8 SerDeInfo

Parameter

Mandatory

Type

Description

name

No

String

Names.

serialization_library

No

String

Class that implements serialization/deserialization.

parameters

No

Map<String,String>

Parameters. The value of key ranges from 1 to 255. The maximum value of value is 4000.

Table 9 SkewedInfo

Parameter

Mandatory

Type

Description

skewed_column_names

Yes

Array of strings

List of skewed columns.

skewed_column_value_location_maps

Yes

Map<String,String>

Mapping between skew values and addresses.

skewed_column_values

Yes

Array<Array<String>>

List of skewed values.

Response Parameters

Status code: 201

Table 10 Response body parameters

Parameter

Type

Description

[items]

Array of Partition objects

List of newly added partitions.

Table 11 Partition

Parameter

Type

Description

catalog_name

String

Catalog name.

catalog_id

String

catalogID.

database_name

String

Database name.

database_id

String

Database ID.

table_name

String

Table name.

table_id

String

Table ID.

partition_id

String

Partition ID.

partition_values

Array of strings

Partition values.

create_time

String

Creation time.

last_access_time

String

Last access time.

parameters

Map<String,String>

Parameter table.

storage_descriptor

StorageDescriptor object

Data storage.

Table 12 StorageDescriptor

Parameter

Type

Description

columns

Array of Column objects

All fields except partition columns.

location

String

Path, for example, obs://location/uri/.

compressed

Boolean

Whether to enable data compression. The default value is false.

input_format

String

Input format.

output_format

String

Output format.

number_of_buckets

Integer

Number of buckets. The default value is 0.

bucket_columns

Array of strings

Bucket field.

sort_columns

Array of Order objects

A list that specifies the bucket sorting method.

serde_info

SerDeInfo object

Serialization/Deserialization information.

parameters

Map<String,String>

Descriptor storage parameter. key can contain 255 characters at most. value can contain a maximum of 4,000 characters.

skewed_info

SkewedInfo object

Information about values (skewed values) that often appear in a column.

stored_as_sub_directories

Boolean

Whether to store data in subdirectories.

Table 13 Column

Parameter

Type

Description

column_type

String

Column type, including array, bigint, binary, boolean, char, date, decimal, double, float, int, interval, map, set, smallint, string, struct, timestamp, tinyint, union, and varchar.

column_name

String

Column name. The value can contain 1 to 767 characters. Only letters, digits, and special characters (_-+*(),) are allowed.

comment

String

Description of a column. Enter a maximum of 4,000 characters.

Table 14 Order

Parameter

Type

Description

column

String

Column name. The value can contain 1 to 767 characters. Only letters, digits, and special characters (_-+*(),) are allowed.

sort_order

Integer

Sorting method. The values are ==1 (ascending order) and ==0 (descending order).

Table 15 SerDeInfo

Parameter

Type

Description

name

String

Names.

serialization_library

String

Class that implements serialization/deserialization.

parameters

Map<String,String>

Parameters. The value of key ranges from 1 to 255. The maximum value of value is 4000.

Table 16 SkewedInfo

Parameter

Type

Description

skewed_column_names

Array of strings

List of skewed columns.

skewed_column_value_location_maps

Map<String,String>

Mapping between skew values and addresses.

skewed_column_values

Array<Array<String>>

List of skewed values.

Status code: 400

Table 17 Response body parameters

Parameter

Type

Description

error_code

String

Error code.

error_msg

String

Error message.

solution_msg

String

Solution.

Status code: 404

Table 18 Response body parameters

Parameter

Type

Description

error_code

String

Error code.

error_msg

String

Error message.

solution_msg

String

Solution.

Status code: 500

Table 19 Response body parameters

Parameter

Type

Description

error_code

String

Error code.

error_msg

String

Error message.

solution_msg

String

Solution.

Example Requests

POST https://{endpoint} /v1/{project_id}/instances/{instance_id}/catalogs/{catalog_name}/databases/{database_name}/tables/{table_name}/partitions/batch-create

{
  "if_not_exist" : "false",
  "partitions" : [ {
    "partition_values" : [ "string" ],
    "create_time" : "2023-05-26T08:03:42.267Z",
    "last_access_time" : "2023-05-26T08:03:42.267Z",
    "parameters" : {
      "additionalProp1" : "string",
      "additionalProp2" : "string",
      "additionalProp3" : "string"
    },
    "storage_descriptor" : {
      "columns" : [ {
        "column_type" : "string",
        "column_name" : "c1",
        "comment" : "string"
      } ],
      "location" : "string",
      "compressed" : false,
      "input_format" : "string",
      "output_format" : "string",
      "number_of_buckets" : 0,
      "bucket_columns" : [ "string" ],
      "sort_columns" : [ {
        "column" : "string",
        "sort_order" : 1
      } ],
      "serde_info" : {
        "name" : "string",
        "serialization_library" : "string",
        "parameters" : {
          "additionalProp1" : "string",
          "additionalProp2" : "string",
          "additionalProp3" : "string"
        }
      },
      "parameters" : {
        "additionalProp1" : "string",
        "additionalProp2" : "string",
        "additionalProp3" : "string"
      },
      "skewed_info" : {
        "skewed_column_names" : [ "string" ],
        "skewed_column_value_location_maps" : {
          "additionalProp1" : "string",
          "additionalProp2" : "string",
          "additionalProp3" : "string"
        },
        "skewed_column_values" : [ [ ] ]
      },
      "stored_as_sub_directories" : false
    }
  } ]
}

Example Responses

Status code: 201

List of newly added partitions.

[ {
  "catalog_name" : "catcd6a92bfc2c74a509dc8501b8124505c",
  "database_name" : "db4cfc99ac3cea421abc157c7bd49f6f16",
  "table_name" : "tblbb79f3733c0747adbb94981b412fe48f",
  "partition_values" : [ "48e1f86dc4394a218b832d94411805fe", "5a2ab07a663c49a98cc606e6ffa0b178" ],
  "create_time" : "2023-05-30T09:06:41.372+00:00",
  "last_access_time" : "2023-05-30T09:06:41.000+00:00",
  "parameters" : {
    "key1" : "value1",
    "transient_lastDdlTime" : "120",
    "classification" : "other"
  },
  "storage_descriptor" : {
    "columns" : [ {
      "column_type" : "string",
      "column_name" : "column_prefix0",
      "comment" : "83e99b4daff04e75b28263bf44495a3c"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix1",
      "comment" : "d947669d90884014b60264b2770849ae"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix2",
      "comment" : "21edd8e92f2448deb036f8e4682ec694"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix3",
      "comment" : "fc25f38e8b694aebb34a2f84ef4a62f7"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix4",
      "comment" : "3443079e616d447f8f35da2014754701"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix5",
      "comment" : "5f7dda02ac8e4813b3eb4858abb7c520"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix6",
      "comment" : "2a6ae2e5bc3f4add99133106f2cd526c"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix7",
      "comment" : "84d50424269a41fbb5528b350e9d0b13"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix8",
      "comment" : "e3f1d2fa80454554a22e8564ec9c4832"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix9",
      "comment" : "f11315794e6d40daa2cb8347c61e5a46"
    } ],
    "location" : "obs://location/test/database/b593fc09871e4c9886ed931a3e51f9e5",
    "compressed" : false,
    "input_format" : "8c470072d43344a3a3bfabd2c7f9d6ec",
    "output_format" : "74ee5be43fb548e7b988390df05134fa",
    "number_of_buckets" : 0,
    "bucket_columns" : [ ],
    "sort_columns" : [ ],
    "serde_info" : {
      "name" : "9b3ea84748854c2d86a71ae37e246293",
      "serialization_library" : "e9632ffbbfbe425d8eac91e765e046e1",
      "parameters" : {
        "e5ae426044054a5f8e362b9b5729dbab" : "33d8296203f94760b1e4b7db55edce0d"
      }
    },
    "parameters" : {
      "7e27b28f439e40bea098dd65bc857334" : "ab893cac08bb4374beb8ff00a02e9a73",
      "c698760d6de74d0c92eee65bea283215" : "8ba9a058a5cc4ebaac6f43c9771e167a"
    },
    "skewed_info" : {
      "skewed_column_names" : [ ],
      "skewed_column_value_location_maps" : { },
      "skewed_column_values" : [ ]
    },
    "stored_as_sub_directories" : false
  }
}, {
  "catalog_name" : "catcd6a92bfc2c74a509dc8501b8124505c",
  "database_name" : "db4cfc99ac3cea421abc157c7bd49f6f16",
  "table_name" : "tblbb79f3733c0747adbb94981b412fe48f",
  "partition_values" : [ "e4af64d55308403a9abf1999afbbe2be", "079f4cd2ee5b4de6922ef8f0c92c82ae" ],
  "create_time" : "2023-05-30T09:06:41.372+00:00",
  "last_access_time" : "2023-05-30T09:06:41.000+00:00",
  "parameters" : {
    "key1" : "value1",
    "transient_lastDdlTime" : "120",
    "classification" : "other"
  },
  "storage_descriptor" : {
    "columns" : [ {
      "column_type" : "string",
      "column_name" : "column_prefix0",
      "comment" : "b40e5a6b6b3c4a538c4b7daa8bc49300"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix1",
      "comment" : "d7de59704feb4218ad6c3899f16f72e3"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix2",
      "comment" : "215d3961dd0c4858b884e50157da1d4b"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix3",
      "comment" : "b8e4888680ca449492be2344a81aba1d"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix4",
      "comment" : "1ac15b288c354834830062e48b04e3c8"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix5",
      "comment" : "5e65189b3f28434c8b96a56fb19b3005"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix6",
      "comment" : "7944cf9158514a96905712d686bc7c18"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix7",
      "comment" : "c6b9396476e541e9ab34a8eb5fda9953"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix8",
      "comment" : "978fe9b8a1d1437391e83553ae361bfc"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix9",
      "comment" : "4f7ed315c7c44c5683500104464f4776"
    } ],
    "location" : "obs://location/test/database/3971910eb87a41ee9d7096d538232c33",
    "compressed" : false,
    "input_format" : "620befc9b9a447fb95ecfb0d33fb636c",
    "output_format" : "fc943b36eaec4d1da525734e56374d32",
    "number_of_buckets" : 0,
    "bucket_columns" : [ ],
    "sort_columns" : [ ],
    "serde_info" : {
      "name" : "9286f98baef44841883166345635faed",
      "serialization_library" : "ac9cdbe5004d468798bc57b4ed10f602",
      "parameters" : {
        "cef1b8d74bdd43459793b00b4deff6ae" : "e66acd0f816047ae9e00d01a4045c340"
      }
    },
    "parameters" : {
      "0546e622cf2042598ed4f31e0b3b3276" : "be46b5accfdc41ba93f50fbe16b782c2",
      "a5b63ea213ab48be8c3c361fe93f75d2" : "c0f1215c944040cfad2e69fbe7cb4b0b"
    },
    "skewed_info" : {
      "skewed_column_names" : [ ],
      "skewed_column_value_location_maps" : { },
      "skewed_column_values" : [ ]
    },
    "stored_as_sub_directories" : false
  }
}, {
  "catalog_name" : "catcd6a92bfc2c74a509dc8501b8124505c",
  "database_name" : "db4cfc99ac3cea421abc157c7bd49f6f16",
  "table_name" : "tblbb79f3733c0747adbb94981b412fe48f",
  "partition_values" : [ "e5ade865888a4bb0a22b71505dd0989d", "788dfdb29db342f488a6155a55875f5b" ],
  "create_time" : "2023-05-30T09:06:41.372+00:00",
  "last_access_time" : "2023-05-30T09:06:41.000+00:00",
  "parameters" : {
    "key1" : "value1",
    "transient_lastDdlTime" : "120",
    "classification" : "other"
  },
  "storage_descriptor" : {
    "columns" : [ {
      "column_type" : "string",
      "column_name" : "column_prefix0",
      "comment" : "a4398b0a8f404ad1a2d8d350efdcd812"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix1",
      "comment" : "c2d17faf55de41ddb9f098a684dd8473"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix2",
      "comment" : "5f2de67835ff49aba7b661712ef059a6"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix3",
      "comment" : "26ce9cc09f3545df80181cbed87d4da2"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix4",
      "comment" : "f5b65b8db66349b3a1f8eace0ea33334"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix5",
      "comment" : "3f9b100dbc2f45d58d6fbcfa63e0d806"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix6",
      "comment" : "22c5f279e9d04cada8bdee6ddbd38af7"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix7",
      "comment" : "7ac1468020094057bf2e1e41f3e035b8"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix8",
      "comment" : "8c8178e2fe694ace80e78d5c985f7165"
    }, {
      "column_type" : "string",
      "column_name" : "column_prefix9",
      "comment" : "3f8d23d533fe44b295067de73a8f1b68"
    } ],
    "location" : "obs://location/test/database/8c351a1aca6b426a936be4453da30c35",
    "compressed" : false,
    "input_format" : "9fb9f9d02b3044febe447a1ad4ca5297",
    "output_format" : "ba8acae75da24d858ebe99f4ef2fce84",
    "number_of_buckets" : 0,
    "bucket_columns" : [ ],
    "sort_columns" : [ ],
    "serde_info" : {
      "name" : "6e4b2479321842d191f1b540b348a4f1",
      "serialization_library" : "22209422df114f589b4ec28d19d86e52",
      "parameters" : {
        "a5c4f39d83d047748dcd1218764c7a1c" : "9e2256acd089474ba469d329d35ace13"
      }
    },
    "parameters" : {
      "997ebfa4c6e343789a92d20f7961c865" : "2a1899624e084fb4b26595469cd23f32",
      "86887611dd7345c88f3de16538cb3587" : "ac4aab973d50429d9803cb418c9d44e1"
    },
    "skewed_info" : {
      "skewed_column_names" : [ ],
      "skewed_column_value_location_maps" : { },
      "skewed_column_values" : [ ]
    },
    "stored_as_sub_directories" : false
  }
} ]

Status code: 400

Bad Request

{
  "error_code" : "common.01000001",
  "error_msg" : "failed to read http request, please check your input, code: 400, reason: Type mismatch., cause: TypeMismatchException"
}

Status code: 401

Unauthorized

{
  "error_code": 'APIG.1002',
  "error_msg": 'Incorrect token or token resolution failed'
}

Status code: 403

Forbidden

{
  "error" : {
    "code" : "403",
    "message" : "X-Auth-Token is invalid in the request",
    "error_code" : null,
    "error_msg" : null,
    "title" : "Forbidden"
  },
  "error_code" : "403",
  "error_msg" : "X-Auth-Token is invalid in the request",
  "title" : "Forbidden"
}

Status code: 404

Not Found

{
  "error_code" : "common.01000001",
  "error_msg" : "response status exception, code: 404"
}

Status code: 408

Request Timeout

{
  "error_code" : "common.00000408",
  "error_msg" : "timeout exception occurred"
}

Status code: 500

Internal Server Error

{
  "error_code" : "common.00000500",
  "error_msg" : "internal error"
}

Status Codes

Status Code

Description

201

List of newly added partitions.

400

Bad Request

401

Unauthorized

403

Forbidden

404

Not Found

408

Request Timeout

500

Internal Server Error

Error Codes

See Error Codes.