Parallel File System Features

Compatibility Between OBS Bucket APIs and Parallel File Systems

You can call some OBS bucket APIs to use parallel file systems. There may be additional requirements when you call these APIs.

For details about the OBS bucket APIs, see Object Storage Service API Reference.

APIs for Basic Bucket Operations

**Table 1** APIs for basic bucket operations
API	Compatible with Parallel File Systems	Differences
Listing buckets	√	The x-obs-bucket-type:POSIX header is required for obtaining the list of parallel file systems.
Creating a bucket	√	The x-obs-fs-file-interface:Enabled header is required for creating a parallel file system.
Listing objects in a bucket	√	-
Obtaining bucket metadata	√	-
Obtaining location of a bucket	√	-
Deleting a bucket	√	-

APIs for Advanced Bucket Settings

**Table 2** APIs for advanced bucket settings
API	Compatible with Parallel File Systems	Differences
Configuring a bucket policy	√	-
Obtaining a bucket policy	√	-
Deleting a bucket policy	√	-
Configuring a bucket ACL	√	-
Obtaining bucket ACL information	√	-
Configuring logging for a bucket	√	-
Obtaining a bucket logging configuration	√	-
Configuring bucket lifecycle rules	√	See Lifecycle Management
Obtaining bucket lifecycle rules	√	-
Deleting bucket lifecycle rules	√	-
Configuring versioning for a bucket	×	-
Obtaining bucket versioning status	×	-
Configuring event notification for a bucket	√	-
Obtaining the event notification configuration of a bucket	√	-
Configuring storage class for a bucket	×	-
Obtaining bucket storage class information	×	-
Configuring cross-region replication for a bucket	×	-
Obtaining the cross-region replication configuration of a bucket	×	-
Deleting the cross-region replication configuration of a bucket	×	-
Configuring tags for a bucket	√	-
Obtaining bucket tags	√	-
Deleting bucket tags	√	-
Configuring bucket storage quota	√	-
Querying bucket storage quota	√	-
Querying information about used space in a bucket	√	-
Configuring bucket inventories	×	-
Obtaining bucket inventories	×	-
Listing bucket inventories	×	-
Deleting bucket inventories	×	-
Configuring a custom domain name for a bucket	√	-
Obtaining the custom domain name of a bucket	√	-
Deleting a custom domain name of a bucket	√	-
Configuring bucket encryption	×	-
Obtaining bucket encryption configuration	×	-
Deleting the encryption configuration of a bucket	×	-
Configuring direct reading for Archive objects in a bucket	√	-
Obtaining direct reading configuration for Archive objects in a bucket	√	-
Deleting direct reading configuration for Archive objects in a bucket	√	-
Configuring mirroring-based back-to-source rules	×	-
Obtaining mirroring-based back-to-source rules	×	-
Deleting mirroring-based back-to-source rules	×	-
Configuring online decompression policies	×	-
Obtaining online decompression policies	×	-
Deleting online decompression policies	×	-
Configuring a default WORM policy for a bucket	×	-
Obtaining the default WORM policy of a bucket	×	-
Configuring static website hosting for a bucket	×	-
Obtaining the static website hosting configuration of a bucket	×	-
Deleting the static website hosting configuration of a bucket	×	-
Configuring bucket CORS	√	-
Obtaining the CORS configuration of a bucket	√	-
Deleting the CORS configuration of a bucket	√	-
Checking bucket OPTIONS	×	-
Checking object OPTIONS	×	-

APIs for Object Operations

**Table 3** APIs for object operations
API	Compatible with Parallel File Systems	Differences
Uploading an object with PUT	√	Headers not supported: x-obs-storage-class, x-obs-website-redirect-location, success-action-redirect, and x-obs-expires Objects uploaded using this API cannot be directly stored in the Infrequent Access or Archive storage class and are stored in the Standard storage class by default. You can later change the storage class by using a lifecycle rule or modifying the metadata. Directories that do not exist in the request path can be automatically created. For example, if you perform the following PUT operation and there is the dir01 directory in the parallel file system but there are no dir02 and dir03 directories, the dir02 and dir03 directories are automatically created. That is, after the command is executed successfully, dir02 and dir03 directories and file01 object are created. PUT /dir01/dir02/dir03/file01 HTTP/1.1
Uploading an object with POST	√	Headers not supported: x-obs-storage-class, x-obs-website-redirect-location, success-action-redirect, and x-obs-expires
Copying an object	√	Data can be replicated only between parallel file systems or buckets that are in the same cluster.
Downloading an object	√	-
Obtaining object metadata	√	-
Deleting an object	√	-
Batch deleting objects	√	-
Restoring Archive objects	√	-
Appending data to an object	×	Parallel file systems do not support this API. You can obtain the file length by calling the API for obtaining object metadata and then call the API for modifying an object to append data.
Configuring object ACL	√	-
Obtaining an object ACL	√	-
Modifying object metadata	√	In a parallel file system, the storage class of a directory cannot be changed. To change the storage class of a file in the directory, modify the metadata of the file or use a lifecycle rule to change the storage class of files in batches.
Modifying an object	√	This API is only for parallel file systems and is not supported for OBS buckets.
Truncating an object	√	This API is only for parallel file systems and is not supported for OBS buckets.
Renaming an object	√	This API is only for parallel file systems and is not supported for OBS buckets.
Configuring object tags	×	-
Obtaining object tags	×	-
Deleting object tags	×	-
Configuring WORM retention for an object	×	-

APIs for Multipart Uploads

**Table 4** APIs for multipart uploads
API	Compatible with Parallel File Systems	Differences
Listing the initiated multipart uploads in a bucket	√	-
Initiating a multipart upload	√	-
Uploading parts	√	-
Copying parts	√	Copying parts is not supported for an appended file.
Listing uploaded parts	√	-
Assembling parts	√	-
Aborting the multipart upload task	√	-

Permissions Configuration

The use cases and main functions of object access control also work on files in parallel file systems. For more information, see Permissions Configuration Guide.

Differences Between File and Object Permission Configurations

To exactly match a specific directory, the resource path in the policy must end with a slash (/). When checking permissions, parallel file systems consider objects as directories. Since objects do not end with a slash (/), a parallel file system will add a slash (/) to the end of objects and then perform a policy matching.

IAM Permission Configuration Examples

Example 1: Grant a user the permissions required to download dir_1, excluding its subdirectories.

In the following configuration, the resource path ends with a slash (/). In such case, a success response can be returned when dir_1 or dir_1/ is contained in the URL of a head request.

Note that this configuration is not applied to subdirectories or files in dir_1. Therefore, a failure response will be returned if a head request is sent to dir_1/file1.

{
    "Version": "1.1",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "obs:object:GetObject",
            ],
            "Resource": [
                "obs:*:*:object:examplebucket/dir_1/",
            ]
        }
    ]
}

Example 2: Grant a user the permissions required to download dir_1 and its subdirectories.

In the following configuration, the resource path uses prefix-based matching and ends with a wildcard (*). In such case, a success response can be returned when a head request is sent to dir_1/file1.

{
    "Version": "1.1",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "obs:object:GetObject",
            ],
            "Resource": [
                "obs:*:*:object:examplebucket/dir_1/*",
            ]
        }
    ]
}

Policy Configuration Examples

Example: Grant a user the permissions required to download dir_1, excluding its subdirectories.

In the following configuration, the resource path ends with a slash (/). In such case, a success response can be returned when dir_1 or dir_1/ is contained in the URL of a head request.

{
    "Statement":[
    {
      "Sid":"test",
      "Effect":"Allow",
      "Principal": {"ID": ["domain/b4bf1b36d9ca43d984fbcb9491b6fce9:user/71f3901173514e6988115ea2c26d1999"]},
      "Action":["*"],
      "Resource":[
        "examplebucket/dir_1/",
      ]
    }
  ]
}

Lifecycle Management

The use cases and main functions of object lifecycle management also work on files in parallel file systems. For more information, see Creating a Lifecycle Rule.

To manage the lifecycle using SDKs, see SDK Overview.

Differences Between File and Object Lifecycle Management

You can configure lifecycle rules to manage files and directories. Directories cannot be transitioned to the Archive storage class, but empty directories can be deleted upon expiration.
Parallel file systems do not support versioning. Therefore, you cannot configure deletion upon expiration or transition between storage classes for historical versions of objects.
Each parallel file system can have a maximum of 20 lifecycle rules.
The time applied for a lifecycle rule to work on a file is when the data of the file was last updated.
Lifecycle rules cannot transition files to the Deep Archive storage.
A lifecycle rule configured for a parallel file system has limits on how many directories it can apply to. It will take longer time to execute the actions defined in the rule if:
1. There are over 100,000 level-1 subdirectories in each directory.
2. There are over 10 million subdirectories (folders) matching the specified prefix in total.
3. There are over 30 million files matching the specified prefix in total.

Notes for Lifecycle Management

If renamed files or files in a renamed directory meet the conditions specified in a lifecycle rule, the rule applies based on the most recent data update, not the renaming time. In addition, the effective time of the lifecycle rule may be delayed for up to seven days.

For a file copy on a client, the lifecycle rule determines when to expire the file copy or transition it to the Archive storage class based on the file copy creation time.
- For example, if a file, src.txt, was created on January 1, 2019, and was then copied to the des.txt file by running the cp -a src.txt des.txt command on September 1, 2019. Then the lifecycle rule calculated when to perform the specified actions on the file copy based on September 1, 2019.
A lifecycle rule scans directories periodically, starting from the deepest directory, and deletes those meeting expiration conditions. Empty directories that meet the conditions will be deleted, while non-empty ones will be ignored. Scan intervals (usually seven days) vary depending on cluster configurations. For this mechanism, a single-level directory that meets the expiration conditions will be deleted 0 to 7 days after it has been emptied. Accordingly, a two-level directory will be deleted 0 to 14 days after it has been emptied. Each time a directory level is added, the waiting time for deletion increases by seven days.
If a file or directory is mistakenly stored under a path whose prefix matches the filter in a lifecycle rule, it must be removed at least 24 hours before the rule is scheduled to apply. If you remove the file or directory less than 24 hours before it will be processed by the lifecycle rule, the file or directory may still be processed.