Help Center/ Object Storage Service/ User Guide/ Data Processing/ Online Decompression
Updated on 2024-10-24 GMT+08:00
View PDF
Share

Online Decompression

You can compress files into a ZIP package, upload it to OBS for auto decompression online, and store it in a specified directory.

Uploading a large number of small files at a time by using a ZIP package saves time and efforts. For batch upload constraints, see OBS Batch Upload.

If the package you uploaded matches the configured decompression policy, it will be automatically decompressed upon upload. A decompression policy does not apply to the ZIP packages that already exist in OBS before the policy is created.

You can create online decompression policies or replicate existing policies from another bucket.

Prerequisites

You have been assigned the Tenant Administrator role.

Constraints

Permissions

  • To configure, obtain, or delete online decompression policies, you must have the Tenant Administrator permission assigned by using IAM.
  • Online decompression requires that you create an IAM agency to delegate OBS to access data in the bucket. The permissions the agency should assign to OBS include obs:object:PutObject, obs:object:GetObject, and obs:object:AbortMultipartUpload.

Regions

  • Online decompression is only available in some regions. For details, see Function Overview.

Package and file size

  • A single ZIP package cannot exceed 1 GB in size.
  • A single ZIP package can contain a maximum of 65,536 files.
  • A single decompressed file cannot exceed 40 GB in size.

Time

  • Decompressing a ZIP package takes 10 minutes at most.

Functions

  • Currently, only ZIP packages are supported.
  • ZIP package names cannot contain Chinese punctuation marks, special characters, or special codes.
  • To decompress the ZIP package that contains other ZIP packages, the event type of the online decompression policy must be set to ObjectCreated:* or ObjectCreated:CompleteMultipartUpload.
  • Currently, no notification will be sent to users after decompression tasks are complete.
  • Currently, only deflate-compressed ZIP packages can be decompressed. Deflate is different from the compression algorithms (such as Store and Normal) provided by WinRAR.
  • The total length of the decompression path plus the name of any decompressed file cannot exceed 512 characters.
  • Currently, encrypted ZIP packages cannot be decompressed.
  • Federated users cannot use the online decompression function. For more information about federated users, see What Are the Relationships Between a Huawei Cloud Account, HUAWEI ID, IAM User, and Federated User? and Identity Providers.

Precautions

  • You are advised to set a precise prefix for a decompression policy. In the same bucket, trigger conditions (including events, prefixes, or suffixes) of different decompression policies cannot overlap with each other.

    For example, there are two decompression policies event-0001 and event-0002 in a bucket. If the prefix of event-0001 is aa, the prefix of event-0002 cannot be aaaa, because aa is contained in aaaa.

  • If the prefix is left blank, the decompression policy applies to all the ZIP packages in the bucket by default. This may trigger cyclic decompression if a package contains other packages.

    For example, package AA.zip contains another package BB.zip. If the prefix is left blank, the system continues to decompress BB.zip after decompressing AA.zip. This issue will not happen if a prefix is set in the policy.

  • You must set a directory for storing the decompressed files. If the directory is not set, decompressed files will be stored in the home directory of the current bucket.
  • You are advised to encode file or folder names using UTF-8. Otherwise, names of decompressed files or folders may contain garbled characters, or the decompression may be interrupted.
  • If you want ZIP packages in the Archive storage class to be automatically decompressed upon upload, ensure that Direct Reading has been enabled for the bucket. ZIP packages in the Deep Archive storage class will not be automatically decompressed after they are uploaded to an OBS bucket.
  • A ZIP package decompression will fail if it takes more than 10 minutes.

Creating an Online Decompression Policy

You can use OBS Console or APIs to create an online decompression policy.

Using OBS Console

  1. In the navigation pane of OBS Console, choose Object Storage.
  2. In the bucket list, click the bucket you want to operate. The Objects page is displayed.
  3. In the navigation pane, choose Data Processing > Online Decompression. The Online Decompression page is displayed.
  4. Click Create.

    Figure 1 Creating an online decompression policy

  5. Configure the online decompression policy. Table 1 describes the related parameters.

    Table 1 Parameter description

    Parameter

    Description

    Policy Name

    Enter a policy name that is easy to remember. The value can contain 1 to 256 characters, and only uppercase letters, lowercase letters, digits, underscores (_), and hyphens (-) are allowed, for example, event_0001.

    Events

    Events that you want to trigger the online decompression policy for. Currently, the following event types are supported:

    • ObjectCreated: all object creation operations, including PUT, POST, COPY, and part assembling
    • Put: object upload using PUT
    • Post: object upload using POST
    • Copy: object copying using COPY
    • CompleteMultipartUpload: assembling of parts in a multipart upload
      NOTE:

      To decompress the ZIP package that contains other ZIP packages, set the event type to ObjectCreated or CompleteMultipartUpload.

    Prefix

    Optional. If this parameter is configured, the decompression policy applies to the packages whose name contains this prefix. The prefix cannot start with a slash (/) or contain double slashes (//), or contain special characters (\ : * ? " < > |). The total length of the prefix and suffix cannot exceed 512 characters.

    • With this parameter configured, ZIP packages whose name contains the specified prefix will trigger online decompression.
    • With this parameter left blank, the decompression policy applies to all the uploaded ZIP packages.
      CAUTION:
      • You are advised to configure a prefix. Otherwise, cyclic decompression may occur if a package contains other packages.
      • The configured prefix must contain all levels of the directory for storing the object.

        For example, there is an object example123 that is stored under bucket/file/example123. If you want example to be the prefix, set the prefix to file/example.

    Suffix

    If this parameter is specified, the decompression policy applies to the packages whose name contains this suffix. Currently, .zip is the default and only value.

    Duplicate Name Processing
    It specifies how the decompressed objects are processed if they have the same names as the existing objects in the bucket.
    • Do not decompress: Retains the existing objects, and does not decompress the objects with the same name.
    • Rename the file: Renames the decompressed objects with the CRC32 value.
    • Overwrite: Overwrites the existing objects in the bucket.

    Decompress To
    Optional, this parameter specifies the path for storing decompressed files. It cannot contain special characters (\:*?\"<>|), start or end with a period (.), or contain two or more consecutive slashes (/). The value can contain 0 to 1,023 characters.
    • With this parameter configured, the path must end with a slash (/). After a ZIP package is decompressed, the decompressed files are stored in the folder with the same name as the path.

      If there is no such a folder in the bucket, OBS automatically creates one for storing the files.

    • With this parameter left blank, decompressed objects are stored in the home directory of the bucket.

    IAM Agency

    Select an IAM agency of OBS, with the OBS OperateAccess permission assigned.

    If no such agency is available, create one.

  6. Click OK. The online decompression policy is created.

Using the API

Setting an Online Decompression Policy

Replicating Online Decompression Policies

You can use OBS Console to replicate online decompression policies.

  1. In the navigation pane of OBS Console, choose Object Storage.
  2. In the bucket list, click the bucket you want to operate. The Objects page is displayed.
  3. In the navigation pane, choose Data Processing > Online Decompression. The Online Decompression page is displayed.
  4. Click Replicate.
  5. Select a replication source, which is the bucket whose online decompression policies you want to replicate.

    • The online decompression policies replicated from a source bucket will not overwrite existing policies in the destination bucket, and any that conflict with the existing ones will not be replicated.
    • Both source and destination buckets must be of the 3.0 version.
    • You can remove the policies that you do not want to replicate.
    • There can be 10 online decompression policies at most in a bucket. If the number of policies you will replicate plus the number of existing policies in the destination bucket exceeds 10, the replication will fail. Before replicating the policies, delete some if necessary.
    Figure 2 Replicating online decompression policies

  6. Click OK to replicate the online decompression policies to the destination bucket.
Parent Topic: Data Processing