Help Center/ MapReduce Service/ User Guide/ Configuring Storage-Compute Decoupling for an MRS Cluster/ Interconnecting an MRS Cluster with OBS Using an IAM Agency/ Configuring the Recycle Bin Directory and OBS Fragment Lifecycle for MRS Cluster Components
Updated on 2026-10-21 GMT+08:00
View PDF

Configuring the Recycle Bin Directory and OBS Fragment Lifecycle for MRS Cluster Components

Scenarios

In MRS 3.2.0-LTS.1 or later, components prevent mis-deletion by default. That is, file data deleted by component users is not directly deleted but stored in the recycle bin directory in the OBS file system. This function is compatible with the native garbage collection mechanism of Hadoop FS, providing additional data protection for OBS-based Hadoop big data systems.

This section describes how to set lifecycle policies for the recycle bin directory and fragments in the OBS file system to automatically clear data periodically.

A recycle bin directory is created for each required user. If a new user is added to the MRS cluster and has permissions to delete component data, you must configure a clearing policy for their recycle bin directory.

  • If your cluster uses decoupled storage and compute, set lifecycle rules for the relevant directories by following this section. Without them, you risk running out of storage space and paying higher storage fees.
  • OBS uses multipart upload. If a data write task fails or is interrupted, file fragments remain in OBS and generate storage charges. To avoid extra costs, set a lifecycle policy for the required directory.

You need to configure lifecycle policies for the recycle bin directories of preset users in the MRS cluster and the recycle bin directories of new users who need accidental deletion prevention. If a low privileged agency is used or only the permission for MRS users to access OBS file system directories is configured by referring to Configuring Fine-Grained OBS Access Permissions for MRS Cluster Users, you will need the operation permission for the recycle bin directory.

Table 1 Directories for which a lifecycle policy needs to be configured

Cluster Version

Directory Type

Component

Directory

How to Create

Versions earlier than MRS 3.3.0-LTS

Recycle bin directories that must be configured by default for each component in an MRS cluster

Hive
  • user/omm/.Trash
  • user/hive/.Trash

If the .Trash folder does not exist, create it on the cluster client as user omm.

Run the following command:

hdfs dfs -mkdir -p obs://Name of the OBS parallel file system where the table is stored/Folder path

Spark
  • user/omm/.Trash
  • user/root/.Trash
  • user/spark2x/.Trash

HetuEngine
  • user/omm/.Trash
  • user/hetuserver/.Trash

HBase
  • user/hbase/.Trash
  • user/omm/.Trash

Recycle bin directories of users who need accidental deletion prevention

Hive/Spark/HetuEngine

user/<New service user>/.Trash

MRS 3.3.0-LTS or later

Default recycle bin directories configured for each component in an MRS cluster

Hive/Spark/HetuEngine

/user/.Trash

For example, if a user with the following permissions has been added to the cluster, you need to create a recycle bin directory clearing rule for the user in the parallel file system:

  • Permissions to delete the HDFS files
  • DROP, INSERT OVERWRITE, and TRUNCATE permissions on Hive tables
  • DROP, TRUNCATE, DELETE, INSERT OVERWRITE, and LOAD OVERWRITE permissions on HetuEngine

The following table lists the MRS component commands that support data restoration after accidental deletion.

Table 2 Commands supporting data restoration after accidental deletion

Component

Commands Supporting Data Restoration After Accidental Deletion

Hive
  • Deleting a table: 
    drop table
  • Clearing table data: 
    truncate table
  • Deleting a table partition: 
    alter table drop partition
  • Overwriting a table or partition: 
    insert overwrite table [partition]
  • Overwriting a directory: 
    insert overwrite directory
  • Overwriting a table or partition: 
    load data overwrite

Spark
  • Deleting a table: 
    drop table
  • Clearing table data: 
    truncate table
  • Clearing a table partition: 
    truncate table table_name partition
  • Deleting a table partition: 
    alter table drop partition
  • Overwriting a table or partition: 
    insert overwrite table [partition]
  • Overwriting a directory: 
    insert overwrite directory
  • Overwriting a table or partition: 
    load data overwrite

HetuEngine
  • Deleting a table: 
    drop table
  • Clearing table data: 
    truncate table
    delete table
  • Deleting a table partition: 
    alter table drop partition
    delete table where [partition]
  • Overwriting a table or partition: 
    insert overwrite table [partition]
  • Overwriting a table or partition: 
    load data overwrite

Hudi
  • Deleting a table: 
    drop table
  • Deleting a table partition:
    • SQL: 
      alter table drop partition
    • API: 
      delete_partition
  • Deleting row-level data:
    • SQL: 
      delete from
    • API: 
      delete
  • Clearing table data: 
    truncate table
  • Overwriting a table:
    • SQL: 
      insert overwrite
      insert overwrite table
    • API: 
      insert_overwrite
      insert_overwrite_table

Configuring the Lifecycle Rule of an OBS Directory

  1. Log in to the OBS console.
  2. Click Parallel File Systems and click the name of the file system used by the current MRS cluster.
  3. In the navigation pane, choose Data Management > Lifecycle Rules. Click Create to create a lifecycle rule for a specified directory. For details about the parameters, see Configuring a Lifecycle Rule.

    Table 3 Parameters for creating a lifecycle rule

    Parameter

    Description

    Example Value

    Status

    Whether to enable the lifecycle rule.

    Enabled

    Rule Name

    Rule name that identifies different lifecycle configurations. You can customize the name.

    rule-test

    Prefix

    Prefix of the objects to which the lifecycle rule applies. Objects with the specified prefix are managed by the lifecycle rule. The prefix cannot start with a slash (/), contain consecutive slashes (/), or include the following special characters: \:*?"<>|. If this parameter is not specified, the rule will take effect for the entire file system.

    WARNING:

    To prevent other service data from being deleted by mistake, you are not advised to use the lifecycle rule configured for the entire file system or high-level directories.

    Generally, the recycle bin directory of MRS components is in the following format. If the folder does not exist, create it.

    user/<Username>/.Trash

    user/omm/.Trash

    Delete Files After (Days)

    The object within the rule configuration scope expires and is automatically deleted by OBS if the number of days since its last update reaches this parameter value. The recommended value is 1 to 7 days.

    2 days

  4. Click OK to complete the lifecycle rule configuration.

    You can click Edit in the Operation column of a lifecycle rule to edit it. You can also click Disable or Enable to disable or enable it.

  5. Repeat the preceding steps to create recycle bin directory clearing rules for all users who have the data deletion permission in the current MRS cluster one by one until all recycle bin directories in the OBS file system are configured.

Configuring a Lifecycle Rule for OBS Fragments

  1. Log in to the OBS console.
  2. Click Parallel File Systems and click the name of the file system used by the current MRS cluster.
  3. Choose Data Management > Lifecycle Rules. On the displayed page, click Create to create a lifecycle rule for OBS fragments.

    It is recommended that you add a lifecycle rule with Scope set to All files for OBS fragments. This rule should be separate from the lifecycle rule you configure for the component recycle bin directory.

    Table 4 Parameters for creating a lifecycle rule

    Category

    Parameter

    Description

    Example Value

    Basic information

    Status

    Whether to enable the lifecycle rule.

    Enabled

    Rule Name

    Rule name that identifies different lifecycle configurations. You can customize the name.

    rule-test1

    Scope

    Scope of objects the lifecycle rule applies to. Configure the scope based on your actual requirements. For example, set it to All files.

    All files

    Current Version

    Status

    Whether to enable the current version. Disable this option.

    Disabled

    Delete Fragments Upon Expiration

    Status

    Whether to enable the Delete Fragments Upon Expiration function. Enable this option.

    Enabled

    Days

    The number of days after which a fragment expires and is automatically deleted by OBS. Set this parameter as required. For example, set it to 7 days.

    7 days
  4. Select the confirmation information and click OK. The lifecycle rule is configured.

    To modify, disable, or enable a lifecycle rule, locate the rule and click Edit, Disable, or Enable in the Operation column.

Parent Topic: Interconnecting an MRS Cluster with OBS Using an IAM Agency