Configuring HDFS Cold and Hot Data Migration

Scenario

The hot and cold data migration tool migrates HDFS files based on the configured policy. A policy is a set of conditional or non-conditional rules. If a file matches the rule set, the tool performs a group of operations for the file.

The hot and cold data migration tool supports the following rules and operations:

• Migration rules:
  • Data is migrated based on the latest access time of the file.
  • Data is migrated based on the file modification time.
  • Data is migrated without conditions.

  Table 1 Rule condition tags

  Condition Tag	Description
  <age operator="lt">	Defines the conditions for changing the age or modification time.
  <atime operator="gt">	Defines the condition for accessing time.

  

  For a manual migration rule, no condition is required.

• Operations:
  • Set the storage policy to a given data tier.
  • Migrate files to another folder.
  • Configure the number of copies for a file.
  • Delete a file.
  • Set a node label.

  Table 2 Behavior types:

  Behavior Type	Description	Required Parameters
  MARK	Determines the data access frequency and set a data storage policy.	<param> <name>targettier</name> <value>STORAGE_POLICY</value> <param>
  MOVE	Sets the data storage policy or NodeLabel and invokes the HDFS Mover tool.	<param> <name>targettier</name> <value>STORAGE_POLICY</value> <param> <param> <name>targetnodelabels</name> <value>SOME_EXPRESSION</value> <param> NOTE: You can set either or both of the parameters.
  SET_REPL	Configures the number of copies for a file.	<param> <name>replcount</name> <value>INTEGER</value> <param>
  MOVE_TO_FOLDER	Moves the file to the target folder. If overwrite is set to true, the target path will be overwritten.	<param> <name>target</name> <value>PATH</value> <param> <param> <name>overwrite</name> <value>true/false</value> <param> NOTE: overwrite is an optional parameter. If this parameter is not set, the default value false is used.
  DELETE	Delete a file.	N/A

Configuration Description

You must periodically invoke the migration tool and perform the following operations in the hdfs-site.xml file on the client:

DefaultDataMovementPolicy has the configuration file default-datamovement-policy.xml. Users need to define all rules based on the age or access time and operations performed in this file. This file must be stored in classpath of the client.

The following is an example of the default-datamovement-policy.xml file:

<policies>
  <policy>
    <fileset>
      <file>
        <name>/opt/data/1.txt</name>
      </file>
      <file>
        <name>/opt/data/*/subpath/</name>
        <excludes>
          <name>/opt/data/some/subpath/sub1</name>
        </excludes>
      </file>
    </fileset>
    <rules>
      <rule>
        <age>2w</age>
        <action>
          <type>MOVE</type>
          <params>
            <param>
              <name>targettier</name>
              <value>HOT</value>
            </param>
          </params>
        </action>
      </rule>
    </rules>
  </policy>
</policies>

Other attributes can be added to the tags used in policies, rules, and behavior operations. For example, name can be used to manage the mapping between the user UI (for example, Hue UI) and tool input XML.

Example: <policy name="Manage_File1">

The tags are described as follows:

For files or folders under the <file> tag, the FileSystem#globStatus API is used. For other files or folders, the GlobPattern class (used by GlobFilter) is used. For details, see the description of supported APIs. For example, for globStatus, /opt/hadoop/* will match everything in the /opt/hadoop folder. /opt/*/hadoop matches all hadoop folders in the subdirectories of the/opt directory.

For globStatus, the glob mode of each path component is matched. For other components, the glob mode is directly matched.

https://hadoop.apache.org/docs/r3.1.1/api/org/apache/hadoop/fs/FileSystem.html#globStatus(org.apache.hadoop.fs.Path)

Behavior Operation Example

• MARK

  <action>
    <type>MARK</type>
    <params>
      <param>
        <name>targettier</name>
        <value>HOT</value>
      </param>
    </params>
  </action>

• MOVE

  <action>
    <type>MOVE</type>
    <params>
      <param>
        <name>targettier</name>
        <value>HOT</value>
      </param>
      <param>
        <name>targetnodelabels</name>
        <value>SOME_EXPRESSION</value>
      </param>
    </params>
  </action>

• SET_REPL

  <action>
    <type>SET_REPL</type>
    <params>
      <param>
        <name>replcount</name>
        <value>5</value>
      </param>
    </params>
  </action>

• MOVE_TO_FOLDER

  <action>
    <type>MOVE_TO_FOLDER</type>
    <params>
      <param>
        <name>target</name>
        <value>path</value>
      </param>
      <param>
        <name>overwrite</name>
        <value>true</value>
      </param>
  
    </params>
  </action>

  

  The MOVE_TO_FOLDER operation only changes the file path to the target folder and does not change the block location. If you want to move a block, you need to configure an independent move policy.

• DELETE

  <action>
    <type>DELETE</type>
  </action>

• When writing an XML file, pay attention to the configuration and sequence of behavior operations. The hot and cold data migration tool executes the rules in the sequence specified in the input XML file.
• If you want to run only one rule based on atime/age, sort the rules in descending order of time and set the idempotent attribute to false.
• If the delete operation is configured for a file set, other rules cannot be configured after the delete operation is performed.
• The -fs option can be used to specify the default file system address of the client.

Audit Logs

The cold and hot data migration tool supports audit logs of the following operations:

• Tool startup status
• Behavior type, parameter details, and status
• Tool completion status

To enable the audit log tool, add the following attributes to the <HADOOP_CONF_DIR>/log4j.property file:

autodatatool.logger=INFO, ADMTRFA
autodatatool.log.file=HDFSAutoDataMovementTool.audit
log4j.logger.com.huawei.hadoop.hdfs.datamovement.HDFSAutoDataMovementTool.audit=${autodatatool.logger}
log4j.additivity.com.huawei.hadoop.hdfs.datamovement.HDFSAutoDataMovementTool-audit=false
log4j.appender.ADMTRFA=org.apache.log4j.RollingFileAppender
log4j.appender.ADMTRFA.File=${hadoop.log.dir}/${autodatatool.log.file}
log4j.appender.ADMTRFA.layout=org.apache.log4j.PatternLayout
log4j.appender.ADMTRFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
log4j.appender.ADMTRFA.MaxBackupIndex=10
log4j.appender.ADMTRFA.MaxFileSize=64MB

For details, see the <HADOOP_CONF_DIR>/log4j_autodata_movment_template.properties file.