Setting HDFS Storage Policies

Function Description

Specify storage policies for a file or folder in HDFS.

Sample Code

1. Set the following parameters in the Hdfs-site.xml file in ${HADOOP_HOME}/etc/hadoop/.

   <name>dfs.storage.policy.enabled</name>
   <value>true</value>

2. Restart HDFS, as shown in Figure 1.
   Figure 1 Restarting HDFS
   
3. Log in to MRS Manager. Choose Service > HDFS > Service Configuration, and set Type to All.
4. Check whether the value of dfs.storage.policy.enabled is true. If it is not, modify the value to true, click Save Configuration, and restart HDFS.
5. View the code.

   The following provides code snippets. For complete codes, see the HdfsMain class in com.huawei.bigdata.hdfs.examples.

      /** 
      * Set storage policies.
      * @param policyName 
      * The policy name can be accepted.
      * <li>HOT 
      * <li>WARM 
      * <li>COLD 
      * <li>LAZY_PERSIST 
      * <li>ALL_SSD 
      * <li>ONE_SSD 
      * @throws IOException  
      */ 
     private void setStoragePolicy(String policyName) throws IOException{
       if(fSystem instanceof DistributedFileSystem) {
         DistributedFileSystem dfs = (DistributedFileSystem) fSystem;
         Path destPath = new Path(DEST_PATH);
         Boolean flag = false;
         mkdir();
         BlockStoragePolicySpi[] storage = dfs.getStoragePolicies();
         for (BlockStoragePolicySpi bs : storage) {
           if (bs.getName().equals(policyName)) {
             flag = true;
           }
           System.out.println("StoragePolicy:" + bs.getName());
         }
         if (!flag) {
           policyName = storage[0].getName();
         }
         dfs.setStoragePolicy(destPath, policyName);
         System.out.println("success to set Storage Policy path " + DEST_PATH);
         rmdir();
       }
       else{
         System.out.println("Storage Policy is only supported in HDFS.");
       }
     }