Managing HDFS Files Using Hue

Scenario

Hue a UI-based file browser function for you to use the HDFS.

The Hue UI is used to view and analyze data such as files and tables. Do not perform high-risk management operations such as deleting objects. If such an operation is required, perform the operation on each component after confirming that the operation does not affect services. For example, use the HDFS client to perform operations on HDFS files and use the Hive client to perform operations on Hive tables.

Accessing File Browser

1. Access the Hue web UI. For details, see Accessing the Hue Web UI.
2. In the left navigation pane, click . The File Browser page is displayed.

   By default, the homepage of File Browser is the home directory of the current login user. On the displayed page, the following information about subdirectories for files in the directory is displayed:

   Table 1 HDFS file attributes

   Attribute	Description
   Name	Name of a directory or file
   Size	File size
   User	Owner of a directory or file
   Group	Group of a directory or file
   Permission	Permission of a directory or file
   Date	Time when a directory or file is created

3. In the search box, enter a keyword. The system automatically searches directories or files in the current directory.
4. Clear the search criteria. The system displays all directories or files.

Creating a New File or Directory

1. On the File Browser page, click New.
2. Select an operation.
   • File: creates a file. Enter a file name and click Create.
   • Directory: creates a directory. Enter a directory name and click Create.

Uploading User Files

1. On the File Browser page, click Upload.
2. In the displayed dialog box for uploading files, click Select files or drag the file to the dialog box.

Managing Files or Directories

1. On the File Browser page, select one or more directories or files.
2. Click Actions. On the menu that is displayed, select an operation.
   • Rename: renames a directory or file.
   • Move: moves a file. In Move to, select a new directory and click Move.
   • Copy: copies the selected files or directories.
   • Change permissions: changes permission to access the selected directory or file.
     • You can grant the owner, the group, or other users with the Read, Write, and Execute permissions.
     • Sticky: indicates that only HDFS administrators, directory owners, and file owners can move files in the directory.
     • Recursive: indicates that permission is granted to subdirectories recursively.
   • Storage policies: indicates the policies for storing files or directories in HDFS.
   • Summary: indicates that the HDFS storage information about the selected file or directory can be viewed.

Storage Policy Definition and Usage

If the value of Hue parameter fs_defaultFS is set to viewfs://ClusterX, the big data storage policy cannot be enabled.

Storage policies on the Hue web UI are classified into the following two types:

• Static Storage Policies

  This is the currently used storage policy.

  According to the access frequency and importance of documents in HDFS, specify a storage policy for an HDFS directory, such as ONE_SSD or ALL_SSD. The files in this directory can be migrated to the storage media.

• Dynamic Storage Policies

  Set rules for an HDFS directory. The system can automatically change the storage policy, change the number of file copies, delete files, or move the file directory based on the latest access time and modification time of files. For details, see Configuring HDFS Cold and Hot Data Migration.

  Before configuring a dynamic storage policy on the Hue web UI, you must set the CRON expressions for cold and hot data migration and start automatic cold and hot data migration on Manager.

  Modify the following NameNode parameters of HDFS. For details, see Modifying Cluster Service Configuration Parameters.

  Parameter	Description	Example Value
  dfs.auto.data.mover.enable	Whether to enable automatic hot and cold data migration. The default value is false.	true
  dfs.auto.data.mover.cron.expression	CRON expression for hot and cold data migration in HDFS, which is used to control the start time of data migration. This parameter is available only when dfs.auto.data.mover.enable is set to true. The default value is 0 * * * *, indicating that the task is executed on the hour.	0 * * * *

  Table 2 describes the expression for modifying the dfs.auto.data.mover.cron.expression parameter. * indicates consecutive time segments.

  Table 2 Parameters in the execution expression

  Column	Description
  1	Minute. The value ranges from 0 to 59.
  2	Hour. The value ranges from 0 to 23.
  3	Date. The value ranges from 1 to 31.
  4	Month. The value ranges from 1 to 12.
  5	Week. The value ranges from 0 to 6. 0 indicates Sunday.

To set storage policies on the web UI, perform the following operations:

1. Log in to FusionInsight Manager.
2. On FusionInsight Manager, choose System > Permission > Manage Role > Create Role.
   1. Set Role Name.
   2. In the Configure Resource Permission area, choose Name of the desired cluster > Hue, select Storage Policy Admin, and click OK. Then, grant the permission to the role.

3. Choose System > Permission > User Group > Create User Group. Set Group Name and click Select and Add Role next to Role. On the displayed page, select the role created in 2 and click OK to add the role to the group.
4. Choose System > Permission > User > Create.
   1. Username: Enter the name of the user to be added.
   2. Set User Type to Human-machine.
   3. Set Password and Confirm Password for logging in to the Hue web UI.
   4. Click Add next to User Group. On the page that is displayed, select the user group created in 3, supergroup, hadoop, and hive, and click OK.
   5. Set Primary Group to hive.
   6. Click Add on the right of Role. On the page that is displayed, select the role created in 2 and System_administrator role, and click OK.
   7. Click OK. The user is added successfully.

5. Access the Hue web UI as the created user. For details, see Accessing the Hue Web UI.
6. In the left navigation tree, click . The File Browser page is displayed.
7. Select the check box of the directory and click Actions on the top of the page. Choose Storage policies.
   Figure 1 Storage policies
   

8. In the dialog box that is displayed, set a new storage policy and click OK.
   • On the Static Storage Policy page, you can set a static storage policy and click Save.
   • On the Dynamic Storage Policy page, you can create, delete, or modify a dynamic storage policy. Table 3 describes the parameters.

     Table 3 Parameters of the dynamic storage policy

     Category	Parameter	Description
     Rule	Last Access to File	Indicates the time when the file is last accessed.
     	Last File Modification	Indicates the time when the file is last modified.
     Operation	Change Number of Copies	Indicates the number of file copies.
     	Modify Storage Policy	Indicates that you can modify storage policies to the following: HOT, WARM, COLD, ONE_SSD, and ALL_SSD.
     	Move to Directory	Indicates that you can move the file to another directory.

     

     • You need to consider whether the rules conflict with each other and whether the rules damage the system when setting rules.
     • When a directory is configured with multiple rules and operations, the rule that is triggered first is located at the bottom of the rule/operation list, and the rules that are triggered later are placed from bottom to top to prevent repeated operations.
     • The system checks whether the files under the directory specified by the dynamic storage policy meet the rules on an hourly basis. If the files meet the rules, the execution is triggered. Execution logs are recorded in the /var/log/Bigdata/hdfs/nn/hadoop.log directory of the active NameNode.

Typical Scenario

On the Hue page, view and edit HDFS files in text or binary mode as follows:

Viewing a File

1. Access the Hue web UI.
2. In the navigation pane on the left, click . The File Browser page is displayed.
3. Click the name of the file you want to view.
4. Click View as binary to switch from the text mode to the binary mode. Click View as file to switch from the binary mode to the text mode.

Editing a file

5. Click Edit File. The file content can be edited.
6. Click Save or Save As to save the file.