Updated on 2024-07-19 GMT+08:00

HDFS on Hue

Hue provides the file browser function for users to use HDFS in GUI mode.

The Hue page is used to view and analyze data such as files and tables. Do not perform high-risk management operations such as deleting objects on the page. If an operation is required, you are advised to perform the operation on each component after confirming that the operation has no impact on services. For example, you can use the HDFS client to perform operations on HDFS files and use the Hive client to perform operations on Hive tables.

How to Use File Browser

Access the Hue web UI.

Click . The File Browser page is displayed. You can perform the following operations:

  • Viewing files or directories

    By default, the directory and files in the directory of the login user are displayed. You can view Name, Size, User, Group,Permission, and Date.

    Click a file name to view the text information or binary data in the text file. The file content can be edited.

    If there are a large number of files and directories, you can enter keywords in the search text box to search for specific files or directories.

  • Creating files or directories

    Click New in the upper right corner. Choose File to create the file. Choose Directory to create a directory.

  • Managing files or directories

    Select the check box of a file or director, and click Actions. In the displayed menu, choose Rename, Move, Copy, and Change permissions to rename, move, copy, or change the file or directory permissions.

  • Uploading files

    Click Upload in the upper right corner and click Select files or drag the file to the window.

How to Use Storage Policies

If the value of Hue parameter fs_defaultFS is set to viewfs://ClusterX, the big data storage policy cannot be enabled.

Storage policies on the Hue web UI are classified into the following two types:

  • Static Storage Policies

    Current storage policy

    According to the access frequency and importance of documents in HDFS, specify a storage policy for an HDFS directory, such as ONE_SSD or ALL_SSD. The files in this directory can be migrated to the storage media.

  • Dynamic Storage Policies

    Set rules for an HDFS directory. The system can automatically change the storage policy, change the number of file copies, or move the file directory.

    Before configuring a dynamic storage policy on the Hue WebUI, you must set the CRON expressions for cold and hot data migration and start automatic cold and hot data migration on Manager.

    Operations:

    Modify the following NameNode parameters of HDFS. For details, see Modifying Cluster Service Configuration Parameters.

    Parameter

    Description

    Example Value

    dfs.auto.data.mover.enable

    Whether to enable automatic hot and cold data migration. The default value is false.

    true

    dfs.auto.data.mover.cron.expression

    CRON expression for hot and cold data migration in HDFS, which is used to control the start time of data migration. This parameter is available only when dfs.auto.data.mover.enable is set to true. The default value is 0 * * * *, indicating that the task is executed on the hour.

    0 * * * *

    Table 1 describes the expression for modifying the dfs.auto.data.mover.cron.expression parameter. * indicates consecutive time segments.

    Table 1 Parameters in the execution expression

    Column

    Description

    1

    Minute. The value ranges from 0 to 59.

    2

    Hour. The value ranges from 0 to 23.

    3

    Date. The value ranges from 1 to 31.

    4

    Month. The value ranges from 1 to 12.

    5

    Week. The value ranges from 0 to 6. 0 indicates Sunday.

To set storage policies on the web UI, perform the following operations:

  1. Log in to FusionInsight Manager.
  1. On FusionInsight Manager, choose System > Permission > Role > Create Role.

    1. Set Role Name.
    2. In the Configure Resource Permission area, choose Name of the desired cluster > Hue, select Storage Policy Admin, and click OK. Then, grant the permission to the role.

  2. Choose System > Permission > User Group > Create User Group. Set Group Name, and click Add next to Role. On the displayed page, select the created role, click OK to add the role to the group, and click OK.
  3. Choose System > Permission > User > Create.

    1. Username: Enter the name of the user to be added.
    2. Set User Type to Human-machine.
    3. Set Password and Confirm Password for logging in to the Hue web UI.
    4. Click Add next to User Group. On the page that is displayed, select the created user group in 3, supergroup, hadoop, and hive, and click OK.
    5. Set Primary Group to hive.
    6. Click Add next to Role. On the page that is displayed, select the created role in 2 and the System_administrator role, and click OK.
    7. Click OK. The user is added successfully.

  4. Access the Hue web UI as the created user.
  5. In the left navigation pane, click . The File Browser page is displayed.
  6. Select the check box of a directory and choose Action on the top of the page. Choose Storage policies.
  7. In the dialog box that is displayed, set a new storage policy and click OK.

    • On the Static Storage Policy page, you can set a static storage policy and click Save.
    • On the Dynamic Storage Policy page, you can create, delete, or modify a dynamic storage policy. Table 2 describes the parameters.
      Table 2 Parameters of the dynamic storage policy

      Category

      Parameter

      Description

      Rule

      Last Access to File

      Indicates the time when the file is last accessed.

      Last File Modification

      Indicates the time when the file is last modified.

      Operation

      Change Number of Copies

      Indicates the number of file copies.

      Modify Storage Policy

      Indicates that you can modify storage policies to the following: HOT, WARM, COLD, ONE_SSD, and ALL_SSD.

      Move to Directory

      Indicates that you can move the file to another directory.

      • You need to consider whether the rules conflict with each other and whether the rules damage the system when setting rules.
      • When a directory is configured with multiple rules and operations, the rule that is triggered first is located at the bottom of the rule/operation list, and the rules that are triggered later are placed from bottom to top to prevent repeated operations.
      • The system checks whether the files under the directory specified by the dynamic storage policy meet the rules on an hourly basis. If the files meet the rules, the execution is triggered. Execution logs are recorded in the /var/log/Bigdata/hdfs/nn/hadoop.log directory of the active NameNode.

Typical Scenarios

On the Hue page, view and edit HDFS files in text or binary mode as follows:

Viewing a File

  1. Access the Hue web UI.
  2. In the left navigation pane, click . The File Browser page is displayed.
  3. Click the name of the file to be viewed.
  4. Click View as binary to switch from the text mode to the binary mode. Click View as file to switch from the binary mode to the text mode.

Editing a file

  1. Click Edit File. The file content can be edited.
  2. Click Save or Save As to save the file.