Help Center/ MapReduce Service/ Developer Guide (Normal_Earlier Than 3.x)/ HDFS Development Guide/ FAQs About HDFS Application Development/ HDFS Shell Commands
Updated on 2024-08-16 GMT+08:00
View PDF
Share

HDFS Shell Commands

HDFS Shell

You can use HDFS shell commands to perform operations on HDFS, such as reading and writing files.

The following describes how to run the HDFS shell commands.

  1. Run the following command to initialize environment variables:

    source /opt/client/bigdata_env

  2. If the Kerberos authentication is enabled for the current cluster, run the following command to authenticate the user. If the Kerberos authentication is disabled for the current cluster, skip this step. The current user is the development user added in Preparing an HDFS Application Development User.

    Human-machine user: kinit MRS cluster user

    For example, kinit hdfsuser.

    Machine-machine user: kinit -kt Authentication credential path MRS cluster user

    For example: kinit -kt /opt/user.keytab hdfsuser.

  3. Run the HDFS shell command.

    The format of the HDFS shell operation is as follows: hadoop fs <args>

    Enter the command. Examples:

    • Viewing the content in the directory: hadoop fs -ls /tmp/input/
    • Creating a directory: hadoop fs -mkdir /tmp/input/new_dir
    • Viewing file content: hadoop fs -cat /tmp/input/file1
    • Clearing files: hadoop fs -rm -r /tmp/input/file1
    • Querying help information about the HDFS commands: hadoop fs --help

    For details about the HDFS commands, visit the following website:

    https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/FileSystemShell.html

Table 1 Commands related to transparent encryption

Scenario

Operation

Command

Description

Hadoop shell command management key

Create a key.

hadoop key create <keyname> [-cipher <cipher>] [-size <size>] [-description <description>] [-attr <attribute=value>] [-provider <provider>] [-help]

The create subcommand creates a new key for the name specified by <keyname> in provider. The provider is specified by the -provider parameter. You can use the -cipher parameter to define a password. The default password is AES/CTR/NoPadding.

The default key length is 128. You can use the -size parameter to define a key length. Any attribute of the attribute=value type can be defined by the -attr parameter. -attr can be defined many times for each attribute.

Perform a rollback.

hadoop key roll <keyname> [-provider <provider>] [-help]

The roll subcommand creates a new version for the key specified in provider. The provider is specified by the -provider parameter.

Delete a key.

hadoop key delete <keyname> [-provider <provider>] [-f] [-help]

The delete subcommand deletes all versions of a key. The key is specified by the <keyname> parameter in provider, and the provider is specified by the -provider parameter. Unless -f is specified, the command needs to be confirmed by the user.

View a key.

hadoop key list [-provider <provider>] [-metadata] [-help]

The list subcommand displays all key names in provider. The provider is configured by the user in the core-site.xml or specified by the -provider parameter. The -metadata parameter displays metadata.