Updated on 2024-08-16 GMT+08:00

HDFS HTTP REST APIs

Function Description

In the REST application development sample code, file operations include creating, reading, writing, appending and deleting files. For details about related APIs, visit http://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/WebHDFS.html.

Preparing an Operating Environment

  1. Install a client. Install the client on the node, for example, to the /opt/client directory. For details about how to install the client, see Client Management in the MapReduce Service User Guide.

    1. You have installed HDFS on the server and confirmed that HDFS is running properly.
    2. The JDK 1.7 or 1.8 has been installed on the client.
    3. Obtain the MRS_HDFS_Client.tar installation package, and run the following commands to decompress the package:

      tar -xvf MRS_HDFS_Client.tar

      tar -xvf MRS_HDFS_ClientConfig.tar

      You are advised to install a client of the same version as the cluster on the server to avoid version incompatibility.

    4. Go to the MRS_HDFS_ClientConfig decompressed folder and run the following command to install the client:

      sh install.sh /opt/client

      In the preceding command, /opt/client is an example user-defined path.

    5. Go to the /opt/client client installation directory and run the following command to initialize the environment variables:

      source bigdata_env

    6. Run the following command to perform user authentication. The following uses user hdfs as an example. You can change the username based on the site requirements (skip this step for a normal cluster).
      kinit hdfs

      The validity period of once kinit is 24 hours. Run the kinit command again when you run the sample application 24 hours later.

    7. Run the following commands to prepare the testFile and testFileAppend files in the client directory. Their contents are "Hello, webhdfs user!" and "Welcome back to webhdfs!", respectively.

      touch testFile

      vi testFile

      Write "Hello, webhdfs user!". Save the file and exit.

      touch testFileAppend

      vi testFileAppend

      Write "Welcome back to webhdfs!". Save the file and exit.

  2. MRS clusters support only HTTPS-based access by default. If the HTTPS service is used, go to 3. If the HTTP service is used (supported only by security clusters), go to 4.
  3. HTTPS-based access is different from HTTP-based access. When you access HDFS using HTTPS, you must ensure that the SSL protocol supported by the curl command is supported by the cluster because SSL security encryption is used. If the cluster does not support the SSL protocol, change the SSL protocol in the cluster. For example, if the curl supports only the TLSv1 protocol, perform the following steps:

    Log in to MRS Manager. Choose Service > HDFS > Service Configuration, and set Type to All. Search for hadoop.ssl.enabled.protocols in the search box, and check whether the parameter value contains TLSv1. If it does not contain TLSv1, add TLSv1 to the hadoop.ssl.enabled.protocols configuration item. Clear the value of ssl.server.exclude.cipher.list. Otherwise, HDFS cannot be accessed using HTTPS. Click Save Configuration and select Restart the affected services or instances. Click Yes and restart the HDFS service.

    TLSv1 has security vulnerabilities. Exercise caution when using it.

  4. Log in to MRS Manager. Choose Service > HDFS > Service Configuration, and set Type to All. Search for dfs.http.policy in the search box, and select HTTP_AND_HTTPS. Click Save Configuration and select Restart the affected services or instances. Click Yes and restart the HDFS service.

Procedure

  1. Log in to MRS Manager, and click Services. Select HDFS and click it to access the HDFS service status page.

    Because webhdfs is accessed through HTTP and HTTPs, you need to obtain the IP address and HTTP and HTTPs ports of the active NameNode.

    1. Click Instance to access the page displayed in Figure 1. Find the host name and IP address of NameNode(hacluster,active).
      Figure 1 HDFS instance
    2. Click Service Configuration to access the page displayed in Figure 2. Find namenode.http.port (9870) and namenode.https.port (9871).

      For versions earlier than MRS 1.9.2, the default values of the preceding ports are 25002 and 25003. For details, see the related port information in MapReduce Service User Guide.

      Figure 2 HDFS service configuration

  2. Create a directory by referring to the following link.

    http://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Make_a_Directory

    Click the link. The page is displayed, as shown in Figure 3.

    Figure 3 Example command for creating a directory

    Go to the installation directory of the client, for example, /opt/client, and create a directory named huawei.

    1. Run the following command to check whether the directory named huawei exists:

      hdfs dfs -ls /

      The command output is as follows:

      linux1:/opt/client # hdfs dfs -ls /
      16/04/22 16:10:02 INFO hdfs.PeerCache: SocketCache disabled.
      Found 7 items
      -rw-r--r--   3 hdfs   supergroup          0 2016-04-20 18:03 /PRE_CREATE_DIR.SUCCESS
      drwxr-x---   - flume  hadoop              0 2016-04-20 18:02 /flume
      drwx------   - hbase  hadoop              0 2016-04-22 15:19 /hbase
      drwxrwxrwx   - mapred hadoop              0 2016-04-20 18:02 /mr-history
      drwxrwxrwx   - spark  supergroup          0 2016-04-22 15:19 /sparkJobHistory
      drwxrwxrwx   - hdfs   hadoop              0 2016-04-22 14:51 /tmp
      drwxrwxrwx   - hdfs   hadoop              0 2016-04-22 14:50 /user

      The huawei directory does not exist in the current path.

    2. Run the command in Figure 3 to create a directory named huawei. Replace <HOST> and <PORT> in the command with the host name or IP address and port number obtained in 1, and enter the huawei directory to be created in <PATH>.

      Replace <HOST> with the host name or IP address. Note that the HTTP and HTTPS ports are different.

      • Run the following command to access HTTP:
        linux1:/opt/client # curl -i -X PUT --negotiate -u: "http://linux1:9870/webhdfs/v1/huawei?op=MKDIRS"

        In the preceding information, linux1 indicates <HOST> and 9870 indicates <PORT>.

      • Command output
        HTTP/1.1 401 Authentication required
        Date: Thu, 05 May 2016 03:10:09 GMT
        Pragma: no-cache
        Date: Thu, 05 May 2016 03:10:09 GMT
        Pragma: no-cache
        X-Frame-Options: SAMEORIGIN
        WWW-Authenticate: Negotiate
        Set-Cookie: hadoop.auth=; Path=/; Expires=Thu, 01-Jan-1970 00:00:00 GMT; HttpOnly
        Content-Length: 0
        HTTP/1.1 200 OK
        Cache-Control: no-cache
        Expires: Thu, 05 May 2016 03:10:09 GMT
        Date: Thu, 05 May 2016 03:10:09 GMT
        Pragma: no-cache
        Expires: Thu, 05 May 2016 03:10:09 GMT
        Date: Thu, 05 May 2016 03:10:09 GMT
        Pragma: no-cache
        Content-Type: application/json
        X-Frame-Options: SAMEORIGIN
        WWW-Authenticate: Negotiate YGoGCSqGSIb3EgECAgIAb1swWaADAgEFoQMCAQ+iTTBLoAMCARKiRARCArhuv39Ttp6lhBlG3B0JAmFjv9weLp+SGFI+t2HSEHN6p4UVWKKy/kd9dKEgNMlyDu/o7ytzs0cqMxNsI69WbN5H
        Set-Cookie: hadoop.auth="u=hdfs&p=hdfs@HADOOP.COM&t=kerberos&e=1462453809395&s=wiRF4rdTWpm3tDST+a/Sy0lwgA4="; Path=/; Expires=Thu, 05-May-2016 13:10:09 GMT; HttpOnly
        Transfer-Encoding: chunked
        {"boolean":true}linux1:/opt/client # 

        The return value {"boolean" :true} indicates that the huawei directory is successfully created.

      • Run the following command to access HTTPS:
        linux1:/opt/client # curl -i -k -X PUT --negotiate -u: "https://linux1:9871/webhdfs/v1/huawei?op=MKDIRS"

        In the preceding information, linux1 indicates <HOST> and 9871 indicates <PORT>.

      • Command output
        HTTP/1.1 401 Authentication required
        Date: Fri, 22 Apr 2016 08:13:37 GMT
        Pragma: no-cache
        Date: Fri, 22 Apr 2016 08:13:37 GMT
        Pragma: no-cache
        X-Frame-Options: SAMEORIGIN
        WWW-Authenticate: Negotiate
        Set-Cookie: hadoop.auth=; Path=/; Expires=Thu, 01-Jan-1970 00:00:00 GMT; Secure; HttpOnly
        Content-Length: 0
        HTTP/1.1 200 OK
        Cache-Control: no-cache
        Expires: Fri, 22 Apr 2016 08:13:37 GMT
        Date: Fri, 22 Apr 2016 08:13:37 GMT
        Pragma: no-cache
        Expires: Fri, 22 Apr 2016 08:13:37 GMT
        Date: Fri, 22 Apr 2016 08:13:37 GMT
        Pragma: no-cache
        Content-Type: application/json
        X-Frame-Options: SAMEORIGIN
        WWW-Authenticate: Negotiate YGoGCSqGSIb3EgECAgIAb1swWaADAgEFoQMCAQ+iTTBLoAMCARKiRARCugB+yT3Y+z8YCRMYJHXF84o1cyCfJq157+NZN1gu7D7yhMULnjr+7BuUdEcZKewFR7uD+DRiMY3akg3OgU45xQ9R
        Set-Cookie: hadoop.auth="u=hdfs&p=hdfs@HADOOP.COM&t=kerberos&e=1461348817963&s=sh57G7iVccX/Aknoz410yJPTLHg="; Path=/; Expires=Fri, 22-Apr-2016 18:13:37 GMT; Secure; HttpOnly
        Transfer-Encoding: chunked
        
        {"boolean":true}linux1:/opt/client # 

        The return value {"boolean" :true} indicates that the huawei directory is successfully created.

    3. Run the following command to check the huawei directory in the path.
      linux1:/opt/client # hdfs dfs -ls /
      16/04/22 16:14:25 INFO hdfs.PeerCache: SocketCache disabled.
      Found 8 items
      -rw-r--r--   3 hdfs   supergroup          0 2016-04-20 18:03 /PRE_CREATE_DIR.SUCCESS
      drwxr-x---   - flume  hadoop              0 2016-04-20 18:02 /flume
      drwx------   - hbase  hadoop              0 2016-04-22 15:19 /hbase
      drwxr-xr-x   - hdfs  supergroup          0 2016-04-22 16:13 /huawei
      drwxrwxrwx   - mapred hadoop              0 2016-04-20 18:02 /mr-history
      drwxrwxrwx   - spark  supergroup          0 2016-04-22 16:12 /sparkJobHistory
      drwxrwxrwx   - hdfs   hadoop              0 2016-04-22 14:51 /tmp
      drwxrwxrwx   - hdfs   hadoop              0 2016-04-22 16:10 /user

  3. Create an upload request to obtain the location where the DataNode address is written in.

    • Run the following command to access HTTP:
      linux1:/opt/client # curl -i -X PUT --negotiate -u: "http://linux1:9870/webhdfs/v1/huawei/testHdfs?op=CREATE"

      In the preceding information, linux1 indicates <HOST> and 9870 indicates <PORT>.

    • Command output
      HTTP/1.1 401 Authentication required
      Date: Thu, 05 May 2016 06:09:48 GMT
      Pragma: no-cache
      Date: Thu, 05 May 2016 06:09:48 GMT
      Pragma: no-cache
      X-Frame-Options: SAMEORIGIN
      WWW-Authenticate: Negotiate
      Set-Cookie: hadoop.auth=; Path=/; Expires=Thu, 01-Jan-1970 00:00:00 GMT; HttpOnly
      Content-Length: 0
      
      HTTP/1.1 307 TEMPORARY_REDIRECT
      Cache-Control: no-cache
      Expires: Thu, 05 May 2016 06:09:48 GMT
      Date: Thu, 05 May 2016 06:09:48 GMT
      Pragma: no-cache
      Expires: Thu, 05 May 2016 06:09:48 GMT
      Date: Thu, 05 May 2016 06:09:48 GMT
      Pragma: no-cache
      Content-Type: application/octet-stream
      X-Frame-Options: SAMEORIGIN
      WWW-Authenticate: Negotiate YGoGCSqGSIb3EgECAgIAb1swWaADAgEFoQMCAQ+iTTBLoAMCARKiRARCzQ6w+9pNzWCTJEdoU3z9xKEyg1JQNka0nYaB9TndvrL5S0neAoK2usnictTFnqIincAjwB6SnTtht8Q16WDlHJX/
      Set-Cookie: hadoop.auth="u=hdfs&p=hdfs@HADOOP.COM&t=kerberos&e=1462464588403&s=qry87vAyYzSn9VsS6Rm6vKLhKeU="; Path=/; Expires=Thu, 05-May-2016 16:09:48 GMT; HttpOnly
      Location: http://linux1:25010/webhdfs/v1/huawei/testHdfs?op=CREATE&delegation=HgAFYWRtaW4FYWRtaW4AigFUf4lZdIoBVKOV3XQOCBSyXvFAp92alcRs4j-KNulnN6wUoBJXRUJIREZTIGRlbGVnYXRpb24UMTAuMTIwLjE3Mi4xMDk6MjUwMDA&namenoderpcaddress=hacluster&overwrite=false
      Content-Length: 0
    • Run the following command to access HTTPS:
      linux1:/opt/client # curl -i -k -X PUT --negotiate -u: "https://linux1:9871/webhdfs/v1/huawei/testHdfs?op=CREATE"

      In the preceding information, linux1 indicates <HOST> and 9871 indicates <PORT>.

    • Command output
      HTTP/1.1 401 Authentication required
      Date: Thu, 05 May 2016 03:46:18 GMT
      Pragma: no-cache
      Date: Thu, 05 May 2016 03:46:18 GMT
      Pragma: no-cache
      X-Frame-Options: SAMEORIGIN
      WWW-Authenticate: Negotiate
      Set-Cookie: hadoop.auth=; Path=/; Expires=Thu, 01-Jan-1970 00:00:00 GMT; Secure; HttpOnly
      Content-Length: 0
      
      HTTP/1.1 307 TEMPORARY_REDIRECT
      Cache-Control: no-cache
      Expires: Thu, 05 May 2016 03:46:18 GMT
      Date: Thu, 05 May 2016 03:46:18 GMT
      Pragma: no-cache
      Expires: Thu, 05 May 2016 03:46:18 GMT
      Date: Thu, 05 May 2016 03:46:18 GMT
      Pragma: no-cache
      Content-Type: application/octet-stream
      X-Frame-Options: SAMEORIGIN
      WWW-Authenticate: Negotiate YGoGCSqGSIb3EgECAgIAb1swWaADAgEFoQMCAQ+iTTBLoAMCARKiRARCZMYR8GGUkn7pPZaoOYZD5HxzLTRZ71angUHKubW2wC/18m9/OOZstGQ6M1wH2pGriipuCNsKIfwP93eO2Co0fQF3
      Set-Cookie: hadoop.auth="u=hdfs&p=hdfs@HADOOP.COM&t=kerberos&e=1462455978166&s=F4rXUwEevHZze3PR8TxkzcV7RQQ="; Path=/; Expires=Thu, 05-May-2016 13:46:18 GMT; Secure; HttpOnly
      Location: https://linux1:9865/webhdfs/v1/huawei/testHdfs?op=CREATE&delegation=HgAFYWRtaW4FYWRtaW4AigFUfwX3t4oBVKMSe7cCCBSFJTi9j7X64QwnSz59TGFPKFf7GhNTV0VCSERGUyBkZWxlZ2F0aW9uFDEwLjEyMC4xNzIuMTA5OjI1MDAw&namenoderpcaddress=hacluster&overwrite=false
      Content-Length: 0

  4. Based on the obtained location information, you can create the /huawei/testHdfs file in HDFS and upload the content in the local testFile file to the testHdfs file.

    • Run the following command to access HTTP:
      linux1:/opt/client # curl -i -X PUT -T testFile --negotiate -u: "http://linux1:9864/webhdfs/v1/huawei/testHdfs?op=CREATE&delegation=HgAFYWRtaW4FYWRtaW4AigFUf4lZdIoBVKOV3XQOCBSyXvFAp92alcRs4j-KNulnN6wUoBJXRUJIREZTIGRlbGVnYXRpb24UMTAuMTIwLjE3Mi4xMDk6MjUwMDA&namenoderpcaddress=hacluster&overwrite=false"

      In the preceding information, linux1 indicates <HOST> and 9864 indicates <PORT>.

    • Command output
      HTTP/1.1 100 Continue
      HTTP/1.1 201 Created
      Location: hdfs://hacluster/huawei/testHdfs
      Content-Length: 0
      Connection: close
    • Run the following command to access HTTPS:
      linux1:/opt/client # curl -i -k -X PUT -T testFile --negotiate -u: "https://linux1:9865/webhdfs/v1/huawei/testHdfs?op=CREATE&delegation=HgAFYWRtaW4FYWRtaW4AigFUfwX3t4oBVKMSe7cCCBSFJTi9j7X64QwnSz59TGFPKFf7GhNTV0VCSERGUyBkZWxlZ2F0aW9uFDEwLjEyMC4xNzIuMTA5OjI1MDAw&namenoderpcaddress=hacluster&overwrite=false"
    • Command output
      HTTP/1.1 100 Continue
      HTTP/1.1 201 Created
      Location: hdfs://hacluster/huawei/testHdfs
      Content-Length: 0
      Connection: close

  5. Open the /huawei/testHdfs file and read content in the file.

    • Run the following command to access HTTP:
      linux1:/opt/client # curl -L --negotiate -u: "http://linux1:9870/webhdfs/v1/huawei/testHdfs?op=OPEN"

      In the preceding information, linux1 indicates <HOST> and 9870 indicates <PORT>.

    • Command output
      Hello, webhdfs user!
    • Run the following command to access HTTPS:
      linux1:/opt/client # curl -k -L --negotiate -u: "https://linux1:9871/webhdfs/v1/huawei/testHdfs?op=OPEN"

      In the preceding information, linux1 indicates <HOST> and 9871 indicates <PORT>.

    • Command output
      Hello, webhdfs user!

  6. Create a request to append a file to obtain the location where the DataNode address of /huawei/testHdfs file is written in.

    • Run the following command to access HTTP:
      linux1:/opt/client # curl -i -X POST --negotiate -u: "http://linux1:9870/webhdfs/v1/huawei/testHdfs?op=APPEND"

      In the preceding information, linux1 indicates <HOST> and 9870 indicates <PORT>.

    • Command output
      HTTP/1.1 401 Authentication required
      Cache-Control: must-revalidate,no-cache,no-store
      Date: Thu, 05 May 2016 05:35:02 GMT
      Pragma: no-cache
      Date: Thu, 05 May 2016 05:35:02 GMT
      Pragma: no-cache
      Content-Type: text/html; charset=iso-8859-1
      X-Frame-Options: SAMEORIGIN
      WWW-Authenticate: Negotiate
      Set-Cookie: hadoop.auth=; Path=/; Expires=Thu, 01-Jan-1970 00:00:00 GMT; HttpOnly
      Content-Length: 1349
      
      HTTP/1.1 307 TEMPORARY_REDIRECT
      Cache-Control: no-cache
      Expires: Thu, 05 May 2016 05:35:02 GMT
      Date: Thu, 05 May 2016 05:35:02 GMT
      Pragma: no-cache
      Expires: Thu, 05 May 2016 05:35:02 GMT
      Date: Thu, 05 May 2016 05:35:02 GMT
      Pragma: no-cache
      Content-Type: application/octet-stream
      X-Frame-Options: SAMEORIGIN
      WWW-Authenticate: Negotiate YGoGCSqGSIb3EgECAgIAb1swWaADAgEFoQMCAQ+iTTBLoAMCARKiRARCTYvNX/2JMXhzsVPTw3Sluox6s/gEroHH980xMBkkYlCnO3W+0fM32c4/F98U5bl5dzgoolQoBvqq/EYXivvR12WX
      Set-Cookie: hadoop.auth="u=hdfs&p=hdfs@HADOOP.COM&t=kerberos&e=1462462502626&s=et1okVIOd7DWJ/LdhzNeS2wQEEY="; Path=/; Expires=Thu, 05-May-2016 15:35:02 GMT; HttpOnly
      Location: http://linux1:9864/webhdfs/v1/huawei/testHdfs?op=APPEND&delegation=HgAFYWRtaW4FYWRtaW4AigFUf2mGHooBVKN2Ch4KCBRzjM3jwSMlAowXb4dhqfKB5rT-8hJXRUJIREZTIGRlbGVnYXRpb24UMTAuMTIwLjE3Mi4xMDk6MjUwMDA&namenoderpcaddress=hacluster
      Content-Length: 0
    • Run the following command to access HTTPS:
      linux1:/opt/client # curl -i -k -X POST --negotiate -u: "https://linux1:9871/webhdfs/v1/huawei/testHdfs?op=APPEND"

      In the preceding information, linux1 indicates <HOST> and 9871 indicates <PORT>.

    • Command output
      HTTP/1.1 401 Authentication required
      Cache-Control: must-revalidate,no-cache,no-store
      Date: Thu, 05 May 2016 05:20:41 GMT
      Pragma: no-cache
      Date: Thu, 05 May 2016 05:20:41 GMT
      Pragma: no-cache
      Content-Type: text/html; charset=iso-8859-1
      X-Frame-Options: SAMEORIGIN
      WWW-Authenticate: Negotiate
      Set-Cookie: hadoop.auth=; Path=/; Expires=Thu, 01-Jan-1970 00:00:00 GMT; Secure; HttpOnly
      Content-Length: 1349
      
      HTTP/1.1 307 TEMPORARY_REDIRECT
      Cache-Control: no-cache
      Expires: Thu, 05 May 2016 05:20:41 GMT
      Date: Thu, 05 May 2016 05:20:41 GMT
      Pragma: no-cache
      Expires: Thu, 05 May 2016 05:20:41 GMT
      Date: Thu, 05 May 2016 05:20:41 GMT
      Pragma: no-cache
      Content-Type: application/octet-stream
      X-Frame-Options: SAMEORIGIN
      WWW-Authenticate: Negotiate YGoGCSqGSIb3EgECAgIAb1swWaADAgEFoQMCAQ+iTTBLoAMCARKiRARCXgdjZuoxLHGtM1oyrPcXk95/Y869eMfXIQV5UdEwBZ0iQiYaOdf5+Vk7a7FezhmzCABOWYXPxEQPNugbZ/yD5VLT
      Set-Cookie: hadoop.auth="u=hdfs&p=hdfs@HADOOP.COM&t=kerberos&e=1462461641713&s=tGwwOH9scmnNtxPjlnu28SFtex0="; Path=/; Expires=Thu, 05-May-2016 15:20:41 GMT; Secure; HttpOnly
      Location: https://linux1:9865/webhdfs/v1/huawei/testHdfs?op=APPEND&delegation=HgAFYWRtaW4FYWRtaW4AigFUf1xi_4oBVKNo5v8HCBSE3Fg0f_EwtFKKlODKQSM2t32CjhNTV0VCSERGUyBkZWxlZ2F0aW9uFDEwLjEyMC4xNzIuMTA5OjI1MDAw&namenoderpcaddress=hacluster

  7. Based on the obtained location information, you can append content in the testFileAppend file to the /huawei/testHdfs file in HDFS.

    • Run the following command to access HTTP:
      linux1:/opt/client # curl -i -X POST -T testFileAppend --negotiate -u: "http://linux1:9864/webhdfs/v1/huawei/testHdfs?op=APPEND&delegation=HgAFYWRtaW4FYWRtaW4AigFUf2mGHooBVKN2Ch4KCBRzjM3jwSMlAowXb4dhqfKB5rT-8hJXRUJIREZTIGRlbGVnYXRpb24UMTAuMTIwLjE3Mi4xMDk6MjUwMDA&namenoderpcaddress=hacluster"

      In the preceding information, linux1 indicates <HOST> and 9864 indicates <PORT>.

    • Command output
      HTTP/1.1 100 Continue
      HTTP/1.1 200 OK
      Content-Length: 0
      Connection: close
    • Run the following command to access HTTPS:
      linux1:/opt/client # curl -i -k -X POST -T testFileAppend --negotiate -u: "https://linux1:9865/webhdfs/v1/huawei/testHdfs?op=APPEND&delegation=HgAFYWRtaW4FYWRtaW4AigFUf1xi_4oBVKNo5v8HCBSE3Fg0f_EwtFKKlODKQSM2t32CjhNTV0VCSERGUyBkZWxlZ2F0aW9uFDEwLjEyMC4xNzIuMTA5OjI1MDAw&namenoderpcaddress=hacluster"

      In the preceding information, linux1 indicates <HOST> and 9865 indicates <PORT>.

    • Command output
      HTTP/1.1 100 Continue
      HTTP/1.1 200 OK
      Content-Length: 0
      Connection: close

  8. Open the /huawei/testHdfs file and read all content in the file.

    • Run the following command to access HTTP:
      linux1:/opt/client # curl -L --negotiate -u: "http://linux1:9870/webhdfs/v1/huawei/testHdfs?op=OPEN"

      In the preceding information, linux1 indicates <HOST> and 9870 indicates <PORT>.

    • Command output
      Hello, webhdfs user!
      Welcome back to webhdfs!
    • Run the following command to access HTTPS:
      linux1:/opt/client # curl -k -L --negotiate -u: "https://linux1:9871/webhdfs/v1/huawei/testHdfs?op=OPEN"

      In the preceding information, linux1 indicates <HOST> and 9871 indicates <PORT>.

    • Command output
      Hello, webhdfs user!
      Welcome back to webhdfs!

  9. List the detailed information about all directories and files in the huawei directory in HDFS.

    LISTSTATUS returns information about all subfiles and folders in a request.
    • Run the following command to access HTTP:
      linux1:/opt/client # curl --negotiate -u: "http://linux1:9870/webhdfs/v1/huawei/testHdfs?op=LISTSTATUS"

      In the preceding information, linux1 indicates <HOST> and 9870 indicates <PORT>.

    • Command output
      {"FileStatuses":{"FileStatus":[
      {"accessTime":1462425245595,"blockSize":134217728,"childrenNum":0,"fileId":17680,"group":"supergroup","length":70,"modificationTime":1462426678379,"owner":"hdfs","pathSuffix":"","permission":"755","replication":3,"storagePolicy":0,"type":"FILE"}
      ]}}
    • Run the following command to access HTTPS:
      linux1:/opt/client # curl -k --negotiate -u: "https://linux1:9871/webhdfs/v1/huawei/testHdfs?op=LISTSTATUS"

      In the preceding information, linux1 indicates <HOST> and 9871 indicates <PORT>.

    • Command output
      {"FileStatuses":{"FileStatus":[
      {"accessTime":1462425245595,"blockSize":134217728,"childrenNum":0,"fileId":17680,"group":"supergroup","length":70,"modificationTime":1462426678379,"owner":"hdfs","pathSuffix":"","permission":"755","replication":3,"storagePolicy":0,"type":"FILE"}
      ]}}

    LISTSTATUS along with the size and startafter parameters will help fetch the subfiles and folders information through multiple requests, thereby avoiding the user interface from becoming slow when there are plenty of child information to be fetched.

    • Run the following command to access HTTP:
      linux1:/opt/client # curl --negotiate -u: "http://linux1:9870/webhdfs/v1/huawei/?op=LISTSTATUS&startafter=sparkJobHistory&size=1"

      In the preceding information, linux1 indicates <HOST> and 9870 indicates <PORT>.

    • Command output
      {"FileStatuses":{"FileStatus":[
      {"accessTime":1462425245595,"blockSize":134217728,"childrenNum":0,"fileId":17680,"group":"supergroup","length":70,"modificationTime":1462426678379,"owner":"hdfs","pathSuffix":"testHdfs","permission":"755","replication":3,"storagePolicy":0,"type":"FILE"}
      ]}}
    • Run the following command to access HTTPS:
      linux1:/opt/client # curl -k --negotiate -u: "https://linux1:9871/webhdfs/v1/huawei/?op=LISTSTATUS&startafter=sparkJobHistory&size=1"

      In the preceding information, linux1 indicates <HOST> and 9871 indicates <PORT>.

    • Command output
      {"FileStatuses":{"FileStatus":[
      {"accessTime":1462425245595,"blockSize":134217728,"childrenNum":0,"fileId":17680,"group":"supergroup","length":70,"modificationTime":1462426678379,"owner":"hdfs","pathSuffix":"testHdfs","permission":"755","replication":3,"storagePolicy":0,"type":"FILE"}
      ]}}

  10. Delete the /huawei/testHdfs file from HDFS.

    • Run the following command to access HTTP:
      linux1:/opt/client # curl -i -X DELETE  --negotiate -u: "http://linux1:9870/webhdfs/v1/huawei/testHdfs?op=DELETE"

      In the preceding information, linux1 indicates <HOST> and 9870 indicates <PORT>.

    • Command output
      HTTP/1.1 401 Authentication required
      Date: Thu, 05 May 2016 05:54:37 GMT
      Pragma: no-cache
      Date: Thu, 05 May 2016 05:54:37 GMT
      Pragma: no-cache
      X-Frame-Options: SAMEORIGIN
      WWW-Authenticate: Negotiate
      Set-Cookie: hadoop.auth=; Path=/; Expires=Thu, 01-Jan-1970 00:00:00 GMT; HttpOnly
      Content-Length: 0
      HTTP/1.1 200 OK
      Cache-Control: no-cache
      Expires: Thu, 05 May 2016 05:54:37 GMT
      Date: Thu, 05 May 2016 05:54:37 GMT
      Pragma: no-cache
      Expires: Thu, 05 May 2016 05:54:37 GMT
      Date: Thu, 05 May 2016 05:54:37 GMT
      Pragma: no-cache
      Content-Type: application/json
      X-Frame-Options: SAMEORIGIN
      WWW-Authenticate: Negotiate YGoGCSqGSIb3EgECAgIAb1swWaADAgEFoQMCAQ+iTTBLoAMCARKiRARC9k0/v6Ed8VlUBy3kuT0b4RkqkNMCrDevsLGQOUQRORkzWI3Wu+XLJUMKlmZaWpP+bPzpx8O2Od81mLBgdi8sOkLw
      Set-Cookie: hadoop.auth="u=hdfs&p=hdfs@HADOOP.COM&t=kerberos&e=1462463677153&s=Pwxe5UIqaULjFb9R6ZwlSX85GoI="; Path=/; Expires=Thu, 05-May-2016 15:54:37 GMT; HttpOnly
      Transfer-Encoding: chunked
      {"boolean":true}linux1:/opt/client # 
    • Run the following command to access HTTPS:
      linux1:/opt/client # curl -i -k -X DELETE --negotiate -u: "https://linux1:9871/webhdfs/v1/huawei/testHdfs?op=DELETE"

      In the preceding information, linux1 indicates <HOST> and 9871 indicates <PORT>.

    • Command output
      HTTP/1.1 401 Authentication required
      Date: Thu, 05 May 2016 06:20:10 GMT
      Pragma: no-cache
      Date: Thu, 05 May 2016 06:20:10 GMT
      Pragma: no-cache
      X-Frame-Options: SAMEORIGIN
      WWW-Authenticate: Negotiate
      Set-Cookie: hadoop.auth=; Path=/; Expires=Thu, 01-Jan-1970 00:00:00 GMT; Secure; HttpOnly
      Content-Length: 0
      HTTP/1.1 200 OK
      Cache-Control: no-cache
      Expires: Thu, 05 May 2016 06:20:10 GMT
      Date: Thu, 05 May 2016 06:20:10 GMT
      Pragma: no-cache
      Expires: Thu, 05 May 2016 06:20:10 GMT
      Date: Thu, 05 May 2016 06:20:10 GMT
      Pragma: no-cache
      Content-Type: application/json
      X-Frame-Options: SAMEORIGIN
      WWW-Authenticate: Negotiate YGoGCSqGSIb3EgECAgIAb1swWaADAgEFoQMCAQ+iTTBLoAMCARKiRARCLY5vrVmgsiH2VWRypc30iZGffRUf4nXNaHCWni3TIDUOTl+S+hfjatSbo/+uayQI/6k9jAfaJrvFIfxqppFtofpp
      Set-Cookie: hadoop.auth="u=hdfs&p=hdfs@HADOOP.COM&t=kerberos&e=1462465210180&s=KGd2SbH/EUSaaeVKCb5zPzGBRKo="; Path=/; Expires=Thu, 05-May-2016 16:20:10 GMT; Secure; HttpOnly
      Transfer-Encoding: chunked
      {"boolean":true}linux1:/opt/client #

The key management system provides the key management service through HTTP REST APIs. For details about the APIs, visit the following website:

http://hadoop.apache.org/docs/r2.7.2/hadoop-kms/index.html

Security hardening has been performed for REST APIs to prevent script injection attacks. The REST APIs cannot be used to create directories and file names that contain the keywords <script, <iframe, <frame, and javascript:.