HDFS Application Development Approach

Scenario

Service operation objects of HDFS are files. File operations in example codes include creating a folder, writing data into a file, appending data to a file, reading data from a file, and deleting a file or folder. HDFS also supports other services including setting file permission. You can learn how to perform other operations on HDFS after learning the example codes in this chapter.

The example codes are described in the following order:

1. Initializing the HDFS.
2. Creating directories.
3. Writing data into a file.
4. Appending data to a file.
5. Reading data from a file.
6. Deleting a file.
7. Deleting directories.
8. Multi-thread tasks.
9. Setting storage policies.
10. Colocation.

Development Guideline

According to the previous scenario description, the following provides the basic operations for HDFS files with read, write, and delete operations on the /user/hdfs-examples/test.txt file as an example:

1. Pass the security certification.
2. Create a FileSystem object: fSystem.
3. Call the mkdir interface in fSystem to create a directory.
4. Call the create interface in fSystem to create an FSDataOutputStream object: out. Use the write method to write data into the object out.
5. Call the append interface in fSystem to create an FSDataOutputStream object: out. Use the write method to append data into the object out.
6. Call the open interface in fSystem to create an FSDataInputStream object: in. Use the read method to read files of the object in.
7. Call the delete interface in fSystem to delete the file.
8. Call the delete interface in fSystem to delete the folder.