Why Is the Storage Type of File Copies DISK When the Tiered Storage Policy Is LAZY_PERSIST?

Question

When the storage policy of the file is set to LAZY_PERSIST, the storage type of the first replica should be RAM_DISK, and the storage type of other replicas should be DISK.

But why is the storage type of all copies shown as DISK actually?

Answer

When a user writes into a file whose storage policy is LAZY_PERSIST, three replicas are written one by one. The first replica is preferentially written into the DataNode where the client is located. The storage type of all replicas is DISK in the following scenarios:

• If the DataNode where the client is located does not have the RAM disk, the first replica is written into the disk of the DataNode where the client is located, and other replicas are written into the disks of other nodes.
• If the DataNode where the client is located has the RAM disk but the parameter dfs.datanode.max.locked.memory is not set or its value is less than the value of dfs.blocksize, the first replica is written into the disk of the DataNode where the client is located, and other replicas are written into the disks of other nodes.

  To set the preceding parameters, log in to Manager, choose Cluster > Services > HDFS > Configurations > All Configurations, and search for the parameters on the displayed page. The parameters are described as follows:

  • dfs.datanode.max.locked.memory: memory size of the DataNode node for caching block replicas. The value of this parameter cannot be greater than the maximum lockable memory of the DataNode. The unit is byte. The default value is 0. The value ranges from 0 to 1800000000000.
  • dfs.blocksize: default block size of a new file. The unit is byte. The value must be a complete number of bytes. For example, if the block size is 128 MB, set the value to 134217728. The value must be a multiple of 512. The default value is 134217728. The value ranges from 512 to 1073741824.