Listing Objects in a Parallel File System (SDK for Java)

Function

You can call ObsClient.listObjects to list objects in a PFS.

If you have any questions during development, post them on the Issues page of GitHub.

Restrictions

To list objects in a PFS, you must be the PFS owner or have the required permission (obs:bucket:ListBucket in IAM or ListBucket in a policy). IAM is recommended for granting permissions. For details, see IAM Custom Policies.
The mapping between OBS regions and endpoints must comply with what is listed in Regions and Endpoints.

Method

obsClient.listObjects(final ListObjectsRequest request)

Request Parameters

**Table 1** List of request parameters
Parameter	Type	Mandatory (Yes/No)	Description
request	ListObjectsRequest	Yes	Explanation: Request parameters for obtaining the object list of a PFS. For details, see Table 2.

**Table 2** ListObjectsRequest
Parameter	Type	Mandatory (Yes/No)	Description
bucketName	String	Yes	Explanation: Bucket name. Restrictions: A bucket name must be unique across all accounts and regions. A bucket name: Must be 3 to 63 characters long and start with a digit or letter. Lowercase letters, digits, hyphens (-), and periods (.) are allowed. Cannot be formatted as an IP address. Cannot start or end with a hyphen (-) or period (.). Cannot contain two consecutive periods (..), for example, my..bucket. Cannot contain periods (.) and hyphens (-) adjacent to each other, for example, my-.bucket or my.-bucket. If you repeatedly create buckets of the same name in the same region, no error will be reported and the bucket attributes comply with those set in the first creation request. Default value: None
prefix	String	No	Explanation: Prefix that the names of objects to list must contain. Restrictions: The value must contain 1 to 1,024 characters. Default value: None
marker	String	No	Explanation: Object name to start with when listing objects in a bucket. All objects following this parameter are listed in alphabetical order. Restrictions: This parameter is only available for listing objects with a single version. Value range: The value must contain 1 to 1,024 characters. Default value: None
maxKeys	int	No	Explanation: The maximum number of objects returned in the response in alphabetical order. Value range: The value ranges from 1 to 1000. If the specified value is beyond this range, only 1,000 objects are returned. Default value: 1000
delimiter	String	No	Explanation: This parameter is used to group object names. If a prefix is specified, objects with the same string from the prefix to the first delimiter are grouped into one CommonPrefixes. If no prefix is specified, objects with the same string from the first character to the first delimiter are grouped into one CommonPrefixes. Assume that a bucket has objects abcd, abcde, and bbcde in it. If delimiter is set to d and prefix is set to a, objects abcd and abcde are grouped into a commonPrefix with abcd as the prefix. If only delimiter is set to d, objects abcd and abcde are grouped into a commonPrefix with abcd as the prefix, and bbcde is grouped separately into another commonPrefix with bbcd as the prefix. For a PFS, if this parameter is not specified, all the contents in the current directory are recursively listed by default, and subdirectories are also listed. In big data scenarios, PFSs usually have multiple directory levels and each directory level has a large number of files. In such case, you are advised to configure [delimiter=/] to list the contents in the current directory but exclude the contents in subdirectories, thereby accelerating the listing. Value range: The value must contain 1 to 1,024 characters. Default value: None
encodingType	String	No	Explanation: Encoding type for some elements in the response. If delimiter, marker, prefix, nextMarker, and objectKey contain control characters that are not supported by the XML 1.0 standard, you can set encodingType to encode delimiter, marker, prefix (including the prefix in commonPrefixes), nextMarker, and objectKey in the response. Value range: url Default value: None. If you leave this parameter blank, encoding is not applied.

Responses

The following table describes the response headers:

**Table 3** ObjectListing
Parameter	Type	Description
bucketName	String	Explanation: PFS name. Restrictions: A PFS name must be unique across all accounts and regions. A PFS name: Must be 3 to 63 characters long and start with a digit or letter. Lowercase letters, digits, hyphens (-), and periods (.) are allowed. Cannot be formatted as an IP address. Cannot start or end with a hyphen (-) or period (.). Cannot contain two consecutive periods (..), for example, my..bucket. Cannot contain periods (.) and hyphens (-) adjacent to each other, for example, my-.bucket or my.-bucket. If you repeatedly create PFSs with the same name in the same region, no error will be reported and the PFS attributes comply with those set in the first creation request. Default value: None
prefix	String	Explanation: File name prefix. Assume that you have the following files: logs/day1, logs/day2, logs/day3, and ExampleObject.jpg. If you specify logs/ as the prefix, logs/day1, logs/day2, and logs/day3 will be returned. If this parameter is left blank and there are no other filtering criteria, all objects in the PFS will be returned. Value range: The value must contain 1 to 1,024 characters. Default value: None
objectSummaries	List<ObsObject>	Explanation: List of objects in the PFS. For details, see Table 4.
commonPrefixes	List<String>	Explanation: List of object name prefixes grouped according to the delimiter parameter (if specified). Value range: The value must contain 1 to 1,024 characters. Default value: None
truncated	boolean	Explanation: Whether all results are returned in the response. A maximum of 1,000 objects can be listed at a time. If the number of objects is greater than 1,000, the objects beyond 1,000 cannot be returned. Value range: true: Not all results are returned. false: All results are returned. Default value: None
marker	String	Explanation: File name which listing starts from. All objects following the specified object are returned in alphabetical order by object name. Restrictions: This parameter is only used for listing objects with a single version. Value range: The value must contain 1 to 1,024 characters. Default value: None
maxKeys	int	Explanation: Maximum number of objects to list. No more than the specified number of objects can be returned in the response in alphabetical order. Value range: The value ranges from 1 to 1000. If the specified value is beyond this range, only 1,000 objects are returned. Default value: 1000
delimiter	String	Explanation: Object names are grouped by this parameter, which is often used with Prefix. If a prefix is specified, objects with the same string from the prefix to the first delimiter are grouped into one CommonPrefixes. If no prefix is specified, objects with the same string from the first character to the first delimiter are grouped into one CommonPrefixes. Assume that a PFS has objects abcd, abcde, and bbcde in it. If delimiter is set to d and prefix is set to a, objects abcd and abcde are grouped into a CommonPrefixes with abcd as the prefix. If only delimiter is set to d, files abcd and abcde are grouped into a CommonPrefixes with abcd as the prefix, and bbcde is grouped separately into another CommonPrefixes with bbcd as the prefix. For a PFS, if this parameter is not specified, all the content in the current directory is recursively listed by default, and subdirectories are also listed. In big data scenarios, PFSs usually have multiple directory levels and each directory level has a large number of objects. In such case, you are advised to configure [delimiter=/] to list the content in the current directory but exclude the content in subdirectories, thereby making the listing more efficient. Value range: The value must contain 1 to 1,024 characters. Default value: None
nextMarker	String	Explanation: Where in the PFS the next listing begins. If not all results are returned, the response contains this parameter to mark the last object listed in the request. In a subsequent request, you can set Marker to the value of this parameter to list the remaining objects. Value range: The value must contain 1 to 1,024 characters. Default value: None
location	String	Explanation: Region where the PFS was created. Restrictions: If the specified endpoint is obs.myhuaweicloud.com, this parameter is not required. If any other endpoints are specified, this parameter is required. Value range: To learn about valid regions and endpoints, see Regions and Endpoints. An endpoint is the request address for calling an API. Endpoints vary depending on services and regions. To obtain the regions and endpoints, contact the enterprise administrator. Default value: If obs.myhuaweicloud.com is used as the endpoint and no region is specified, cn-north-1 (the CN North-Beijing1 region) is used by default.
statusCode	int	Explanation: HTTP status code. Value range: A status code is a group of digits that can be 2xx (indicating successes) or 4xx or 5xx (indicating errors). It indicates the status of a response. For more information, see Status Code. Default value: None
responseHeaders	Map<String, Object>	Explanation: Response header list, composed of tuples. In a tuple, the String key indicates the name of the header, and the Object value indicates the value of the header. Default value: None

**Table 4** ObsObject
Parameter	Type	Description
objectKey	String	Explanation: Object name. An object is uniquely identified by an object name in a PFS. An object name is a complete path of the object that does not contain the PFS name. For example, if the address for accessing the object is examplebucket.obs.ap-southeast-1.myhuaweicloud.com/folder/test.txt, the object name is folder/test.txt. Value range: The value must contain 1 to 1,024 characters. Default value: None
bucketName	String	Explanation: PFS name. Restrictions: A PFS name must be unique across all accounts and regions. A PFS name: Must be 3 to 63 characters long and start with a digit or letter. Lowercase letters, digits, hyphens (-), and periods (.) are allowed. Cannot be formatted as an IP address. Cannot start or end with a hyphen (-) or period (.). Cannot contain two consecutive periods (..), for example, my..bucket. Cannot contain periods (.) and hyphens (-) adjacent to each other, for example, my-.bucket or my.-bucket. If you repeatedly create PFSs with the same name in the same region, no error will be reported and the PFS attributes comply with those set in the first creation request. Default value: None
objectContent	InputStream	Explanation: The data stream of the object. Default value: None
owner	Owner	Explanation: Object owner. This parameter contains the domain ID and name of the object owner. For details, see Table 5. Default value: None
metadata	ObjectMetadata	Explanation: Object metadata. Value range: Object metadata. For details, see Table 7. Default value: None

**Table 5** Owner
Parameter	Type	Mandatory (Yes/No)	Description
id	String	Yes	Explanation: Account (domain) ID of the bucket owner. Value range: To obtain the account ID, see How Do I Get My Account ID and User ID? Default value: None
displayName	String	No	Explanation: Account name of the owner. Value range: To obtain the account name, see How Do I Get My Account ID and User ID? Default value: None

**Table 6** StorageClassEnum
Constant	Default Value	Description
STANDARD	STANDARD	Standard storage class
WARM	WARM	Infrequent Access storage class.
COLD	COLD	Archive storage class.

**Table 7** ObjectMetadata
Parameter	Type	Mandatory (Yes/No)	Description
contentLength	Long	No	Explanation: Object size. Restrictions: The object size in a single upload ranges from 0 to 5 GB. To upload objects larger than 5 GB, multipart uploads should be used. Default value: If this parameter is not specified, the SDK automatically calculates the size of the object.
contentType	String	No	Explanation: MIME type of the object file. MIME type is a standard way of describing a data type and is used by the browser to decide how to display data. Value range: See What Is Content-Type (MIME)? (Java SDK) Default value: If this parameter is not specified, the SDK determines the file type based on the suffix of the file name and assigns a value to the parameter. For example, if the suffix of the file name is .xml, the file is an application/xml file. If the suffix is .html, the file is a text/html file.
contentEncoding	String	No	Explanation: Content-Encoding header in the response. It specifies which encoding is applied to the object. Default value: None
contentDisposition	String	No	Explanation: Provides a default name for the requested object. When the object with the default name is being downloaded or accessed, the content is displayed as part of a web page in the browser or as an attachment in a download dialog box. Default value: None
cacheControl	String	No	Explanation: Cache-Control header in the response. It specifies the cache behavior of the web page when an object is downloaded. Default value: None
contentLanguage	String	No	Explanation: Language or language combination for visitors to customize and use. For details, see the definition of ContentLanguage in the HTTP protocol. Default value: None
expires	String	No	Explanation: The time a cached web page object expires. Restrictions: The time must be in the GMT format. Default value: None
contentMd5	String	No	Explanation: Base64-encoded MD5 value of the object data. It is provided for the OBS server to verify data integrity. The OBS server will compare this MD5 value with the MD5 value calculated based on the object data. If the two values are not the same, HTTP status code 400 is returned. Restrictions: The MD5 value of the object must be Base64 encoded. If the MD5 value is not specified, the OBS server will not verify the MD5 value of the object. Value range: Base64-encoded 128-bit MD5 value of the request body calculated according to RFC 1864. Example: n58IG6hfM7vqI4K0vnWpog== Default value: None
storageClass	StorageClassEnum	No	Explanation: Storage class of an object that can be specified at object creation. If you do not specify this header, the object inherits the storage class of the PFS. Value range: See Table 6. Default value: None
webSiteRedirectLocation	String	No	Explanation: If the PFS is configured with website hosting, the request for obtaining the object can be redirected to another object in the PFS or an external URL. This parameter specifies the address the request for the object is redirected to. The request is redirected to another object in the same PFS: WebsiteRedirectLocation:/anotherPage.html The request is redirected to an external URL: WebsiteRedirectLocation:http://www.example.com/ Restrictions: The value must start with a slash (/), http://, or https:// and cannot exceed 2 KB. OBS only supports redirection for objects in the root directory of a PFS. Default value: None
nextPosition	long	No	Explanation: Start position for the next append upload. Value range: 0 to the object length, in bytes. Default value: None
appendable	boolean	No	Explanation: Whether the object is appendable. Value range: true: The object is appendable. false: The object is not appendable. Default value: None
userMetadata	Map<String, Object>	No	Explanation: User-defined metadata of the object. To define it, you can add a header starting with x-obs-meta- in the request. In Map, the String key indicates the name of the user-defined metadata that starts with x-obs-meta-, and the Object value indicates the value of the user-defined metadata. To obtain the user-defined metadata of an object, use ObsClient.getObjectMetadata. For details, see Obtaining Object Metadata (SDK for Java). Restrictions: An object can have multiple pieces of metadata. The size of the metadata cannot exceed 8 KB in total. When you call ObsClient.getObject to download an object, its user-defined metadata will also be downloaded. Default value: None

Code Examples

This example lists objects in parallel file system examplebucket.

    
     
       
       
         import com.obs.services.ObsClient;
import com.obs.services.exception.ObsException;
import com.obs.services.model.*;
import java.io.*;

public class ListPosixObjects{
    // Configure the maximum number of objects that can be listed.
    private static int maxKey = 1000;
    private static String bucketName = "examplebucket";
    private static ObsClient obsClient;
    public static void main(String[] args) throws IOException, InterruptedException {
       // Hard-coded or plaintext AK and SK are risky. For security purposes, encrypt your AK and SK and store them in the configuration file or environment variables. In this example, the AK and SK are stored in environment variables for identity authentication. Before running the code in this example, configure environment variables ACCESS_KEY_ID and SECRET_ACCESS_KEY_ID.
        // Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
        String ak = System.getenv("OBS_ACCESS_KEY_ID");
        String sk = System.getenv("OBS_SECRET_ACCESS_KEY_ID");
        // (Optional) If you are using a temporary AK/SK pair and a security token to access OBS, you are advised not to use hard coding, which may result in information leakage.
        // Obtain an AK/SK pair and a security token using environment variables or import them in other ways.
        String securityToken = System.getenv("OBS_SECURITY_TOKEN");
        // Obtain an endpoint using environment variables or import it in other ways.
        String endPoint = System.getenv("OBS_ENDPOINT");
        // Create an ObsClient instance.
        try {
            obsClient = new ObsClient(ak, sk, endPoint);
            // List objects in the parallel file system.
            ListObjectsRequest request = new ListObjectsRequest(bucketName);
            // Configure the maximum number of objects that can be listed.
            request.setMaxKeys(maxKey);
            // Set delimiter to / to list the content only in the current directory. Content in subdirectories is not listed.
            request.setDelimiter("/");
            listObjects(request);
        } catch (ObsException e) {
            // Request failed. Print the HTTP status code.
            System.out.println("HTTP Code: " + e.getResponseCode());
            // Request failed. Print the server-side error code.
            System.out.println("Error Code:" + e.getErrorCode());
            // Request failed. Print the error details.
            System.out.println("Error Message: " + e.getErrorMessage());
            // Request failed. Print the request ID.
            System.out.println("Request ID:" + e.getErrorRequestId());
            System.out.println("Host ID:" + e.getErrorHostId());
        } catch (Exception e) {
            // Print other error information.
            e.printStackTrace();
        }
    }

    static void listObjectsByPrefix(ObjectListing result) throws ObsException {
        for (String prefix : result.getCommonPrefixes()) {
            System.out.println("Objects in folder [" + prefix + "]:");
            ListObjectsRequest request = new ListObjectsRequest(bucketName);
            // Configure the maximum number (1 to 1,000) of objects to include in the response.
            request.setMaxKeys(maxKey);
            // Set delimiter to / to list the content only in the current directory. Content in subdirectories is not listed.
            request.setDelimiter("/");
           // Return the listing result based on the configured object prefix.
            request.setPrefix(prefix);
            listObjects(request);
        }
    }

    static void listObjects(ListObjectsRequest request) {
        ObjectListing result;
        do {
            result = obsClient.listObjects(request);
            for (ObsObject obsObject : result.getObjects()) {
                // Print a listed object.
                System.out.println("\t" + obsObject.getObjectKey());
                // Print the object owner.
                System.out.println("\t" + obsObject.getOwner());
            }
            // Configure the position where the listing starts.
            request.setMarker(result.getNextMarker());
            // List folders recursively.
            listObjectsByPrefix(result);
        } while (result.isTruncated());
    }
}  

        

      

    
   