Listing Objects in a Parallel File System (SDK for Go)

Function

This API lists some or all of the objects in a parallel file system (PFS). You can configure the prefix, number, and start position to list objects that meet specified criteria. Returned objects are listed in lexicographical order by object name.

Restrictions

A maximum of 1,000 objects can be listed for each API call.
To list objects in a PFS, you must be the PFS owner or have the required permission (obs:bucket:ListBucket granted using IAM or ListBucket granted using a policy). For details, see Introduction to OBS Access Control, IAM Custom Policies, and Creating a Custom Bucket Policy.
The mapping between OBS regions and endpoints must comply with what is listed in Regions and Endpoints.

Method

func (obsClient ObsClient) ListObjects(input *ListObjectsInput) (output *ListObjectsOutput, err error)

Request Parameters

**Table 1** List of request parameters
Parameter	Type	Mandatory (Yes/No)	Description
input	*ListObjectsInput	Yes	Explanation: Input parameters for listing objects in a PFS. For details, see Table 2. Restrictions: None Value range: None Default value: None

**Table 2** ListObjectsInput
Parameter	Type	Mandatory (Yes/No)	Description
Bucket	string	Yes	Explanation: PFS name. Restrictions: A PFS name must be unique across all accounts and regions. A PFS name: Must be 3 to 63 characters long and start with a digit or letter. Lowercase letters, digits, hyphens (-), and periods (.) are allowed. Cannot be formatted as an IP address. Cannot start or end with a hyphen (-) or period (.). Cannot contain two consecutive periods (..), for example, my..bucket. Cannot contain a period (.) and a hyphen (-) adjacent to each other, for example, my-.bucket or my.-bucket. If you repeatedly create parallel file systems with the same name in the same region, no error will be reported and the parallel file system properties comply with those set in the first creation request. Value range: None Default value: None
Marker	string	No	Explanation: All objects following the specified object are returned in lexicographical order by object name. Restrictions: This parameter is only used for listing objects with a single version. Value range: The value must contain 1 to 1,024 characters. Default value: None
Prefix	string	No	Explanation: Prefix that the object names to be listed must contain. Assume that you have the following objects: logs/day1, logs/day2, logs/day3, and ExampleObject.jpg. If you specify logs/ as the prefix, logs/day1, logs/day2, and logs/day3 will be returned. If this parameter is left blank and there are no other filter criteria, all objects in the PFS will be returned. Value range: The value must contain 1 to 1,024 characters. Default value: None
MaxKeys	int	No	Explanation: The maximum number of objects returned in the response in lexicographical order. Value range: The value ranges from 1 to 1000. If the specified value is greater than 1000, only 1,000 objects are returned. Default value: 1000
Delimiter	string	No	Explanation: Object names are grouped by this parameter, which is often used with Prefix. If a prefix is specified, objects with the same string from the prefix to the first delimiter are grouped into one CommonPrefixes. If no prefix is specified, objects with the same string from the first character to the first delimiter are grouped into one CommonPrefixes. Assume that a PFS has objects abcd, abcde, and bbcde in it. If delimiter is set to d and prefix is set to a, objects abcd and abcde are grouped into a CommonPrefixes with abcd as the prefix. If only delimiter is set to d, files abcd and abcde are grouped into a CommonPrefixes with abcd as the prefix, and bbcde is grouped separately into another CommonPrefixes with bbcd as the prefix. For a PFS, if this parameter is not specified, all the contents in the current directory are recursively listed by default, and subdirectories are also listed. In big data scenarios, PFSs usually have multiple directory levels and each directory level has a large number of objects. In such case, you are advised to configure [delimiter=/] to list only the content in the current directory, without the content in subdirectories, to make the listing more efficient. Restrictions: None Value range: The value must contain 1 to 1,024 characters. Default value: None
EncodingType	string	No	Explanation: Encoding type for some elements in the response. If Delimiter, Marker, Prefix, CommonPrefixes, NextMarker, and Key contain control characters (special characters) that are not supported by the XML 1.0 standard, set this parameter to url. Restrictions: None Value range: url Default value: None. If you do not configure this parameter, encoding is not applied.

Responses

**Table 3** List of returned results
Parameter	Type	Description
output	*ListObjectsOutput	Explanation: Response parameters for listing objects in a PFS. For details, see Table 4. Value range: None
err	error	Explanation: The error message returned by the server. Value range: None

**Table 4** ListObjectsOutput
Parameter	Type	Description
StatusCode	int	Explanation: HTTP status code. Value range: A status code is a group of digits that can be 2xx (indicating successes) or 4xx or 5xx (indicating errors). It indicates the status of a response. For more information, see Status Code.
RequestId	string	Explanation: Request ID returned by the OBS server. Value range: None
ResponseHeaders	map[string][]string	Explanation: HTTP response headers. Value range: None
Location	string	Explanation: Region where the PFS is located. Restrictions: If the specified endpoint is obs.myhuaweicloud.com, this parameter is not required. If any other endpoints are specified, this parameter is required. Value range: To learn about valid regions and endpoints, see Regions and Endpoints. An endpoint is the request address for calling an API. Endpoints vary depending on services and regions. To obtain the regions and endpoints, contact the enterprise administrator. Default value: If obs.myhuaweicloud.com is used as the endpoint and no region is specified, cn-north-1 (the CN North-Beijing1 region) is used by default.
Name	string	Explanation: PFS name. Restrictions: A PFS name must be unique across all accounts and regions. A PFS name: Must be 3 to 63 characters long and start with a digit or letter. Lowercase letters, digits, hyphens (-), and periods (.) are allowed. Cannot be formatted as an IP address. Cannot start or end with a hyphen (-) or period (.). Cannot contain two consecutive periods (..), for example, my..bucket. Cannot contain a period (.) and a hyphen (-) adjacent to each other, for example, my-.bucket or my.-bucket. If you repeatedly create parallel file systems with the same name in the same region, no error will be reported and the parallel file system properties comply with those set in the first creation request. Value range: The value can contain 3 to 63 characters.
Prefix	string	Explanation: Object name prefix, which is consistent with that set in the request. Assume that you have the following objects: logs/day1, logs/day2, logs/day3, and ExampleObject.jpg. If you specify logs/ as the prefix, logs/day1, logs/day2, and logs/day3 will be returned. If this parameter is left blank and there are no other filter criteria, all objects in the PFS will be returned. Value range: The value can contain 1 to 1,024 characters.
Marker	string	Explanation: Object name to start with for listing objects. All objects following the specified object are returned in lexicographical order by object name. Restrictions: This parameter is only used for listing objects with a single version. Value range: The value can contain 1 to 1,024 characters.
Delimiter	string	Explanation: Object names are grouped by this parameter, which is often used with Prefix. If a prefix is specified, objects with the same string from the prefix to the first delimiter are grouped into one CommonPrefixes. If no prefix is specified, objects with the same string from the first character to the first delimiter are grouped into one CommonPrefixes. Assume that a PFS has objects abcd, abcde, and bbcde in it. If delimiter is set to d and prefix is set to a, objects abcd and abcde are grouped into a CommonPrefixes with abcd as the prefix. If only delimiter is set to d, files abcd and abcde are grouped into a CommonPrefixes with abcd as the prefix, and bbcde is grouped separately into another CommonPrefixes with bbcd as the prefix. For a PFS, if this parameter is not specified, all the contents in the current directory are recursively listed by default, and subdirectories are also listed. In big data scenarios, PFSs usually have multiple directory levels and each directory level has a large number of objects. In such case, you are advised to configure [delimiter=/] to list only the content in the current directory, without the content in subdirectories, to make the listing more efficient. Value range: The value must contain 1 to 1,024 characters.
MaxKeys	int	Explanation: The maximum number of objects returned in the response in lexicographical order. Value range: The value ranges from 1 to 1000. If the specified value is greater than 1000, only 1,000 objects are returned. Default value: 1000
IsTruncated	bool	Explanation: Whether all results are returned in the response. A maximum of 1,000 objects can be listed at a time. If the number of objects is greater than 1,000, the objects beyond 1,000 cannot be returned. Value range: true: Not all results are returned. false: All results are returned.
NextMarker	string	Explanation: Object name to start with for the next object listing request. If not all results are returned, the response contains this parameter to mark the last object listed in the request. In a subsequent request, you can set Marker to the value of this parameter to list the remaining objects. Value range: The value can contain 1 to 1,024 characters.
Contents	[]Content	Explanation: List of objects in the PFS. For details, see Table 5. Value range: None
CommonPrefixes	[]string	Explanation: List of object name prefixes grouped according to the Delimiter parameter (if specified). Value range: The value can contain 1 to 1,024 characters.
EncodingType	string	Explanation: Encoding type for some elements in the response. If Delimiter, Marker, Prefix, CommonPrefixes, NextMarker, and Key contain control characters (special characters) that are not supported by the XML 1.0 standard, set this parameter to url. Value range: url

**Table 5** Content
Parameter	Type	Description
Key	string	Explanation: Object name. An object is uniquely identified by an object name in a PFS. An object name is the complete path of the object without the PFS name. For example, if the address for accessing the object is examplebucket.obs.ap-southeast-1.myhuaweicloud.com/folder/test.txt, the object name is folder/test.txt. Value range: The value can contain 1 to 1,024 characters.
LastModified	time.Time	Explanation: Time when the object was last modified, in UTC. Value range: UTC time
ETag	string	Explanation: Base64-encoded, 128-bit MD5 value of an object. ETag is the unique identifier of the object contents and is used to determine whether the contents of an object are changed. For example, if the ETag value is A when an object is uploaded and is B when the object is downloaded, this indicates the contents of the object are changed. The ETag reflects changes only to the contents of an object, not its metadata. Objects created by the upload and copy operations have unique ETags after being encrypted using MD5. Restrictions: If an object is encrypted using server-side encryption, the ETag is not the MD5 value of the object. Value range: The value must contain 32 characters.
Size	int64	Explanation: Object size in bytes. Value range: The value ranges from 0 TB to 48.8 TB, in bytes.
Owner	Owner	Explanation: Object owner. This parameter contains the domain ID and name of the object owner. For details, see Table 7. Value range: None
StorageClass	string	Explanation: Object storage class. Value range: See Table 6.

**Table 6** StorageClassType
Constant	Default Value	Description
StorageClassStandard	STANDARD	OBS Standard Features low access latency and high throughput and is used for storing massive, frequently accessed (multiple times a month) or small objects (< 1 MB) requiring quick response.
StorageClassWarm	WARM	OBS Infrequent Access Used for storing data that is semi-frequently accessed (fewer than 12 times a year) but is instantly available when needed.
StorageClassCold	COLD	OBS Archive Used for storing rarely accessed (once a year) data.

**Table 7** Owner
Parameter	Type	Mandatory (Yes/No)	Description
ID	string	Yes if used as a request parameter	Explanation: Account (domain) ID of the owner Value range: To obtain the account ID, see How Do I Get My Account ID and User ID?

Sample Code

OBS does not involve the concept of folders. All elements in parallel file systems are objects. Folders are actually objects whose sizes are 0 and whose names end with a slash (/). When you set a folder name as the prefix, objects in this folder will be listed. This example lists objects in a folder.

     package main
import (
    "fmt"
    "os"

    obs "github.com/huaweicloud/huaweicloud-sdk-go-obs/obs"
)
func main() {
    // Obtain an AK/SK pair using environment variables or import an AK/SK pair in other ways. Using hard coding may result in leakage.
    // Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
    ak := os.Getenv("AccessKeyID")
    sk := os.Getenv("SecretAccessKey")
    // (Optional) If you use a temporary AK/SK pair and a security token to access OBS, you are advised not to use hard coding to reduce leakage risks. You can obtain an AK/SK pair using environment variables or import an AK/SK pair in other ways.
    // securityToken := os.Getenv("SecurityToken")


    // Enter the endpoint corresponding to the bucket. CN-Hong Kong is used here as an example. Replace it with the one currently in use.
    endPoint := "https://obs.ap-southeast-1.myhuaweicloud.com"




    // Create an obsClient instance.
    // If you use a temporary AK/SK pair and a security token to access OBS, use the obs.WithSecurityToken method to specify a security token when creating an instance.
    obsClient, err := obs.New(ak, sk, endPoint/*, obs.WithSecurityToken(securityToken)*/)
    if err != nil {
        fmt.Printf("Create obsClient error, errMsg: %s", err.Error())
    }
    input := &obs.ListObjectsInput{}
    // Specify a bucket name.
    input.Bucket = "examplebucket"
    // Specify the maximum number of objects to be returned. 100 is used as an example. Returned objects are listed in alphabetic order. The default value is 1000.
    input.MaxKeys = 100
    // Specify a delimiter for grouping object names. A slash (/) is used as an example.
    input.Delimiter = "/"
    // List objects in the bucket.
    output, err := obsClient.ListObjects(input)
    if err == nil {
        fmt.Printf("List objects under the bucket(%s) successful!\n", input.Bucket)
        fmt.Printf("RequestId:%s\n", output.RequestId)
        for index, val := range output.Contents {
            fmt.Printf("Content[%d]-OwnerId:%s, ETag:%s, Key:%s, LastModified:%s, Size:%d\n",
                index, val.Owner.ID, val.ETag, val.Key, val.LastModified, val.Size)
        }
        return
    }
    fmt.Printf("List objects under the bucket(%s) fail!\n", input.Bucket)
    if obsError, ok := err.(obs.ObsError); ok {
        fmt.Println("An ObsError was found, which means your request sent to OBS was rejected with an error response.")
        fmt.Println(obsError.Error())
    } else {
        fmt.Println("An Exception was found, which means the client encountered an internal problem when attempting to communicate with OBS, for example, the client was unable to access the network.")
        fmt.Println(err)
    }
}
 
 
  

Helpful Links

Parent topic: Parallel File Systems (SDK for Go)

Previous topic: Listing Parallel File Systems (SDK for Go)

Next topic: Direct Reading