Help Center/ Object Storage Service/ SDK Reference/ Go/ Parallel File System APIs (SDK for Go)/ Listing Objects in a Parallel File System (SDK for Go)
Updated on 2024-09-05 GMT+08:00

Listing Objects in a Parallel File System (SDK for Go)

Function

This API lists some or all of the objects in a parallel file system (PFS). You can configure the prefix, number, and start position to list objects that meet specified criteria. Returned objects are listed in alphabetical order by object name.

Restrictions

Method

func (obsClient ObsClient) ListObjects(input *ListObjectsInput) (output *ListObjectsOutput, err error)

Request Parameters

Table 1 List of request parameters

Parameter

Type

Mandatory (Yes/No)

Description

input

*ListObjectsInput

Yes

Explanation:

Input parameters for listing objects in a PFS. For details, see Table 2.

Table 2 ListObjectsInput

Parameter

Type

Mandatory (Yes/No)

Description

Bucket

string

Yes

Explanation:

PFS name.

Restrictions:

  • A PFS name must be unique across all accounts and regions.
  • A PFS name:
    • Must be 3 to 63 characters long and start with a digit or letter. Lowercase letters, digits, hyphens (-), and periods (.) are allowed.
    • Cannot be formatted as an IP address.
    • Cannot start or end with a hyphen (-) or period (.).
    • Cannot contain two consecutive periods (..), for example, my..bucket.
    • Cannot contain a period (.) and a hyphen (-) adjacent to each other, for example, my-.bucket or my.-bucket.
  • If you repeatedly create PFSs with the same name in the same region, no error will be reported and the PFS attributes comply with those set in the first creation request.

Default value:

None

Marker

string

No

Explanation:

Object name to start with when listing objects in a bucket. Objects after the specified object are returned in alphabetical order.

Restrictions:

This parameter is only used for listing objects with a single version.

Value range:

The value can contain 1 to 1,024 characters.

Default value:

None

Prefix

string

No

Explanation:

Name prefix that the objects to be listed must contain

Assume that you have the following objects: logs/day1, logs/day2, logs/day3, and ExampleObject.jpg. If you specify logs/ as the prefix, logs/day1, logs/day2, and logs/day3 will be returned. If this parameter is left blank and there are no other filter criteria, all objects in the PFS will be returned.

Value range:

The value can contain 1 to 1,024 characters.

Default value:

None

MaxKeys

int

No

Explanation:

The maximum number of objects returned in the response in alphabetical order

Value range:

The value ranges from 1 to 1000. If the specified value is greater than 1000, only 1,000 objects are returned.

Default value:

1000

Delimiter

string

No

Explanation:

Object names are grouped by this parameter, which is often used with Prefix. If a prefix is specified, objects with the same string from the prefix to the first delimiter are grouped into one CommonPrefixes. If no prefix is specified, objects with the same string from the first character to the first delimiter are grouped into one CommonPrefixes.

Assume that a PFS has objects abcd, abcde, and bbcde in it. If delimiter is set to d and prefix is set to a, objects abcd and abcde are grouped into a CommonPrefixes with abcd as the prefix. If only delimiter is set to d, files abcd and abcde are grouped into a CommonPrefixes with abcd as the prefix, and bbcde is grouped separately into another CommonPrefixes with bbcd as the prefix.

For a PFS, if this parameter is not specified, all the contents in the current directory are recursively listed by default, and subdirectories are also listed. In big data scenarios, PFSs usually have multiple directory levels and each directory level has a large number of objects. In such case, you are advised to configure [delimiter=/] to list only the content in the current directory, without the content in subdirectories, to make the listing more efficient.

Value range:

The value can contain 1 to 1,024 characters.

Default value:

None

EncodingType

string

No

Explanation:

Encoding type for some elements in the response. If Delimiter, Marker, Prefix, CommonPrefixes, NextMarker, and Key contain control characters (special characters) that are not supported by the XML 1.0 standard, set this parameter to url.

Value range:

url

Default value:

None. If you leave this parameter blank, encoding is not applied.

Responses

Table 3 List of returned results

Parameter

Type

Description

output

*ListObjectsOutput

Explanation:

Response parameters for listing objects in a PFS. For details, see Table 4.

err

error

Explanation:

Error messages returned by the API

Table 4 ListObjectsOutput

Parameter

Type

Description

StatusCode

int

Explanation:

HTTP status code

Value range:

A status code is a group of digits indicating the status of a response. It ranges from 2xx (indicating successes) to 4xx or 5xx (indicating errors). For more information, see Status Code.

Default value:

None

RequestId

string

Explanation:

Request ID returned by the OBS server

Default value:

None

ResponseHeaders

map[string][]string

Explanation:

HTTP response headers

Default value:

None

Location

string

Explanation:

Region where the PFS is located.

Restrictions:

If the specified endpoint is obs.myhuaweicloud.com, this parameter is not required. If any other endpoints are specified, this parameter is required.

Value range:

To learn about valid regions and endpoints, see Regions and Endpoints. An endpoint is the request address for calling an API. Endpoints vary depending on services and regions. To obtain the regions and endpoints, contact the enterprise administrator.

Default value:

If obs.myhuaweicloud.com is used as the endpoint and no region is specified, cn-north-1 (the CN North-Beijing1 region) is used by default.

Name

string

Explanation:

PFS name.

Restrictions:

  • A PFS name must be unique across all accounts and regions.
  • A PFS name:
    • Must be 3 to 63 characters long and start with a digit or letter. Lowercase letters, digits, hyphens (-), and periods (.) are allowed.
    • Cannot be formatted as an IP address.
    • Cannot start or end with a hyphen (-) or period (.).
    • Cannot contain two consecutive periods (..), for example, my..bucket.
    • Cannot contain a period (.) and a hyphen (-) adjacent to each other, for example, my-.bucket or my.-bucket.
  • If you repeatedly create PFSs with the same name in the same region, no error will be reported and the PFS attributes comply with those set in the first creation request.

Value range:

The value can contain 3 to 63 characters.

Prefix

string

Explanation:

Object name prefix, which is consistent with that set in the request.

Assume that you have the following objects: logs/day1, logs/day2, logs/day3, and ExampleObject.jpg. If you specify logs/ as the prefix, logs/day1, logs/day2, and logs/day3 will be returned. If this parameter is left blank and there are no other filter criteria, all objects in the PFS will be returned.

Value range:

The value can contain 1 to 1,024 characters.

Default value:

None

Marker

string

Explanation:

Object name to start with for listing objects. All objects following the specified object are returned in alphabetical order by object name.

Restrictions:

This parameter is only used for listing objects with a single version.

Value range:

The value can contain 1 to 1,024 characters.

Default value:

None

Delimiter

string

Explanation:

Object names are grouped by this parameter, which is often used with Prefix. If a prefix is specified, objects with the same string from the prefix to the first delimiter are grouped into one CommonPrefixes. If no prefix is specified, objects with the same string from the first character to the first delimiter are grouped into one CommonPrefixes.

Assume that a PFS has objects abcd, abcde, and bbcde in it. If delimiter is set to d and prefix is set to a, objects abcd and abcde are grouped into a CommonPrefixes with abcd as the prefix. If only delimiter is set to d, files abcd and abcde are grouped into a CommonPrefixes with abcd as the prefix, and bbcde is grouped separately into another CommonPrefixes with bbcd as the prefix.

For a PFS, if this parameter is not specified, all the contents in the current directory are recursively listed by default, and subdirectories are also listed. In big data scenarios, PFSs usually have multiple directory levels and each directory level has a large number of objects. In such case, you are advised to configure [delimiter=/] to list only the content in the current directory, without the content in subdirectories, to make the listing more efficient.

Value range:

The value can contain 1 to 1,024 characters.

Default value:

None

MaxKeys

int

Explanation:

The maximum number of objects returned in the response in alphabetical order.

Value range:

The value ranges from 1 to 1000. If the specified value is greater than 1000, only 1,000 objects are returned.

Default value:

1000

IsTruncated

bool

Explanation:

Whether all results are returned in the response. A maximum of 1,000 objects can be listed at a time. If the number of objects is greater than 1,000, the objects beyond 1,000 cannot be returned.

Value range:

true: Not all results are returned.

false: All results are returned.

Default value:

None

NextMarker

string

Explanation:

Where in the bucket the next listing begins. If not all results are returned, the response contains this parameter to mark the last object listed in the request. In a subsequent request, you can set Marker to the value of this parameter to list the remaining objects.

Value range:

The value can contain 1 to 1,024 characters.

Default value:

None

Contents

[]Content

Explanation:

List of objects in the PFS. For details, see Table 5.

CommonPrefixes

[]string

Explanation:

List of object name prefixes grouped according to the Delimiter parameter (if specified)

Value range:

The value can contain 1 to 1,024 characters.

Default value:

None

EncodingType

string

Explanation:

Encoding type for some elements in the response. If Delimiter, Marker, Prefix, CommonPrefixes, NextMarker, and Key contain control characters (special characters) that are not supported by the XML 1.0 standard, set this parameter to url.

Value range:

url

Default value:

None

Table 5 Content

Parameter

Type

Description

Key

string

Explanation:

Object name. An object is uniquely identified by an object name in a PFS. An object name is the complete path of the object without the PFS name.

For example, if the address for accessing the object is examplebucket.obs.ap-southeast-1.myhuaweicloud.com/folder/test.txt, the object name is folder/test.txt.

Value range:

The value can contain 1 to 1,024 characters.

Default value:

None

LastModified

time.Time

Explanation:

Time when the object was last modified.

Value range:

UTC time

Default value:

None

ETag

string

Explanation:

Base64-encoded, 128-bit MD5 value of an object. ETag is the unique identifier of the object contents that is used to determine whether the contents of an object are changed. For example, if the ETag value is A when an object is uploaded and becomes B when the object is downloaded, this indicates the contents of the object were changed. The ETag reflects changes of an object, not of the metadata. Objects created by the upload and copy operations have unique ETags after being encrypted using MD5.

Restrictions:

If an object is encrypted using server-side encryption, the ETag is not the MD5 value of the object.

Value range:

The value must contain 32 characters.

Default value:

None

Size

int64

Explanation:

Object size in bytes

Value range:

The value ranges from 0 TB to 48.8 TB, in bytes.

Default value:

None

Owner

Owner

Explanation:

Object owner. This parameter contains the domain ID and name of the object owner. For details, see Table 7.

Default value:

None

StorageClass

string

Explanation:

Object storage class

Value range:

See Table 6.

Default value:

None

Table 6 StorageClassType

Constant

Default Value

Description

StorageClassStandard

STANDARD

OBS Standard

Features low access latency and high throughput and is used for storing massive, frequently accessed (multiple times a month) or small objects (< 1 MB) requiring quick response.

StorageClassWarm

WARM

OBS Infrequent Access

Used for storing data that is semi-frequently accessed (fewer than 12 times a year) but is instantly available when needed.

StorageClassCold

COLD

OBS Archive

Used for storing rarely accessed (once a year) data.

Table 7 Owner

Parameter

Type

Mandatory (Yes/No)

Description

ID

string

Yes if used as a request parameter

Explanation:

Account (domain) ID of the owner

Value range:

To obtain the account ID, see How Do I Get My Account ID and User ID?

Default value:

None

Code Examples

This example lists all files in a PFS examplebucket using pagination.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
// This example lists all files in examplebucket using pagination.
package main
import (
    "fmt"
    "os"
    obs "github.com/huaweicloud/huaweicloud-sdk-go-obs/obs"
)
func main() {
    // Obtain an AK/SK pair using environment variables or import an AK/SK pair in other ways. Using hard coding may result in leakage.
    // Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
    ak := os.Getenv("AccessKeyID")
    sk := os.Getenv("SecretAccessKey")
    // (Optional) If you use a temporary AK/SK pair and a security token to access OBS, you are advised not to use hard coding, which may result in information leakage. You can obtain an AK/SK pair using environment variables or import an AK/SK pair in other ways.
    securityToken := os.Getenv("SecurityToken")
    // Enter the endpoint corresponding to the region where the bucket is located. CN-Hong Kong is used here as an example. Replace it with the one currently in use.
    endPoint := "https://obs.ap-southeast-1.myhuaweicloud.com"
    // Create an obsClient instance.
    // If you use a temporary AK/SK pair and a security token to access OBS, use the obs.WithSecurityToken method to specify a security token when creating an instance.
    obsClient, err := obs.New(ak, sk, endPoint, obs.WithSecurityToken(securityToken))
    if err != nil {
        fmt.Printf("Create obsClient error, errMsg: %s", err.Error())
    }
    input := &obs.ListObjectsInput{}
    // Specify the name of the PFS.
    input.Bucket = "examplebucket"
    // Specify the maximum number of objects to be returned in alphabetic order. The default value is 1000. 100 is used as an example.
    input.MaxKeys = 100
    // Specify an encoding type. url is used as an example. If the objects to list contain special characters, this parameter must be passed.
    input.EncodingType = "url"
    for {
        // List objects in the PFS.
        output, err := obsClient.ListObjects(input)
        if err != nil {
            fmt.Printf("List objects under the bucket(%s) fail!\n", input.Bucket)
            if obsError, ok := err.(obs.ObsError); ok {
                fmt.Println("An ObsError was found, which means your request sent to OBS was rejected with an error response.")
                fmt.Println(obsError.Error())
            } else {
                fmt.Println("An Exception was found, which means the client encountered an internal problem when attempting to communicate with OBS, for example, the client was unable to access the network.")
                fmt.Println(err)
            }
            os.Exit(-1)
        }
        // Print the result. By default, 100 records are returned at a time.
        fmt.Printf("List objects under the bucket(%s) successful!\n", input.Bucket)
        for index, val := range output.Contents {
            fmt.Printf("Content[%d]-OwnerId:%s, ETag:%s, Key:%s, LastModified:%s, Size:%d\n",
                index, val.Owner.ID, val.ETag, val.Key, val.LastModified, val.Size)
        }
        if output.IsTruncated {
            input.Marker = output.NextMarker
        } else {
            break
        }
    }
}