Deleting Fragments (SDK for Go)

Function

Deleting fragments refers to deleting the parts generated by incomplete multipart uploads. If a multipart upload task is interrupted or fails, unassembled parts are left in the bucket, occupying storage space and incurring fees. You can use the fragment deletion function to delete these unnecessary parts.

OBS SDKs provide the following methods for deleting fragments:

Single deletion: You can call the AbortMultipartUpload API to delete the fragments of a single incomplete multipart upload.
Batch deletion: You can call the ListMultipartUploads API to list all incomplete multipart uploads and then delete their fragments in batches.

Restrictions

To abort a multipart upload, you must be the bucket owner or have the required permission (obs:object:AbortMultipartUpload in IAM or AbortMultipartUpload in a bucket policy). For details, see Introduction to OBS Access Control, IAM Custom Policies, and Configuring an Object Policy.
The mapping between OBS regions and endpoints must comply with what is listed in Regions and Endpoints.
Deleting fragments will permanently delete data and the deleted data cannot be recovered. Exercise caution when performing this operation.
After a multipart upload task is cleared, the uploaded parts will be deleted.

Method - Deleting Fragments of a Single Multipart Upload

func (obsClient ObsClient) AbortMultipartUpload(input *AbortMultipartUploadInput) (output *BaseModel, err error)

Method – Listing Multipart Uploads

func (obsClient ObsClient) ListMultipartUploads(input *ListMultipartUploadsInput) (output *ListMultipartUploadsOutput, err error)

Request Parameters

**Table 1** List of request parameters for deleting fragments of a single multipart upload
Parameter	Type	Mandatory (Yes/No)	Description
input	*AbortMultipartUploadInput	Yes	Explanation: Input parameters for aborting a multipart upload. For details, see Table 2.

**Table 2** AbortMultipartUploadInput
Parameter	Type	Mandatory (Yes/No)	Description
Bucket	string	Yes	Explanation: Bucket name Restrictions: A bucket name must be unique across all accounts and regions. A bucket name: Must be 3 to 63 characters long and start with a digit or letter. Lowercase letters, digits, hyphens (-), and periods (.) are allowed. Cannot be formatted as an IP address. Cannot start or end with a hyphen (-) or period (.). Cannot contain two consecutive periods (..), for example, my..bucket. Cannot contain a period (.) and a hyphen (-) adjacent to each other, for example, my-.bucket or my.-bucket. If you repeatedly create buckets of the same name in the same region, no error will be reported and the bucket attributes comply with those set in the first creation request. Default value: None
Key	string	Yes	Explanation: Object name. An object is uniquely identified by an object name in a bucket. An object name is a complete path that does not contain the bucket name. For example, if the address for accessing the object is examplebucket.obs.ap-southeast-1.myhuaweicloud.com/folder/test.txt, the object name is folder/test.txt. Value range: The value must contain 1 to 1,024 characters. Default value: None
UploadId	string	Yes	Explanation: Multipart upload ID, for example, 000001648453845DBB78F2340DD460D8 Value range: The value must contain 32 characters. Default value: None

**Table 3** List of request parameters for batch listing multipart uploads
Parameter	Type	Mandatory (Yes/No)	Description
input	*ListMultipartUploadsInput	Yes	Explanation: Input parameters for listing multipart uploads. For details, see Table 4.

**Table 4** ListMultipartUploadsInput
Parameter	Type	Mandatory (Yes/No)	Description
Bucket	string	Yes	Explanation: Bucket name Restrictions: A bucket name must be unique across all accounts and regions. A bucket name: Must be 3 to 63 characters long and start with a digit or letter. Lowercase letters, digits, hyphens (-), and periods (.) are allowed. Cannot be formatted as an IP address. Cannot start or end with a hyphen (-) or period (.). Cannot contain two consecutive periods (..), for example, my..bucket. Cannot contain a period (.) and a hyphen (-) adjacent to each other, for example, my-.bucket or my.-bucket. If you repeatedly create buckets of the same name in the same region, no error will be reported and the bucket attributes comply with those set in the first creation request. Default value: None
Prefix	string	No	Explanation: Name prefix that the objects in the multipart uploads to be listed must contain Assume that you have the following objects: logs/day1, logs/day2, logs/day3, and ExampleObject.jpg. If you specify logs/ as the prefix, the multipart uploads of logs/day1, logs/day2, and logs/day3 will be returned. If you leave this parameter blank, all multipart uploads in the bucket will be returned. Restrictions: The value must contain 1 to 1,024 characters. Default value: None
MaxUploads	int	No	Explanation: Maximum number of listed multipart uploads Restrictions: If the specified value is greater than 1000, only 1,000 multipart uploads are returned. Value range: An integer from 1 to 1000 Default value: 1000
Delimiter	string	No	Explanation: Object names are grouped by this parameter, which is often used with Prefix. If a prefix is specified, objects with the same string from the prefix to the first delimiter are grouped into one CommonPrefixes. If no prefix is specified, objects with the same string from the first character to the first delimiter are grouped into one CommonPrefixes. Assume that a bucket has objects abcd, abcde, and bbcde in it. If delimiter is set to d and prefix is set to a, objects abcd and abcde are grouped into a CommonPrefixes with abcd as the prefix. If only delimiter is set to d, objects abcd and abcde are grouped into a CommonPrefixes with abcd as the prefix, and bbcde is grouped separately into another CommonPrefixes with bbcd as the prefix. Value range: The value must contain 1 to 1,024 characters. Default value: None
KeyMarker	string	No	Explanation: Object name after which multipart upload listing begins Value range: Value of NextKeyMarker in the response body of the last request Default value: None
UploadIdMarker	string	No	Explanation: Upload ID after which the multipart upload listing begins Restrictions: This parameter is valid only when used with KeyMarker. If both parameters are specified, multipart uploads with IDs greater than the specified UploadIdMarker for the specified KeyMarker are listed. Value range: The value of NextUploadIdMarker in the response body of the last request Default value: None
EncodingType	string	No	Explanation: Encoding type for Key in the response. If Key in the response contains control characters that are not supported by the XML 1.0 standard, you can specify this parameter to encode Key. Value range: url Default value: None. If you leave this parameter blank, encoding is not applied to Key.

Responses

**Table 5** Returned results of a single deletion
Parameter	Type	Description
output	*BaseModel	Explanation: Returned results. For details, see Table 6.
err	error	Explanation: Error messages returned by the API

**Table 6** BaseModel
Parameter	Type	Description
StatusCode	int	Explanation: HTTP status code Restrictions: None Value range: A status code is a group of digits that can be 2xx (indicating successes) or 4xx or 5xx (indicating errors). It indicates the status of a response. For more information, see Status Code. Default value: None
RequestId	string	Explanation: Request ID returned by the OBS server Restrictions: None Value range: None Default value: None
ResponseHeaders	map[string][]string	Explanation: HTTP response headers Restrictions: None Value range: None Default value: None

**Table 7** Returned results of a batch deletion
Parameter	Type	Description
output	*ListMultipartUploadsOutput	Explanation: Returned results. For details, see Table 4.
err	error	Explanation: Error messages returned by the API

**Table 8** ListMultipartUploadsOutput
Parameter	Type	Description
StatusCode	int	Explanation: HTTP status code Value range: A status code is a group of digits that can be 2xx (indicating successes) or 4xx or 5xx (indicating errors). It indicates the status of a response. For more information, see Status Code. Default value: None
RequestId	string	Explanation: Request ID returned by the OBS server Default value: None
ResponseHeaders	map[string][]string	Explanation: HTTP response headers Default value: None
Bucket	string	Explanation: Bucket name Restrictions: A bucket name must be unique across all accounts and regions. A bucket name: Must be 3 to 63 characters long and start with a digit or letter. Lowercase letters, digits, hyphens (-), and periods (.) are allowed. Cannot be formatted as an IP address. Cannot start or end with a hyphen (-) or period (.). Cannot contain two consecutive periods (..), for example, my..bucket. Cannot contain a period (.) and a hyphen (-) adjacent to each other, for example, my-.bucket or my.-bucket. If you repeatedly create buckets of the same name in the same region, no error will be reported and the bucket attributes comply with those set in the first creation request. Value range: The value must contain 3 to 63 characters.
KeyMarker	string	Explanation: Object name after which multipart upload listing begins. This parameter is consistent with that set in the request. Value range: The value must contain 1 to 1,024 characters. Default value: None
UploadIdMarker	string	Explanation: Upload ID after which the multipart upload listing begins, which is consistent with that set in the request Value range: The value must contain 1 to 32 characters. Default value: None
NextKeyMarker	string	Explanation: Object name to start with for the next multipart upload listing request. NextKeyMarker is returned when not all the objects are listed. You can set KeyMarker to this value in the next request to list the remaining multipart uploads. Value range: The value must contain 1 to 1,024 characters. Default value: None
NextUploadIdMarker	string	Explanation: Upload ID to start with for the next multipart upload listing request. This parameter is used together with NextKeyMarker. NextUploadIdMarker is returned when not all the objects are listed. You can set UploadIdMarker to this value in the next request to list the remaining multipart uploads. Value range: The value must contain 1 to 32 characters. Default value: None
MaxUploads	int	Explanation: Maximum number of listed multipart uploads, which is consistent with the same parameter in the request Restrictions: If the specified value is greater than 1000, only 1,000 multipart uploads are returned. Value range: An integer from 1 to 1000 Default value: 1000
IsTruncated	bool	Explanation: Whether all results are returned in the response Value range: true: Not all results are returned. false: All results are returned. Default value: None
Prefix	string	Explanation: Prefix that the object names in the multipart uploads to be listed must contain. This parameter is consistent with that set in the request. Assume that you have the following objects: logs/day1, logs/day2, logs/day3, and ExampleObject.jpg. If you specify logs/ as the prefix, the multipart uploads of logs/day1, logs/day2, and logs/day3 will be returned. If you leave this parameter blank, all multipart uploads in the bucket will be returned. Restrictions: The value must contain 1 to 1,024 characters. Default value: None
Delimiter	string	Explanation: A character used to group object names in multipart uploads, which is consistent with that set in the request. This parameter is often used with Prefix. If a prefix is specified, objects with the same string from the prefix to the first delimiter are grouped into one CommonPrefixes. If no prefix is specified, objects with the same string from the first character to the first delimiter are grouped into one CommonPrefixes. Assume that a bucket has objects abcd, abcde, and bbcde in it. If delimiter is set to d and prefix is set to a, objects abcd and abcde are grouped into a CommonPrefixes with abcd as the prefix. If only delimiter is set to d, objects abcd and abcde are grouped into a CommonPrefixes with abcd as the prefix, and bbcde is grouped separately into another CommonPrefixes with bbcd as the prefix. Value range: The value must contain 1 to 1,024 characters. Default value: None
Uploads	[]Upload	Explanation: List of multipart uploads. For details, see Table 5.
CommonPrefixes	[]string	Explanation: List of object name prefixes grouped according to the Delimiter parameter (if specified) Value range: The value must contain 1 to 1,024 characters. Default value: None
EncodingType	string	Explanation: Encoding type for Key in the response. If Key in the response contains control characters that are not supported by the XML 1.0 standard, you can specify this parameter to encode Key. Value range: url Default value: None. If you leave this parameter blank, encoding is not applied to Key.

**Table 9** Upload
Parameter	Type	Description
Key	string	Explanation: Object name. An object is uniquely identified by an object name in a bucket. An object name is a complete path that does not contain the bucket name. For example, if the address for accessing the object is examplebucket.obs.ap-southeast-1.myhuaweicloud.com/folder/test.txt, the object name is folder/test.txt. Value range: The value must contain 1 to 1,024 characters. Default value: None
UploadId	string	Explanation: Multipart upload ID, for example, 000001648453845DBB78F2340DD460D8 Value range: The value must contain 1 to 32 characters. Default value: None
Initiator	Initiator	Explanation: Initiator of the multipart upload. For details, see Table 6.
Owner	Owner	Explanation: Account ID of the object owner. For details, see Table 7.
StorageClass	StorageClassType	Explanation: Storage class of the object to be uploaded Value range: See Table 8. Default value: None
Initiated	time.Time	Explanation: Time when the multipart upload is initiated Restrictions: The time must be in the ISO8601 format, for example, 2018-01-01T00:00:00.000Z. Sample code: *time.Now().Add(time.Duration(24) time.Hour) Default value:** None

**Table 10** Initiator
Parameter	Type	Mandatory (Yes/No)	Description
ID	string	Yes if used as a request parameter	Explanation: Account (domain) ID of the initiator Value range: To obtain the account ID, see How Do I Get My Account ID and User ID? Default value: None
DisplayName	string	No	Explanation: Account name of the initiator Restrictions: The account name can contain 6 to 32 characters and must start with a letter. Only letters, digits, hyphens (-), and underscores (_) are allowed. Default value: None

**Table 11** Owner
Parameter	Type	Mandatory (Yes/No)	Description
ID	string	Yes if used as a request parameter	Explanation: Account (domain) ID of the owner Value range: To obtain the account ID, see How Do I Get My Account ID and User ID?

**Table 12** StorageClassType
Constant	Default Value	Description
StorageClassStandard	STANDARD	OBS Standard Features low access latency and high throughput and is used for storing massive, frequently accessed (multiple times a month) or small objects (< 1 MB) requiring quick response.
StorageClassWarm	WARM	OBS Infrequent Access Used for storing data that is semi-frequently accessed (fewer than 12 times a year) but is instantly available when needed.
StorageClassCold	COLD	OBS Archive Used for storing rarely accessed (once a year) data.

Example of Deleting Fragments of a Single Multipart Upload

This example aborts a multipart upload in bucket examplebucket. The object name is example/objectname and the upload task ID is 0000000xxxxxxxx.

     package main

import (
    "fmt"
    "os"

    obs "github.com/huaweicloud/huaweicloud-sdk-go-obs/obs"
)
func main() {
    //Obtain an AK/SK pair using environment variables or import the AK/SK pair in other ways. Using hard coding may result in leakage.
    //Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
    ak := os.Getenv("AccessKeyID")
    sk := os.Getenv("SecretAccessKey")
    // (Optional) If you use a temporary AK/SK pair and a security token to access OBS, you are not advised to use hard coding, which may result in information leakage. You can obtain an AK/SK pair using environment variables or import an AK/SK pair in other ways.
    // securityToken := os.Getenv("SecurityToken")


    // Enter the endpoint corresponding to the bucket. CN-Hong Kong is used here as an example. Replace it with the one currently in use.
    endPoint := "https://obs.ap-southeast-1.myhuaweicloud.com"




    // Create an obsClient instance.
    // If you use a temporary AK/SK pair and a security token to access OBS, use the obs.WithSecurityToken method to specify a security token when creating an instance.
    obsClient, err := obs.New(ak, sk, endPoint/*, obs.WithSecurityToken(securityToken)*/)
    if err != nil {
        fmt.Printf("Create obsClient error, errMsg: %s", err.Error())
    }
    input := &obs.AbortMultipartUploadInput{}
    // Specify the bucket name.
    input.Bucket = "examplebucket"
    // Specify the object (example/objectname as an example) to upload.
    input.Key = "example/objectname"
    // Specify the multipart upload ID (00000188677110424014075CC4A77xxx as an example).
    input.UploadId = "00000188677110424014075CC4A77xxx"
    // Abort the multipart upload.
    output, err := obsClient.AbortMultipartUpload(input)
    if err == nil {
        fmt.Printf("Abort multipart upload successful!\n")
        fmt.Printf("RequestId:%s\n", output.RequestId)
        return
    }
    fmt.Printf("Abort multipart upload fail!\n")
    if obsError, ok := err.(obs.ObsError); ok {
        fmt.Println("An ObsError was found, which means your request sent to OBS was rejected with an error response.")
        fmt.Println(obsError.Error())
    } else {
        fmt.Println("An Exception was found, which means the client encountered an internal problem when attempting to communicate with OBS, for example, the client was unable to access the network.")
        fmt.Println(err)
    }
}
 
 
  

Example of Deleting Fragments of Multipart Uploads in Batches

This example aborts all multipart uploads in the bucket examplebucket.

package main
import (
  "fmt"

    obs "github.com/huaweicloud/huaweicloud-sdk-go-obs/obs"
  "sync"
)
type CleanupResult struct {
  SuccessCount int
  FailedCount  int
  Errors       []error
  mutex        sync.Mutex
}
func (r *CleanupResult) addSuccess() {
  r.mutex.Lock()
  defer r.mutex.Unlock()
  r.SuccessCount++
}
func (r *CleanupResult) addError(err error) {
  r.mutex.Lock()
  defer r.mutex.Unlock()
  r.FailedCount++
  r.Errors = append(r.Errors, err)
}
func main() {
  //Obtain an AK/SK pair using environment variables or import the AK/SK pair in other ways. Using hard coding may result in leakage.
  //Obtain an AK/SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.


  // Enter the endpoint corresponding to the bucket. CN-Hong Kong is used here as an example. Replace it with the one currently in use.
  endPoint := "https://obs.ap-southeast-1.myhuaweicloud.com"




  // Create an obsClient instance.
  // If you use a temporary AK/SK pair and a security token to access OBS, use the obs.WithSecurityToken method to specify a security token when creating an instance.
  obsClient, err := obs.New(ak, sk, endPoint /*, obs.WithSecurityToken(securityToken)*/)
  if err != nil {
    fmt.Printf("Create obsClient error, errMsg: %s", err.Error())
  }
  // Specify the bucket name.
  bucket = "examplebucket"
  var result CleanupResult
  // List all incomplete multipart uploads using pagination.
  marker := ""
  for {
    listInput := &obs.ListMultipartUploadsInput{}
    listInput.Bucket = bucket
    listInput.Prefix = ""
    listInput.MaxUploads = 1000
    if marker != "" {
      listInput.KeyMarker = marker
    }
    output, err := obsClient.ListMultipartUploads(listInput)
    if err != nil {
      fmt.Printf("List multipart uploads failed: %v\n", err)
      return
    }
    // Concurrently clear multipart upload tasks.
    if len(output.Uploads) > 0 {
      var wg sync.WaitGroup
      for _, upload := range output.Uploads {
        wg.Add(1)
        go func(u obs.Upload) {
          defer wg.Done()
          input := &obs.AbortMultipartUploadInput{}
          input.Bucket = bucket
          input.Key = u.Key
          input.UploadId = u.UploadId
          _, abortErr := obsClient.AbortMultipartUpload(input)
          if abortErr != nil {
            fmt.Printf("Failed to cleanup %s (UploadId: %s): %v\n",
              u.Key, u.UploadId, abortErr)
            result.addError(abortErr)
          } else {
            fmt.Printf("Cleaned up upload: %s (UploadId: %s)\n",
              u.Key, u.UploadId)
            result.addSuccess()
          }
        }(upload)
      }
      wg.Wait()
    }
    // Check whether there is more data.
    if !output.IsTruncated {
      break
    }
    marker = output.NextKeyMarker
  }
  fmt.Printf("\nCleanup completed for bucket: %s\n", bucket)
  fmt.Printf(" Total cleaned up: %d\n", result.SuccessCount)
  fmt.Printf(" Failed: %d\n", result.FailedCount)
  if result.FailedCount > 0 {
    fmt.Printf("\nFailed uploads:\n")
    for i, err := range result.Errors {
      fmt.Printf(" %d. %v\n", i+1, err)
    }
  }
}

Helpful Links

Parent topic: APIs Related to Multipart Upload (SDK for Go)

Previous topic: Aborting a Multipart Upload (SDK for Go)

Next topic: Object ACLs (SDK for Go)