Appendable Upload
Scenarios
After a simple upload, form-based upload, or multipart upload is complete, the uploaded object can be read but cannot be modified. If the object's content changes, you must upload a new object with the same name to overwrite the original one.
There will be real-time streams when you upload logs, video surveillance data, live streaming data, and others. Using any of the foregoing uploads may cause problems such as complex software architecture, high server loads from frequent client requests, and high data latency. To solve such problems, you can use the appendable upload, which allows you to append data to the end of an object in a specific bucket. If there is no object with the same key value in the bucket, a new object will be created.
With append uploads, newly generated video data can be appended to the same object in real time. The client only needs to periodically read the current object length and compare it with the length obtained during the previous read. If there is new data that can be read, the client will read the newly uploaded data. In this way, the architecture can be simplified and its scalability can be enhanced.
Objects created through append uploads are appendable objects, while those created through simple PUT uploads are normal objects.
Constraints
- The last modification time of the object is updated each time an appendable upload is performed.
- When SSE-C is used, append upload works the same as multipart-upload initiation. You must include the required SSE-C request headers (for example, x-obs-server-side-encryption), and all subsequent append uploads must also include these headers.
- When SSE-KMS is used, you only need to include the required SSE-KMS request headers (for example, x-obs-server-side-encryption) for the first upload, and only if no object with the same name exists in the bucket. Subsequent append uploads do not include these headers.
- Each append operation must not exceed the maximum object size of 5 GB.
- Each appendable object can be appended for up to 10,000 times.
- Append uploads are not supported for objects in the Archive or Deep Archive storage class.
- If cross-region replication is configured for a bucket, appendable upload API cannot be called.
- Appendable upload is not available for parallel file systems.
- If you perform the PUT operation on an existing appendable object, the appendable object is overwritten by the newly uploaded object and the object type changes to normal. If you perform the PUT operation on a normal object, an error occurs.
- An appendable object will be changed to a normal object after being copied. An appendable object cannot be copied and saved as an appendable object.
- Upload callback is not available for appendable objects.
- If a bucket has WORM enabled, an append upload on this bucket will fail, with a 403 error returned.
Important Notes
- If the file you want to append data to does not exist, calling the AppendObject API will create an appendable file.
- If a bucket has WORM enabled, an append operation on this bucket will fail, with a 403 error returned.
- The ETag returns the hash value of the data to be uploaded, not the hash value of the entire object.
- Suppose there is a file you want to append data to:
- If the file is appendable and the position where data will be appended matches the file length, the data can be directly appended to the end of the file. If the position does not match the file length, the PositionNotEqualToLength exception will be returned.
- If the file is not appendable, for example, it is a normal object, the ObjectNotAppendable exception is returned.
Prerequisites
Ways to Append Data
You can use SDKs or APIs to append objects.
FAQ
Are There Any File Format Constraints for Append Uploads?
No. Append upload processes data as a binary stream, so it does not impose any requirements on the file format.
Does Append Upload Support a Lock Mechanism for Concurrent Operations?
OBS does not support this capability. To prevent concurrent access to an object, you must implement a lock mechanism at the application layer.
What is your overall rating for this page?
Thank you very much for your feedback. We will continue working to improve the documentation.See the reply and handling status in My Cloud VOC.
For any further questions, feel free to contact us through the chatbot.
Chatbot