Verifying Data Consistency

obsutil supports data consistency verification using the vmd5 parameter.

For example, to upload the test.txt file drive D on a Windows OS to bucket mytestbucket, run the following command to enable consistency verification:

obsutil cp D:\test.txt obs://mytestbucket/test.txt -vmd5

The object is uploaded after the verification passes, and Upload successfully is displayed in the command output.

By default, MD5 verification is disabled on OBS Browser+. To enable MD5 verification and upload objects through OBS Browser+, perform the following steps:

1. Log in to OBS Browser+.
2. Click in the upper right corner of displayed page and click Advanced Settings.
3. Select MD5 Verification. For details, see Figure 3.
   Figure 3 Configuring MD5 verification
   

4. Click OK.
5. Select the bucket to which the file is to be uploaded and upload the file.
   • If the MD5 verification is successful, the file will be uploaded successfully.
   • If the MD5 verification fails, the file upload will fail and the failure cause will be displayed on the task management page: failed to verify the file MD5 value.

OBS provides SDKs for multiple programming languages such as Java and Python. You can specify the Content-MD5 parameter to enable consistency verification when uploading objects through any OBS SDK. For details about how to calculate and set the MD5 value of an object, see the setObjectMetadata API in the OBS SDK of the programming language that you use.

For example, to upload file text.txt from drive D on a Windows OS to bucket mytestbucket by using OBS Java SDK, you can use the following sample code to verify data consistency based on the MD5 value:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

String endPoint = "https://your-endpoint";
// Hard-coded or plaintext AK and SK are risky. For security purposes, encrypt your AK and SK and store them in the configuration file or environment variables. In this example, the AK and SK are stored in environment variables for identity authentication. Before running the code in this example, configure environment variables ACCESS_KEY_ID and SECRET_ACCESS_KEY_ID.
// Obtain an AK and SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
String ak = System.getenv("ACCESS_KEY_ID");
String sk = System.getenv("SECRET_ACCESS_KEY_ID");
// Create an ObsClient instance.
ObsClient obsClient = new ObsClient(ak, sk, endPoint);
// Compute and set the MD5 value.
ObjectMetadata metadata = new ObjectMetadata();
File file = new File("D:\\text.txt");
FileInputStream fis = new FileInputStream(file);
InputStream is = (InputStream)fis;
String contentMd5 = obsClient.base64Md5(is);
metadata.setContentMd5(contentMd5);
// Upload the file with the MD5 value.
obsClient.putObject("mytestbucket", "text.txt", file, metadata);

• The MD5 value of an object must be a Base64-encoded digest.
• The OBS server will compare this MD5 value with the MD5 value obtained by object data calculation. If the two values are not the same, the upload fails and the HTTP status code 400 is returned. If the two values are consistent, the object is successfully uploaded and the HTTP status code 200 is returned.

obsutil supports data consistency verification using the vmd5 parameter.

For example, to download the test.txt file from bucket mytestbucket to a Windows PC, perform the following steps to enable data consistency verification:

1. Run the following command to check whether the object to be downloaded has MD5 information:

   obsutil stat obs://test-bucket/test.txt

   • If the returned information contains MD5 information, as shown in the following figure, go to 2.

     

   • If MD5 information is not contained, consistency verification cannot be performed when the object is downloaded.

2. Run the following command to download the object:

   obsutil cp obs://mytestbucket/test.txt D:\test.txt -vmd5

   • The message highlighted in the following figure is displayed in the command output when the object is downloaded and the consistency verification passes.

     

   • If the object does not have an MD5 value, the object can be successfully downloaded but data consistency is not verified. The message highlighted in the following figure is displayed in the command output.

     

By default, MD5 verification is disabled on OBS Browser+. To enable MD5 verification and download objects from OBS Browser+, perform the following steps:

1. Log in to OBS Browser+.
2. Click in the upper right corner of displayed page and click Advanced Settings.
3. Select MD5 Verification. For details, see Figure 4.
   Figure 4 Configuring MD5 verification
   

4. Click OK.
5. Select the bucket from which you want to download the object and download the object.
   • If the MD5 verification is successful, the file will be downloaded successfully.
   • If the MD5 verification fails, the file download will fail and the failure cause will be displayed on the task management page: failed to verify the file MD5 value.

The verification compares the MD5 value in the custom metadata of the object to be downloaded with that of the downloaded object to check whether the downloaded object is consistent with the original object.

1. This method requires that the custom metadata of the object contains its original MD5 value.
2. For details about the sample code for setting an MD5 value in the custom metadata during upload, see Method 3: Using OBS SDKs to Verify Data Consistency During Upload.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

String endPoint = "https://your-endpoint";
// Hard-coded or plaintext AK and SK are risky. For security purposes, encrypt your AK and SK and store them in the configuration file or environment variables. In this example, the AK and SK are stored in environment variables for identity authentication. Before running the code in this example, configure environment variables ACCESS_KEY_ID and SECRET_ACCESS_KEY_ID.
// Obtain an AK and SK pair on the management console. For details, see https://support.huaweicloud.com/intl/en-us/usermanual-ca/ca_01_0003.html.
String ak = System.getenv("ACCESS_KEY_ID");
String sk = System.getenv("SECRET_ACCESS_KEY_ID");
// Create an ObsClient instance.
final ObsClient obsClient = new ObsClient(ak, sk, endPoint);
// Obtain the MD5 value of the object.
ObjectMetadata metadata = obsClient.getObjectMetadata("mytestbucket", "test.txt");
String md5Origin = metadata.getUserMetadata("contentMd5");
// Compute the MD5 value of the downloaded object.
Obsobject obsobject = obsClient.getObject("mytestbucket", "test.txt");
String md5Download = obsClient.base64Md5(obsobject.getObjectContent());
// Compare the MD5 values.
if(md5Origin.contentEquals(md5Download))
    System.out.println("Object MD5 validation passes!\n");
else
    System.out.println("Object MD5 validation failed!\n");