Using Mirroring-Based Back to Source to Retrieve Data

1. In the navigation pane of OBS Console, choose Object Storage.
2. In the bucket list, click the bucket you want to operate. The Objects page is displayed.
3. In the navigation pane, choose Back to Source.
4. Click Create.
   Figure 4 Creating a back-to-source by mirroring rule
   

5. Configure parameters by referring to Table 2.

   Table 2 Parameters for configuring a back-to-source by mirroring rule

   Parameter	Description
   Resource Type	The type of the origin server Public: The origin server data comes from public object storage. Private: The origin server data comes from private object storage of some cloud vendors.
   Back-to-Source Condition	The conditions that trigger the back-to-source rule The back-to-source rule is triggered when the following conditions are met: a. The requested object starts with the specified file name prefix. b. An HTTP status code 404 is returned because the object is not found in the bucket. The file name prefix: Cannot exceed 1,023 characters. Cannot contain or overlap with any other file name prefix specified for an existing rule. Can be left blank, which means that the rule applies to all files that do not meet the conditions of other back-to-source rules configured for the bucket. A bucket can have only one back-to-source rule that does not have a file name prefix specified. Example: The file name prefix is set to 123/. A request for 123/456.txt (that is not found in OBS) triggers OBS to pull 123/456.txt from the origin server.
   Add Prefix or Suffix	When OBS requests data from the origin server, it adds the specified prefix or suffix to the name of the object requested by the client. However, the object returned to OBS and the client retains its original name, without the added prefix or suffix. Example: The prefix is set to 123. A request for abc.txt triggers OBS to pull 123abc.txt from the origin server. However, the object is saved in OBS and returned to the client under its original name abc.txt.
   Replace Prefix With	When OBS requests data from the origin server, it uses the characters specified here to replace the file name prefix defined in the back-to-source condition. However, the object returned to the client retains its original name. Example: The file name prefix is set to 123 as the back-to-source condition and abc is used to replace the prefix. In this case, a request for 123456.txt triggers OBS to fetch abc456.txt from the origin server. However, the object is saved in OBS and returned to the client under its original name 123456.txt.
   Source URL	The address of the origin server. Both active and standby sites can be configured. During the back-to-source process, OBS preferentially pulls data from the active site, or polls all active sites if multiple are available. If there are two or more active sites, when the first attempt to an active site fails and the retry conditions are met, OBS will retry another active site. There must be at least one active site. Up to five active sites can be configured. If data retrieval from all active sites fails, OBS will pull data from standby sites. Example site address: http://Source domain name/Static path or https://Source domain name/Static path Source domain name indicates the domain name of the origin server. If the origin server is a bucket accessible over HTTP, use the bucket's domain name as Source domain name. If the origin server is a private bucket hosted by another cloud vendor, use the regional domain name as Source domain name. At present, only private buckets of some cloud vendors are supported. Static path indicates the directory that stores the target file. For example, if the static path is 123/, the target file is in the 123/ directory. The source URL can be an origin server IP address, and it must be a public IP address.
   Retry Condition	When a retry is triggered 4XX and specific status codes starting with 4 cannot be configured together. The same applies to 5XX and specific status codes starting with 5. A maximum of 20 status codes can be configured.
   Include request strings	After this option is enabled, query parameters in the request URL will be passed to the origin server.
   Redirect requests to the returned location	After this option is enabled, if redirection is configured for the origin server, the resources will be requested from the location included in the 3xx response returned by the origin server and be saved to OBS. Each request supports up to 10 redirects.
   Redirect without Referer	If redirection has been configured for the origin server, enabling this option will remove the Referer header from requests during the redirection process.
   HTTP Header Pass Rule	You can specify the HTTP header parameters that can be passed to the origin server when a request sent to OBS triggers the back-to-source by mirroring rule. Example for Configuring an HTTP Header Pass Rule gives a configuration example. Pass all parameters/Pass specified parameters: Set the HTTP header parameters that can be passed. Do not pass specified parameters: Set the HTTP header parameters that cannot be passed. If a client's request includes the specified headers, OBS will not pass them to the origin server. If a header is specified for both the pass and do-not-pass categories, it is deemed as a do-not-pass parameter. Configure custom parameters: You can set a custom value for a specified header. If a client request includes that header, OBS overrides it with the custom value before passing it to the origin server.
   IAM Agency	An IAM agency is required to delegate OBS to pull data from the origin server. The agency must include the obs:object:PutObject, obs:object:GetObject, obs:bucket:ListBucket, and obs:object:AbortMultipartUpload permissions. If no such IAM agency is available, create one by referring to Creating an Agency for Back to Source.
   Redirection Settings	After a redirection address is configured, if OBS fails to pull the requested resource from the origin server, OBS returns a 302 status code along with the redirection address to the client. The client then requests the resource from that address. Redirection Match: Forward match: Back to source by redirection will be performed if the returned status code is any of those you specified. Reverse match: Back to source by redirection will be performed if the returned status code is not among those you specified. Redirection URL: http://Source domain name/Static path or https://Source domain name/Static path A source domain name is the domain name of the origin server to which requests are redirected. Static path indicates the directory that stores the target file. For example, if the static path is 123/, the target file is in the 123/ directory.

6. Click OK. After the rule is created, it appears in the list on the back to source page.

Creating a Back-to-Source by Mirroring Rule

You can only use OBS Console to replicate a back-to-source by mirroring rule.

1. In the navigation pane of OBS Console, choose Object Storage.
2. In the bucket list, click the bucket you want to operate. The Objects page is displayed.
3. In the navigation pane, choose Back to Source.
4. Click Replicate.
5. Select a replication source, which is bucket whose back-to-source rules you want to replicate.
   

   • The back-to-source rules replicated from a source bucket will not overwrite existing rules in the destination bucket, and any that conflict with the existing ones will not be replicated.
   • The version of both source and destination buckets must be 3.0.
   • Before replication, you can change the source URL. For details, see Table 2.
   • You can remove the rules that you do not want to replicate.
   • There can be five back-to-source rules at most in a bucket. If the number of rules you will replicate plus the number of existing rules in the destination bucket exceeds five, the replication will fail. Before replicating the rules, delete some if necessary.
   Figure 5 Replicating back-to-source rules
   

6. Click OK to replicate the rules to the destination bucket.

Example 1

Assume that you created a bucket bucket01 in the CN East-Shanghai1 region. The bucket has two origin servers configured: A (https://example1.com) and B (https://example2.com). The origin pull scenario is as follows:

• If a client accesses a file that does not exist in bucket01/dir1, OBS will pull the file from the example1 directory on origin server A.
• If the dir1 directory does not exist on origin server A, OBS will pull the file from the example1 directory on origin server B.
• OBS requests the file from the returned redirection address if redirection has been configured for origin servers A and B.

Table 3 describes how to configure a back-to-source rule that matches this scenario.

After this rule is configured, the access process is as follows:

1. A client makes the first access to https://bucket1.obs.cn-east-3.myhuaweicloud.com/dir1/example.txt.
2. If the example.txt file does not exist in the dir1 directory in bucket01, OBS requests the file from origin server A (https://example1.com/example1/example.txt).
   • If origin server A returns a 404 status code, OBS requests the file from origin server B (https://example2.com/example1/example.txt).
   • If origin server A or B returns a 3xx status code, OBS sends a request to the redirection address returned by the origin server. After obtaining the file, OBS saves the file as dir1/example1/example.txt to bucket01 and returns the file to the client.

Example 2

Assume that you created a bucket bucket02 in the CN East-Shanghai1 region. The bucket has one origin server configured: A (https://example1.com). The origin pull scenario is as follows:

• If a client accesses a file that does not exist in bucket02/example1, OBS will pull the file from the example1 directory on origin server A.
• Query strings in the request URL can be passed to the origin server.
• HTTP headers (header1, header2, and header3) included in the request URL can be passed to the origin server.

  Table 4 describes how to configure a back-to-source rule that matches this scenario.

  Table 4 Configuring a back-to-source rule

  Parameter	Description
  Back-to-Source Condition	Set the file name prefix to the example1 directory in bucket bucket02.
  Source URL	Set the first column to https and the second column to example1.com. Leave the third column blank. Figure 6 shows the settings.
  Carry Request String	Enable this option.
  HTTP Header Pass Rule	Select Pass specified parameters and add the HTTP headers header1, header2, and header3. Figure 7 shows the settings.

  Figure 6 Source URL
  
  Figure 7 HTTP Header Pass Rule
  

  After this rule is configured, the access process is as follows:

  1. A client makes the first access to https://bucket02.obs.cn-east-3.myhuaweicloud.com/example1/example.png?name=yun&production=obs.
  2. If the example.png file does not exist in the example1 directory in bucket02, OBS requests the file from origin server A (https://example1.com/example1/example.png?name=yun&production=obs).
  3. The origin server returns the example.png file to OBS according to the passed name=yun&production=obs parameter.
  4. OBS names the obtained file example1/example.png and stores it to bucket02.

     If the request also contains the HTTP headers header1, header2, and header3, OBS will also pass them to origin server A.

Assume that the parameters are set as shown in Figure 8.

Based on the preceding configuration, if the header of the request sent to OBS is as follows:

GET /ObjectName HTTP/1.1 
Host: bucketname.obs.region.myhuaweicloud.com
aaa:aaa
bbb:bbb
ccc:ccc

After the back-to-source rule is triggered, OBS sends the following request to the origin server:

GET /ObjectName HTTP/1.1 
Host: source.com 
aaa:aaa
ccc:111

Notes for passing HTTP headers during back to source

• HTTP headers that can be passed from an origin server to a client:
  • Content-Type
  • Content-Language
  • Content-Encoding
  • Content-Disposition
  • Cache-Control
  • Expires
• HTTP headers that cannot be passed from a client to an origin server:
  1. HTTP headers starting with the prefix below:

     x-obs-

  2. All standard HTTP headers, including:
     • Content-Length
     • Authorization2
     • Authorization
     • Range
     • Date

1. In the Create Back-to-Source Rule dialog box on OBS Console, click View Agencies to jump to the Agencies page on the IAM console.
2. Click Create Agency.
3. Enter an agency name.
4. Select Cloud service for the Agency Type.
5. Select Object Storage Service (OBS) for Cloud Service.
6. Set a validity period.
7. Click Next.
   

   The console for creating an agency has the new and old editions. The following steps use the new edition.

8. Click Done. In the displayed dialog box, click Authorize.
9. If there is already a policy meeting your requirements, go to 11. Otherwise, click Create Policy in the upper right corner. Then, specify a policy name and choose Visual editor for Policy View.
   Figure 9 Authorizing an agency
   

10. Configure Policy Content as follows:
    1. Select Allow.
    2. For the service, select Object Storage Service (OBS).
    3. For actions, under ReadOnly, select obs:object:GetObject, under Read/Write, select obs:object:PutObject and obs:object:AbortMultipartUpload, and under ListOnly, select obs:bucket:ListBucket.
    4. For Resources, select All.
    5. Click Next.

11. Select the policy and click Next.
12. On the Select Scope page, select All resources for Scope and click OK.
    

    All resources is the recommended option for Scope. In this policy, the All resources option means that OBS can use all resources, including those in enterprise projects, region-specific projects, and global services under the account based on assigned permissions.