Configuring URL Validation to Prevent Unauthorized Access to Your Data

Scenarios

Some rogue websites may steal links from other websites to enrich their content without any costs. Link stealing hurts the interests of the original websites and it is also a strain on their servers. URL validation is designed to address this issue.

In HTTP, the Referer field allows websites and web servers to identify where people are visiting them from. URL validation of OBS utilizes this Referer field. The idea is that once you find that a request to your resource is not originated from an authorized source, you can have the request blocked or redirected to a specific web page. This way, OBS prevents unauthorized access to data stored in buckets.

Referers can be configured using a whitelist or blacklist.

Referer rules are as follows:

• The length of a whitelist or blacklist cannot exceed 1,024 characters.
• Referer format:
  • You can enter multiple referers, each in a line.
  • The referer parameter supports asterisks (*) and question marks (?). An asterisk works as a wildcard that can replace zero or multiple characters, and a question mark (?) can replace a single character.
  • If the referer header field contains http or https during download, the referer must contain http or https.
• If Whitelisted Referers is left blank but Blacklisted Referers is not, all websites except those specified in the blacklist are allowed to access data in the target bucket.
• If both Whitelisted Referers and Blacklisted Referers have referers, only those specified in the whitelist that are not in the blacklist are allowed to access the target bucket.

If Whitelisted Referers overlaps with Blacklisted Referers, the overlapped referers are forbidden. For example, if both Whitelisted Referers and Blacklisted Referers contain https://www.example.com, requests from this website will be blocked.

• If Whitelisted Referers and Blacklisted Referers are both left blank, all websites are allowed to access data in the target bucket by default.
• Before determining whether a user has the four types of permissions (read, write, ACL read, and ACL write) for a bucket or objects in the bucket, check whether this user complies with the URL validation principles of the Referer field.
• Referer can be a wildcard domain name.

Whitelist and blacklist setting methods:

• Whitelist settings

  By setting a whitelist, you can allow requests from the websites in the whitelist, but deny those from the websites that are not in the whitelist.

  For the requests that are initialized from a browser's address box, you can add the ${null} field to Referer of Condition to specify whether to allow the HTTP requests with a blank referer.

  To configure a whitelist, refer to the following policy:

  "Statement":[ 
      {"Sid": "1", 
       "Effect": "Allow", 
       "Principal": {"ID":["*"]}, 
       "Action": "*", 
       "Resource":["bucket/*"], 
      }, 
      {"Sid":"2", 
       "Effect":"Deny", 
       "Principal":{"ID":["*"]}, 
       "Action":["*"], 
       "Resource":["bucket/*"],
       "Condition":{ 
           "StringNotEquals": 
           {"Referer":["http://www.example01.com","${null}"]} 
        } 
      } 
  ]

  If you set a whitelist in this way, only the requests whose referer is set to www.example01.com or left blank can work on resources in buckets.

• Blacklist settings

  To configure a blacklist, refer to the following policy:

  "Statement":[ 
      {"Sid":"1", 
       "Effect":"Deny", 
       "Principal":{"ID":["*"]}, 
       "Action":["*"], 
       "Resource":["bucket/*"],                              
       "Condition":{ 
           "StringEquals": 
              {"Referer":["http://www.example01.com","http://www.example02.com"]} 
        } 
      } 
  ]

  If you set a blacklist in this way, requests whose referer is set to www.example01.com or www.example02.com cannot work on resources in buckets.

Prerequisites

Static website hosting has been enabled.

Ways to Configure URL Validation

You can use OBS Console or APIs to configure URL validation.