URL Validation

Scenarios

Some rogue sites may steal links from other sites to enrich their content without any costs. Link stealing hurts the interests of the original websites and it is also a strain on their servers. OBS provides URL validation to solve this problem.

In HTTP, the Referer field allows websites and web servers to identify where people are visiting them from. URL validation of OBS utilizes this Referer field. The idea is that once you find that a request to your resource is not originated from an authorized source (for example, a URL), you can have the request blocked or redirected to a specific web page. This way, OBS prevents unauthorized access to data stored in buckets.

Such authorization is controlled using a whitelist and a blacklist.

Referer setting rules:

The length of a whitelist or blacklist cannot exceed 1,024 characters.
Referer format:
- You can enter multiple referers, each in a line.
- The referer parameter supports asterisks (*) and question marks (?). An asterisk works as a wildcard that can replace zero or multiple characters, and a question mark (?) can replace a single character.
- If the referer header field contains http or https during download, the referer must contain http or https.
If there are websites configured in the blacklist referer, but no websites in the whitelist referer, all websites except those on the blacklist are allowed to access the target bucket.
If there are websites configured in the whitelist referer, only the websites on the whitelist but not on the blacklist are allowed to access the target bucket, regardless of whether there are websites configured in the blacklist referer or not.

If a website is configured in both the whitelist and blacklist referers, access from this website will be forbidden. For example, if https://www.example.com is configured in both Whitelisted Referers and Blacklisted Referers, access requests from https://www.example.com will be blocked.

If Whitelisted Referers and Blacklisted Referers are both left blank, all websites are allowed to access data in the target bucket by default.
Before determining whether a user has the four types of permissions (Read, Write, ACL View, and ACL Edit) for a bucket or objects in the bucket, check whether this user complies with the URL validation principles of the Referer field.

Whitelist and blacklist setting methods:

Whitelist settings
By setting a whitelist, you can allow requests from the websites in the whitelist, but deny those from the websites that are not in the whitelist.

For the requests that are initialized from a browser's address box, you can add the ${null} field to Referer of Condition to specify whether to allow the HTTP requests with a blank referer.

To configure a whitelist, refer to the following policy:
```
"Statement":[ 
    {"Sid": "1", 
     "Effect": "Allow", 
     "Principal": {"ID":["*"]}, 
     "Action": "*", 
     "Resource":["bucket/*"], 
    }, 
    {"Sid":"2", 
     "Effect":"Deny", 
     "Principal":{"ID":["*"]}, 
     "Action":["*"], 
     "Resource":["bucket/*"],
     "Condition":{ 
         "StringNotEquals": 
         {"Referer":["http://www.example01.com","${null}"]} 
      } 
    } 
]
```
If you set a whitelist in this way, only the requests whose referer is set to www.example01.com or left blank can work on resources in buckets.

Blacklist settings

To configure a blacklist, refer to the following policy:

"Statement":[ 
    {"Sid":"1", 
     "Effect":"Deny", 
     "Principal":{"ID":["*"]}, 
     "Action":["*"], 
     "Resource":["bucket/*"],                              
     "Condition":{ 
         "StringEquals": 
            {"Referer":["http://www.example01.com","http://www.example02.com"]} 
      } 
    } 
]

If you set a blacklist in this way, requests whose referer is set to www.example01.com or www.example02.com cannot work on resources in buckets.