Configuring URL Validation to Prevent Unauthorized Access to Your Data

Scenarios

Some rogue websites may steal links from other websites to enrich their content without any costs. Link stealing hurts the interests of the original websites and it is also a strain on their servers. URL validation is designed to address this issue.

In HTTP, the Referer field allows websites and web servers to identify where people are visiting them from. URL validation of OBS utilizes this Referer field. The idea is that once you find that a request to your resource is not originated from an authorized source, you can have the request blocked or redirected to a specific web page. This way, OBS prevents unauthorized access to data stored in buckets.

Referers can be configured using a whitelist or blacklist.

Referer Rules

• The total length of the Referers in a whitelist or blacklist cannot exceed 1,024 characters.
• Referer format:
  • You can enter multiple Referers, with each one on a separate line.
  • A Referer can contain wildcards (*) and question marks (?). A wildcard can match zero or more characters and a question mark (?) can match a single character.
  • If the Referer header contains http or https during a download, the Referer configuration must also contain http or https.
• If the Referer whitelist is empty while the blacklist is not, all websites except those listed in the blacklist are allowed to access the target bucket.
• If the Referer whitelist is not empty, regardless of whether the blacklist is empty or not, only websites that appear in the whitelist and do not appear in the blacklist are allowed to access the target bucket.
• If a Referer appears in both the whitelist and the blacklist, that Referer is denied. For example, if both the whitelist and the blacklist contain https://www.example.com, requests from that website will be blocked.
• If both the Referer whitelist and blacklist are empty, all websites are allowed to access the target bucket by default.
• Before determining whether a user has the four types of permissions (read, write, ACL read, and ACL write) for a bucket and its objects, check whether this user complies with the URL validation rules defined in the Referer field.
• The Referer supports wildcard domain names.

Important Notes

• Risk of bypassing URL validation via CDN caching: If both URL validation and CDN acceleration domain names are configured for an OBS bucket, an unauthorized request to OBS may directly hit the CDN cache node and obtain the cached content without triggering OBS validation. To ensure complete and effective protection, you must also configure URL validation rules on CDN. For details, see Filtering Requests from Specific Referers.
• Impact of browser Referrer-Policy: Modern browsers support the Referrer-Policy response header, which allows websites to control how the Referer header is sent in cross-domain requests. Under certain policies, such as no-referrer, the browser suppresses the Referer header, and OBS processes it as a null Referer. When configuring URL validation, you need to take this behavior into consideration (for example, when configuring whitelisted Referers, enable Null referer is allowed).
• Video playback compatibility: When a browser uses the native <video> tag to play videos, it sends requests to the video server to obtain media data. The Referers of these requests may be null or point to other pages. Therefore, OBS must allow access from null Referers (when configuring whitelisted Referers, enable Null referer is allowed). Otherwise, video playback will fail.

Prerequisites

Static website hosting has been enabled.

Ways to Configure URL Validation

You can use OBS Console or APIs to configure URL validation.

Example Scenarios

The following are common application scenarios of URL validation: