ST-2026-237 · APIs & integration
Cloudflare Browser Run crawl endpoint enforces Content Signals use directives
Cloudflare Browser Run's /crawl endpoint now accepts a contentUse declaration and rejects a request with HTTP 400 when a target site's robots.txt specifies a more restrictive Content Signals use level.
Previous state
Before the 31 August 2026 change, the governed Browser Run /crawl API surface did not expose the contentUse request parameter or the described rejection behavior for a stricter robots.txt Content Signals use directive.
Current state
The /crawl endpoint accepts contentUse values of reference or full, defaults to full, and returns HTTP 400 when the target site's robots.txt sets a more restrictive use level than the request declares.
Affected users
Who needs to care
Developers and automated systems using Cloudflare Browser Run /crawl to retrieve third-party web content under site-owner Content Signals policies.
Required response
What to do
Set contentUse to the maximum intended use level, handle HTTP 400 policy rejections, and avoid treating a rejected crawl as a transport or authentication failure.
Evidence boundary
What the source does not prove
Cloudflare proves the contentUse parameter, reference/full values, full default and stricter-robots-policy rejection for Browser Run /crawl. It does not establish legal rights beyond the declared protocol or change unrelated Cloudflare crawling APIs.
Lifecycle history
Dated event sequence
- Content Signals enforcement added
Cloudflare added contentUse handling and stricter robots.txt use-level rejection to Browser Run /crawl.
Evidence ledger
First-party sources
- 01Cloudflare — Crawl endpoint now respects the Content Signals use directive
Official Cloudflare developer changelog · 2026-08-31
Open official source ↗