How to Check AI Search Crawler Access Through Robots.txt and WAFs
Check declared crawler policy and actual edge responses separately, verify authentic requests, and review narrowly scoped access corrections.
The short answer
Check crawler access in layers: identify the search agent, read the robots policy that applies to the URL, inspect the actual response path, and review edge or WAF events. Keep declared permission, a synthetic request, and an authenticated provider request distinct. A crawler can be allowed in robots.txt yet receive a challenge from the edge. Correct the specific access problem without weakening unrelated security controls.
Identify the agent and its purpose
Consult the provider's current crawler documentation before testing. Record whether the agent supports search indexing or retrieval, model training, or a user-triggered visit. Those purposes can have separate controls. For OpenAI, OAI-SearchBot and GPTBot have independent robots policies; allowing one should not be treated as a statement about the other.
Choose a public URL the relevant search experience should be able to retrieve. Test the canonical page and any necessary redirect path. An agent's access to your homepage says little about a guide behind a different policy or template. Keep the intended publishing decision alongside the test, so a deliberate restriction is not automatically called a defect.
Read robots rules and indexing controls separately
Inspect the applicable user-agent group and path rule in robots.txt. Evaluate the actual URL path, including meaningful variants, rather than assuming a broad allow overrides every matching rule. Preserve the policy content and observation time in the finding. Review a proposed policy edit before applying it.
Robots.txt governs crawling behavior for compliant agents; it is not authentication and is not a universal instruction to remove a URL from an index. Inspect page-level robots metadata and relevant response headers separately. Your audit should explain whether it found a crawl restriction, an indexing directive, or a request that did not return the intended page.
Reference: Google Search Central: robots.txt guidance
Inspect the edge response, not just the browser
Compare a normal public fetch with a synthetic request using the documented agent string. Record response status, final URL, headers, and enough body text to recognize the page or challenge. A status of 200 is not sufficient when the body asks the visitor to complete a security check.
Review WAF or edge events for the specific path and request conditions when you have authorized access. If verifying real crawler traffic, follow the provider's published IP or verification guidance. A user-agent name can be copied by unrelated traffic. Do not grant broad access based only on that string or disable protections across the entire website.
Reference: Perplexity: search crawlers and user agents
Retest a narrow correction
Suppose a hypothetical public guide is permitted by robots.txt but an edge rule challenges the search request path. The appropriate task names the affected URL, rule, intended provider verification, and rollback procedure. An authorized owner reviews a narrow correction while leaving login, admin, and unrelated protections intact.
Repeat the policy and response checks after delivery. Where available, compare verified provider request evidence before and after the change. Keep visibility follow-up separate: an engine may need time to revisit a page and may still choose other sources. The access checker should report what the observed request received, not promise that the next answer will cite your website.
| Layer | Check | Interpretation |
|---|---|---|
| Policy | Matching robots directives | Declared permission for compliant crawling |
| Page | Robots metadata and headers | Page-level indexing or preview instructions |
| Edge | Status, challenge body, and WAF event | The observed request's delivery path |
| Identity | Provider verification evidence | Whether a logged request was authentic |
| Follow-up | Comparable page and answer checks | Delivery outcome and later visibility observations |
Your next steps
- Record the agent's documented search, training, or user-request purpose.
- Evaluate policy against the intended public URL and preserve the observed rules.
- Inspect body content, redirects, and edge challenges alongside HTTP status.
- Verify real request identity before changing allowlists.
- Review and retest a narrow correction with a rollback path.
Common questions
Why does the browser work while a crawler test fails?
The request path can differ in cookies, IP identity, headers, JavaScript, or edge policy. Preserve the failing response and investigate the matching rule. A successful browser visit proves only that browser's access path.
Can robots.txt protect a private page?
No. Use proper authentication and access controls for private information. Robots policies communicate crawl preferences to compliant agents; they do not prevent arbitrary clients from requesting public URLs.
Sources & further reading
Official documentation supporting this guide. Worked examples are illustrative unless explicitly identified as measured results.