AI Crawler Audit: A Practical Website Checklist

Choose representative public pages and agents

Start with the homepage, a product or feature page, a current guide, and a page with a different template. Include the sitemap and robots file as discovery and policy evidence. Prioritize pages tied to important buyer questions rather than collecting a large number of low-value URLs. Record the intended canonical address for each page.

Build the agent list from current provider documentation. OpenAI separates OAI-SearchBot search retrieval from GPTBot training use. Perplexity publishes its crawler guidance and request-verification information. A training crawler result is not a direct measure of search visibility. Document the purpose of each test before combining results into an audit summary.

Reference: OpenAI: overview of search and training crawlers

Inspect responses and published policy

Check HTTP status, redirect destination, content type, and whether a useful page body arrives without authentication. Review the relevant robots directives and meta or response-header indexing controls. A 200 status can still contain an edge challenge or empty shell; a visual browser pass is insufficient evidence of usable crawler content.

Compare browser and synthetic agent responses without treating a spoofed user agent as an authentic provider request. Edge policies can consider IP addresses, request history, and other conditions. If a test fails, preserve the response and identify the narrow rule that caused it. Never disable broad security controls merely to obtain a green audit tile.

curl -sS -L -D /tmp/page-headers.txt https://example.com/guide -o /tmp/page.html
# Inspect status, final URL, content type, and the returned page text.
# A changed User-Agent is a synthetic test, not proof of a real crawler visit.

Compare initial content with the rendered page

Inspect the server response for the main answer, headings, links, and any structured data the page intentionally publishes. Compare it with the browser-rendered page to identify information added only after JavaScript or interaction. Preserve the difference and the resource failures that may explain it.

Google documents its own rendering process and notes that not all bots execute JavaScript. Use that as a reason to test the actual page and retrieval context rather than assume all engines behave like a browser. When a key answer is absent from the initial response, investigate server rendering or an equivalent public content path.

Reference: Google Search Central: JavaScript SEO basics

Verify a correction with the same evidence

In a hypothetical audit, a guide returns normal content to a browser but a challenge to one synthetic crawler request. Server logs show no verified provider request in the observation window. The finding supports an access-path investigation; it does not yet prove that an engine tried and failed to crawl the page. Keep those evidence levels separate.

After a reviewed correction, rerun the same URL and relevant access checks, then inspect verified request logs if available. Use the provider's published verification mechanism instead of trusting the user-agent string alone. Record live content, policy, and response evidence. A successful audit establishes the checks that passed; it does not guarantee indexing, a recommendation, or a citation.

Reference: Perplexity: search crawlers and user agents

Common questions