An access and eligibility checklist for AI search

A public page can be reachable in your browser yet inaccessible to a crawler, absent from an index or unselected for a particular answer. These are different conditions. Start by naming the product and mode: Google AI Overviews, AI Mode and ChatGPT search do not have one shared inclusion rule.

For supporting links in Google AI features, Google requires indexing and eligibility for a Search snippet. OpenAI documents its own search crawler, OAI-SearchBot. Meeting an access requirement creates an opportunity for discovery, not a guarantee of selection.

Inspect the target URL without a logged-in session. Look for login walls, redirects, error responses and hosting-layer challenges. Confirm that the relevant facts are present as readable text, not only as an image. Check whether required scripts or resources are blocked and whether an infrastructure rule rejects the relevant crawler.

A successful browser visit is not sufficient evidence that an automated fetch succeeded. If you control the site’s logs, inspect actual requests and use the provider’s published verification guidance rather than trusting a user-agent string alone. Keep private content protected; GEO is not a reason to expose confidential documents.

Google explains that robots.txt manages crawler access and is not a reliable way to keep an HTML URL out of search. A blocked URL may still appear without a description. A crawler must be able to access a page to see a noindex directive. Access controls, crawl rules and indexing directives solve different problems.

OpenAI documents independent search and training settings: allowing OAI-SearchBot while disallowing GPTBot is a supported distinction. ChatGPT-User represents some user-triggered access and is not the control for automatic search inclusion. Review each documented purpose instead of applying one broad “AI bot” policy.

Use URL Inspection to investigate the exact page in Google Search Console. Review indexing status and whether the intended page is represented, then inspect robots and preview controls relevant to snippets. Google’s AI guidance states that no additional technical requirement or special schema is needed beyond its Search requirements.

Keep useful internal links, visible content and structured data aligned. Avoid diagnosing every missing citation as an indexing failure: an eligible page can be passed over for a question even when it is indexed and available.

Illustrative investigation: a company’s integration guide has no observed AI citations. A crawler request receives a hosting challenge, while the owner’s browser sees the page. That establishes an access issue worth correcting, not a weak-content diagnosis.

After the access rule is corrected, record the response and indexing evidence separately. Then rerun the original discovery prompts under comparable conditions. If no citation appears, keep investigating relevance and evidence; do not claim the fix failed simply because access did not immediately produce a citation.

• Target URL and the assistant or search surface being investigated.
• Public response, redirect destination and relevant access rules.
• Provider-specific crawler controls and verification evidence.
• Google indexing and snippet eligibility where applicable.
• Prompt, date, mode, answer and displayed links.

Google reports AI-feature traffic within the Web search type in Search Console. That traffic is useful context, but it is not a standalone measure of owned inline-citation share. Treat access, eligibility, answer inclusion and referral visits as separate stages.