AI access and guidance
What robots.txt and the AI guidance files tell AI crawlers they may do.
What it tells you
Part of AI readability. These checks read what the brand's site tells AI crawlers and agents: which of them may come in, and the files written for them, such as llms.txt.
Checks
AI bots allowed in robots.txt
robots.txt is missing or empty. Crawling still works, an absent file means everything is permitted, but there is nowhere to declare the sitemap or to control AI crawlers.
robots.txt blocks an answering bot, the crawlers that fetch a page in order to cite it. This is the direct way to be absent from AI answers.
robots.txt blocks a training crawler. Usually deliberate: most publishers now refuse training while admitting answering bots.
Read the documentation ›WebMCP detected
No WebMCP tools found in the page. WebMCP lets a page offer actions to an assistant directly; adoption is early.
Read the documentation ›MCP manifest present and valid
No /.well-known/mcp.json. MCP is how an assistant discovers tools a site exposes rather than only reading its pages.
The MCP manifest at /.well-known/mcp.json is malformed, so an assistant cannot discover the tools this site offers.
Read the documentation ›ACP discovery present and valid
No /.well-known/acp.json. Agentic Commerce is how a shopping agent discovers checkout; the standard is still emerging.
The Agentic Commerce discovery document at /.well-known/acp.json is malformed, so agents that look for it cannot use it. The standard is still emerging.
Read the documentation ›ai.txt present
No /.well-known/ai.txt. An early convention for stating AI usage terms; no major provider has committed to reading it.
Read the documentation ›TDM reservation present and valid
No /.well-known/tdmrep.json. TDMRep is the W3C way to state a mining policy; crawlers that honour it use their default without one.
The text and data mining reservation at /.well-known/tdmrep.json is malformed. Crawlers that honour it cannot read the site's stated policy, so they fall back to their own default.
Read the documentation ›llms.txt present
The state of llms.txt and its variants. Reported for completeness only, the AI crawlers do not fetch this file, and no major provider has committed to reading it.
Read the documentation ›No noai directive
A noai tag asks AI tools not to use the page, which keeps it out of AI answers that respect it.
No deprecated ai-plugin.json
/.well-known/ai-plugin.json is present but the ChatGPT plugin format it belongs to is retired. MCP replaced it.
Read the documentation ›Page source readable
Whether the page’s raw HTML could be read, which is what most AI crawlers see.
Seen in the app on: Technical Audit
See how the brand's site does on these checks.Related guides
- How the audit score is built
One score out of 100 for the brand's site: how well AI models can reach and read it, and how sound it is, with AI readability counting most.
- AI readability and site health
AI readability is whether AI models can reach and understand the brand's pages; site health is how fast, safe and sound they are.
Still stuck? Contact support ›