AI crawler
A bot an AI company sends to read web pages, for answers, for training or on a user's behalf.
A site's robots.txt says which crawlers may read it. Blocking the crawlers that feed answers keeps the brand's pages out of those answers.
In GeckoCheck: Technical Audit
Related terms
AI readability
Whether AI models can reach and understand the brand's pages.
llms.txt
A proposed file that lists a site's key pages for AI models. No major AI provider has committed to reading it.
Still stuck? Contact support ›