Why AI crawler access matters
ChatGPT, Claude, Perplexity, Gemini and Copilot can only cite pages their crawlers can read. Each company runs several bots: some build AI search indexes (OAI-SearchBot, Claude-SearchBot, PerplexityBot), some fetch a page when a user asks (ChatGPT-User, Claude-User), and some collect data for model training (GPTBot, ClaudeBot, CCBot, Google-Extended).
Many sites block these bots by accident, through a copied robots.txt, a CDN “block AI bots” switch, or a security plugin. Others want to block training but stay visible in AI search. This tool shows the real state for each crawler so you can decide on purpose.
How to use the checker
- Enter any page of your site. We read robots.txt, the page's meta robots tags and X-Robots-Tag header.
- We also request the page with the user agent of the main AI crawlers to spot firewall or server blocks.
- Adjust robots.txt or your CDN settings, then check again.
Examples
Stay in AI search, opt out of training
User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: OAI-SearchBot Allow: /
Accidental block
A rule like “User-agent: * / Disallow: /” left from a staging site blocks every AI crawler and search engine. The checker shows the exact robots.txt line.
Firewall block
robots.txt allows ClaudeBot but the server answers 403 to its user agent. That usually means a CDN or security plugin rule; allow verified AI bots there.
Frequently asked questions
If I block GPTBot, will my site disappear from ChatGPT?
Not necessarily. GPTBot is used for training. ChatGPT search uses OAI-SearchBot and live answers use ChatGPT-User. You can block one and allow the others.
What is Google-Extended?
A robots.txt token, not a separate crawler. Disallowing it stops Google from using your content to train Gemini models. It does not remove you from Google Search or AI Overviews, which rely on Googlebot.
Do AI crawlers respect robots.txt?
The major ones (OpenAI, Anthropic, Google, Perplexity, Apple, Common Crawl) say they do. Some less known bots do not, which is why the firewall check matters if you want a hard block.
What do noai and noimageai mean?
They are meta robots or X-Robots-Tag values that ask AI systems not to use the page text or images for training. Support varies, so combine them with robots.txt rules.
Is the checker free?
Yes, no account needed. We make a handful of requests to the page you enter and store nothing.
Related free tools
llms.txt generator and validator
Generate an llms.txt file for your site from its sitemap, or validate an existing /llms.txt against the llmstxt.org format.
AI-friendly content checker
Check whether a page is easy for ChatGPT, Perplexity, Gemini and Google AI Overviews to read, quote and cite, with a score and concrete fixes.
robots.txt checker
Check whether a URL can be crawled by search engines and list the sitemaps declared in robots.txt.