What robots.txt does
robots.txt is a text file at the root of a site, for example https://example.com/robots.txt, that tells crawlers which paths they may fetch. One wrong Disallow line can stop Google from crawling a whole section, and it is one of the most common causes of pages dropping out of search.
This checker downloads the robots.txt of the URL's site, applies its rules for the user agent you choose (Googlebot by default) and tells you whether the URL is allowed or blocked, quoting the exact rule. It also lists the declared sitemaps and checks the first one, shows any crawl-delay and flags lines it does not recognise.
Rules are matched the way Google does it: the most specific (longest) matching path wins, and when an Allow and a Disallow rule are equally specific, Allow wins.
How to test a URL against robots.txt
- Paste the URL of the page you want to test.
- Keep Googlebot as the user agent or type another one, such as Bingbot or GPTBot.
- Run the check to see whether the URL is allowed or blocked, and by which rule.
- Review the sitemaps and unrecognised lines, then fix the file on your server if needed.
Common robots.txt patterns
Block the cart and account area
User-agent: * / Disallow: /cart / Disallow: /account. Keeps crawlers out of pages with no search value without touching products or categories.
Allow one folder inside a blocked one
Disallow: /wp-admin/ with Allow: /wp-admin/admin-ajax.php. The longer Allow rule wins for that one file.
Accidentally blocking the whole site
Disallow: / under User-agent: * blocks everything. It is often left over from a staging site; this checker shows it as the blocking rule for every URL.
Frequently asked questions
Does blocking a page in robots.txt remove it from Google?
No. robots.txt controls crawling, not indexing. A blocked URL can still be indexed if other pages link to it. To keep a page out of results, allow crawling and add a noindex meta tag or header.
Where must robots.txt be located?
At the root of each host and protocol, for example https://www.example.com/robots.txt. A file in a subfolder is ignored, and each subdomain needs its own file.
Does Google follow crawl-delay?
No, Googlebot ignores crawl-delay. Some other crawlers, such as Bingbot, respect it. Google adjusts its crawl rate automatically based on how your server responds.
What happens if there is no robots.txt?
If the file returns 404, crawlers treat the whole site as allowed. Persistent server errors (5xx) on robots.txt can make Google pause crawling the site, so make sure it returns 200 or 404.
Can I block AI crawlers with robots.txt?
Yes. Most AI crawlers, such as GPTBot, ClaudeBot or Google-Extended, follow robots.txt rules for their user agent. Test their names here to see what your current file allows.
Related free tools
On-page SEO checker
Analyse one page: title, meta description, headings, canonical, images, structured data and common issues.
Schema markup generator (JSON-LD)
Generate structured data for products, articles, local businesses and FAQs.
SERP preview
See how your page shows up in Google and check title and description length.