Robots Txt Tester

What Is robots.txt?

The robots.txt file is a plain text file placed at the root of your website (https://example.com/robots.txt) that tells web crawlers which parts of your site they're allowed to access.

It's part of the Robots Exclusion Protocol (REP) — an informal but universally respected standard that well-behaved bots follow voluntarily. It's not technically enforceable (a malicious scraper can ignore it), but all major search engine crawlers — Google, Bing, Yandex, DuckDuckGo — honor it.

Anatomy of a robots.txt File

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /admin/public-page.html

User-agent: Googlebot
Disallow: /staging/

User-agent: AdsBot-Google
Disallow: /

Sitemap: https://example.com/sitemap.xml

User-agent: — Which bot this rule applies to. * means all bots. Named bots (like Googlebot) get their own specific rules.

Disallow: — Paths the bot should not crawl. An empty Disallow: means "allow everything" (this is the same as having no rule at all).

Allow: — Overrides a broader Disallow. Useful when you want to block a folder but allow one specific file inside it.

Sitemap: — Points crawlers to your sitemap file. Not part of the original protocol but universally supported.

Common Robots.txt Mistakes

Using robots.txt to hide sensitive content robots.txt is public — anyone can read it. Disallowing /admin/ effectively tells the world that your admin panel is at /admin/. For actual security, use authentication. robots.txt only controls crawling, not access.

Blocking CSS and JavaScript Googlebot needs to render your pages to understand them. If your CSS and JS files are blocked, Google sees a broken page with missing styles. This can hurt your rankings. Avoid:

# Breaks Google's rendering
Disallow: /static/css/
Disallow: /static/js/

Blocking your own sitemap Some CMS configurations accidentally block *.xml files, which prevents crawlers from reading the sitemap. Check that your sitemap URL is accessible.

Wrong path format Paths must start with /. Disallow: admin/ (without the leading slash) is invalid.

Conflicting rules When Allow and Disallow rules both match a URL, the most specific rule wins. If they're the same length, Allow wins. Many webmasters expect the opposite — test carefully.

User-Agent Specific Rules

Some bots have specific user-agent strings you might want to target:

If you want to block AI training scrapers without affecting search engines:

User-agent: GPTBot
Disallow: /

User-agent: Claude-Web
Disallow: /

What robots.txt Does NOT Do

It does not prevent indexing. A page can be crawled (allowed by robots.txt) but not indexed (with a noindex tag). And critically, a page can be indexed without being crawled if another site links to it — Google may index the URL based purely on the backlink. For true noindex, use the <meta name="robots" content="noindex"> tag, not robots.txt.

It does not affect pages already in the index. If Google already indexed a page, blocking it in robots.txt prevents future crawling but doesn't remove it from the index. Use the URL removal tool in Google Search Console for that.

It is not instant. Changes to robots.txt take time to take effect — until Googlebot next visits and rereads the file, the old rules apply.

How to Use the Tester

1. Paste your robots.txt content (or enter your domain to fetch it automatically)
2. Enter a URL to test
3. Enter a user-agent (or leave as * for the default)
4. See instantly whether that URL is Allowed or Blocked

This is particularly useful when you've set up complex rules with multiple Allow/Disallow pairs and want to verify the behavior before pushing changes to your live site.

Related Search Queries

To help users find exactly what they are looking for, this tool is also optimized for searches like: robots.txt tester, robots.txt validator, check robots.txt, test robots txt, google robots txt checker, verify robots txt, robots txt file checker.