robots.txt Tester

How to use the robots.txt Tester

1. Paste your robots.txt content into the input area.
2. Enter the URL path you want to test (e.g., /admin/settings).
3. Specify the user-agent to test against (e.g., Googlebot or *).
4. View the result: allowed or blocked, along with the specific rule that matched.
5. Adjust your robots.txt rules and re-test until the results match your intent.

robots.txt is a request, not a security measure

Well-behaved crawlers (Googlebot, Bingbot) respect robots.txt rules, but the file itself is publicly readable by anyone and provides no actual access control. It doesn't prevent a user, a malicious bot, or anyone with the direct URL from visiting a "disallowed" page; it only asks compliant crawlers not to index it. If a page contains sensitive information, robots.txt is the wrong tool entirely. Use authentication or a noindex meta tag (which prevents indexing) combined with actual access restriction if the content itself needs to stay private.

Frequently asked questions

Does blocking a page in robots.txt remove it from Google's search results?

Not necessarily and not immediately. Robots.txt prevents crawling, but a page already indexed can still appear in search results (often with no description) until Google recognizes the block; use a noindex meta tag instead if the goal is specifically to remove a page from search results.

Can I have different rules for different crawlers?

Yes. User-agent blocks let you target specific bots (e.g., Googlebot, Bingbot) with different rules than your general User-agent: * block.

Why is a URL I "blocked" still showing as allowed in the test?

Check for a more specific Allow rule overriding a broader Disallow, or confirm the path syntax matches exactly. Robots.txt path matching is case-sensitive and doesn't support the same wildcard behavior in every crawler implementation.