robots.txt Tester

Paste your robots.txt content, enter a URL path and user-agent, and instantly see whether the path is allowed or blocked. The tester follows Google's longest-match specification and highlights the specific rule that matched.

User-Agent: *
Allow/
Disallow/admin/
Disallow/api/
Disallow/private/*
User-Agent: Googlebot
Allow/api/public/
Disallow/api/
Sitemap: https://example.com/sitemap.xml

How it works

The tester parses robots.txt content into structured blocks of user-agent directives and their associated Allow/Disallow rules. When you test a URL, it first finds the most specific matching user-agent block (exact match takes priority over the wildcard *).

Within the matched block, all rules are evaluated against the URL path. Following Google's specification, the longest matching pattern wins. This means a longer Allow path can override a shorter Disallow. The * wildcard in paths matches any sequence of characters, and $ anchors the match to the end of the URL.

If no rule matches the URL, it is allowed by default. The parsed rules view shows all directives organized by user-agent block, with the matched rule highlighted for easy identification.