BUY WEBSITES
BUY DOMAINS
LOGIN
SIGNUP
Robots.txt Tester
A robots.txt tester checks whether a specific URL is allowed or blocked for a given crawler before a mistake in the file quietly removes pages from search. Robots.txt does not work line by line in order: Google picks the single most specific user-agent group, then applies the rule whose path is the longest match, with an Allow beating a Disallow of equal length. A rule that looks like it blocks a page can be overridden by a longer Allow, and a stray wildcard can block far more than intended.
This free, browser-based tool parses a pasted robots.txt into its user-agent groups, supports the * wildcard and $ end-anchor, and returns a clear ALLOWED or BLOCKED verdict with the exact directive that decided it. Nothing is fetched and nothing is uploaded, so it is safe for staging files and unpublished rules.
/cart/checkout
*
🔒 Private: everything runs in your browser. Nothing you paste is uploaded.
How to use the robots.txt tester
Paste your robots.txt on the left, enter the path or full URL to check, set the crawler (Googlebot by default), and the verdict updates with the deciding directive and the user-agent group that was used.
The longest matching rule wins, not the first one
The most common robots.txt mistake is assuming rules are read top to bottom and the first match applies. Google instead compares every Allow and Disallow in the selected group and keeps the one whose path string is longest, because a longer path is more specific. So Disallow: /wp-admin/ blocks that folder, but Allow: /wp-admin/admin-ajax.php is longer and wins for that one file. This tool measures each pattern, shows which rule was longest, and highlights it in the parsed group so the override is obvious rather than guessed.
Wildcards and the dollar end-anchor change everything
Two special characters control matching. An asterisk * matches any run of characters, so /*?sessionid= blocks every URL containing that query parameter anywhere. A dollar sign $ anchors the end of the path, so /*.php$ blocks page.php but not page.php?id=2, because the query makes the path no longer end in .php. Misplacing either one is how a single line accidentally blocks a whole site. The tester applies both rules exactly as Google does so the real effect is visible before deployment.
The right crawler picks a different group
Robots.txt can hold several user-agent groups, and a crawler obeys only the one most specific to its name. Google ignores the * group entirely if a named group like Googlebot exists, even when the wildcard group has stricter rules. That means a page can be blocked for generic bots yet allowed for Googlebot, or the reverse. Set the crawler field to the exact bot in question, and the tool shows which group was selected and whether it matched by name or fell back to *, so the verdict reflects the right crawler.
Robots.txt tester frequently asked questions
Q1How does Google decide between conflicting Allow and Disallow rules?
Google does not use rule order. Within the matching user-agent group it compares every Allow and Disallow that matches the path and applies the one with the longest path string, since a longer path is more specific. If an Allow and a Disallow match with the same length, the Allow wins. This tester implements that exact logic and shows the deciding directive.
Q2Does robots.txt support wildcards and the dollar sign?
Yes, for major crawlers including Google and Bing. An asterisk matches any sequence of characters, and a dollar sign anchors the end of the URL path. So a rule like Disallow followed by a star, a dot, php, and a dollar sign blocks PHP files but not the same files carrying a query string. The tester supports both characters when computing each verdict.
Q3Does this robots.txt tester fetch my live file from the server?
No. It only analyses the robots.txt text you paste into the box and never makes a network request. That makes it safe for testing draft, staging, or unpublished rules, because nothing you enter is fetched, uploaded, logged, or stored. Everything runs locally in your browser with client-side JavaScript, even with the connection disabled.
Q4Why is my page blocked for one crawler but allowed for another?
Because each crawler obeys only the most specific user-agent group that matches its name, and ignores the wildcard group entirely when a named group exists. A page disallowed under the star group can still be allowed under a named Googlebot group, or the opposite. Set the crawler field to the exact bot you care about, and the tool reports which group it selected.
Q5Does blocking a URL in robots.txt remove it from Google?
Not directly. Robots.txt stops compliant crawlers from fetching a page, but a blocked URL can still appear in results from external links, just without a description. To keep a page out of the index, allow crawling and use a noindex tag or an authentication wall instead. Blocking in robots.txt prevents crawling, which is a different goal from removing a page from the index.
