BUY WEBSITES
BUY DOMAINS
LOGIN
SIGNUP
Robots.txt Generator
A robots.txt generator builds the small text file that tells search engine and AI crawlers which parts of a site they may request, written in the exact User-agent, Disallow, Allow, and Sitemap syntax the Robots Exclusion Protocol expects. One stray character or a misplaced wildcard can quietly block an entire site from Google, so producing the directives from a tool that assembles them correctly is safer than hand-typing them into a live file.
This generator builds the file live as rules are added: stack User-agent groups, drop in Allow and Disallow paths, set an optional crawl-delay, point at one or more sitemaps, and toggle named AI crawlers such as GPTBot and ClaudeBot on or off. It is free and runs entirely in the browser, so nothing typed into it is uploaded, logged, or stored anywhere.
Sitemap lines are emitted once, at the end of the file, as Google recommends.
🔒 Private: everything runs in your browser. Nothing you paste is uploaded.
How to use the robots.txt generator
Start from a preset, edit the rule groups until the live preview reads the way you want, then copy or download the file and place it at the root of your domain.
Start from a preset, then refine the rule groups
The fastest way to a correct file is to pick a preset and adjust it. Allow all emits a single group with an empty Disallow:, the canonical way to permit full crawling; Block all writes Disallow: /; and WordPress default blocks /wp-admin/ while keeping admin-ajax.php reachable so front-end features keep working. From there, add a rule group per crawler you want to treat differently, give each a User-agent name (or * for every bot), and add Allow or Disallow lines. The preview rebuilds on every keystroke so a mistake is visible immediately.
Get the path syntax and ordering right
Paths are case-sensitive and matched from the start of the URL, so /Admin/ and /admin/ are different rules. A trailing slash like /private/ blocks the folder and everything beneath it, while * matches any sequence and $ anchors the end of a URL, for example Disallow: /*.pdf$. When Allow and Disallow rules overlap, modern crawlers obey the most specific match rather than file order, which is why the WordPress preset can disallow a folder yet allow one file inside it. Leave a Disallow path blank to permit everything for that user-agent.
Block AI crawlers and declare your sitemap
The AI section adds a dedicated group with Disallow: / for each crawler you check, including GPTBot, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, CCBot, Bytespider, Amazonbot, and Applebot-Extended; the master toggle flips all of them at once. These directives are honoured voluntarily, so they deter compliant bots rather than enforce a hard block. Finally, list each XML sitemap on its own line. The tool prints Sitemap: entries once at the very end of the file, with absolute URLs, exactly where the major engines expect to find them.
Robots.txt generator frequently asked questions
Q1Where do I put the robots.txt file?
The file must sit at the root of each host it governs, reachable at https://yourdomain.com/robots.txt, and be served as plain text. A robots.txt in a subfolder is ignored. Each subdomain and each protocol needs its own file, so the www and non-www versions, and any staging host, are treated separately. Place the downloaded file there and confirm it loads in a browser before relying on it.
Q2Does robots.txt actually stop a page from being indexed?
No. A Disallow rule asks compliant crawlers not to fetch a URL, but Google can still index a blocked URL it discovers through links and show it without a snippet. To keep a page out of search results, allow crawling and add a noindex meta tag or X-Robots-Tag header instead, because a crawler must be able to read the page to see the noindex instruction in the first place.
Q3Can robots.txt block AI bots like GPTBot or ClaudeBot?
It can request that they stop. Major AI crawlers including GPTBot, ClaudeBot, Google-Extended, and CCBot publish user-agent names and state that they respect robots.txt, so a Disallow rule for those names discourages them from collecting your content. Compliance is voluntary, though, so a bot that ignores the protocol is not stopped by it. For a hard block you need server-level rules such as firewall or user-agent filtering.
Q4Is the Crawl-delay directive supported by Google?
Google ignores Crawl-delay and manages its own crawl rate through Search Console settings instead. Bing, Yandex, and some other crawlers do read it, where the number is the minimum seconds to wait between requests. Add it only if a specific bot is hitting the server too hard, and keep the value modest, because a large delay can slow how quickly your new pages are discovered and crawled.
Q5Does this generator send my rules anywhere?
No. The file is assembled entirely in your browser with client-side JavaScript, and nothing you type into the user-agent, path, sitemap, or AI fields is uploaded, logged, or stored. The Copy and Download buttons work on the text already in the page, so the tool functions offline once loaded, which makes it safe to use for unreleased or confidential site structures.
