One per line
User-agent: * Allow: / Disallow: /admin Disallow: /api Disallow: /private Sitemap: https://example.com/sitemap.xml
How to use
01
Set the crawler switches
Tick Allow all crawlers to keep the site open, or tick Block AI crawlers to add ready-made rules for GPTBot, ClaudeBot, Google-Extended, and CCBot in one click.
02
List paths and extras
Type one path per line in the Disallow paths box, add your Sitemap URL, and set an optional Crawl-delay in seconds if bots hammer your server.
03
Copy the finished file
Hit the copy button next to Generated robots.txt and upload the text to your domain root, so it answers at yoursite.com/robots.txt.
Why Robots.txt Generator
- Configure crawler rules and this robots txt generator writes valid syntax with no typos.
- One-click block for AI crawlers like GPTBot, ClaudeBot, Google-Extended, and CCBot.
- Set your own disallow paths for a custom robots txt, then copy the whole file in one go.
- Includes the sitemap declaration automatically.
- Free with no limit. A free robots txt generator with no sign up.
Common questions
Does the Robots.txt Generator handle AI crawlers like GPTBot?
Yes. One checkbox appends a block section covering GPTBot, ClaudeBot, Google-Extended, and CCBot, each with its own User-agent group and a Disallow rule for the whole site. You can leave regular search bots untouched while shutting out model training crawlers.
Where does my finished robots.txt file have to live?
At the root of the host, nowhere else. Crawlers request /robots.txt on each hostname, so every subdomain needs its own copy. After you create robots txt online here, upload the result so it resolves at https://yourdomain.com/robots.txt.
Can I block only certain folders in the generated robots.txt?
Yes, that's the whole point of the Disallow paths box. Enter /admin, /api, or any prefix, one per line, and each becomes its own Disallow line. Combined with Allow for the root, this custom robots txt generator setup leaves everything open except what you name.
What does the Crawl-delay field in this robot generator do?
It adds a Crawl-delay line with the number of seconds you enter. Bing and Yandex respect it as a pause between requests, while Googlebot ignores the directive entirely. Leave it empty unless a specific bot is genuinely overloading your server.
Is the Sitemap line in robots.txt actually necessary?
It's optional but cheap insurance. The Sitemap declaration tells any crawler where your URL list lives, including ones you never registered with. The tool appends it automatically whenever the Sitemap URL field is filled in.
Why use a robots txt generator online instead of writing the file by hand?
Because the syntax fails silently. A missing colon or a Disallow typed as Dissallow doesn't throw an error anywhere, crawlers just skip the line. This robot txt file generator writes each directive for you, so what you copy is always parseable.
Will Disallow rules from this generator remove pages from Google?
Not by themselves. Disallow stops crawling, not indexing, so a blocked URL can still appear in results if other sites link to it. To deindex a page, allow crawling and add a noindex meta tag on the page instead.
After I upload the generated robots.txt, what's the quickest way to verify it?
Point the Robots.txt Reader at your domain. It fetches the live file and lays out every group by crawler, so you can confirm the rules you built here arrived at the server intact.