Robots.txt Reader

Google removed its tester from Search Console, and this fills the gap. It fetches the current file from any domain and parses it the way a crawler does, grouped by user-agent, without needing the site verified in your account.

It surfaces them visually. Rules that landed under the wrong user-agent, an empty Disallow that permits everything, or a group with no rules at all stand out once the file is parsed into cards. It won't rewrite anything, but seeing the structure is usually enough to spot the bug.

Every user-agent named in the file, whether that's the wildcard, Googlebot, Bingbot, GPTBot, or something obscure. The tool doesn't filter against a known list; it reports exactly the groups the site published.

No. What people casually call robot text is a convention, not access control. Well-behaved bots like Googlebot honor it, scrapers often don't. If you need to genuinely block access, use authentication or firewall rules on the server itself.

Three things: that the file still loads at all, since a 404 here means no rules apply; that no new Disallow line covers pages you want indexed; and that the Sitemap declaration still points at a live URL. A quick robot check after each release takes about ten seconds.

Because consecutive User-agent lines share the rules that follow them. The parser keeps those together on one card, matching how real crawlers interpret the file: Googlebot and Bingbot stacked above a single Disallow both obey it.

Completely. It's a normal fetch of a public file, the same thing your browser does if you type the URL directly. Reading how large sites structure their rules is one of the fastest ways to learn the format.

You'll get a load error, since there's nothing at /robots.txt to fetch. Build one first: you can create robots txt online with the Robots.txt Generator, upload it to your root, then come back here and confirm it parses cleanly.