Robots.txt Tester

A robots.txt file is a plain text file placed at the root of a website (e.g. https://example.com/robots.txt) that provides instructions to web crawlers about which pages or sections of the site they are allowed or not allowed to access. It follows the Robots Exclusion Protocol. While well-behaved crawlers (Googlebot, Bingbot) respect robots.txt, it is not a security measure — the file is publicly accessible and malicious bots may ignore it.

Disallow in robots.txt prevents crawlers from accessing (fetching) a URL. Noindex is a meta tag or HTTP header directive that allows crawlers to fetch the page but instructs them not to include it in search results. Disallow is more aggressive — if Googlebot cannot crawl a page, it also cannot read any noindex tag on it. If you want to remove a page from search results, noindex is preferred; Disallow keeps the page from being crawled but does not guarantee de-indexing of already-discovered URLs.

Yes — robots.txt supports per-crawler rules using the User-agent directive. Use User-agent: * to apply rules to all crawlers, or specify a bot by name (e.g. User-agent: Googlebot, User-agent: GPTBot) to apply rules only to that bot. Rules for a specific User-agent take precedence over * rules for that crawler. You can also use User-agent: GPTBot with Disallow: / to prevent AI training crawlers from accessing your content.

Robots.txt itself has no direct impact on page speed. For SEO, robots.txt can affect crawl efficiency — blocking irrelevant pages (admin areas, search result pages, session URLs) from being crawled preserves crawl budget for your important content. However, blocking CSS and JavaScript files that are needed to render pages can hurt how well Googlebot renders and understands your site. Always test that blocking rules do not inadvertently prevent crawlers from accessing resources needed for proper page rendering.