Robots.txt and Sitemap Generator
The robots.txt file tells crawlers which parts of a site to skip. The sitemap lists the pages you want found. Together they steer how search engines and AI systems read a site. Since 2023, AI companies publish separate crawler names for model training and for search answers, so a site can allow one and block the other.
Everything above runs in your browser and saves to this device only. Set the paths to keep out of search, choose which AI crawlers to block, list the site’s URLs, and download both files.
How it works
The robots.txt starts with a rule for all crawlers, adds a block rule for each AI crawler you tick, and ends with the sitemap location.
Disallow: /admin/
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xml
The sitemap follows the sitemaps.org protocol: one host, at most 50,000 URLs per file, with an optional last-modified date per URL. Robots.txt is a request that reputable crawlers honour. Private pages need a login to stay private.
Frequently Asked Questions
What is a robots.txt file?
A robots.txt file is a text file at the root of a website that tells crawlers which paths they may fetch. Reputable crawlers honour it. Private pages need a login to stay private.
How do I block AI crawlers in robots.txt?
Add a user-agent group for each AI crawler with Disallow: /. Training crawlers include GPTBot, ClaudeBot, CCBot, and Google-Extended. Answer crawlers include OAI-SearchBot, Claude-SearchBot, and PerplexityBot, and blocking them removes the site from AI search answers.
What is an XML sitemap?
An XML sitemap is a file that lists the URLs you want search engines to find, with optional last-modified dates. It follows the sitemaps.org protocol, holds up to 50,000 URLs per file, and is referenced from robots.txt.
Does robots.txt remove a page from Google?
Only partly. A page blocked in robots.txt can still appear if other sites link to it. Use a noindex tag to keep a page out of search results.
Is this Robots.txt and Sitemap Generator free?
Yes. The Robots.txt and Sitemap Generator is free and opens straight away in any modern browser, including on a phone. Everything you enter is processed in your browser and stays on your device.