Robots.txt & Sitemap Generator
What It Is
This tool generates two essential SEO files for any website: a robots.txt file that controls crawler access to your site, and an XML sitemap that helps search engines discover and prioritize your pages. Both files are created client-side with no server upload, giving you full control over the output before deploying to production.
How It Works
The robots.txt section follows the Robots Exclusion Protocol: each crawler (identified by User-agent) gets a set of Disallow and Allow directives specifying which URL paths are blocked or permitted. The optional Crawl-delay directive throttles aggressive bots, and the Sitemap directive points crawlers to your XML sitemap. The sitemap section builds a valid <urlset> document where each <url> entry contains a <loc> (the page URL), an optional <changefreq> hint, and an optional <priority> value between 0.0 and 1.0.
Worked Example
Robots.txt input: User-agent: * with Disallow: /admin, Allow: /public, and Sitemap: https://example.com/sitemap.xml. Output: A standard robots.txt allowing all crawlers, blocking the /admin path, explicitly allowing /public, and pointing to the sitemap. Sitemap input: One entry with https://example.com/, changefreq weekly, priority 1.0. Output: A valid XML sitemap document ready for upload to your server root.
Common Mistakes
- Using robots.txt for security. Robots.txt only blocks well-behaved crawlers — malicious bots and scrapers ignore it entirely. Do not rely on it to hide sensitive pages. Use authentication or server-side access controls instead.
- Listing pages in robots.txt that you also want indexed. If you disallow a URL in robots.txt, crawlers cannot access it to see
noindextags or other metadata. The page may still appear in search results with missing or incorrect snippets. - Setting every sitemap URL to priority 1.0. When every page has the same priority, the hint is meaningless. Reserve 1.0 for your homepage and highest-value landing pages, and scale other pages down to 0.5–0.8.
- Forgetting to update the sitemap after adding new pages. An outdated sitemap is almost as bad as no sitemap. Regenerate and re-submit your sitemap whenever you publish significant new content.
Frequently Asked Questions
Q:What is a robots.txt file and why do I need one?
A robots.txt file tells search engine crawlers which parts of your website they are allowed or disallowed to crawl. It lives at the root of your domain (e.g., https://example.com/robots.txt) and follows the Robots Exclusion Protocol. Every production website should have one to prevent crawlers from wasting quota on admin pages, duplicate content, staging environments, or internal search results. Without it, search engines will crawl everything they can reach.
Q:What is an XML sitemap and how does it help SEO?
An XML sitemap is a machine-readable file that lists all important URLs on your website along with metadata like last modification date, change frequency, and priority. Search engines use it to discover pages they might miss during regular crawling. This is especially valuable for new sites with few external backlinks, sites with deep content hierarchies, or pages that are only reachable through JavaScript navigation.
Q:Can a robots.txt block a page from appearing in Google?
No. A robots.txt disallow directive prevents crawling but does not prevent indexing. If a page is disallowed in robots.txt but still gets discovered through external links, Google may index it without seeing the content — showing only the URL and a "description not available" snippet. To truly block a page from search results, use a <code>noindex</code> meta tag or HTTP <code>X-Robots-Tag</code> header instead.
Q:Where should I place my robots.txt and sitemap.xml files?
The robots.txt file must be placed at the root of your domain: <code>https://example.com/robots.txt</code>. The sitemap can live anywhere on your site but is conventionally placed at <code>https://example.com/sitemap.xml</code>. You can also specify the sitemap location inside your robots.txt file using the <code>Sitemap:</code> directive and submit it directly to Google Search Console and Bing Webmaster Tools for faster indexing.
Q:What do changefreq and priority values mean in a sitemap?
Changefreq (always, hourly, daily, weekly, monthly, yearly, never) is a hint to search engines about how often a page is likely to change — it is not a guarantee and modern crawlers largely ignore it. Priority (0.0 to 1.0) lets you hint which pages are more important relative to others on your site, with 1.0 being the highest. These are only hints and do not directly influence ranking, but they help crawlers allocate their crawl budget more intelligently.