Robots.txt Generator
Generate formatted robots.txt directives to guide search crawlers and specify sitemap locations.
Run tool
Create clean directives for web crawlers with this online robots.txt generator. The generator constructs plain-text files that direct how automated search bots explore your site structure. You can specify a targeted user agent such as Googlebot or apply directives to every crawler using the standard wildcard symbol. When full site access is enabled, the tool creates an open rule allowing crawlers to scan all pages. If you disable universal access, you can supply individual path rules line by line to restrict private sections such as administrative panels or internal search queries. Leaving the path field empty while access is disabled produces a global block directive. You can also include an optional crawl-delay value to request a pacing interval between crawler page fetches, as well as an absolute URL to your XML sitemap to assist bot navigation. Review the generated directives and copy the text directly into your root directory.
Instructions
- Enter a specific crawler name in the User Agent field, or keep the asterisk to apply directives to all search bots.
- Choose whether to permit full crawling access with the Allow All toggle.
- If full access is turned off, enter the directory paths you wish to block, placing each path on its own line.
- Add your full sitemap web address to include a sitemap reference for search engines.
- Optionally enter a crawl delay value in seconds to request a rate limit from supporting crawlers.
- Copy the generated plain text and paste it into the robots.txt file located at your domain root.
Examples
- Setting User Agent to * with Allow All enabled produces a file containing 'User-agent: *' and an empty 'Disallow:' line.
- Setting User Agent to Googlebot, unchecking Allow All, and entering '/admin/' and '/checkout/' produces separate Disallow lines for each directory alongside the specified bot header.
- Unchecking Allow All without providing paths adds 'Disallow: /' to block crawling on the entire site.
FAQ
Where should the generated text be placed on a website?
The text should be saved in a file named robots.txt and uploaded directly to the root directory of your website domain.
What happens if Allow All is unchecked without entering any paths?
When full access is disabled and no specific paths are provided, the generator outputs a single Disallow directive for the root slash, requesting that bots avoid crawling the site.
Does this tool verify that entered sitemap URLs are live?
No, the tool inserts the supplied text directly into the sitemap directive without performing live web requests or URL validation.
Do all search engine crawlers support the crawl-delay directive?
Crawler support varies across search providers, as some major engines manage crawl rates through their respective webmaster consoles rather than robots.txt directives.