How to Generate a robots.txt File

Choose the rule inputs

A robots.txt file presents crawler directives for a site, including the user agent addressed by a rule, whether access remains open, which paths receive disallow directives, and whether the text includes a sitemap or crawl-delay entry. The Robots.txt Generator turns these choices into robots.txt text that you can inspect before deciding how to use it on a site.

Begin with the purpose of the rule rather than with the optional fields. The user-agent input identifies the group at the start of the text, and its empty state uses *. The allow-all setting determines whether the result contains an empty Disallow directive or path-based directives. A sitemap value and a crawl-delay value can be added when those lines belong in the intended file.

Prepare each value in advance so the generated lines are easy to compare with your plan. Put separate disallowed paths on separate lines, leave optional fields blank when their entries are not wanted, and keep the distinction between an open rule and a rule targeting selected paths clear.

Build the directive set

1. Open the Robots.txt Generator and identify the rule you want to create.

2. Leave the user-agent field empty to use the default *, or enter the value that should appear in the rule. The generated text begins with a User-agent line containing the supplied value.

3. Leave allow-all enabled when the result should contain an empty Disallow directive. This option starts enabled.

4. For selected disallowed locations, turn allow-all off and enter one path per line in the disallow-paths field. For example, put /private on one line and /draft on another. Each line is trimmed, and blank lines are ignored. If no usable path remains while the option is off, the result contains Disallow: /.

5. Enter a sitemap value when that entry belongs in the text. Non-whitespace input adds a blank line and a Sitemap line using the supplied value.

6. Enter a non-whitespace crawl-delay value when that line is needed. The result adds a Crawl-Delay line using the supplied value.

7. Generate the text and compare its displayed lines with your intended user agent, access choice, paths, sitemap value, and crawl-delay value. Resolve any later decision about using the text on a site separately from creating it.

Check what the lines mean

Treat the result as generated robots.txt text, not as a prediction of what a crawler will do. The opening line identifies the supplied user-agent value, or * when the optional field was left empty. An enabled allow-all choice appears as an empty Disallow directive. A disabled choice produces one Disallow directive for each usable path that remains after blank lines are ignored.

The fallback Disallow: / has a specific meaning within the displayed result: allow-all was disabled, but no usable disallow path was retained. A nonblank sitemap value adds a separate Sitemap line after a blank line, while a nonblank crawl-delay value adds its corresponding line. The output therefore reflects the values entered and the documented whitespace handling.

The reviewed behavior does not establish URL validation, numeric validation, standards compliance, crawler enforcement, or a guaranteed change to crawling or indexing. Read values and line order carefully before using the text elsewhere, especially when an empty field or whitespace-only entry was involved.

Worked example

You want a Googlebot rule that excludes two draft-related locations while adding a supplied sitemap value to the generated text.

Enter Googlebot, turn allow-all off, place /private and /draft on separate lines, enter the descriptive sitemap value sitemap entry, leave crawl delay blank, and generate the text. Check the opening line, both path directives, the blank-line separation, and the sitemap line.

The result begins with a User-agent line for Googlebot, followed by Disallow: /private and Disallow: /draft. After a blank line, it contains a Sitemap line using the entered sitemap value.

Limitations

• The documented behavior covers text generation and line handling, but it does not establish input validation, crawler enforcement, standards compliance, or a guaranteed crawling or indexing outcome.

Common errors

• If allow-all is disabled and the path field contains only blank lines, those entries are ignored and the result uses Disallow: /. Add a trimmed path on its own line when you intend to target selected locations.

FAQ

What happens if I leave the user-agent field empty?

When the user-agent field is empty, the optional input uses *, and that value appears in the opening User-agent line. A supplied value appears there instead, so inspect the first line after generation.

How do I generate disallow rules for specific paths?

When allow-all is enabled, the result includes an empty Disallow directive. After you turn it off, each usable nonblank path becomes a separate directive; without a usable path, the result uses Disallow: /.

Can I add a sitemap and crawl delay to the output?

When either field contains non-whitespace text, its corresponding Sitemap or Crawl-Delay line is added. The documented behavior does not establish validation or crawler enforcement for those supplied values.