robots.txt Guide for SEO: Rules, Examples, and Common Mistakes

A robots.txt file tells participating web crawlers which parts of a site they are allowed or discouraged from crawling. It is simple to create, but a single broad rule can hide important site sections from crawling.

For developers and site owners, the safest approach is to keep the file understandable and test it whenever routes or deployment rules change.

Where robots.txt lives

The standard location is at the root of a host:

https://example.com/robots.txt

A rule on one hostname does not automatically define behavior for every other hostname or subdomain.

Use Duck Cloud's robots.txt Checker to read and summarize a public site's directives.

Basic syntax

A simple file might look like:

User-agent: *
Disallow: /admin/
Disallow: /internal/
Sitemap: https://example.com/sitemap.xml

User-agent identifies the crawler group. Disallow specifies paths that should not be crawled by matching crawlers.

Keep rules deliberate. A pattern that looks harmless can affect more URLs than expected.

The dangerous rule: Disallow everything

This rule blocks participating crawlers from the entire site:

User-agent: *
Disallow: /

It is sometimes used on staging sites, but it becomes a serious production problem if copied during deployment.

After launching a new site or changing hosting, always verify the live robots.txt.

robots.txt is not access control

A crucial security rule: robots.txt does not protect private information.

The file itself is public. Anyone can read the paths listed inside it. Sensitive admin routes, customer documents, private APIs, and internal files need real authentication and authorization.

Do not treat Disallow as a password.

Crawling and indexing are different concepts

Blocking crawling does not necessarily mean a URL can never appear in search results. Search engines may discover a URL through external or internal links even when they cannot crawl its content.

For pages that must not appear in search, use the appropriate indexing controls and make sure the crawler can access the page long enough to see them. For private content, use authentication.

Do not block assets your pages need

Modern crawlers render pages using CSS, JavaScript, images, and other resources. Broadly blocking asset directories can make it harder to understand the rendered page.

Before blocking a directory such as /assets/, /static/, or framework-generated paths, confirm what it contains.

Use specific rules

Prefer narrow rules over broad rules.

Instead of:

Disallow: /api

understand whether that could also match paths you did not intend to block.

Document why each blocked path exists. That helps prevent old rules from surviving after the original feature is removed.

Add your sitemap location when appropriate

A Sitemap directive can point crawlers toward your XML sitemap:

Sitemap: https://example.com/sitemap.xml

Make sure the sitemap URL itself works and returns the correct content type and response status.

Use the Website Status Checker or HTTP Header Checker when validating the sitemap endpoint.

Check production after every major migration

robots.txt problems often appear during:

• staging-to-production deployments;
• domain migrations;
• framework rewrites;
• reverse-proxy changes;
• CDN rule changes;
• multilingual route changes;
• CMS migrations.

After launch, retrieve the live file from the public hostname instead of checking only the repository copy.

Common robots.txt mistakes

Accidentally blocking the whole site

Usually caused by a staging configuration reaching production.

Blocking CSS or JavaScript

This can reduce a crawler's ability to render the page correctly.

Using robots.txt for sensitive data

The path remains public and may still be discoverable.

Forgetting a subdomain

www.example.com, example.com, and app.example.com may each need their own configuration.

Keeping obsolete rules

A blocked directory may no longer exist, while a new route that should be blocked is missing.

Assuming it fixes duplicate content

URL canonicalization, redirects, canonical tags, and site architecture are separate concerns.

robots.txt deployment checklist

1. Open the live /robots.txt.
2. Check it with the robots.txt Checker.
3. Confirm the intended production hostname.
4. Look for Disallow: /.
5. Review blocked framework and asset paths.
6. Confirm the sitemap URL works.
7. Verify important public pages are crawlable.
8. Protect private content with authentication, not crawler directives.
9. Repeat the check after major deployments.

A good robots.txt file is usually boring. Keep it small, specific, documented, and aligned with the routes that actually exist on your live site.