Robots.txt Generator

robots.txt is a plain text file placed at the root of your domain (e.g. example.com/robots.txt) that tells well-behaved search engine crawlers which parts of your site they may or may not access. It's a request, not an enforcement mechanism, malicious bots can ignore it entirely, so it shouldn't be relied on to keep sensitive content private.

Disallowing a path in robots.txt prevents crawling, but a disallowed URL can still occasionally appear in search results (without a snippet) if it's linked from elsewhere. To reliably keep a specific page out of search results, use a noindex meta tag on that page instead (see our Meta Tag Generator), robots.txt and noindex serve different purposes and are often used together.