Generate a robots.txt file to control which parts of your site search engine crawlers can access.
Google AdSense Banner
This area will contain advertisements after approval.
Google AdSense Banner
This area will contain advertisements after approval.
robots.txt is a plain text file placed at the root of your domain (e.g. example.com/robots.txt) that tells well-behaved search engine crawlers which parts of your site they may or may not access. It's a request, not an enforcement mechanism, malicious bots can ignore it entirely, so it shouldn't be relied on to keep sensitive content private.
Disallowing a path in robots.txt prevents crawling, but a disallowed URL can still occasionally appear in search results (without a snippet) if it's linked from elsewhere. To reliably keep a specific page out of search results, use a noindex meta tag on that page instead (see our Meta Tag Generator), robots.txt and noindex serve different purposes and are often used together.
Say you set User-agent to *, list /admin and /cart as disallowed paths, leave allowed paths blank, set your sitemap URL to https://example.com/sitemap.xml, and set a crawl-delay of 10. The output is: User-agent: *, Disallow: /admin, Disallow: /cart, Crawl-delay: 10, then a blank line followed by Sitemap: https://example.com/sitemap.xml, note each path automatically gets a leading slash added if you forget it.
Assuming a Disallow rule keeps a page out of search results entirely, it only blocks crawling, a linked disallowed URL can still surface (without a snippet). Use a noindex meta tag on the page itself if the goal is to keep it out of search results.
Relying on robots.txt to hide sensitive or private content, it's a publicly readable file and a request, not an access control, well-behaved crawlers respect it but nothing enforces it, so it should never substitute for real authentication.
Forgetting that most crawl-delay directives are ignored by Google (which paces crawling automatically), it's mainly honored by some other search engines and bots, don't rely on it to solve a Google-specific server load problem.
It must be placed at the root of your domain, accessible at https://yourdomain.com/robots.txt, robots.txt files placed anywhere else are not recognized by crawlers.
The asterisk (*) is a wildcard meaning the rules apply to all crawlers. You can add separate User-agent blocks with different rules for specific crawlers if needed.
Google AdSense Banner
This area will contain advertisements after approval.