Robots.txt Generator

What Is a Robots.txt File?

A robots.txt file is a plain text file placed in your website's root directory that instructs search engine crawlers which pages they are allowed or not allowed to access. It follows the Robots Exclusion Protocol (REP) and is one of the first files that bots check when visiting your site. Configuring it correctly can have a significant impact on your site's crawl budget and overall SEO performance.

Key Directives Explained

User-agent: Specifies which crawler the rules apply to. Use an asterisk (*) to target all crawlers. Disallow: Prevents crawlers from accessing specific paths. Allow: Overrides a disallow directive for a specific sub-path. Sitemap: Points crawlers to your sitemap location. Crawl-delay: Tells crawlers to wait a set number of seconds between requests to reduce server load.

Common Robots.txt Mistakes to Avoid

Never disallow your CSS and JS files as this can prevent Google from rendering your pages properly. Don't use robots.txt to hide sensitive content — it's publicly viewable and doesn't provide real security. Always test your robots.txt using Google Search Console's Robots.txt Tester to verify your rules work as expected before uploading them to your server.

Where to Place Your Robots.txt File

Upload the generated file to the root of your domain — for example, https://yourwebsite.com/robots.txt. Each subdomain requires its own robots.txt file. After uploading, visit the URL in your browser to verify it's accessible and correctly formatted. You can also use Google Search Console to monitor how Googlebot is respecting your crawl rules.