WordPress tools

Robots.txt Generator

Generate a WordPress-optimized robots.txt — block sensitive WP paths, point crawlers to your XML sitemap and protect crawl budget in one click.

Block /wp-admin/ Block /wp-includes/ Sitemap directive Yoast / Rank Math / AIOSEO Allow admin-ajax.php Copy & download
Get started free Sign in

Free · No credit card · 50 credits/day

What gets included and why

🚫

Block /wp-admin/

Prevents crawlers from indexing admin pages. Keeps crawl budget for public content. Note: admin-ajax.php is explicitly allowed as plugins require it.

🔧

Block /wp-includes/

WordPress core PHP, JS and CSS files have no SEO value. Blocking prevents crawl budget waste on hundreds of internal core files.

🗺️

Sitemap directive

Adds Sitemap: https://yoursite.com/sitemap_index.xml so all crawlers (not just Googlebot) know where to find your XML sitemap.

🔌

SEO plugin sitemap URLs

Pre-configured sitemap paths for Yoast SEO (/sitemap_index.xml), Rank Math (/sitemap_index.xml), AIOSEO (/sitemap.xml) and WP core (/wp-sitemap.xml).

🔒

Block sensitive files

Dissuades crawlers from requesting /wp-login.php, /wp-cron.php and /xmlrpc.php — file paths that should not appear in search indexes.

✅

Allow public content

Correctly configured Allow: / and Allow: /wp-content/uploads/ ensures your posts, pages and media remain fully crawlable and indexable.

Frequently asked questions

What should a WordPress robots.txt file contain?

Allow all crawlers to access public content (User-agent: * / Allow: /); block /wp-admin/ (except /wp-admin/admin-ajax.php); block /wp-includes/; block /wp-content/plugins/ and /wp-content/themes/; block /xmlrpc.php, /wp-login.php, /wp-cron.php; include a Sitemap: directive pointing to your XML sitemap. Place the file at example.com/robots.txt (root domain).

Does robots.txt actually protect sensitive WordPress files?

No. robots.txt is a convention, not a security mechanism. It instructs well-behaved crawlers which URLs to skip but does NOT prevent access. For actual security: block wp-admin/ in .htaccess, block xmlrpc.php, use HTTPS, set wp-config.php permissions to 640. Use robots.txt for crawl budget management and sitemap declaration only.

Where is the robots.txt file in WordPress?

WordPress can serve a virtual robots.txt (no physical file — generated in memory). Yoast, Rank Math and AIOSEO can edit this via Settings → SEO → Tools → File Editor. A physical file at /public_html/robots.txt takes precedence. SEO plugins may not be able to edit a physical file — use FTP/SFTP or your hosting file manager.

Should I block wp-content/uploads/ in robots.txt?

No. This directory contains images, PDFs and media files. Blocking prevents Google Image Search indexing and can prevent Google from rendering your pages correctly. Only block specific subdirectories within uploads if you have private document directories. Use X-Robots-Tag HTTP header for fine-grained control over specific file types.

Related WordPress tools

More tools for WordPress SEO and security.

.htaccess Generator

Server-level access rules that actually block sensitive files (unlike robots.txt).

 Login Exposure Checker

Verify that wp-login.php and xmlrpc.php are not exposed.

 Slug Generator

Generate clean, SEO-optimized URL slugs for your WordPress posts.

Protect your crawl budget in one click

Free account. 50 credits per day. Access to 75+ tools instantly.

Create free account →