What is the recommended robots.txt file for WordPress?

A standard WordPress robots.txt includes: `User-agent: *`, `Disallow: /wp-admin/`, `Allow: /wp-admin/admin-ajax.php`, and a link to your XML sitemap.

Why must /wp-admin/admin-ajax.php be allowed in WordPress?

Many WordPress themes and plugins use `admin-ajax.php` to load dynamic frontend elements, content tabs, and interactive widgets. Blocking it prevents Googlebot from rendering the full webpage accurately.

Should I block /wp-includes/ and /wp-content/plugins/ in robots.txt?

No! Modern search engines render pages like a browser. Blocking `/wp-includes/` or `/wp-content/` prevents Googlebot from loading essential CSS stylesheets and JavaScript files.

How do I edit robots.txt in WordPress without FTP?

You can edit your virtual robots.txt directly inside the WordPress dashboard using plugins like Yoast SEO (Tools > File editor) or RankMath (General Settings > Edit robots.txt).

Does WordPress create a robots.txt file automatically?

WordPress dynamically generates a basic virtual robots.txt file if no physical file exists at the root. Creating a custom physical or plugin-managed file allows you to add custom disallow rules and sitemap directives.

Why should /wp-admin/ be disallowed in WordPress robots.txt?

Disallowing `/wp-admin/` prevents search crawlers from wasting time indexing login panels, admin dashboards, and internal system files that have no search value.

How do I link my Yoast or RankMath sitemap in WordPress robots.txt?

Add `Sitemap: https://yourdomain.com/sitemap_index.xml` at the bottom of your `robots.txt` file.

SEO Tools Cluster

WordPress Robots.txt Generator (Optimized WP Crawler Rules)

Generate production-ready robots.txt rules optimized for WordPress, WooCommerce, Yoast, and RankMath.

TL;DR: WordPress Robots.txt Generator builds tailored crawler rules for WordPress sites, allowing crucial AJAX endpoints while disallowing admin folders and cart sessions.

WordPress Robots.txt Generator

WP Best-Practice Rules

WordPress Site URL

SEO Plugin / Sitemap Type

Included WordPress Directives:

Disallow: /wp-admin/ (Protects admin backend)
Allow: /wp-admin/admin-ajax.php (Keeps AJAX frontend functional)
Allow: /wp-content/uploads/ (Allows indexing media)
Keeps CSS & JS assets unblocked for Google mobile rendering

Generated WP robots.txt

What Is the Recommended Robots.txt Configuration for WordPress Websites?

The recommended robots.txt configuration for WordPress disallows the `/wp-admin/` backend folder while explicitly allowing `/wp-admin/admin-ajax.php` so search engines can execute frontend JavaScript functions. For WooCommerce sites, it also disallows `/cart/`, `/checkout/`, and `/my-account/` to preserve crawl budget.

How to Use the WordPress Robots.txt Generator

Our WordPress Robots.txt Generator is engineered for rapid SEO workflow, high-performance in-browser execution, and complete data privacy.

Select your SEO plugin (Yoast SEO, RankMath, All in One SEO, or Core WP Sitemap).
Toggle WooCommerce protection if your site runs an online store.
Confirm the default allow exception for `/wp-admin/admin-ajax.php`.
Input your custom WordPress sitemap URL (e.g., `https://example.com/sitemap_index.xml`).
Copy the generated text into your WordPress SEO plugin's file editor or physical root directory.

Key Capabilities & SEO Best Practices

Instant In-Browser Execution: Fast, zero-latency analysis and tag generation right inside your browser.
Complete Data Confidentiality: Work with unpublished article drafts, keywords, and metadata safely on your local device.
Standards-Compliant Output: Generates clean, Google-approved HTML tags, XML protocols, and Schema.org JSON-LD markup.

100% In-Browser Privacy Guarantee

All SEO data analysis, text parsing, tag generation, and metrics calculation happen entirely on your device inside your browser using client-side JavaScript. None of your keywords, unpublished drafts, URLs, or metadata are ever transmitted or stored on any server.

Frequently Asked Questions

: A standard WordPress robots.txt includes: `User-agent: *`, `Disallow: /wp-admin/`, `Allow: /wp-admin/admin-ajax.php`, and a link to your XML sitemap.
: Many WordPress themes and plugins use `admin-ajax.php` to load dynamic frontend elements, content tabs, and interactive widgets. Blocking it prevents Googlebot from rendering the full webpage accurately.
: No! Modern search engines render pages like a browser. Blocking `/wp-includes/` or `/wp-content/` prevents Googlebot from loading essential CSS stylesheets and JavaScript files.
: You can edit your virtual robots.txt directly inside the WordPress dashboard using plugins like Yoast SEO (Tools > File editor) or RankMath (General Settings > Edit robots.txt).
: WordPress dynamically generates a basic virtual robots.txt file if no physical file exists at the root. Creating a custom physical or plugin-managed file allows you to add custom disallow rules and sitemap directives.
: Disallowing `/wp-admin/` prevents search crawlers from wasting time indexing login panels, admin dashboards, and internal system files that have no search value.
: Add `Sitemap: https://yourdomain.com/sitemap_index.xml` at the bottom of your `robots.txt` file.

Cite This Tool

Referencing this SEO tool in a digital marketing case study, research paper, or guide? Copy a citation below.

NexLove.org. (2026). WordPress Robots.txt Generator [Software]. https://nexlove.org/tools/wordpress-robots-txt

“WordPress Robots.txt Generator.” NexLove.org, 2026, nexlove.org/tools/wordpress-robots-txt.

NexLove.org (2026) WordPress Robots.txt Generator. Available at: https://nexlove.org/tools/wordpress-robots-txt (Accessed: 2026).

@misc{nexlove_wordpress_robots_txt,
  title = {WordPress Robots.txt Generator},
  author = {{NexLove.org}},
  year = {2026},
  url = {https://nexlove.org/tools/wordpress-robots-txt}
}

Embed This Tool

Add the live WordPress Robots.txt Generator to your own website with this lightweight responsive iframe:

<iframe src="https://nexlove.org/embed/wordpress-robots-txt.html" width="100%" height="650" style="border:1px solid #e2e8f0;border-radius:12px" title="WordPress Robots.txt Generator — NexLove.org" loading="lazy"></iframe>

Related SEO Tools & Utilities

Free Robots.txt Generator

Meta Robots Tag Generator

Browse all SEO Tools