How To Create Robot Txt File For WordPress

1. Create a Robots.txt File Manually

If your website does not already have a robots.txt file, you can easily create one using a basic text editor.

1. Open Notepad (Windows) or TextEdit (Mac). Ensure the file is saved as plain text.
2. Add these standard WordPress rules:

   User-agent: *
   Disallow: /wp-admin/
   Allow: /wp-admin/admin-ajax.php
   Sitemap: https://yourwebsite.com/sitemap.xml

   • User-agent: * – Applies the rules to all search engine bots.
   • Disallow: /wp-admin/ – Prevents bots from crawling your administrative backend.
   • Allow: /wp-admin/admin-ajax.php – Ensures AJAX functionality remains accessible (essential for many themes).
   • Sitemap: – Points search engines to your XML sitemap for efficient indexing.
3. Save the file exactly as robots.txt.
4. Upload the file to your website’s root directory (usually public_html/) using an FTP client like FileZilla or through your hosting cPanel > File Manager.

2. Edit an Existing Robots.txt File

If a robots.txt file already exists in your root directory, you can modify it directly to update your crawling preferences.

1. Log in to your cPanel or connect via FTP.
2. Navigate to the root directory (public_html/).
3. Locate the robots.txt file, right-click, and select Edit.
4. Input your changes and save the file to apply them immediately.

3. Test Your Robots.txt File

After uploading or editing your file, it is vital to verify that it is configured correctly.

• Browser Check: Visit https://yourwebsite.com/robots.txt in your browser to confirm the file is live and readable.
• Google Search Console: Use the Robots.txt Tester under Settings in Google Search Console. This tool identifies errors or warnings and shows you exactly how Googlebot perceives your rules.

4. Advanced Robots.txt Rules

For more granular control, you can add specific instructions to block folders or files:

• Block a specific folder:

  Disallow: /private-folder/

• Block a specific file:

  Disallow: /secret-page.html

• Allow everything (use with caution):

  User-agent: *
  Disallow:

Final Thoughts

A properly configured robots.txt file ensures that search engines focus on your most important content while ignoring administrative or private files. Always double-check your syntax, as a small error can accidentally block your entire site from search results.