How to Edit Robots.txt on WordPress

Posted on 18. Jul, 2009 by Isaac Yassar in WordPress

robotsAre you looking for a way to edit the robots.txt of your WordPress powered blog (own domain publishing), the easy way?

You see, robots.txt is important part of your site because it handles how search engine bots work in their indexing process. Such as what to and not to index, who can and cannot index, etc. When a crawler bot comes to your domain, it will look for your robots.txt first. Afterward it will do its job based on your rules on the text file, which location is on your root domain (like www.your-website.com/robots.txt).

By default, WordPress automatically generate a robots.txt in case a bot needs it. However, the text file is actually non-existent. It is only a result of an internal WordPress script. If you look into your domain storage, you won’t find any file namely robots.txt there. Trust me, I’ve tried it.

Therefore, manual uploading of a robots.txt file (from your hardisk) is not the natural way here. There’s got to be some other way.

Yes there are, using plugins. Many WordPress plugins out there help you in customizing your robots.txt. Some are simple, some are sophisticated. Now the question is, which one?

KB Robots.txt is really the answer. It is easy to use and gets right to the business. After installing and activating it, you just need to edit your robots.txt from your WordPress Dashboard >> Settings >> Robots.txt. It also has a simple but useful guide (samples), in case you are new to this robotic thing.

To download the plugin right now, click here.

I have tried some sophisticated plugins before, but unfortunately, they usually offer advanced settings (that don’t really matter), while forgetting the simple but important ones. I spent much time on those plugins but I couldn’t get what I wanted. Thank God, I finally found the right plugin to help me deal with my robots.txt fast and let me concentrate on building a better blog.

So, what do you think? ;-)

Tags: plugin, robots.txt, WordPress

17 Responses to “How to Edit Robots.txt on WordPress”

  1. djupi

    18. Jul, 2009

    Thanks for the info. I don’t know that Wordpress automatically create robots.txt.
    I’ll try this plugin.

    Reply to this comment

  2. Isaac Yassar

    18. Jul, 2009

    @djupi
    You’re welcome my friend. Glad I can inform it to an IT staff, now I feel smarter ;-)

    Reply to this comment

  3. Tech-Freak Stuff

    05. Sep, 2009

    I knew about it ,but not about the plugin. I hope using the plugin will be helpful in editing the robots.txt eventually helping for better SEO.

    Reply to this comment

    • Isaac Yassar

      06. Sep, 2009

      Now Robots Meta plugin also offer simple editing of robots.txt . It can be great an alternative to KB Robots plugin.

      Reply to this comment

  4. Your Guide

    05. Oct, 2009

    I was searching this for a long time. As you said I didn’t find any robots.txt file in my hosting. I would use plugin now. Thanks a ton.

    Reply to this comment

  5. Karen

    07. Nov, 2009

    Thank you so much for this great review! I just found out about robots.txt it was just tech-eeze to me…now I am paying attention! thanks!

    Reply to this comment

  6. Isaac Yassar

    09. Dec, 2009

    You’re welcome folks. It’s my pleasure to share this information to you.

    Reply to this comment

  7. angrezy

    15. Dec, 2009

    thanks for the plugin i was searching for this for a while.

    Reply to this comment

  8. Luca

    22. Jan, 2010

    I love you! :)

    Reply to this comment

  9. Tommy Smith

    20. Feb, 2010

    I was just searching my files because the Google Webmaster Tools was showing it wasn’t allowed. I found your site, installed the plugin, and fetched the site as googlebot…….Success!

    Thank you so much for the information.

    Reply to this comment

  10. karl

    31. Mar, 2010

    hey thanks for the tips!

    Reply to this comment

  11. Mahtab

    27. Apr, 2010

    Thanks for your great contents, from long time I was suffering from same things that’s robots.txt file I did not found in my root directory but as address of url shows. I was very upset that there was my sitemap.xml was disallow.

    Now I am going to install your suggested plugin.

    Warm regards,

    Reply to this comment

  12. David

    02. May, 2010

    I used to think I had a fair bit of programming skill, but I am in awe of seeing some cool programming in action — I think it’s cool that sometimes a robots.txt file doesn’t exist as a discrete file, but is generated as needed by php.

    Reply to this comment

  13. Paul

    20. Jun, 2010

    Great advice! I installed a blog with the “opt out” to reference content and then changed my mind. This plugin is great.

    For text to add though- try:

    http://www.askapache.com/seo/updated-robotstxt-for-wordpress.html

    Reply to this comment

  14. Djinny

    13. Jul, 2010

    Thank you so much for this post! That plugin is just what I needed.

    Reply to this comment

  15. Lisa

    28. Jul, 2010

    This plugin is essential to any website, I was confused about Robot.txt but your explanation and examples were what I needed.

    Reply to this comment

Leave a Reply