Before proceeding let’s know a bit about Why to add or Set Robot.txt file ? The robots.txt file is the key to control the crawlers activity on your website/blog, this helps you to keep some directories of blog out of range of crawling while allowing some, Adding Robots file makes your WordPress blog more relevant as it block un-relevant url from indexing to the search engines, in easy words by adding robots file, your blog commands google bot `Hey Robot ! don’t index my Harkuchh.com/wp-admin that’s not important to index !`
Now let’s see how to add Robots file to your blog, Believe me I’m telling you the best and easiest way to add Robot file to your blog, there are some alternative ways to add robot file via plugin but using much plugins are not good, so here is the simple guidance to add Robots.txt to a blog, it can be added through cPanel, Don’t worry it’s easy, let’s start..
- Go to cPanel
- Click on File Manager under File Section and open Home Directory by clicking Go.
- Click on Public_Html at Left side of the Screen
- Now click on New File from the top of the screen
- Name it as robots.txt and click on Create New File (file create)
- Now click on new robots.txt file
- Then click Edit Icon and then click on `Edit` again
What to add in Robots.Txt file
By adding Allow and Disallow commands in robots file we tell the crawler exactly what to crawl and what to not, so here we Disallow all those thing to crawl which are not important to be on Google or any search engine and which are having very important and critical info about the blog, like it’s directories, various parts and components of blog, the thing which we need to exclude from crawling in a normal WordPress blog are given here..
Sitemap: http://Harkuchh.com/sitemap.xml
Sitemap: http://Harkuchh.com/sitemap-image.xml
User-agent: Mediapartners-Google
Disallow:
User-Agent: *
Allow: /
User-agent: *
# disallow all files in these directories
Disallow: /cgi-bin/
Disallow: /blog/wp-admin/
Disallow: /blog/wp-includes/
Disallow: /blog/wp-content/plugins/
Disallow: /blog/wp-content/themes/
Disallow: /blog/wp-content/upgrade/
Disallow: /blog/page/
disallow: /blog/*?*
Disallow: /blog/comments/feed/
Disallow: /blog/tag
Disallow: /blog/author
Disallow: /blog/trackback
Disallow: /blog/*trackback
Disallow: /blog/*trackback*
Disallow: /blog/*/trackback
Disallow: /blog/*.html/$
Disallow: /blog/feed/
Disallow: /blog/xmlrpc.php
Disallow: /blog/?s=*
Disallow: *?wptheme
Disallow: ?comments=*
Disallow: /blog/?p=*
Disallow: /blog/search?
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/
Disallow: /page/
Disallow: /*?*
Disallow: /comments/feed/
Disallow: /tag
Disallow: /author
Disallow: /trackback
Disallow: /*trackback
Disallow: /*trackback*
Disallow: /*/trackback
Disallow: /*.html/$
Disallow: /feed/
Disallow: /xmlrpc.php
Disallow: /?s=*
Disallow: /?p=*
Disallow: /search?
User-agent: Googlebot-Image
Allow: /wp-content/uploads/
User-agent: Adsbot-Google
Allow: /
User-agent: Googlebot-Mobile
Allow: /
Copy and paste the complete given list of commands to your robots.txt file and just make a small change –add your blog’s URL at the place of mine Harkuchh.com. Don’t ignore to add robots file to your blog because it just not keep your blog’s critical infosafe instead it also helps in better SEO of your blog.
Read this post in Hindi.
Hey, I'm
very informative content i wil do it today
thanks buddy………!!!!!!!!!!!!!!
Yeah Vipin this is really important but many people don’t know about this so don’t forget to share it on all social platforms, thank you.. :)
Please keep updating these kind of posts. These are helping me a lot.. thanks for the information. :)
Nice posts … Sir
isn’t that the all the content will be blocked???
No our posts will not be blocked neither our Photos and only these things are supposed to be on Google for best and long term results.. :)
i was waiting for this post.will u please help me to improve my website rank..my website is pappuappu.com
very useful post sir…..
sir , i worked as u say in your this post , but their is no effect in robots comments on my site,maximum comments , i got from robots, i also installed captcha plugin in wp , but their is no effect on robots comments.
give me a solid solution.
Sir Robots.txt file is not for stoping spam comments :) it’s for giving robots a direction and for stoping unusual comments u must use Akismet as well, and for help just check my Akismet post in Blogging category.
If we want to block some certain page on my website what should i write in the robots.txt file
Just add the url of that post with Disallow :)
What is the difference between xml sitemaps and robots.txt file? Is there any default robots file present in the file manager , before we add our own ?
xml sitemap is to pin Google that there is a new post over your blog where as Robots.txt file tells up what to scroll and what to not to index on Google page for visitors and there is not any default Robot file we need to create it .