Before proceeding let’s know a bit about Why to add or Set Robot.txt file ? The robots.txt file is the key to control the crawlers activity on your website/blog, this helps you to keep some directories of blog out of range of crawling while allowing some, Adding Robots file makes your WordPress blog more relevant as it block un-relevant url from indexing to the search engines, in easy words by adding robots file, your blog commands google bot `Hey Robot ! don’t index my Harkuchh.com/wp-admin that’s not important to index !`
Now let’s see how to add Robots file to your blog, Believe me I’m telling you the best and easiest way to add Robot file to your blog, there are some alternative ways to add robot file via plugin but using much plugins are not good, so here is the simple guidance to add Robots.txt to a blog, it can be added through cPanel, Don’t worry it’s easy, let’s start..
- Go to cPanel
- Click on File Manager under File Section and open Home Directory by clicking Go.
- Click on Public_Html at Left side of the Screen
- Now click on New File from the top of the screen
- Name it as robots.txt and click on Create New File (file create)
- Now click on new robots.txt file
- Then click Edit Icon and then click on `Edit` again
What to add in Robots.Txt file
By adding Allow and Disallow commands in robots file we tell the crawler exactly what to crawl and what to not, so here we Disallow all those thing to crawl which are not important to be on Google or any search engine and which are having very important and critical info about the blog, like it’s directories, various parts and components of blog, the thing which we need to exclude from crawling in a normal WordPress blog are given here..
# disallow all files in these directories
Copy and paste the complete given list of commands to your robots.txt file and just make a small change –add your blog’s URL at the place of mine Harkuchh.com. Don’t ignore to add robots file to your blog because it just not keep your blog’s critical infosafe instead it also helps in better SEO of your blog.
Read this post in Hindi.