Smart Traffic SEO Tip #81 – Robots.txt: Is it Still Useful?
Last year we posted about dealing with duplicate content issues and we discussed the use of robots.txt file as an option when dealing with duplicates created by pages like product search results. Search Engines like Google are continuously evolving so is this still applicable for 2011?
Yes, it still is but fewer websites are using it, preferring the use of 301 redirects and Canonical tags. This is because it affects the flow of link juices to the site’s pages. Each case is different but generally Robots.txt is the last option for duplicates where you cannot use 301 redirection, canonical tag, meta noindex tag or even the parameter handling tool from Google Webmaster Tool (GWT).
For very big websites, it is sometimes hard to keep track of duplicates and some pages are not even that important. In this case, you can use robots.txt file. It also comes in handy when blocking generated duplicate pages that you have no control of.
To summarise, robots.txt file still has some advantages. You just need to make sure that you know and check carefully the pages you are blocking.
