A Reminder to Webmasters: Check your Robots.txt File’s Health
Googler John Mueller posted on his Google+ profile a reminder to webmasters. He said, “…things to keep in mind when working on a big website: If you have a giant robots.txt file, remember that Googlebot will only read the first 500kb”. He also added, “If your robots.txt is longer, it can result in a line being truncated in an unwanted way.”



