Googler John Mueller posted on his Google+ profile a reminder to webmasters. He said, “…things to keep in mind when working on a big website: If you have a giant robots.txt file, remember that Googlebot will only read the first 500kb”. He also added, “If your robots.txt is longer, it can result in a line being truncated in an unwanted way.”

With all the many updates that Google has rolled out in the last month (e.g. Search Plus Your World, Page Layout Algorithm), which has caused quite a fuss and not a tiny bit stress to online marketers and the search optimization community, we sometimes get so preoccupied with the big changes that we tend to forget a few basics. It is a good thing that there are proactive Googlers out there who remind us of these things. Anyhow, since the subject of Robots.txt limitations has been brought up, we thought it necessary to have a refresher post covering this topic.

Robots.txt Specifications

Here are some basic things about the Robots.txt file as per Google’s specifications:

• The robots.txt file must be in the top-level directory of the host. Crawlers will not check for robots.txt files inside subdirectories.
• The URL for the robots.txt file is case-sensitive. You should always name it as lower case “robots.txt”.

Below image shows example of valid robots.txt URLs.

Image Courtesy of Google

• The expected file format is plain text encoded in UTF-8. If a character encoding is not a subset of UTF-8 or is not compliant to its standard, this may result in the contents of the file being parsed incorrectly. In this case we have to be mindful of the text editor being used.
• Google currently enforces a file size limit of 500KB. Content after the maximum limit will be ignored. Note that the size limit may vary across various crawlers.
• The “disallow” directive specifies paths that must not be accessed by the designated crawlers while the “allow” directive specifies paths that may be accessed. When no path is defined, the directive will be ignored.
• When specifying a sitemap or sitemaps to your robots.txt file, the absolute or full URL must be used. However, the sitemap URL does not necessarily have to be on the same host.

The robots.txt file may appear to be a simple tool but is actually a powerful and useful addition to one’s site since it can be essential for the purposes of SEO. However, if not done properly, this could result in your site being removed from the face of Google. So go ahead and check your robots.txt file’s health.

To learn more as to how Google handles the robots.txt file, check out this link: http://code.google.com/web/controlcrawlindex/docs/robots_txt.html

Google has launched a change in algorithm that looks at the layout of a webpage bringing in factors highlighted in user research . This improvement calls the attention of webmasters who have placed excessive ads above the fold on their sites. It yet another update to the algorithm which values visitor experience over profit, which may also show some correlation to the quality of the on-page content.

What has been improved on this update?

Google uses a visual system, much like the one that shows image previews in SERPs. This time, the algorithm update is focused on looking for signs of an ad, penalising websites with too many above the fold. It should affect 1% of searches globally.

How will these improvements affect SEO?

(1) If you have already been following the guidelines in Google’s Panda update last year, this improvement will no longer be threatening on your part since one of the key points there was to take a specific look at the abusive use of ads.

(2) From SEO’s perspective, the updated algorithm serves as housecleaning tool which improves the user’s search engine experience by bringing up the quality content on the top of the search results while the putting ad-loaded sites beneath.

(3) According to Matt Cutts: “This algorithmic change does not affect sites that place ads above-the-fold to a normal degree, but affects sites that go much further to load the top of the page with ads to an excessive degree or that make it hard to find the actual original content on the page”. This means that having ads on the site will not harm your webpage; they just don’t want you to overdo it, especially for content appearing above the fold.

For your convenience, Google has provided browser size tool to show you how your website appears in various screen resolutions to help you measure the scale of your ads in your site.

Move Forward & Take Some Actions
(1) Check out the site preview via Google SERPs to see one possible way Google might be judging how much content is above the fold. (2) Review your content and try to see if any of the headers can be reduced to bring more content above the fold. (3) Make sure to embed ads into relevant content below the fold. Take note this might also improve the CTR & ad engagement levels of your site. (4) Just to be safe, consistently check the traffic of your pages if they are still stable and dynamically increasing its position.

Overall, the best advice for publishers and website owners is to continue focusing on delivering the best possible user experience on your websites.

Google’s unveiling of  “Search Plus Your World”  last week signifies a new look SERP for 2012. This integration of Google Search with Google+ data means that if you are signed into a Google Account, your results will now be personalised with updates, posts, photo, video and other media from within your circles.

The search results page has introduced a new toggle on the upper right of the search results page where you can see what your search results look like.

At a single click you can see the personalised view of your search results. If toggled to No Personal Results, it means that Google will no longer display results from your friends, no private information and no personalization of results based on your Web History will be shown.

With Search plus Your World enabled you can do the following:
Personal Results – allows you to find personal content shared to you by people who matter. This includes blog entries as well as Google+ photos and posts. Both your own and those shared specifically with you will be visible on your results page.

Profiles in Search – allows you to immediately find the people whom you are close to or interested in following. You can even connect with them instantly from Google SERPS.

People and Pages – allows you to connect directly to Google+ with people who frequently discuss the topics you are interested in and gives you the chance to interact with them.

So how is this new innovation changing SERP appearance? Just take a look at these cool features.

Green arrow shows that there are “50 personal results and 419,000 other results” that have been found. Then some of those 50 results will be combined into the first page of results being shown.

Yellow arrow shows that when the button with a person icon is shown dark, it means results are being personalized.

Purple arrow shows the new toggle that Google has introduced. It allows you to switch between personalized and non-personalized results.

So What Does Search Plus Your World mean for SEO?

This is yet another major step towards a completed personalised search experience, following in from the introduction of Google+1. This might be great news to users, as it moves towards individuals seeing results that they trust, recognise, and favour – but for SEOs and webmasters it means that generating organic traffic may become not just about rankings and more about the overall marketing and onsite experience. More and more, social media optimisation will be an important factor in reaching out to and retaining visitors. This also moves towards a SERP landscape where understanding and utilising Google Products can give you a significant advantage.

We’ll be keeping track of Search Plus Your World over the coming weeks to give you further information on it’s impact to SEO.