Earlier this week Google announced the roll-out of its new search infrastructure named “Caffeine”.  Google claim that it provides  “50 percent fresher results” than it’s previous index.

A long time in the coming Google rolled out a public test in August last year against one of their data centers finishing it’s testing in November 2009. By all accounts it is expected that it has been rolled out worldwide to all data centers.

Much like any search engine, when performing a search on Google you’re not actually performing a search of the live web but a snapshot of the internet that known as the index. A search index is “like the list in the back of a book, helps you pinpoint exactly the information you need.”

The previous search index was broken into a number of layers with some layers with it’s main layer taking in the region of two weeks to refresh. Analysing the entire stored snapshot of the web (index) before making those pages available to be searched against .

Due to the ever increasing size of the internet and growth of content including images, videos and real-time updates (such as micro blogging) Google decided to improve the infrastructure to meet the demand of more current and relevant search results.

As opposed to it’s previous method of analysing the entire web before refreshing it’s top layer, Caffeine continuously updates it’s index analysing smaller portions of the web meaning that we can “find fresher information than ever before — no matter when or where it was published” .

For years the two major SEO don’t dos were building your site purely in Flash or using AJAX. Since major search engines such as Yahoo! and Google have received a sanctioned Flash reader from Adobe, they have both improved their results on reading SWF files. The next major battle is crawling AJAX websites.

Many website owners will have had the experience of Google indexing URLs that have only ever appeared in JavaScript so we know that there has always been some support, even though the message in the SEO community has always been that JavaScript is not crawlable.

For the less technical of us, the goals outlined in Google’s proposal for making AJAX crawlable are:

• Minimal changes are required as the website grows
• Users and search engines see the same content (no cloaking)
• Search engines can send users directly to the AJAX URL (not to a static copy)
• Site owners have a way of verifying that their AJAX website is rendered correctly and thus that the crawler has access to all the content

Let’s hope that there is more guidance on crawlability of AJAX than there is for Flash, otherwise many of you will be finding parts of your site that you genuinely do not wish to be crawled and indexed appear in search results.