When I first started full time using the Internet (2004) to try and make money I would ask lots of questions of my friends. After awhile I started becoming annoying to one particular friend. (Adam) Adam started sending me links to google pages  to read. It was right around then I discovered everything you need to work online is right their on the Internet for free. Google, Yahoo and most search engines provide all the tools you need right on their sites. You just need to look for it.

 I like digging into how search engines crawl through websites and how the Internet works in general. It is very important to understand just happens to a websites once it is live. Live being when it gets its first back link and crawled. Basically when Google or any search engine first crawls a site the bot has different ways to go on your site. So we must provide it with a path to follow to insure all pages get indexed. 

On the Google blog their is a post with a white paper of study done by a intern named Uri Schonfeld. Uri presented his paper named, SITEMAPS: Above and Beyond the Crawl of Duty. Cool name ya? The study is about sitemaps, robot text and how they effect the way a website is crawled. The path Search Engines take on a website.

SITEMAPS: Above and Beyond the Crawl of Duty clearly show the hierarchy of tracking XML Data & Web Data & Session XML Extraction and Crawling. See how Goggles Search Main Pipeline Architecture looks and lead the search engine in the right direction. LINK

We’ve been hearing the term “FRESH” a lot lately. Uri’s paper also explains how Search Engines use sitemaps to keep Search Engine index’s Fresh. Google has always said in the Web Master Guideline to keep content Original, Authoritative and Relevant. We can now include the term “FRESH”. Uri’s paper clearly shows how Search Engines can get bored with a sites content if it’s not FRESH.

A Sitemap is one of the most important parts of a Website. The sitemape is what tells the Bots whats going on with your website or blog. It shows how fresh it is! aswell as hidden pages. Webmasters can find free Sitemap Generating Tools simply by searching Google. WordPress Bloggers can have it easy. My favorite WP Sitemap Plug here. LINK. Once you have a sitemap.xml you will need to upload the file to your server and then submit it to the SE’s.

Read Uri Schonfeld’s study and then see how sitemaps can help your site rank better. Once you read this paper you will see just how search engines work and maybe why your site is or isnt doing good.