The Webmasters Tool Box

This is an article about Sitemaps and Google Webmaster Tools.

This article tells you why you need a Sitemap, and how Google Webmaster Tools uses the Sitemap, but it will not tell you how to make a Sitemap or how to use Google Webmaster Tools. There are plenty of good articles that explain the mechanics of Sitemaps and Webmaster Tools; but sometimes it is good to understand the big picture, without worrying about details.

In a Nutshell
In a nutshell, when people search the web using Google, you want them to find your Site. But if Google does not have the right information, it will not return links to your Site. So the Google search engine needs to store and access information about your Site.

To better understand this concept, let us look at an example.

Suppose someone is a dog lover. This person wants Google to find links to pages with dogs. More specifically, suppose this person types “Boston Terrier” into the Google search box, and hits return.

So now the Google search engine has two problems to solve. First, it wants to find site with the “Boston Terrier” content. Second, it does not want to return unpopular sites. It wants to return a site people are interested in, otherwise people will stop using Google and start using another search engine.

So the Google search engine really as two problems to solve — content and popularity. This article discusses content. The topic of popularity (better known as “ranking”) can be found in many other articles. Search on keyword SEO (Search Engine Optimization).

How Google Keeps Track of Content
How does the Google search engine keep track of everything on the Web?

Well, the truth is, the Google search engine does not keep track of EVERYTHING on the Web; but it tries to.

Someplace (probably more than one place) Google Corporation has an enormous number of big Server machines with lots of hard drives and backup media, just for keeping lists of what is on the World Wide Web. But the problem (actually opportunity) for Google is that web content keeps changing. Each day trillions pieces of new information are added to the Web.

So the Google search engine is constantly combing through the web in a process called “crawling”.

Here is the definition of “crawling”, according to Google’s Webmaster help page:

“Crawling is the process by which Googlebot discovers new and updated pages to be added to the Google index.”

Google goes on to say the following:

We use a huge set of computers to fetch (or “crawl”) billions of pages on the web. The program that does the fetching is called Googlebot (also known as a robot, bot, or spider). Googlebot uses an algorithmic process: computer programs determine which sites to crawl, how often, and how many pages to fetch from each site.

Going back to our Boston Terrier search example, it is important to understand that every time the Google search engine finds a site with a lot of information about Boston Terriers, it indexes the site under the key phrase “Boston Terrier”.

Another Way Google Tracks Content
As described in the previous section, the Google search engine is constantly looking at sites on the Web to build indices of Web content. But there is still a lot of content that the Google search engine cannot process.

For instance, if you submit a video to YouTube, the Google search engine cannot (yet!) watch the video and create index data based on the content of the Video.

Also, more sophisticated Web sites build their pages dynamically, meaning that pages and content of pages change, depending upon user input. It is difficult or impossible for the search engine to build an index of such a site.

So there is some content that the Google search engine cannot find or index by crawling. Still, the authors of such sites want people to be able to find their content and the operators of the search engines want to be able to find the content. Therefore, Sitemaps were invented.

A Sitemap is a protocol webmasters use to tell search engines about Web pages that are available for crawling. Below is a very simple example of a Sitemap. Again, we leave it to other documents and articles to give detailed technical descriptions. Suffice it to say, the information in the Sitemap helps the Google search engine to find pages on your site.

Submitting the Sitemap
At the beginning of this article, I implied that Sitemaps and Google Webmaster Tools go together like milk and cookies. Here is why — once your Sitemap is ready, you register your Site with Google Webmaster Tools and then submit your Sitemap.

You join Google Webmaster Tools pretty much the same way you join any other Google Application Service. Just go to the Google Webmaster Tools site and sign in with an existing Google Account Name/Password or create a new one. Then follow the instructions for registering your Site and submitting your Sitemap.

If you have followed reasonable rules of SEO (Search Engine Optimization) in creating your site and your site has good quality content then you are off and running. Of course, Google makes different types of reports available. “Demystifying Google Webmaster Tools Reports” is a series of articles to get you up to speed.

Other Search Engines
Google is not the only search engine. Both Microsoft Bing and Yahoo Search have tools for registering your Site and submitting Sitemaps. Check them out too.