Definition: Search Engine Spiders
Tuesday, January 8th, 2008By: Gabriel Gervelis
What is a search engine spider (robot)?
When researching search engine optimization knowing what a search engine spider is, and what they do is very important. A search engine spider is a computer program used by search engines to find and index web pages.
Think of a search engine spider as a pawn in a chess game. A spider is a search engines information gatherer as a pawn is the solder in the game of chess.
What The Search Engine Spiders Do:
The search engine sends the spiders out into the internet to research web pages. Spiders are programmed by the search engines to bring back bits of information about the web page.
Search Engine Spiders are also know as:
- Crawlers
- Agents
- Bots
- Robots
Examples of “Bits of Information”
- Main words within the webpage/website (Keyword density)
- Main words are in the webpages/websites title
- Text is used in the links (if any) that point to the webpage/website
The spiders are programmed to find hundreds of bits of information within the webpage/website. This gathered information lets the search engine know how to index the webpage/website. Based on point values given to each bit of information, the search engine can determine what webpage ranks where on the result listing.
Other Sources of Information about Search Engine Spiders:
Search Engine Spider Simulator - This tool lets you see what a search engine spider sees when it comes to your webpage.
Search Engine Spider - Taming the beast’s definition of a search engine spider with additional information.
Google’s Source - Google’s source of definitions of a search engine spider.
Good luck with your SEO ventures!
