Spiders and stop words.
When a spider or crawler encounters stop words, they will immediately leave your website. Information they gathered from the website won't be saved in their database. Meaning your website will not be indexed. If your website is already indexed in a searched engine, the crawler will return to check for updates, however if it encounters stop words when it returns, your potential rankings in the search engine could be adversely effected.

So what are what are spiders ? When you enter an inquiry at a search engine site, it is checked against the search engine's index of all the web pages it has analyzed. Leading url's are then returned as hits, with the best results at the top.

Spiders are also called "crawlers", "knowledge-bots","knowbots" or"bots". Spiders are used by search engines to crawl the Web via the Internet,in a methodical, automated manner. They visit websites scan databases, and maintain the search engine database keeping web pages up to date. Spiders acquire new pages, update known pages, and delete obsolete ones. The data acquired is then incorporated into the "home" database. Crawlers can also be used for automating maintenance tasks on a web site, such as checking links or validating HTML code. Also, crawlers can be used to gather specific types of information from Web pages, such as harvesting e-mail addresses (usually for spam).

Web spiders/crawlers are one type of "bot", or software agent. Generally it begins with a list of URL's to visit. When it visits these URL's, it identifies all the hyperlinks in the page and adds them to the list of URL's to visit, browsing the Web according to a set of standards.

Spiders crawling the Web
Spiders or crawlers scour the web looking for specific types of information. Spiders need to be fed a set of "seed" pages to be able to do their work. Seeds are the addresses of web pages that the spider visits first. Spiders analyze each page they land on, searching for links to other pages. Should a spider find a new link, it goes to that page, looking for more links. Given the right set of seed pages eventually a spider can in theory visit every public page on the Internet.

Search engines like Google would have no way of finding information for you were it not for spiders. Google's spiders constantly roam the web, searching for pages to put in Google's database. When the user enters a search, Google reviews its database for pages containing the words you typed. Pages that appear in the search results had to first be found by a Google spider.

Just as you might expect, spammers also use spiders! Spammer spiders, also called spambots, look for e-mail addresses targeting them for spam. The best way to avoid spammers, is to keep your e-mail address off the web then spam spiders can't find it. If you absolutely need to post your address there are ways to hide it from spambots without preventing people from corresponding with you. To prevent the spambots from finding your address, there are methods like email obfuscation and scrambling work. These methods change email addresses into forms that spambots can't recognize. There are a number of ways to hide your email addresses using javascript. How do you decide which to use? Use all of them or you could also write your own! Why? Eventually spammers will identify the most popular method used to hide addresses; designers of email harvesters will incorporate detection of the method identified to circumvent the script.

A very basic example is:

<script language=javascript>
< !--
document.rite("no spam" + "@" + "yoursitename.com");
//-->
< /script>

As with all things there is a disadvantage to this method: visitor's will not be able to see the address if their browser does not support Javascript.

Supplementary Spiders and Deep Crawlers
All spiders have the ability to locate pages to add to their web page indexes, even when those pages have not been submitted to them. But some crawlers are better than others. Some search engines are more likely to do a "deep crawl" and gather many pages from your web site, even if these pages were never submitted. Generally the larger a search engine's index is, the more likely it will list many pages per site.

Search engines do not, nor should they, rely entirely on the "deep crawl" because it only occurs approximately on a monthly basis. If they did so, their indexes would become outdated quickly in the rapidly growing Web. To stay current, Google launches various supplemental fresh crawls. These crawlers skim the Web more shallowly and frequently than the deep crawl. The purpose of these supplementary spiders is not to update the entire index, rather just to freshen it by updating the content of some sites. Google does not divulge its fresh-crawling schedules or targets, but through close observation Webmasters at least have an indication of the crawl's schedule.

Deep crawl and fresh crawl are widely used terms in the online marketing field. These terms are used to distinguish between the thorough spidering of the Web and various intermediate crawls such as those run by Google.

For the most part, targets are determined by automatic processes built into the spider's programming, but we cannot ignore the human element in the process which also directs the spider to specific destinations.