How do Search Engines find Web Pages?

Different methods are used by different search engines, but programs using the variations of the spider method have thus far proved the most effective at indexing large quantities of content and regularly updating these included pages and discovering new ones.

Spider methods refer to the search engine sending out electronic detectives referred to as spiders to find web pages and classify them. Search engines create an index of the web pages it has “spidered”. That index is in constant flux, being updated, hourly, daily, weekly and monthly depending upon the ability of the spiders to detect a new web page or a change in a previously indexed web page.

One of the most immediate goals of search engine marketing is to get the pages one wishes to drive traffic to included in the indices of the major search engines. Two years ago, Yahoo was offering results from Google’s index. This gave Google’s search technology the brand recognition that Yahoo had been building for years.

Today, these industry leaders compete for the eyes and clicks of potential consumers by building the most complete index of the web and properly determining which minuscule chunk is the one we are looking for when we enter a few choice words.

Read more: Being Search Engine Relevant

How do Internet Search Engine companies make money?

Like many businesses, search engines have more than one source of revenue and are working to develop new revenue streams to solidify their profitability. Advertising, lead generation and sponsored web page positioning are the top three source of revenue for the majority of search engines. In the future, paid for content in the form of subscriptions to valuable or timely information may provide a new source for search engine revenue.

One of the most popular forms of paid advertising is pay-per-click (PPC) or cost-per-click (CPC). This is a type of lead generation model where the web site owner pays the search engine each time someone clicks on the web site owner’s ad and is referred to the owner’s website.

PPC/CPC has become the primary revenue stream for Google, accounting for 90% of its revenue since they launched their pay-per-click advertising model in February 2002. Businesses use a bidding model to determine how highly they rank in “sponsored listings”, the clearly marked advertisements that are displayed above, below and to the side of natural or organic results from the search engine’s index.

Generating advertising revenue through Pay-Per-Click, or PPC, or Cost-Per-Click, or CPC, earned Google $9.288US million in 2004. Since visitors primarily choose their next destination with a click from the page they are viewing, portal sites like MSN™ and Yahoo were built to deliver traffic to advertisers.

Another type of advertising involves graphical banner advertisements where advertisers are interested in the number of visitors that the website displaying the banner tracks on a regular basis. These are known as impression based advertisements as in each visitor that loaded the page was given the impression the banner suggests. Banner ads are often paid for by a set monthly fee determined by the size and placement of an advertisement.

This has evolved as more websites seek to build traffic from a variety of sources. Rather than pay a set monthly fee, advertisers were willing to pay a premium for placement and pay only for traffic delivered to their website. The company most successful with this model was Overture™, who successfully sued Google for infringing on Overture’s patented Pay-Per-Click technology in 2004. The $328US million settlement paid to Overture in the form of Google stock did little to dampen Google’s operations, cash-laden as they were from one of the most successful initial offerings of public stock in recent years.

Google has announced future services that may have content charges associated with them therefore opening another potential revenue stream. The content Google has talked about so far are access to University library materials from several well recognized organizations, specifically Stanford University, Harvard University, the University of Michigan, the University of Oxford and the New York Public Library.

Read more: How do Search Engines find Web Pages?

The Tools of the Search Engine Marketing Trade

Search Engine Marketing, or SEM, refers to the determination of the appropriate steps to take to achieve the highest possible visibility in search engine results pages (SERPs). The first step has been to determine what keywords are relevant to the searchers interest and the web owners’ offering.

SE Marketers use tools to see the words search engine users use to search. The number of user searching for a given keyword phrase is also in flux, as people using search engines become more sophisticated in the art of searching. However, since there is an ever increasing supply of inexperienced web users, the glut of traffic on very common, general search terms will remain a pot-of-gold for e-businesses now and in the future. SE Marketers try to help web site owners avoid using keywords that may describe their offerings accurately but are unlikely to be words that searchers will use.

Next, one must “re-invent” their web pages in order to make them the best possible result for the search words that are expected to draw search engine users interested in their product or service. This may involve writing new copy for the web pages, creating a new page to discuss keywords and how they relate to their business in depth, or completely reorganize websites to allow search engine software to accurately evaluate the website’s relevancy to a group of keyword phrases.

SEM may be referred to as the research needed to develop an intelligent search engine focused marketing strategy. Search Engine Optimization, or SEO, is then the actions taken to optimize websites for the search engines so the client website will obtain the best possible positioning in terms of relevancy and importance.

Read more: What does it mean to have 3PRIME in your corner?

Being Search Engine Relevant

In all search activities whether library book classification systems, data managements systems or even a filing cabinet, the function of searching is not independent from the function of being searched. If you want someone to find your ad in the classified section of the newspaper or Yellow Pages you want to do things to stand out.

Internet Marketing has changed nothing in this regard. Web page owners can do things to help search engines find them and evaluate their relevancy and importance. The only difference is that the web page owner is attracting computer programs that are highly secret in their operation. Web page owners are not attracting human beings with the current knowledge we have of how humans look for what they want, and evaluate whether they think the result relevant to their search.

The secondary concern, not to be overlooked, is the conversion of a visitor towards your desired action whether that is an email address entry, catalog request, an information form completion or a purchase.

The most common method to becoming noticed and positively evaluated by a search engine spider is to use keywords on a web page that will help the spider know what it has found and classify it accordingly. But identification is just one dimension. Search engines are very sensitive to importance. After all they sell their service based upon search engine result relevancy and comprehensiveness. Search engines want the searcher to experience the best possible result. If they fail in this even once, that user is very likely to try another search engine, delivering traffic and associated ad revenue to a competitor.

With the internet quickly becoming the world wide locator for everything from restaurants, books, articles, schools, parts, clothing and much more, being found on the internet is a marketing advantage comparable to putting a store in the right shopping center and hanging out the most attractive shingle.

Big money is spent on the right commercial real estate location and big money is starting to be spent on getting the right internet location. Of course, the difference is that internet locations are not static but change with every search so web site owners need to work constantly to make sure to gain and maintain the best possible position.

Read More: Search Engine Marketing (SEM)

Who is on top in the Search Engine Business?

Measuring the greatest search engine depends upon what you think makes a search engine great. Public opinion would indicate that is the greatest boasting almost 82 million unique users each month spanning the globe. Google is consistently regarded as delivering the most relevant results. They certainly make an effort to make their search service applicable to web enabled businesses through search term advertising.

Perhaps their greatest asset is their drive to index more and more useful information each week. Since mid-2004, Google has launched a series of new projects including the scanning and indexing of university libraries like Stanford University and finding ways to offer video results for searches conducted concerning television programming.

Google is working hard because other deep pocketed corporations like Yahoo! ™ and Microsoft™ are pushing more capital towards developing their search services. Yahoo!, throughout the dot-com bubble, maintained their profitability by expanding their search engine from a service for the digitally literate, to a consumer product advertised at major sporting events and during prime time television.

Yahoo! continues to develop consumer friendly services such as Local search, geared towards supplying the searcher with the contact information of a business near the physical location they are searching from.

Microsoft Corporation has taken a new interest in the potential revenue of search. As a recognized industry leader in computer technology, a long history of profitability, and immense brand recognition, Microsoft has gone from offering search results from other search engine’s indices to developing its own search algorithms and creating its own index of web pages. With that accomplished, Microsoft will undoubtedly try to win search engine market share from the current leader, Google.

Measuring market share is difficult because the same people may use a variety of search engines to discover the answers to the questions they seek. Having watched the incoming traffic of my clients and seeing where they receive their traffic from, I feel confident providing a few statistics. The following numbers are the percentage of visitors that are referred by each search engine:

Google: 30-60%
Yahoo: 5-20%
MSN: 5-20%
AOL: 3-15%

The ranges are given to illustrate that the amount of traffic entering your website might have everything or nothing to do with search engines. Since our goal is to maximize keyword qualified traffic from search engines, these are the results that a website optimized to receive search traffic might expect to receive. Keep in mind that there are many other referrers besides search engines both on and off the web that drive traffic to websites.

For example, businesses might receive the majority of traffic to their website through offline marketing efforts such as direct mail with their website location as part of the mailing, television advertisements promoting their website location, or simply using existing brand names as their website location as in™. Other examples of traffic generators other than search engine traffic include industry specific directory listings, online yellow pages listings and links from sites that would like their audience to benefit from the information, products and services offered on businesses websites.

Another noteworthy quality enhancing Google’s status as the leader in their pack is their commitment to global relevance. Fully 50% of traffic on comes from outside the United States. What Google has done to segment both their traffic and their massive index of web pages into regionally focused indexes is supply mirrors of their homepage search form. Two such examples are and, the Canadian and Hong Kong mirrors, respectively.

In both cases, the purpose is to cater to the fact that what users in one part of the world may find most relevant may not be as useful to users in other areas.

Read more: How have Internet Marketing Techniques changed Internet Marketing?

What is an Internet Search Engine?

A search engine is a website that uses a series of algorithms, or specific computer programs, to find a selection of the most relevant websites based upon a search criteria. Most search engine users rely upon simple keyword searches to find web pages about their topic of interest.

Depending on the purpose or ability of the search engine, users may enter a combination of keywords as their search criteria. The word keyword has been adopted for a variety of purposes but in the context of this article, a keyword is any single term such as “car” while a keyword phrase is a more specific phrase made up of more general keywords, such as “new car loan”. As the internet grows in size and in variety of uses, search engines continually change to find web pages faster, and deliver the most relevant search results to the searcher.

A website is made up of web pages which is what we all view when using a client browser such as Netscape, Internet Explorer or Mozilla FireFox. With the number of web pages growing by the day, the best estimate available as to the size of the internet is the number of web pages contained in the largest search engine’s index. When a search engine performs a keyword search, it must have a set of possible results. We refer to this as its index. Google™ currently boasts the largest search engine index, proudly announcing on their homepage that, as of this writing, they are searching over 8 billion distinct web pages.

This is not, by any means, the full extent of the Internet as companies like Google hasten to break new ground in areas like indexing telephone directories and public information. Even with this “limited” set of possible search results, finding the page or pages you want fast and with more competitive relevancy is the mission of search engines. If a search engine is not know for delivering relevant results when searched, the engine will have no users conducting searches and generating traffic to resulting websites. It is the massive amounts of traffic delivered from search engine results that generates revenue for businesses who in turn seek to earn more potential customers from search engine queries.

Because of the popularity of Search engines and their applications for E-Business, new ideas involving or catering to search engines are making their appearance on the web every week.

Read more: Who is on top in the Search Engine Business?

3PRIME Products

As we are interested in bringing our flavor of customer interaction and service excellence beyond our core services, 3 Prime will also be applying our methods to our own e-commerce storefront.

Some of the products we are considering are:

2010 3PRIME Press Releases

3PRIME Builds New Website

E-mail Print PDF

NORTH HAVEN, Connecticut (PRBuzz) July 9, 2010 — 3PRIME is building a new website for Hamden, Connecticut based Yankee Appraisal & Title Services, LLC. The new website will enable customers to easily order services online using a credit card or PayPal account at Yankee Appraisal & Title Services specializes in residential real estate appraisals, title searches, deed recording, and flood determination…

Read the rest of the release here

CT Web Solutions Company 3PRIME Partners with Spectrum Marketing

E-mail Print PDF

(PRBuzz) July  16, 2010 — The Spectrum Marketing Group, a family owned business located in New Bedford, Massachusetts, has entered into a strategic partnership with 3PRIME. The Spectrum Marketing Group has over twenty years of experience working in marketing, advertising, publishing and direct mailing. Recently, they received a new face for their online relations thanks to a site redesign by 3PRIME…

Read the rest of the release here

Spectrum Marketing Group

Spectrum and 3PRIME Partner to Develop Web Real Estate

We’re happy to announce that 3PRIME has recently partnered with the Spectrum Marketing Group, a family owned business located in New Bedford, Massachusetts. The Spectrum Marketing Group has over twenty years of experience working in marketing, advertising, publishing and direct mailing. And thanks to a site redesign by 3PRIME they’ve received a new face to their online presence.

It’s a mutually beneficial partnership that the Spectrum Marketing Group sees as an opportunity to provide their clientele with additional services, “Even if someone already has a website, 3PRIME can help develop, or redevelop the existing site to be more effective, user friendly and optimized for search engine results.”

At 3PRIME we understand that choosing partners is the key to business development. Spectrum’s experience with the Real Estate Book makes them a natural choice for us to expand our service to the real estate industry. We’re thrilled by the prospect of working on future ventures with Spectrum and other like minded companies.

Case Study: Cleanup, online since 1997, has consistently provided quality leads to private attorneys and law firms by way of their online visibility. Their search engine pay-per-click (PPC) advertising campaign and their natural search engine rankings bring constant traffic to their homepage search function, allowing visitors to freely search their database of listed law offices through a 3 step process, resulting in a short list of legal category specific lawyers that can advise on specific areas of law and provide legal services to the visitor. Perhaps the most important quality of their online attorney search is that visitors can designate their region, city or state, being as locale specific as they choose to find a local or regional a practice as they need or desire.

In August 2005, due to algorithmic changes by Google, the leading search engine both in terms of daily searches and traffic delivered to websites listed in their mammoth database of potential search results, disappeared from Google’s index. Along with losing their natural search ranks, Google no longer reported that the AttorneyFind domain existed in their database of active websites. While was still online, and had never ceased to function properly, the search giant no longer acknowledged their existence or experience in the market.

We became aware of this when we were contacted by AttorneyFind’s website administrator to provide an explanation and solution to their quandary. What follows is a report of what we determined to be the explanation for their disappearance, what we did to resolve this issue and what resulted from our actions.

What was wrong?

Google provides some explanation of what consists of acceptable and unacceptable activities and content for websites that want to be included in their online database of potential search results. These Webmaster Guidelines have remained fairly consistent over the years and can be reviewed here:

AttorneyFind had in the past worked with what, at the time, were the standard for search engine optimization practices. The additions they made to their website were intended to enhance their homepage as a prime destination for web users searching for law firm contact information. We analyzed the massive index of pages that had been generated to improve AttorneyFind’s search engine rankings and determined that they were now in violation of Google’s guidelines. Specifically, the domain contained over 1000 pages created solely for the benefit of search engine software called spiders. Here are the 2 most relevant quality guidelines to AttorneyFind’s position:
1) Avoid hidden text or hidden links.
2) Don’t load pages with irrelevant words.

AttorneyFind’s homepage contained 30 hidden links targeting 25 pages generated to use Google’s algorithmic appreciation of anchor text, or clickable link text, to AttorneyFind’s advantage. The pages that were linked to from the homepage and the pages they in turn linked to were loaded with irrelevant words and links that in the past may have earned a prominent place in Google search results for their relevant keywords and keyword phrases, but were now detrimental to their very inclusion in Google’s search index. 3Prime was glad to quickly address this situation and make a concerted effort to regain AttorneyFind’s valuable search engine real estate.

What we did

AttorneyFind needed to do all of the following, which we split into 3 distinct phases.

Phase 1: Site Cleanup – Rid the domain of pages violating Google’s Guidelines.
1) Remove the hidden links from the homepage
2) Remove the 1000+ pages that violated Google quality guidelines
3) Remove comment tags containing keywords and keyword phrases
4) Remove unnecessary keyword density in HTML attribute tags
5) Replace hidden links with visible navigation text links

Phase 2: Search Engine Optimization – Optimize the website by replacing removed pages with pages that could be used to provide informational content to visitors and search engines.
1) Convert HTML pages to ASP for easier site maintenance and future enhancement
2) Consolidate JavaScript, which is not read by search engine spiders to supporting pages for improved spiderability
3) Ensure no broken links or images
4) Create 2 press releases to alert search and news engines to AttorneyFind’s revision
5) Generate an XML formatted Google-friendly SiteMap to alert Google to the removed pages and locations of new pages

Phase 3: Submission – Submit AttorneyFind’s new, cleaned up website to Google for re-inclusion using Google’s new SiteMap generation and submission function (Beta).
1) Submit the XML formatted document we created designating all the pages of to be included in Google’s search engine index
2) Submit and distribute the 2 press releases as widely as possible.

It took 76 hours over 3 days to complete these tasks but all proved worthwhile.

What results were achieved

Our tasks were completed Friday by the close of business at 5pm EST. On Sunday, Google’s spiders revisited the homepage and included it in their index of potential search results. had earned back many of its natural search positions, with some reduction in ranking such as #1 to #3, and now had much better website architecture to move forward with in the future.

Interestingly enough, Google still reported many of the pages that were removed as being a part of their index. As the site is respidered, we expect these pages to be removed from Google’s index, and be replaced with the well-formed, search engine friendly pages that we created so that the website would be in a good position for future improvement.

Using search engine position monitoring software, we checked for AttorneyFind’s presence in all of Google datacenters and were proud to see that each Google datacenter had the same results. Our goal of returning to the Google fold was achieved.

View our rankings results online at:
AttorneyFind Cleanup

Update: October 2, 2005 is once again #1 in Google for their most important, highest traffic term, “Attorney”.

About 3PRIME

3PRIME, with operations based in New Haven, CT, offers search engine marketing services to small and medium sized businesses. Consisting of 3 experts, 3Prime can help your business earn prime visibility in the major search engines through a process of website goal analysis, strategic planning, aesthetic graphic design and layout modifications, advanced web programming and link popularity building.