With Google (11%) and Yahoo! (4%) leading,  today's search engines are responsible for directing 18% of all traffic to e-commerce & classified sites. Is it high time to take a re- look at Search Engine Optimization?

In very simple terms, a Search Engine comprises of a suite of software that decides what is "appropriate" (or "inappropriate") for inclusion in the result set of any keyword search.

The speed & completeness of each search depends on the underlying database on which the search is based. This is a humongous effort of constantly crawling through trillions of web pages and applying rules to decide whether to include their links, index them (based on keywords found) & even rank them. 

Over the past few years, a number of companies have sprung up offering what is called "Search Engine Optimization" (SEO). These firms try to improve the visibility and rank of web pages in web search results - by deploying what can be termed as "insider knowledge" about the logic each search engine uses to include/exclude links that it encounters during its crawling process.

Again, it is reasonable to assume that all search engines will not deploy the same logic - therefore all search engines are not in total agreement about what is acceptable and appropriate for inclusion.

Manipulating The System
Most of the work of SEOs involves first understanding & then manipulating the system for the benefit of their clients. In the process, some exercises cross the border into the gray area. Over a period, it is these transgressions that have given the SEO business a "not so clean" reputation.

The SEO’s job is to impress the web crawler software to include his client’s URLs at a relatively higher position. To do so, it must understand the logic of the various web crawler softwares. And simulate a scenario that will make its clients' web pages more "acceptable".

One of the simplest (& earliest!) tricks was to repeatedly re-submit a web-site to boost relevancy. Today, most search engines can track such patterns, ignore multiple re-submissions, and even flag such a site as a spam site!

Another early trick was to set up "gateway" web pages & "dummy" domain names which point to certain URLs. This is considered as "inappropriate" by most major search engines. (As of now, Google for one, senses such activities and is likely to remove such sites entirely from its index.)

Another method is to create "shadow" domains (owned by the SEO or a 3rd party) that re-direct users to a site by using (what can only be called) deceptive methods.

Then there is "invisible text", "invisible links" and "non-compliant HTML" to manipulate relevancy. For example, use of multiple titles (which goes against HTML standards) to increase relevancy. Or using a title that does not reflect the content of the page. Or using hidden elements ("invisible" to crawler software) to reveal links to the crawler software at runtime.

There is also the technique of machine generated code (where specific code is written to produce keyword specific "gateway" pages) - pages which are automatically optimized for specific search engines.

And mirrored content techniques - by which web-page content is duplicated on different domains, just to inflate relevancy!

Caveat Emptor
There can be no doubt of the importance of SEOs, who use their expertise to enhance "visibility" of  their clients' web-pages. But, because the techniques involve methods to work around the logic of search engine software, manipulating existing rules, there are chances of transgression into gray areas. Which in turn, could lead to loss of accountability as well as reliability.

What is heartening to note is that over a period of time, more & more of such activities are being deemed as "inappropriate" by the entire search engine industry.