Using Statistics to Detect Search Engine Spam

Using Statistics to Detect Search Engine Optimization Spam
An example of an application of statistical methods to detect web spam is presented in the paper "Spam, Damn Spam and Statistics" by Dennis Fetterly, Mark Manasse and Marc Najork from Microsoft. They used two sets of pages downloaded from the Internet. The first set was crawled repeatedly from November 2002 to February 2003 and consisted from 150 million URLs. For each page the researches recorded HTTP status, time of download, document length, number of non-markup words, and a vector indicating the changes in page content between downloads. A sample of this set (751 pages) was inspected manually and 61 spam pages were discovered, or 8.1% of the set with a confidence interval of 1.95% at 95% confidence.
Another set was crawled between July and September 2002 and comprises 429 million pages and 38 million HTTP redirects. For this set the following properties were recorded: URL, URLs of outgoing links; for the HTTP redirects - the source and the target URL. 535 pages were manually inspected and 37 of them were identified as spam (6.9%).

The research concentrates on studying the following properties of web pages:

URL properties, including length and percentage of non-alphabetical characters (dashes, digits, dots etc.). *
Host name resolutions.
Linkage properties.
Content properties.
Content evolution properties.
Clustering properties.

SEO - How Cloaking Sabotages SEO

One of the biggest newbie mistakes made by people first trying out search engine optimization on their websites is to try and cloak it. Most people don’t even realize that they are practicing cloaking when they do it and even perceive it as a legitimate form of Search Engine Optimization but the problem is that the big search engines will penalize and blacklist your site for doing it.

In a nutshell, cloaking is the practice of using keywords and metatags to show a search engine spider a different page then what your human visitor to your site is going to see. The purpose of doing this is to drive visitors to a site. An example would be a psychic site that uses the name of famous psychics such Sylvia Browne or John Edwards to promote its services even though they sell no books or services by those famous psychics who have high keyword popularity. Some high-ranking pages are cloaked simply to prevent others from stealing the underlying code but this is no longer such a problem as it is not just keyword content or metatags that the search engine algorithms look at nowadays when they go to index a site.

SEO - The Downfall of SEO Content

You can search engine optimize your website or blog all you want with keyword rich copy but if the information you are providing is stale, repetitive, robotic or plagiarized then you are not likely to attract many visitors. Furthermore most search engine spiders and robots will penalize you for plagiarizing and rewriting content from elsewhere by not including your pages in its search engines.

Another reason that your SEO content could be failing is not necessarily that you have committed the sin of plagiarism or being out of touch, but you may have chosen to search engine optimize copy for a product that has a glutted market. If a lot of people are writing about the product then it means you have failed to properly define your niche market in terms of how it will make a profit. Trying to make a profit in a glutted market is often useless and it is also a bane to SEO marketing as the keywords that are used to market glutted products are often too general and too popular to bring you the targeted customers that you need.