Search and you will find.

Search engines have improved without you noticing.  How did they do it?  Jim Giles, Ferris Jabr and New Scientist try to find out.

Anyone can publish on the web, but it would be better if some people didn’t; the world does not need another site that provides advice on how to unlock an iPhone or find cheap car insurance.  Now new evidence shows that search engines have upped their game to make sure their results are not dominated by such low-quality sites.

Search engines are meant to pick out high-quality sites amid the sea of knock-offs, but even they get overwhelmed.  As recently as March, for example, the first 10 results from a Google search for “how to organize your desktop” contained nine links to pages churned out by “content farms” –  websites that publish reams of articles, often of dubious quality, that aim simply to attract clicks and advertising dollars.

That prompted New Scientist to ask computer scientist Richard McCreadie at the University of Glasgow, UK, to look into the issue.  The results show that Google and Microsoft have won a major victory in the fight against such content farms.  In the process they may have inflicted serious pain on two organizations often cited as providers of content farm material: Seed, a project from AOL, appears to have stopped commissioning content from freelance writers, though the firm declined to comment.  And Santa Monica-based Demand Media has seen its stock price fall be over 50% since it went public earlier this year for $1.5 billion.

Most of the credit for such changes has been given to Google, which this February announced that it had updated its search algorithm in a bid to prioritize sites that publish original and well-researched material.  It won’t provide details, but many site owners noticed that the update detected and penalized sites that publish multiple near-identical articles, a favorite tactic of content farms.  For example, the traffic flowing from search engines to eHow, a Demand Media site, dropped 20% after the update.  Google has also enlisted users to keep up the fight – it says that it now demotes sites that people choose to exclude from search results.

To test how successful Google and Microsoft’s Bing have been at fending off content-farmed results, McCreadie ran 50 search queries known to be a target of content farmers, such as “how to train for a marathon” in March and August this year.  Then he paid people to examine the results for links to low-quality sites, where “low quality” was defined as uninformative sites whose primary function appears to be displaying adverts.

The results are striking.  In the case of the marathon query, sites that contained lists of generic tips, such as “invest in a good pair of running shoes” were present in the top 10 in March but had disappeared by August, while high-quality sources, such as Runner’s World magazine, now appear near the top.  Similar trends were found throughout the 50 queries.

The story is far from over.  Search engines play such a prominent role in our lives – Google alone handles over a billion queries a day – that companies constantly vie for the top places in the results.  Sites can rank highly by producing authoritative material, but this is expensive, so there will always be those looking for a cheap short cut.  Content farms may be out for now, but with billions of dollars hanging in the balance, it’s just a matter of time before the next battle commences.

# # #

We build strategies and everything that goes with them.

Some of the largest organizations in the world, including many in the mortgage and finance industries, trust us with the most important aspects of their business. From defining clients’ brands and identities to developing ongoing campaigns in a variety of media, we provide the communications and measurement tools to move them forward. Applying our experience and dedication to the media and the message, bloomfield knoble handles every detail of our clients’ strategic marketing initiatives.