On its Bing It On site, Microsoft claims people prefer Bing to Google search by almost 2 to 1. Bing the other one, say the academics. Freakonomics binged it on and wasn't happy. (Credit: Screenshot by Chris Matyszczyk/CNET) Googling is one of those habits. You know, like flossing in the living room or changing your underpants every other day. Yet Microsoft's search engine has Binged on and on that it is, in fact, better. Its own Web site insists that real people prefer Bing's results by a ratio of almost 2 to 1. Such a claim tends to invite skepticism. The vast minds at Freakonomics decided to take their skepticism to another level -- the one of actually trying to replicate the Bing results. Their expectations were that people might not be able to tell the difference between the two searches. Their findings were quite searching. Using a similar sample size to Microsoft's own claimed research (around 1,000), Freakonomics enlisted the help of four Yale law students -- people of obviously unimpeachable integrity. The next step was, just as with the Bing It On Challenge, to perform five searches and see which results respondents preferred. In the experiment, Freakonomics found that 53 percent chose the Google results, 41 percent chose Microsoft's, and 6 percent just couldn't make their minds up. This might be summarized as not exactly 2 to 1. And not exactly in favor of Bing. Freakonomics didn't freak out. It tried some other experiments to see if it could find parameters along which there was a clear preference for Bing. It found none. It found a clear preference for Google or, at best, a tie. In conclusion, it described Microsoft's claims as "fishy." More Technically Incorrect Drone crashed, almost hit me, says Manhattan businessman First Samsung Galaxy Gear, Note 3 ad: Annie Lennox, puppet love GPS gets elderly couple stuck in mud for 2 days Apple now has 10 percent of all corporate cash At last! A Putin vs. Zombie Rasputin video game I contacted Microsoft to ask whether it, too, smelled a little rotting clam in its claims. Bing's behavioral scientist, Matt Wallaert, offered this statement: "The professor's analysis is flawed and based on an incomplete understanding of both the claims and the Challenge. The Bing It On claim is 100 percent accurate, and we're glad to see we've nudged Google into improving their results. Bing It On is intended to be a lightweight way to challenge people's assumptions about which search engine actually provides the best results. Given our share gains, it's clear that people are recognizing our quality and unique approach to what has been a relatively static space dominated by a single service." On reading this, I found myself relatively static. What about the Freakonomics analysis was flawed? What didn't it understand about Microsoft's claims? And what makes the challenge "lightweight?" Is this scientific language "not entirely serious?" Or perhaps even "not at all scientific?" I confess that I have twice tried the Bing It On challenge. Once when the site came out. And once outside a newly opened Microsoft store. Both times, Google won. Does that mean I am abnormal? Or might it mean that the Freakonomics research reflects a greater reality than does Bing's own?
On its Bing It On site, Microsoft claims people prefer Bing to Google search by almost 2 to 1. Bing the other one, say the academics.
Freakonomics binged it on and wasn't happy.
(Credit: Screenshot by Chris Matyszczyk/CNET)
Googling is one of those habits.
You know, like flossing in the living room or changing your underpants every other day.
Yet Microsoft's search engine has Binged on and on that it is, in fact, better. Its own Web site insists that real people prefer Bing's results by a ratio of almost 2 to 1.
Such a claim tends to invite skepticism.
The vast minds at Freakonomics decided to take their skepticism to another level -- the one of actually trying to replicate the Bing results.
Their expectations were that people might not be able to tell the difference between the two searches. Their findings were quite searching.
Using a similar sample size to Microsoft's own claimed research (around 1,000), Freakonomics enlisted the help of four Yale law students -- people of obviously unimpeachable integrity.
The next step was, just as with the Bing It On Challenge, to perform five searches and see which results respondents preferred.
In the experiment, Freakonomics found that 53 percent chose the Google results, 41 percent chose Microsoft's, and 6 percent just couldn't make their minds up.
This might be summarized as not exactly 2 to 1. And not exactly in favor of Bing.
Freakonomics didn't freak out. It tried some other experiments to see if it could find parameters along which there was a clear preference for Bing. It found none. It found a clear preference for Google or, at best, a tie.
In conclusion, it described Microsoft's claims as "fishy."
More Technically Incorrect
- Drone crashed, almost hit me, says Manhattan businessman
- First Samsung Galaxy Gear, Note 3 ad: Annie Lennox, puppet love
- GPS gets elderly couple stuck in mud for 2 days
- Apple now has 10 percent of all corporate cash
- At last! A Putin vs. Zombie Rasputin video game
I contacted Microsoft to ask whether it, too, smelled a little rotting clam in its claims.
Bing's behavioral scientist, Matt Wallaert, offered this statement: "The professor's analysis is flawed and based on an incomplete understanding of both the claims and the Challenge. The Bing It On claim is 100 percent accurate, and we're glad to see we've nudged Google into improving their results. Bing It On is intended to be a lightweight way to challenge people's assumptions about which search engine actually provides the best results. Given our share gains, it's clear that people are recognizing our quality and unique approach to what has been a relatively static space dominated by a single service."
On reading this, I found myself relatively static.
What about the Freakonomics analysis was flawed? What didn't it understand about Microsoft's claims? And what makes the challenge "lightweight?" Is this scientific language "not entirely serious?" Or perhaps even "not at all scientific?"
I confess that I have twice tried the Bing It On challenge. Once when the site came out. And once outside a newly opened Microsoft store.
Both times, Google won.
Does that mean I am abnormal? Or might it mean that the Freakonomics research reflects a greater reality than does Bing's own?
