“The worst part for me is that after I finished the panel, I sent an email to my son, who’s in high school, and said look, in case you hear anything about Google and Bing at the high school, don’t panic …. I’ll be coming back home, and I will talk to you and mom and if you’re interested, I’ll explain more details. I just worry that he’ll hear ‘Bing what?’ at school,” said Shum, corporate vice president of Bing development.
It’s been four days now since the news of Google’s “sting operation” came out, first reported here in Search Engine Land. Since then, we’ve had Bing’s denial, Google’s denial of Bing’s denial, Bing’s further denial, Google’s further denial of Bing’s denial and even comedian Stephen Colbertgot in on the action. What more is there to say?
Quite a bit, I think, in terms of explaining Bing’s side of the story, beyond what’s just been in the dueling blog posts from each side. I talked with Shum, along with Bing’s director Stefan Weitz, after the Bing event concluded Tuesday and with a follow-up call yesterday.
Watching Users Isn’t Copying
Is Microsoft really denying that it copies Google’s results, when its own posts make it clear that it does get clickstream data that will include activity from Google?
“We’re not copying but watching users,” Shum said.
Weitz added, “The word ‘copy’ has a very specific connotation, and it’s wrong. We get the clickstream. We’re going to see it. We may choose to show it or not.”
Of Clickstreams & Surfstreams
Clickstream? What clickstream? Is this the “copying” that Google is talking about? Clearing up what the clickstream is, and how Bing uses is, it critical in understanding Bing’s defense.
Google alleges that Bing monitors what people search for on its site, if they have Internet Explorer equipped with certain features. Bing doesn’t dispute this. But Bing isn’t just monitoring what happens at Google. It monitors what people do as they travel across the entire web.
The “clickstream” might be more accurately described as the “surfstream.” Bing is able to see what people are browsing — at least what people who are using the Bing Toolbar, if they have that installed in their browser, or if they are using Internet Explorer with the Suggested Sites feature switched on.
Go to Amazon, and Bing may see that. Do a search at Amazon, and Bing may see the search you did. Click on a result after searching at Amazon, and Bing can tell what you saw. Leave Amazon and head to a new site, and Bing also sees that.
Extracting Search Activity From The Surfstream
One of the places that many people surf to on the web is Google. If they go there and do a search, Bing can tell this because it’s fairly easy to examine the URL of a Google search page and know that a search was conducted. In fact, you can even see the exact word that was searched for, as you can see in the example below:
It’s also fairly easy to guess at what someone clicked on from the results after doing a search. You just look to see the URL of the page that was surfed to immediately after a search was conducted.
Bing could, if it wanted, build a pure Google “signal” this way. Any time Bing sees a Google search, it would then look at the next page that a particular person requested. By doing this, it could get a sense of what Google users consider to the the best pages that Google itself lists for particular searches.
Bing Has Search Signal, Not Google Signal
Bing says it does NOT do this. It says there is no Google specific search signal that it being used, no list of all the popular pages as selected just by Google users. Instead, it has a “search signal” based on searching activity observed across a range of sites.
For example, if you did a search on Amazon, Bing might detect that. A search on eBay might get spotted. A search on Yahoo, that also might get extracted. Any number of searches might be identified. Bing would associate the next page you went to after doing those searches as being a possible “answer” to those searches.
This means that, in the end, Bing has crowdsourced the most popular pages linked to search terms from a wide range of search engines. That’s clever. This search signal, which isn’t purely from Google but does contain many Google searches (because Google’s is searched on so much), is then added into the wide range of other signals that Bing uses to rank pages.
“We aggregate the information,” Shum said. “The entire clickstream gets weighted along with different signals,” he explained. “For head queries, we have more signals. For tail queries, we have less. For the Google ‘synthetic’ queries [done for the Google sting operation], we have nothing.”
The Search Signal Doesn’t Overrule Other Signals
Let’s step back. There’s no “Google signal” but rather a “search signal” that Google activity is mixed into. That search signal is further mixed in with other ranking signals, a dilution that is crucial to why Bing so strongly rejects Google’s claim that it is copying its results. Most of Bing’s searches — popular so-called “head” searches — are not heavily influenced by the search signal, Bing says. There are many other signals that come into play.
“Movies” would be an example of a head search. That’s a search that’s done by thousands of people each day, and a search topic with lots of “signal” to measure. Bing can determine if pages are relevant to that term based on the words that appear on the page itself; how many people link to some of these pages with the word “movies” in the link; how authoritative those links seem to be, and more.
Bing can also examine how people click on its own results that it lists in response to that search. If a page doesn’t get as many clicks as would be expected, it might get dropped further down on the list. Pages that get more clicks than expected can be a sign that users find them more relevant than Bing’s ranking algorithm did originally, and so that can be used to boost them.
In addition to these and other signals, Bing can also turn to the surfstream to perhaps give a boost to pages that it sees are well visited by Internet Explorer users.
“Bombilate” would be an example of a “tail” search, a search that’s rarely done, and on an unusual topic. Here, there are fewer signals to assess. Only a few pages might be found. These pages might have few links pointing at them. Since the term isn’t searched for often, measuring clicks on Bing’s own results might not help much much. This is a case where the surfstream — as well as the search signal within it — might count more.
Even Google’s Test Proves The Search Signal Is Weak
Both Google and Bing agree that in some unique cases, the surfstream that Bing collects can cause Bing to reflect the same or oddly similar results to Google. Google considers this copying. Bing doesn’t — and thinks it’s so rare that Google is overblowing things with its accusations of copying
Indeed, Google’s own test also proves what Bing’s saying — that the surfstream is a weak signal. In a worse case situation, where Google was providing the sole signal to Bing for about 100 nonsensical terms, Bing used the signal only about 9% of the time.
That’s incredibly low. Google didn’t know why it was so low, when I asked about this as part of my original story. Bing says the reason is because even in the rare cases where Google is the sole signal, there still might not be enough confidence for its algorithm to consider using that signal.
Overall, the test seems to back up what got Google to start investigating in the first place — that using the surfstream might cause the occasional search where Bing seem to be copying Google but nothing widespread.
Didn’t Google Find Bing To Be More Bing-Like?
Then again, Google said in October that it found statistical evidence that Bing suddenly became more Google-like. More listings in the first page of results of both search engines seemed to match, as did more of the number one results. Google assumed this was because Bing started using a Google signal more heavily.
Not so, Bing told me. In October, Bing says it rolled out a new ranking algorithm plus a new experimental system called “Aether” that allows them to test changes in their ranking methodology. That’s what caused the bump that Google saw, not some sudden use of the surfstream, Bing said.
Indeed, this was another question Google had no clear answer for, when I asked in my original report. Google could see that Bing may have been using surfstream data from Google as far back as May 2010 (and Bing itself says it’s been used for years). What did it take until months later for the relevancy rise to happen? Google didn’t know, other than to guess maybe Bing decided to make heavier use of the data it was gaining.
How Bing Doesn’t Copy Google
If I could go back and change only one thing in my original story, I’d have made the headline “Google: Bing Is Cheating, Copying SOME Of Our Search Results.” I explained at length in the article itself that most of Bing’s results were unique. But the headline was taken by some to mean that Bing copied all of Google’s results.
That’s not the case. Even Google would readily admit that Bing’s results are not an exact copy of its results. I certainly never assumed anyone would think ALL of Bing’s results were being copied, but this might be a case where I’m too close to search. Anyone who knows search well would immediately understand that they both have different results overall.
To better explain, let me show two cases of rare queries where Bing isn’t copying Google. First is “pontneddfechan,” a small village in South Wales:
Click on the image to make it larger, if you’d like. This isn’t a popular query. This isn’t a popular term. The arrows tell the story, even if you don’t enlarge the image. They show where the same results appear on both Google and Bing.
At the top, the first arrow actually points to the results count, already a key difference. Google has found about 71,000 matches to Bing’s 9,000.
Beyond that, only 4 of the top 10 results in Bing’s list match Google’s. Only one of them is in exactly the same spot — Wikipedia, at number one. Maybe Bing learned that from Google. Maybe Bing was just following the law that requires Wikipedia to be at the top of all search results. It’s hardly conclusive. But there’s no doubt to me that this is a rare query that, if Bing is learning from Google, it sure doesn’t seem to be making much use of that visibly.
Here’s another one. This time, it’s a misspelling of “bombilate,” a rare word I cited above. I searched for “bombilete,” instead:
Remember what kicked this entire dispute off. Google spotting that Bing was returning good answers for a misspelled word — with strong evidence Bing did this because of the surfstream data it was collecting from Google.
Well, above is the same situation where Bing gets a misspelled word right — a link to a definition of the correctly spelled word at the top of the list. But it’s very unlikely it figured this out from Google, given that for the misspelling, Google doesn’t auto-correct the word nor provide the same answer.
So Why The Dispute?
If it’s easy to show that Bing doesn’t do wholescale copying, if it’s easy to find “tail” searches where there’s no evidence of copying, then why has Google gone to the extraordinary lengths of accusing its chief rival of being a cheater?
As I wrote in my original article, it’s because Google does see evidence that Bing is learning ways to improve its results especially for unusual queries and spelling corrections where it says huge amounts of engineering time are spent getting the right answers.
To Google, that it’s not wholescale copying doesn’t make the issue less of a concern. For Google, Bing seems to be cherrypicking the best stuff (though Google hasn’t shown more than one example of this).
For Bing, the opposite. The signal comes from searches across the web, not just from Google — and is so little is used that it’s not a big deal.
Why Not Exclude Google? Can You Even Exclude It?
That leads to a key question. If the data is so little used, of so little importance, why doesn’t Bing just drop it or exclude Google as part of the collection? At the very least, it will stop Google from constantly yapping about it.
“We’re not going to stop using that signal, unless it messes up relevancy. It doesn’t make sense to exclude that large amount of traffic from our usage set,” Weitz said.
Weitz lobs back how Bing’s believes that Google uses data from the Google Toolbar and Google’s Chrome browser to improve its results in similar ways that Google does, by collecting clickstream data from across the web. Bing sees no reason to limit itself.
Weitz also lobs back Bing’s view that Google has no problem copying some of Bing’s user experience changes:
“Is development of user experience any less worth of protection than the spelling algorithm? They are somehow holding it up as some pinnacle of engineering prowness. It [spelling correction] is hard to do,” Weitz said. But so is user experience work, he points out. “A lot of core cost goes to do a great UX.”
As I thought about the issue more, it occurred to me that Bing might not even be able to exclude seeing some of Google’s search results. Google’s results are distributed through a wide number of partners. Some of them, Google won’t even disclose (Google probably powers Ask.com’s organic results, but neither side will confirm or deny this). Even if Bing excluded data from Google’s own sites, Google’s results from ISP partners, portal partners and others might still flow through.
As For The Google Toolbar
Meanwhile, I’m on my third day of waiting to hear back from Google about just what exactly it does with its own toolbar. Now that the company has fired off accusations against Bing about data collection, Google loses the right to stay as tight-lipped as it has been in the past about how the toolbar may be used in search results.
Google’s initial denial that it has never used toolbar data “to put any results on Google’s results pages” immediately took a blow given that site speed measurements done by the toolbar DO play a role in this. So what else might the toolbar do?
Reflecting On The Past Week
After I wrote my original article, I came to the conclusion that Bing shouldn’t be monitoring Google in the way it seemed it was. I still have some unease that in certain situations, Bing isn’t really learning from user behavior but instead simply having those users report back to it about what Google is doing.
I’d still prefer a way that Bing isn’t improving its results in those types of cases — but we also really don’t know how widespread those might be. Perhaps the most important thing is Bing’s claim that it is not using a Google-specific signal. That’s reassuring.
I certainly have a better understanding of Bing’s side of the story now. That’s something I could have had as part of my original article, but Bing deliberately chose not to talk in detail before that story was written. Just as Google has played a PR game with the timing of its release, Bing has played its own PR game with the timing of a response.
I’ll probably come back in a day or two with some further reflections on the entire PR dance that’s been going on, as well as the issue as a whole. I’ve literally been in the front row of all of this, and some of the speculation is so far off the mark as to be laughable.
Was Google trying to put a shadow over Bing’s event and distract from questions about its relevancy? Did Bing hope that not talking in depth as part of my original article would give it more time to concoct a better defense?
Relevancy Remains The Issue
As I said in my original article, these questions on timing aren’t the issue. The relevancy of search results is the issue. Is Google better or worse than it was? Is Bing better or worse than it was? Is Google better than Bing? Is Bing better than Google? That’s what’s important.
The current dispute cause both sides to lose. Other than anecdotal examples, no one knows whether Google has gotten worse in relevancy. But some critics are already seizing on Google’s accusations as a sign that something must be wrong. Meanwhile, Bing has improved over time — but pulling in a search signal as it does will cause some to conclude that it is unfairly improving on the back of Google.
It’s More Than PR
Perhaps we’ll get some detente going forward. But I’ll close with one last observation. PR is not leading this dispute. It’s following behind. This dispute is happening because real engineers at Google felt there was a deep injustice going on — as reflected in the quote from Google’s Amit Singhal in my original article. I’ve known Singhal for years. I’ve never seen him speak like this before. It’s not because Google PR told him to. It’s because he’s fundamentally bothered by what he’s seen — as are members of his team.
This dispute is also happening because real engineers at Bing feel there’s a deep injustice going on — as reflected in the quote from Harry Shum above. Bing’s worked incredibly hard to build a search engine that’s worthy of respect. Now here’s Google suggesting that Bing has simply cheated its way to relevancy. It’s not not a game for Shum. Microsoft PR didn’t plant a story of him talking to his son. In fact, he originally told me it and specifically said it wasn’t for publication. He was just sharing something personal.
I went back and asked Shum I could include it, because I thought the story was a good illustration of what’s really driving all this — not PR maneuvering but fundamental questions about what’s fair and not in the creation of search results. We don’t know all the answers. Some of them are worth exploring, especially in terms of the bigger picture of search relevancy. And hopefully, they can be explored without the assumption that everything is just a PR exercise.
If you haven’t already, please be sure to also read my original post from earlier this week, which provides more context and background on some of the things I’ve covered above. You’ll find it here: