Well, Watson beat the human champions in the first game of the Jeopardy! face off between man and machine, with a score of $35,734 to $10,400 for Brad Rutter and $4,800 for Ken Jennings. But Watson’s developers were puzzled by his flub in the Final Jeopardy! segment. The category was US Cities, and the answer was: “Its largest airport was named for a World War II hero; its second largest, for a World War II battle.” The two human contestants wrote “What is Chicago?” for its O’Hare and Midway, but Watson’s response was a lame “What is Toronto???”
How could the machine have been so wrong? David Ferrucci, the manager of the Watson project at IBM Research, explained during a viewing of the show on Monday morning that several things probably confused Watson. First, the category names on Jeopardy! are tricky. The answers often do not exactly fit the category. Watson, in his training phase, learned that categories only weakly suggest the kind of answer that is expected, and, therefore, the machine downgrades their significance. The way the language was parsed provided an advantage for the humans and a disadvantage for Watson, as well. “What US city” wasn’t in the question. If it had been, Watson would have given US cities much more weight as it searched for the answer. Adding to the confusion for Watson, there are cities named Toronto in the United States and the Toronto in Canada has an American League baseball team. It probably picked up those facts from the written material it has digested. Also, the machine didn’t find much evidence to connect either city’s airport to World War II. (Chicago was a very close second on Watson’s list of possible answers.) So this is just one of those situations that’s a snap for a reasonably knowledgeable human but a true brain teaser for the machine.
The mistake actually encouraged Ferrucci. “It’s goodness,” he said. Watson knew it did not know that right answer with any confidence. Its confidence level was about 30%. So it was right about that. Moreover, Watson has learned how the categories work in Jeopardy! It understands some of the subtleties of the game, and it doesn’t make simplistic assumptions. Think about how Watson could be used in medicine, as a diagnostic aid. A patient may describe to a doctor a certain symptom or a high level of pain, which, on the surface, may seem to be an important clue to the cause of the ailment. But Watson may know from looking at a lot of data that that symptom or pain isn’t the key piece of evidence, and could alert the doctor to be aware of other factors.
(By the way, there are many fields where Watson could help out. IBM general counsel Robert Weber describes how Watson might be used in the legal profession in a guest blog posting on The National Law Journal Web site. Anne K. Altman, general manager, Global Public Sector, talks about how Watson could be helpful to government in a posting on Government Technology magazine’s blog.)
Another encouraging sign: Watson bet intelligently, just $947, so it still won the game by a wide margin. “That’s smart,” Ferrucci said. “You’re in the middle of the contest. Hold onto your money. Why take a risk?”
Watson may not have much of a sense of humor, but Ferrucci sure does. He wore a Toronto Blue Jays jacket to the Jeopardy! viewing.
Here are some explanations of how Watson plays J
Here’s how Watson knows what it knows from Jon Lenchner:
Here’s a post on Watson’s wagering strategies from Gerald Tesauro:
Here’s some info on how Watson sees, hears and speaks from Dave Gondek: