Smartest Machine on Earth

Smartest Machine on Earth

PBS Airdate: February 9, 2011

NARRATOR: The stage is set for a showdown the world will be watching. All eyes will be on one competitor, but he's not breaking a sweat. We're not talking nerves of steel, more like a rock-solid memory of silicon.

WATSON: Hello, my name is WATSON.

NARRATOR: For the first time in history, a computer will compete in the ultimate brain game.

ANNOUNCER (Jeopardy!): This is Jeopardy!

NARRATOR: For the scientists who created WATSON, the excitement and tension are palpable.

DAVID FERRUCCI (IBM Research): It's going to be edge-of-your-seat. It's going to be nerve-wracking.

NARRATOR: A four-year odyssey comes down to this.

DAVE FERRUCCI: What really is going to happen? And you just don't know. You don't know.

NARRATOR: There are more than reputations at stake. Some say a WATSON victory could signal a new age, a revolution in artificial intelligence.

LUIS VON AHN (Carnegie Mellon University): It used to be the case that intelligence was chess, right? If you can play chess, that's intelligence.

NARRATOR: But taking on spoken language and ferocious word-play means engaging by a whole new set of rules. It requires more than just memory to match wits with humans.

TODD CRAIN(Training Game Host): WATSON?

LUIS VON AHN: Just understanding the question is a big deal.

NARRATOR: How will WATSON do it?

NOVA gets unique access to the making of WATSON, shadowing the IBM team working around the clock, training and test-driving WATSON,...

TODD CRAIN: Comedies directed by Blake Edwards. WATSON?

WATSON: What is The Pink Panther?

DAVE FERRUCCI: We couldn't write rules for every combination of word or phrases or context.

NARRATOR: ...refining powerful new tools of computing,...

DAVE FERRUCCI: We took a huge jump with machine learning. We were like, "Woooh! Yes, he got it!"

NARRATOR: ...and creating a machine that can learn from its mistakes.

WATSON: What is mosquito?

TODD CRAIN (Training Game Host): No!

WATSON: Holiday?

TODD CRAIN: No.

WATSON: What is artificial sweetener?

TODD CRAIN: Come on, now. No.

DAVE FERRUCCI: We don't understand the question. We don't understand the category.

DAVE GONDEK (IBM Research): Yeah, that was one of the tensest days I've ever had.

DAVE FERRUCCI: You fall flat on your face. Did I expect to get fired? No, but...maybe.

We came back and the game was neck and neck.

TODD CRAIN: Wordsworth said they "soar, but never roam." WATSON?

WATSON: What is skylark?

TODD CRAIN: That is correct.

NARRATOR: Is WATSON ready to face the game's all time champions?

KEN JENNINGS (Jeopardy! Champion): This is too daunting a task for a computer.

HARRY FRIEDMAN (Jeopardy!): I think we've gone from impressed to blown away.

NARRATOR: When man takes on machine who will win?

WATSON: What is Crete?

TODD CRAIN: Yes!

NARRATOR: NOVA takes you inside the world of artificial intelligence and the quest to build a computer like none before it.

ALEX TREBEK (Jeopardy!): Good luck, WATSON.

WATSON: Smartest Machine on Earth, right now, on NOVA!

NARRATOR: Dave Ferrucci is a nervous parent.

He's spent the last four years building a revolutionary new computer, and it's about to face its biggest test, in front of an audience of millions.

WATSON: Hello, my name is WATSON. I hope we have a good game today, but first I have to test my voice.

NARRATOR: When the cameras roll, the computer, called WATSON, will make history, as it competes on the popular quiz show Jeopardy!

ANNOUNCER (Jeopardy!): This is Jeopardy!

This is Jeopardy!

DAVE FERRUCCI: It's frightening, right? It's a different experience. It's a very different experience for a scientist to sit here and have this happen live.

NARRATOR: Ferrucci has reason to fear. WATSON is playing for a million-dollar jackpot against the game's toughest competitors: Brad Rutter, Jeopardy's biggest money winner, and Ken Jennings, famous for winning 74 consecutive games.

KEN JENNINGS: There's some contestant's pride—I want to beat my human competition—but you know, as a species, I would like mankind to beat the big bad computer.

WATSON: Let's finish "Leaders of World War II."

TODD CRAIN: All right!

NARRATOR: This big bad computer is the culmination of four years' intensive work. IBM has put WATSON through hundreds of practice games, with a stand-in host and real contestants.

TODD CRAIN: After Germany invaded the Netherlands, this queen, her family and cabinet fled to London. Maria?

MARIA (IBM Jeopardy! Practice Contestant) Who is Beatrix?

TODD CRAIN: No. WATSON?

WATSON: Who is Wilhelmina?

TODD CRAIN: That is correct.

ERIC BROWN (IBM Research): It's a human standing there, with their carbon and water, versus the computer with all of its silicon and its main memory and its disc.

NARRATOR: It seems like it should be easy for the computer to win, with its enormous memory and processing power, but the human brain makes an intimidating opponent, especially on Jeopardy.

DAVE GONDEK: Jeopardy questions are tricky. They have puns in them. They have little jokes in them.

LUIS VON AHN: Just understanding the question is a pretty big deal.

TODD CRAIN: This trusted friend was the first non-dairy powdered creamer. WATSON?

WATSON: What is milk?

TODD CRAIN: No. Maria?

MARIA: What is Coffee-mate^®?

TODD CRAIN: Thank you!

DAVE FERRUCCI: Humans communicate very fluently in, you know, natural language, and that's where computers struggle dramatically.

NARRATOR: Now, WATSON is on the verge of conquering that challenge.

TODD CRAIN: A garment worn by a child, perhaps aboard an operatic ship. WATSON?

WATSON: What is pinafore?

TODD CRAIN: Yes! How did you get that?

NARRATOR: If WATSON wins on Jeopardy, it will be a major breakthrough in a quest that's gone on for decades: the audacious dream to build a machine as smart as a person, the quest for artificial intelligence, A.I.

ELEKTRO (Westinghouse Electric Corporation /Video clip): I am Elektro the Robot. My brain is bigger than yours.

TERRY WINOGRAD (Stanford University): When we started doing A.I., the goal was, "Why can't we build a person?" We all know how to make people; that's easy. What if we could build one out of silicon?

NARRATOR: Pioneers drew their inspiration from the world of science fiction.

MARVIN MINSKY (Massachusetts Institute of Technology): As a child, Isaac Asimov turns up. So, here, I'm an adolescent, and, and the robots, intelligent machines are part of my life.

ROBBY THE ROBOT (Forbidden Planet/Video Clip): At your service!

MARVIN MINSKY: Let's, let's see about getting them built.

NARRATOR: In the early days, computers grew rapidly more powerful, quickly mastering complex equations.

MARVIN MINSKY: The first programs we wrote at M.I.T. solved problems that only very educated people could solve, like problems in calculus and then algebra.

NARRATOR: The computer pioneers thought they were on a fast track to building human-like intelligence.

CLAUDE SHANNON (Massachusetts Institute of Technology/Film Clip): I confidently expect that within 10 or 15 years, we will find, emerging from the laboratories, something not too far from the robot of science fiction fame.

PAT WINSTON (Massachusetts Institute of Technology): In the beginning, we thought, "Well, maybe 10 or 15 years and we'll have something that's really smart."

TERRY WINOGRAD: In the beginning, people really were amazed at how much computers could do. When you see something that's improving very fast, you simply assume it will continue improving that fast, indefinitely.

NARRATOR: In the '60s, confidence was so high, it inspired one of the most iconic film characters of all time.

KEIR DULLEA as DAVE (2001: A Space Odyssey/Film Clip): Hello, Hal, do you read me? Do you read me, Hal?

DOUGLAS RAIN as HAL (2001: A Space Odyssey/Film Clip): Affirmative, Dave. I read you.

RODNEY BROOKS (Massachusetts Institute of Technology): When I was a kid, I saw 2001: A Space Odyssey, and Hal was just the best thing ever!

DAVE (2001: A Space Odyssey/Film Clip): Open the pod bay doors, please, Hal.

HAL (2001: A Space Odyssey/Film Clip): I'm sorry, Dave. I'm afraid I can't do that.

RODNEY BROOKS: You know, it was a murdering psychopath,...

DAVE (2001: A Space Odyssey/Film Clip): Hal? Hal?

RODNEY BROOKS: But, it was intelligent, could talk to people, could see people, could lip-read, could do all this stuff, a machine that could do that, and I'd never even seen a real computer, at that time.

NARRATOR: More than 40 years after the creation of HAL, no real computer or robot has been able to interact with humans as seamlessly as Hollywood imagines.

WILLIAM HURT (Lost in Space/Film Clip): Tell me, what is love?

NARRATOR: The problem is our own human computer, the brain, a complex entity that's defied any attempts at replication.

LUIS VON AHN: We just had no idea how sophisticated the brain was.

NARRATOR: The computer has always been...is king when it comes to calculation and processing huge amounts of data.

TEACHER: Pen, as in the pen. What's the middle letter?

SCHOOL CHILDREN: E!

TEACHER: Excellent.

NARRATOR: But simple skills that humans master early in life, like understanding language or recognizing objects, continue to baffle researchers.

TOM MITCHELL: You know, people vastly misjudged how subtle we are when we're intelligent. People just hugely underestimated that.

NARRATOR: But the dream of building a computer that could talk and match wits with humans never really died, and a few years ago a new plan was hatched, sparked by an unlikely event.

ANNOUNCER (Jeopardy!): This is Jeopardy!

ALEX TREBEK: In Wagner's operas, this eldest Valkyrie is stereotypically dressed in a horned helmet and breastplate.

KEN JENNINGS: Who is Brí¼nnhilde?

NARRATOR: In 2004, Ken Jennings' 74-game winning streak on Jeopardy set the country abuzz, and caught the eye of an IBM executive while out to dinner.

CHARLES LICKEL (IBM Research): All of a sudden, the entire restaurant cleared out to the bar that I am sitting at, to go see Ken Jennings play Jeopardy.

ALEX TREBEK: This Grand Ole Opry comedy star used to wear a straw hat with the $1.98 price tag still attached.

KEN JENNINGS: Who is Minnie Pearl?

NARRATOR: Charles Lickel wondered if a computer could ever play as well as Ken Jennings. So he pitched the idea to some of IBM's top scientists.

CHARLES LICKEL: For the ones that knew Jeopardy, they said, "Charles, that's just too hard." Um, I think the, the prevailing view was, the, these questions were difficult to understand, difficult to even comprehend what was being asked.

DAVE GONDEK: Yeah, I was like, "No way!" I was like, "No way!"

NARRATOR: But one researcher, Dave Ferrucci, was intrigued.

DAVE FERRUCCI: My view was, maybe this isn't as completely impossible as we think it is.

NARRATOR: For over 40 years, Jeopardy has been pop culture's I.Q. test. Clues are given as answers, and contestants have to respond in the form of a question.

ALEX TREBEK: It's a larger vessel that guards and supplies smaller ones. Christina.

CHRISTINA (Jeopardy Contestant):What is the mother ship?

KEN JENNINGS: The show's central conceit is a little syntactic reversal, whereby they give you an answer and you supply a question, you don't say "George Washington," you say "Who is George Washington?"

NARRATOR: To win, contestants need to be human encyclopedias.

KEN JENNINGS: It's essentially everything under the Sun. You know the categories at the same second Alex tells the folks at home the categories.

ALEX TREBEK: One, you have to have a broad knowledge, because we have 13 categories on each show.

Kate, start.

KATE ( Jeopardy Contestant): "P.h." for 400.

ALEX TREBEK: For the record, Thomas Edison invented the first practical one of these 1877.

NARRATOR: Contestants also have to be fast.

ARIEL( Jeopardy Contestant): What is the phonograph?

ALEX TREBEK: Good!

NARRATOR: They typically have three seconds or less to come up with an answer.

ALEX TREBEK: The mortar and pestle is a symbol of this profession. Ariel?

ARIEL : What is a pharmacist?

ALEX TREBEK: Pharmacist is right.

NARRATOR: To compete on Jeopardy, IBM's computer must have an enormous knowledge base, because it will not be connected to the internet. But the far bigger challenge for the machine will be understanding clues which can be extremely convoluted or obscure.

ALEX TREBEK: You'll find this flower before "Pickle Bottom" in a line of handbags and bedding.

That would be petunia. Back to you, Ariel.

KEN JENNINGS: There will be a lot of puns. There'll be double meanings. And these are things that computers historically are terrible at.

NARRATOR: Human language is a minefield for computers. Consider this sentence.

DAVE FERRUCCI: How's it go? "I shot an elephant wearing my pajamas." Was I wearing the pajamas? Was the elephant wearing the pajamas? So there are different interpretations, different ways to parse the sentence. The word shot. What's really going on there? There's already ambiguity in there. Could actually be shooting, sort of a hunting shooting; if I'm a photographer and I'm immersed in that context, I may interpret that as shooting with a camera. Which one did I mean? You have to look at the context.

NARRATOR: But a computer has no context. It's just an electronic brain in a box.

In 2006, Ferrucci tackles this challenge, along with the best and brightest programmers from IBM and the country's top A.I. labs.

To start, they run a test.

DAVE GONDEK: We had a, um existing state of the art system that people had worked on for a number of years, and we tried applying that to the Jeopardy challenge.

NARRATOR: They feed one of IBM's most sophisticated computer programs hundreds of Jeopardy questions, like this one: "In 1698, this comet discoverer took a ship called the Paramour Pink on the first purely scientific sea voyage." The correct answer is "Who is Edmond Halley?" The computer says, "Who is Peter Sellers?"

The computer ran a search through a million documents, looking for key words from the clue. It homed in on a description of one of The Pink Panther films, in which one character was a paramour, or mistress. The star of movie? Peter Sellers.

It's probably the last answer a human would come up with, but it's typical for computers. The team has a long way to go.

Just how far becomes clear when they compare the computer to the best human players. They create a graph called the "Cloud." Each dot represents a Jeopardy champion's performance. Jennings is at the top.

DAVE GONDEK: What you see is a cloud, around...they answer around 50 percent, the winners do. And they get around 90 percent correct.

NARRATOR: And where is the computer in this cloud?

DAVE GONDEK: If you ask it to answer all the questions, it would be giving you 10 percent of the questions right. You can't go on Jeopardy like that. I mean, the best humans are 90 percent, 92 percent. We weren't even close.

NARRATOR: To win at Jeopardy, the team will need a whole new way to tackle human language, one that takes advantage of the computer's basic strengths.

At its electronic core, a computer speaks a very simple language: binary code, on or off. But with that simple code, it can follow instructions and solve complex problems once reserved for intellectual giants.

LUIS VON AHN: It used to be the case that intelligence was chess, right? If you can play chess, that's intelligence.

NARRATOR: Computers have mastered the game.

LUIS VON AHN: Chess is easy for computers, because the rules are very well defined and very clear.

NARRATOR: The rules of chess are relatively simple: a board of 64 squares; each piece—pawn, knight, queen—can move a certain way; and there's a single goal: take out your opponent's king.

For humans it is the ultimate game of strategy.

RODNEY BROOKS: The way computers play chess is not at all the way people play chess. We humans look at the board and have conceptual ideas like, "Control the center," "Attack on the right." Very different from the way computers play chess.

NARRATOR: A chess-playing computer looks at virtually every possible move it could make and every response, every way the game could play out.

RODNEY BROOKS: Computers play chess through searching a tree of moves down to a very deep level, looking ahead on every possible path. But they do it by brute force, by going 20, 30, 40 moves ahead and seeing all the bad things that can happen. A person can't look that many moves ahead, broadly.

NARRATOR: This is the power behind the most famous chess game in the history of A.I., when, in 1997, another IBM computer, named Deep Blue, beat the reigning world champion, Gary Kasparov.

NATURAL SOUND CORRESPONDENT (1997): The chess world champion walked away from the match, never looking back at the computer that had just beaten him.

NARRATOR: The victory makes Deep Blue look pretty smart, but is it?

PATRICK WINSTON: Deep Blue, it's only acting as if it's intelligent, and it's not really intelligent in the way that we humans are.

RODNEY BROOKS: It's good at one thing. It's playing chess. It can't do anything else. It has no other understanding of the world. It's just about chess moves.

NARRATOR: This lack of understanding has hampered every computer program that's tackled human language.

A perfect example is a program from the 1960s, called Eliza.

TERRY WINOGRAD (Stanford University): Eliza was one of the first programs that had anything resembling human conversation. It was a dialogue. You typed things in, it typed things back.

ELIZA VOICE: How do you do? Please tell me your problem.

TERRY WINOGRAD: "I am feeling sad."

Then it types back, "Did you come to me because you are feeling sad?"

NARRATOR: Eliza was programmed to respond like a psychiatrist, but it had no real insight. Instead, it followed simple rules and rearranged key phrases.

TERRY WINOGRAD: So if I say, "I'm dead," it responds, "Do you enjoy being dead?" It doesn't have any understanding that dead is a different kind of condition. It really is just doing this sort of fill-in-the-blanks kind of pattern matching.

NARRATOR: Anyone who tried to solve the language problem hit the same brick wall: the computer's profound ignorance of what we take for granted every day.

LUIS VON AHN: There is just so much more that we know that we don't know we know. I mean, just, we know all kinds of stuff, like you press the up button in the elevator, that means it's going to go up. Or, milk is white, or water is wet. I mean, there's just stuff that we know that we don't even realize we know. That's one of the things that makes it hard.

NARRATOR: All the common sense knowledge a human brain collects naturally seems much too complex to program into a computer. But that hasn't stopped one scientist from trying.

DOUG LENAT (Cycorp, Inc.): So we have actually manually entered about 6,000,000 rules. That's about three percent of what it's going to need to know in terms of actually spanning what you and I would call human common sense.

NARRATOR: For the last 25 years, Doug Lenat has been leading a team trying to create human-like intelligence by teaching a computer common sense, rule by rule. The program is called Cyc, and at headquarters, the walls are covered with logic diagrams.

DOUG LENAT: In a way, the, the magic of this, the power of this is if you just tell it each rule, one by one, by one, and you give it general logical reasoning capabilities, that's all you need to do.

NARRATOR: So far, Cyc has 6,000,000 rules and can answer a lot of common sense questions. Like this one:

DOUG LENAT: Can a can cancan?

NARRATOR: Cyc says no, and it explains why.

DOUG LENAT: Here, it's essentially saying the reason why cans can't cancan is that cans are inanimate objects, and it knows

that cancan dancing requires at least partially having a brain and using it.

It's just enough for Cyc to get the right answer for the right reason.

NARRATOR: Twenty-five years ago, many experts considered rules and logic the best hope for building artificial intelligence, but it's become clear these alone are not enough.

TERRY WINOGRAD: It's not just a matter of piling in more and more stuff. There are basic principles that we didn't understand, and putting in more and more stuff doesn't get you basic principles.

NARRATOR: At IBM, as Dave Ferrucci and his team tackle the Jeopardy challenge, they know that facts and rules are just the beginning.

DAVE FERRUCCI: We couldn't, couldn't write rules for every combination of word and phrases and context.

NARRATOR: They need a new system, more fluid and flexible, to navigate the twists and turns of many different kinds of Jeopardy questions.

They name their system WATSON, after IBM founder Thomas Watson.

The electronic WATSON consists of 2,800 processors, that's like 6,000 high-end home computers. All together it's the size of 10 refrigerators.

The team starts filling his memory banks with about 10 million documents—most downloaded from the internet—because when WATSON plays Jeopardy, he must stand alone, just like his human competitors.

DAVE FERRUCCI: All kinds of content, okay? Encyclopedia, dictionaries, thesauri, books, plays, you name it.

NARRATOR: The entire World Book Encyclopedia, Wikipedia, the Internet Movie Database, much of the New York Times archive, and the Bible are just some of WATSON's resources.

And to build on WATSON's foundation of data and rules, the team turns to a powerful tool in the computing world. It's called machine learning.

SEBASTIAN THRUN (Stanford University): Machine learning is just like human learning from examples.

LUIS VON AHN: Before, people would just write rules, write rules by hand. Nowadays, it is all based on examples.

NARRATOR: To understand how machine learning works, consider for a moment the letter A. What if you had to describe it to a computer? It's a real problem faced by the U.S. Postal Service, whose computers must decipher all kinds of addresses, printed and handwritten.

LUIS VON AHN: We all know what an A looks like, I know when I see it, but there is just way too many different types of As. There are fonts where the A is just a triangle pointing up, that is an A.

Pretty quickly, you realize there is no simple set of rules that you can write down, currently, for a program to determine whether a letter is an A or not.

NARRATOR: Humans might not be able to come up with the rules that reliably identify all kinds of As, but, it turns out, a computer can do it for itself, if you give it enough examples.

LUIS VON AHN: The way you do it, you just get an A, send it to the program and say that is an A. Here is another A, different one. That is an A. Here is another A. This is a different one. That is an A. Then you would give another example, and you would give another example, and you would do that a million times.

NARRATOR: The computer hunts for patterns among all those examples, and it finds them. So, the next time it meets a letter A, even one it hasn't seen before, it will recognize it. This is machine learning, and it's a crucial element of WATSON's programming.

The team trains WATSON, but here, instead of letters, the examples are tens of thousands of old Jeopardy questions, along with a cheat sheet of all the correct answers. Using machine learning, WATSON will hunt for patterns between the type of question, the correct answer, and the kinds of evidence that support that answer.

DAVE GONDEK: Now, we do this over thousands of questions, so we come up with some way to weigh the evidence on average, so that we come up with the right answer.

NARRATOR: Now when he's faced with a brand new question, WATSON uses what he learned from these patterns, and declares his confidence in each possible answer.

DAVE FERRUCCI: And in the end, we get a list that says, "Here's the top answer, and we're 75 percent sure it's right."

NARRATOR: WATSON has now become a complex architecture of rules, raw data and machine learning that enables him to use statistics to choose the right answer.

To test out this system, the team scours the halls for IBM employees who can play Jeopardy, and everyone squeezes into a conference room.

CHRIS WELTY (IBM Research): In 1978, New Jersey Monthly reporter Steven Levy famously found this man's brain. WATSON?

WATSON: What is Einstein?

CHRIS WELTY: Albert Einstein's. Yes.

The Fifth Amendment says that private property shall not be taken for public use without this.

WATSON: What is "just compensation?"

CHRIS WELTY: Yes!

DAVE FERRUCCI: We took that huge jump with machine learning.

CHRIS WELTY: WATSON, with a commanding lead of 24,863.

DAVE FERRUCCI: We saw a huge jump in performance, and we were, like, "Whooo!

NARRATOR: Up to now, appearing on the TV show has only been a dream, but WATSON is performing so well, Dave Ferrucci decides it's time to call Jeopardy.

In December, 2009, Jeopardy producers arrive at IBM to size up Dave Ferrucci's new creation. Like any human contestant, WATSON must audition to earn his spot on the show.

DAVE FERRUCCI: We spent all this time, you know, developing this system, and pushing its capabilities. Then here you are, sitting here, all the executives are there,...

HARRY FRIEDMAN: You hear "computer" you think, "Uh, well, of course a computer should have all the answers." You hear about Q&A technology, well isn't this just a big search engine?

DAVE FERRUCCI: ...and they're waiting to see, you know, what really is going to happen. And you just don't know, you don't know.

NARRATOR: To impress the executives, IBM builds a makeshift studio, hires comedian Todd Crain to act as game show host, and brings in former TV contestants.

DAVE GONDEK: Yeah, that was one of the tensest days I've ever had. Because we'd never seen it play against Jeopardy players.

And, I remember, like, the day before, we're tuning everything, and I was putting in the best strategy that we had, I was putting the best stuff that we had, and I thought, "Well, this is just going to kill them!"

MIRANDA (Former Jeopardy! Contestant): What is "The Cat's in the Cradle?"

TODD CRAIN: That is correct.

DAVID (Former Jeopardy! Contestant): What is "I Am the Walrus?"

TODD CRAIN: Yes!

MIRANDA : What is "Crocodile Rock?"

TODD CRAIN: Yes!

DAVE GONDEK: You know, they were just like professional athletes, you know, and it was a really tough few games for us.

NARRATOR: In the first round, it seems that WATSON is auditioning not for a game show but a sit-com.

WATSON: L underscore underscore underscore underscore O for 1,000.

CARRIE (Former Jeopardy! Contestant): Whoa.

NARRATOR: There are suddenly unexpected bugs that need fixing.

DAVE GONDEK: We, we weren't dealing with Roman numerals well. So it'll say, like, Henry V, we would say, "Henry Vee."

TODD CRAIN: In 1682, he came to the throne at the age of 10, along with his weak-minded half-brother, Ivan V. WATSON?

WATSON: What is Peter?

TODD CRAIN: More specific?

WATSON: What is Peter 'I'?

TODD CRAIN: No! Carrie or David? Carrie!

CARRIE : Who is Peter the Great?

NARRATOR: In "Final Jeopardy," where contestants must place bets and write down the answer, things only get worse. Under the category "Flags," the clue is, "In a policy begun in 2002 as a symbol of the war on terrorism, U.S. Navy ships fly the 18th century flag with this four-word motto."

TODD CRAIN: You need to know a little bit something about18th century flags. David, let's see if you did. What is the four-word motto we're looking for, David?

DAVID : What is, "Don't tread on me?"

TODD CRAIN: That is correct! Let's see if WATSON got it right. "What is September 11, 2001 attacks?"

CARRIE : Whoa!

NARRATOR: WATSON did not recognize the word "motto," and after scanning through millions of documents, he found the word terrorism associated with September 11th so frequently that seemed like the best answer.

By the time they break for lunch, it's humans two, WATSON zero, and it's not clear if WATSON will ever be ready for primetime.

DAVE FERRUCCI: This was taking a risk, for me, in the sense that we're sitting here and saying, "You know what? I think this is possible." And then you fall flat on your face, so... "We're never going to believe Ferrucci again."

Did I expect to get fired? No, but...maybe.

NARRATOR: But after lunch, the producers are treated to a different side of WATSON.

DAVE FERRUCCI: We came back, and the third game was neck and neck, incredibly competitive.

TODD CRAIN: In Act Three of an 1846 Verdi opera, this Scourge of God is stabbed to death by his lover, Odabella. WATSON?

WATSON: What is Attila?

TODD CRAIN: Be more specific?

WATSON: What is Attila the Hun?

TODD CRAIN: Thank you, very much; Attila the Hun. I'll take that.

NARRATOR: That afternoon, WATSON climbs back in the game.

TODD CRAIN: Wordsworth said these "soar, but never roam."

WATSON: What is skylark?

TODD CRAIN: It's a device clamped to the wheel of a parked car. WATSON?

WATSON: What is boot?

TODD CRAIN: Be more specific?

WATSON: What is Denver boot?

TODD CRAIN: That is correct.

This African American folklore laborer: "Before I let that steam drill beat me down, I'll die with my hammer in my hand." WATSON!

WATSON: What is John Henry?

TODD CRAIN: That is correct! Select again.

NARRATOR: It may appear that WATSON has redeemed himself, but the producers are troubled by his erratic performance. Their verdict: WATSON isn't strong enough for Jeopardy, at least, not yet.

Why is WATSON so erratic?

To understand his weaknesses, you have to appreciate the complexity of the task. Consider this clue: "Keanu Reeves had a Nokia phone, but it took a landline to slip in and out of this, the title of a 1999 sci-fi flick." The correct response is, "What is The Matrix?" But how can WATSON figure that out?

First, he breaks down the clue into grammatical parts, identifying key words and phrases. Then, WATSON's powerful search engines churn through millions of documents, including the Internet Movie Database.

DAVID GONDEK: What we do next is we take these documents and we pull out candidate answers. And we'll pull out "Keanu Reeves." That could be a candidate. We'll pull out "Nokia;" we'll pull out "The Matrix."

NARRATOR: Other movies starring Keanu Reeves also become possible answers.

DAVID GONDEK: We'll pull out "The Matrix 2;" we'll pull out "Speed," "Bill & Ted's Excellent Adventure," all this stuff.

KEANU REEVES (Bill & Ted's Excellent Adventure/Film Clip): Whoa!

NARRATOR: And WATSON pulls out other famous sci-fi flicks, like "Blade Runner."

DAVE FERRUCCI: And it generates hundreds of possible answers.

NARRATOR: With hundreds of choices, how can WATSON pick the one answer that's correct?

DAVE FERRUCCI: The next thing that WATSON is going to do, it's going to take those answers and say, "Well, let's assume all of them might be right." So, these are its competing hypotheses.

NARRATOR: WATSON starts considering evidence for and against each candidate, using rules like "A movie is sometimes called a flick."

DAVE GONDEK: And we'll look at things like, well it's looking for a flick. Is this candidate answer a flick? Is The Matrix a flick? Yes. Is Speed a Flick? Yes. Is Keanu Reeves a flick? No.

Okay, so we're starting to learn something.

NARRATOR: Within a matter of milliseconds, WATSON analyzes every possible answer in hundreds of different ways and scores each piece of evidence behind every answer in the list.

That's a lot of scores.

DAVE GONDEK: The problem is you have all these different scorers and they don't agree. Some of the scorers are going to say "The Matrix" is the right answer. Some of the scorers are going to say "Keanu Reeves" is the right answer. Some are going to think "Matrix 2" is the right answer.

NARRATOR: And a lot of scorers think "Blade Runner" is the right answer, because it shows up so often as a sci-fi flick.

DAVE GONDEK: So you need someone at the end who is going to listen to all these votes and decide what's going to be the best answer.

NARRATOR: This is where WATSON's machine learning kicks in. Having studied thousands of other Jeopardy questions and their correct answers, WATSON has learned what evidence is important and what's not.

DAVE FERRUCCI: What machine learning will start to do is learn how to weigh them differently. Say, "Wait, questions like this, calling on a phone, not calling on a phone, not so important. This other stuff, "Do I have a sci-fi movie? Is the person named a character in that movie?" Very, very important for questions like this.

NARRATOR: In this case, he successfully weighs the evidence and identifies sci–fi flicks from 1999, starring Keanu Reeves. So, he picks the one answer matching all those elements, "The Matrix."

WATSON's elaborate system doesn't always work, but without machine learning, he wouldn't stand a chance.

Machine learning isn't just important for WATSON. It's driving a revolution in computing. It plays a major role in computer models that predict the weather, days in advance. And all those recommendations you get from Amazon or Netflix? No human is writing up rules about your likes and dislikes. Instead, computers are comparing your preferences to millions of other customers and finding patterns and learning about you.

Today machine learning is conquering many problems once thought too complex for computers. Like speech recognition.

TOM MITCHELL (Carnegie Mellon University): In the earlier days, people decided that they would try to program computers to recognize speech.

Which word am I saying now?

RESEARCHER (Video/Clip): Pick up the big block at the right side.

NARRATOR: In the '60s, this voice-directed block world was the height of technology. Computers could be programmed to recognize the audio signals of specific words and phrases.

RESEARCHER (Video/Clip): Pick up the big block at the right side.

Pick up every small block.

NARRATOR: But they had to be re-programmed for every new speaker, because everyone's speech is slightly different.

TOM MITCHELL: Even though it is very easy for you and I to recognize the word "ice cube..."

BOY's VOICE: Ice cube.

TOM MITCHELL: ...it is very difficult for us to write down the rules that would allow a computer to look at the microphone signal and see that is "ice cube."

MANY VOICES: Ice cube.

NARRATOR: But now, computers are trained with millions of examples of human speech.

TOM MITCHELL: Here is the microphone signal, and this is the word "ice cube."

WOMAN'S VOICE: Ice cube.

TOM MITCHELL: Here is another one.

BOY'S VOICE: Ice cube.

TOM MITCHELL: You end up with much more successful speech recognition systems.

NARRATOR: Today, speech recognition software, though not perfect, is remarkably accurate and getting better all the time.

TOM MITCHELL: All the ones we have today are based on machine learning, simply because it works the best.

NARRATOR: And some programs are taking it a step further,...

ALEX WAIBEL (Carnegie Mellon University): Where do you come from?

NARRATOR: ...not only transcribing your speech,...

CUSTOMER : Shanghai.

NARRATOR: ...but translating it into a foreign language as well.

COMPUTER: Shanghai.

ALEX WAIBEL: It's very nice to meet you.

COMPUTER: (Translation of "It's very nice to meet you into Chinese)

NARRATOR: Every language has so much ambiguity, double meanings and metaphors,...

ALEX WAIBEL: What are your specials today?

NARRATOR: ...accurate computer translation seemed impossible with the old rules-based approach.

COMPUTER: Today's special is roast beef fried rice.

ALEX WAIBEL: All of these interactions are so complex that you couldn't, in your lifetime, write all these rules It's just too enormous, too daunting an effort.

My wife asked me to buy some crackers.

STORE EMPLOYEE: (Asks question in Chinese)

COMPUTER TRANSLATOR: What kind of crackers?

ALEX WAIBEL: Rice crackers.

NARRATOR: Alex Waibel fed a computer millions of examples of English texts, together with their translations into about a dozen different languages. Now he's got a program that can run on your phone or iPad...

ALEX WAIBEL: Do you often shop here?

COMPUTER TRANSLATOR: Do you often shop here?

SHOPPER: (Replies in Chinese) Every day .

NARRATOR: ...and translates on the go.

Machine learning has been so successful, mastering more and more tasks once only done well by humans.

ALEX WAIBEL: The computer program translates between languages.

NARRATOR: Some researchers believe it may be a crucial building block for making real artificial intelligence.

TOM MITCHELL: There are two ways of building intelligence: you either know how to write down the recipe or you let it grow itself. And it is pretty clear that we don't know how to write down the recipe. Machine learning is all about giving it the capability to grow itself.

NARRATOR: Some people find the idea of a machine that can learn threatening, but when WATSON's having a bad day, it's hard to imagine him threatening anyone.

DAVE GONDEK: Did you see the "Daily Double?"

DAVE FERRUCCI: No.

DAVE GONDEK: So, we were way ahead. We were almost locking out.

TODD CRAIN: The other "Daily Double!"

DAVE GONDEK: We get the "Daily Double," there's like two or three clues left. And so what do we do? We bet big so we can lock them out.

WATSON: I'll wager $5,200.

TODD CRAIN: We-e-el!

NARRATOR: The team is nervous. They know WATSON will never make the cut on Jeopardy, unless he can stop making dumb mistakes.

TODD CRAIN: Here's your clue.

DAVE GONDEK: It was about letters, and it was, "A woman wrote to this '40s artist...

TODD CRAIN: In the late '40s, a mother wrote to this artist that his picture "Number 9" looked like some of her son's finger-painting.

DAVE GONDEK: We answered with "Rembrandt."

WATSON: Who is Rembrandt?

TODD CRAIN: Really?

NARRATOR: Although WATSON recognizes most dates, he doesn't know that the '40s refer to the nineteen forties.

DAVE GONDEK: It's a '40s artist. We answered with "Rembrandt," so there's a time, a time problem.

TODD CRAIN: Very bad. We all know that it was Jackson Pollock.

NARRATOR: WATSON loses that game.

DAVE GONDEK: We double wrong and "Final Jeopardy!"

NARRATOR: And his opponents show him no mercy.

MEHRUN (Former Jeopardy! Contestant?): We both beat him!

MARIA: Good for you!

MEHRUN: Humans! Whooo!

DAVE FERRUCCI: Sometimes the reason something is the right answer is very obvious to a human, like, for example, it may be asking for a she or a he.

TODD CRAIN: This first lady was born Thelma Catherine Ryan, on March 16, 1912, in Nevada. WATSON?

WATSON: Who is Richard Nixon?

DAVE FERRUCCI: Oh, here you go! Right?

TODD CRAIN: Patricia?

PATRICIA (Former Jeopardy! Contestant?): Who is Pat Nixon?

TODD CRAIN: That is correct. Richard Nixon was never a First Lady.

DAVE FERRUCCI: I want to understand what went on there.

DAVE GONDEK: Our new gender stuff is not in the system.

CHRIS WELTY: People take offense at being called the wrong gender. WATSON doesn't care about stuff like that. It's making statistical judgments based on how pieces of evidence have gone together in questions and answers that we've given it.

TODD CRAIN: The two famous comedians' noses that make impressions.

WATSON: What is Jimmy Durante?

TODD CRAIN: More specific?

WATSON: Sorry, all I know is "What is Jimmy Dur-an-tay?"

TODD CRAIN: Saying it slower doesn't make it right.

DAVE FERRUCCI: Now, you hear how Todd Crain makes fun of the computer? Now, I had my kids, I had them sign the confidentiality agreement and come in and see a couple of these games. And their comment was, "Why does the host make fun of WATSON, Daddy?"

TODD CRAIN: This what-are-you-doing? Web site's name also refers to a type of nervous laugh.

WATSON: What is "evil laugh?"

TODD CRAIN: Ho ho! No!

In terms of comedy duos, he is the best straight man in the business,...

You're going to kick yourself. Twitter!

...because he doesn't, he doesn't get it. He doesn't get why his inappropriate answer is funny, and you can't ask for better writing than that.

DAVE FERRUCCI: Once in a while, okay, we can all take a joke, but over and over again? And WATSON's defenseless, right? So he's making fun of and criticizing a defenseless computer that represents people with real feelings and real families. All right, maybe if I don't have any feelings, my kids have feelings.

NARRATOR: And feelings are intensified by the fact that there's just one more month before the Jeopardy producers will return to make a decision.

TODD CRAIN: WATSON, you have control.

NARRATOR: The pressure is on to boost WATSON's strengths and eliminate his weaknesses. His strengths are obvious. He dominates when it comes to purely factual questions.

ERIC BROWN (IBM Research): WATSON usually does very well at these factoid questions, looking up facts, you know? History, geography, entertainment.

TODD CRAIN: Clark Gable was happy to see him come in and finish directing Gone with the Wind. WATSON?

WATSON: Who is Victor Fleming?

TODD CRAIN: That is correct!

WATSON, you choose again.

WATSON: For 1,600, please.

TODD CRAIN: His Westerns include Wagon Master and Fort Apache. WATSON?

WATSON: Who is John Ford?

TODD CRAIN: Good for 1,600.

WATSON: Who is Mike Nichols?

TODD CRAIN: Good for 12. Last clue on the board for $800, here it is; Revenge of..., in 1978 was the fifth in this series of comedies directed by Blake Edwards.

WATSON: What is The Pink Panther?

TODD CRAIN: I just want to check, your buzzers are working? Just wanted to, wanted to check in and wake you from your nap.

NARRATOR: WATSON has made it into the cloud, but nowhere near championship level. Ferrucci must come up with a plan to somehow step up his performance, before the Jeopardy producers return.

The team has spent almost four years, enormous intellectual and emotional energy, and tens of millions of dollars on developing WATSON. But all this effort isn't just for a machine that can play Jeopardy. IBM has much bigger goals.

DAVE FERRUCCI: I'm already looking, sort of, beyond Jeopardy. I'm thinking, "Where can we go from here?"

NARRATOR: Ferrucci imagines a day when WATSON might perform much like the computer in Star Trek.

DAVE FERRUCCI: The captain just starts talking to the computer, and they say, "Computer!"

WILLIAM SHATNER (As Captain Kirk/Star Trek/Film Clip): Computer.

COMPUTER (Star Trek/Film Clip): Ready.

WILLIAM SHATNER (As Captain Kirk/Star Trek/Film Clip): Could a storm of such magnitude cause a power surge in the transporter circuits...

DAVE FERRUCCI: It's an information-seeking tool, right? That's capable of understanding your question...

WILLIAM SHATNER (As Captain Kirk/Star Trek/Film Clip) : ...creating a momentary inter-dimensional contact with a parallel universe?

DAVE FERRUCCI: ...and dialoguing with you to make sure that you get what you want.

COMPUTER (Star Trek/Film Clip): Affirmative.

DAVE FERRUCCI: Do you have one of those? I, I, I don't have one of those, right?

NARRATOR: You don't need to be a starship commander to find this helpful.

In a world overflowing with data, intelligent expert systems that can answer vital questions have been the Holy Grail. Now, we could be close to a WATSON, M.D.

DAVE FERRUCCI: Well, think of it this way, there's a bunch of information: new diagnoses, new treatment options, new discoveries. Can anybody keep that all in their head all at once?

NARRATOR: A machine that could access and organize all that information could help doctors analyze symptoms and wade through piles of medical journals.

DAVE FERRUCCI: It changes the paradigm in which we work with computers. That's the vision behind this.

WATSON: How am I doing? I hope we will have a good game today, but first I have to test my voice.

NARRATOR: Before that vision can be fulfilled, before WATSON can even compete on Jeopardy, the team needs to get more bugs out of his system.

One of WATSON's most embarrassing weaknesses is he cannot hear. Instead, WATSON receives each Jeopardy clue as an electronic text message, at the same moment his competitors see it on the board. As a result, he doesn't know the other contestants' answers.

TODD CRAIN: Only the female of this "equine" pest, of the family Tabanidae, feeds on blood; the male feeds on nectar.

BILL (Former Jeopardy! Contestant?) : What's a mosquito?

TODD CRAIN: No. WATSON?

WATSON: What is mosquito?

TODD CRAIN: No!

HARVEY (Former Jeopardy! Contestant?): What's a horsefly?

TODD CRAIN: Yes! Thank you for not saying mosquito! Good job. Good for $2,000, Harvey.

NARRATOR: Ferrucci and the team have been working furiously to boost WATSON's performance. As part of their plan, WATSON will now receive correct answers electronically, after they're revealed. If the fix works, it will be in the nick of time.

DAVE FERRUCCI: Hey, how are you?

NARRATOR: The Jeopardy producers are back and they're about to determine WATSON's fate.

DAVE GONDEK: This was a measure of our progress, and we wanted to hear, "Yeah, you're there. You've made it."

NARRATOR: The new and improved WATSON gets his first big test with a category called "Celebrations of the Month."

TODD CRAIN: Administrative Professionals Day and National C.P.A.s Goof-Off Day.

WATSON: What is "holiday?"

TODD CRAIN: No! That's not even close, really.

NARRATOR: WATSON fails, because he doesn't understand that in this category, the answer must be the name of a month, something his human competition quickly figures out.

TODD CRAIN: Arthur?

ARTHUR (Former Jeopardy! Contestant?): What is April?

TODD CRAIN: What is April? April 18th is...

DAVE FERRUCCI: I don't understand. I don't understand why we don't, we don't understand the question. We don't understand the category, basically.

TODD CRAIN: D-Day Anniversary and Magna Carta Day.

NARRATOR: But now he electronically receives these correct responses.

TODD CRAIN: Arthur!

ARTHUR : What is June?

NARRATOR: Can he learn from the answers?

TODD CRAIN: National Philanthropy Day and All Souls' Day.

DAVE GONDEK: What WATSON does here is, it sees that, "Well, all the answers I've seen have been the month in which the thing in the clue occurs."

TODD CRAIN: Matt?

MATT (Former Jeopardy! Contestant): What is November?

TODD CRAIN: Good for four.

DAVE GONDEK: So then it knows, in the next clue, to look for, um, "What month does this thing occur in?"

MATT: "Celebrations" for six.

TODD CRAIN: "Celebrations" for six. National Teacher Day and Kentucky Derby Day. WATSON?

WATSON: What is May?

DAVE GONDEK: Yes! He got it! He figured it out!

It took us four!

NARRATOR: Thanks to the team's efforts, WATSON is soaring higher in the cloud and now approaching the level of champions like Ken Jennings.

WATSON is on a roll.

TODD CRAIN: Choose again.

WATSON: "You're Tripping," for 1,600.

TODD CRAIN: Jacques Cartier found this largest island of the Hochelaga Archipelago while searching for gold. WATSON!

WATSON: What is Montreal?

TODD CRAIN: Yes.

WATSON: Who is Zebulon Pike?

TODD CRAIN: Good.

WATSON: What is Providence?

Aquarius?

Texas?

Beau Brummell?

TODD CRAIN: WATSON?

WATSON: What is "ciao?"

TODD CRAIN: That was right and cute all at the same time!

NARRATOR: The Jeopardy executives have finally seen enough.

HARRY FRIEDMAN: I think we've gone from impressed to blown away.

TODD CRAIN: Very nicely done, WATSON!

VOICE OFFSTAGE: Quiet!

NARRATOR: Finally, WATSON will get his chance on Jeopardy: a computer, playing against human champions,...

ALEX TREBEK: They say it can think...

NARRATOR: ...in a game that is a very symbol of intelligence.

ALEX TREBEK: ...but can it think like a Jeopardy champion?

NARRATOR: It will be a contest the world has never seen.

ALEX TREBEK: Good luck, WATSON.

NARRATOR: But does this mean that the dream of artificial intelligence is coming true?

RODNEY BROOKS: So artificial intelligence, to me, is trying to get computers to do stuff that, if people did them, you'd say, "Oh, they're demonstrating their "peopleness." That's what makes humans humans, that stuff they're doing."

NARRATOR: But without experience or emotion, can a computer like WATSON ever learn and understand the world the way we humans do, from early childhood on?

PAT WINSTON: Right now, no machine can understand the meaning of a play, what it means to be King Lear or Macbeth or Hamlet... "to be or not to be: that is the question."

No machine can understand the parables of the Judeo-Christian Bible. All they can do is shovel through data and find regularities.

NARRATOR: But for Dave Ferrucci, that kind of understanding was never the goal.

DAVE FERRUCCI: It's not going to emerge as a human, because it doesn't connect the information to human experience, to human cognition. When you think about a great symphony, and when a human sits down with that, that music is affecting that human on an emotional level. The computer doesn't have that human experience, doesn't have that human emotion. It's not human, it's a computer.

NARRATOR: WATSON may never experience the world the way we do, but with his enormous knowledge base, his skill at interpreting language, and his ability to learn...

TODD CRAIN: WATSON?

WATSON: What is May?

DAVE GONDEK: He figured it out!

NARRATOR: ...could he actually be considered intelligent?

SEBASTIAN THRUN: Oh, my god. It is more intelligent than the average Jeopardy player in answering Jeopardy questions. That's impressively intelligent.

WATSON: Good afternoon, Mr. Trebek. I've been waiting for this moment for a very, very, very long time.

NARRATOR: Finally, it's show time. WATSON is now taking the stage, where his intelligence will be put to the ultimate test in front of millions of Jeopardy viewers.

DAVE GONDEK: I think I dream about Jeopardy questions now. I have nightmares about Jeopardy questions.

NARRATOR: Has the team done enough?

They're about to find out, as WATSON meets the world's two best Jeopardy players: Brad Rutter and Ken Jennings.

DAVE GONDEK: We've never had this caliber of a player. And there's a reason why Ken won 74 games in a row; there's a reason why Brad has never been beat by a human.

NARRATOR: The whole team has been waiting four years for this moment.

ALEX TREBEK: Let's play Jeopardy! Here we go!

Four-letter word for the iron fitting on the hoof of a horse or a card-dealing box in a casino.

WATSON: What is shoe?

ALEX TREBEK: You are right. You get to pick.

NARRATOR: At first, man and machine look evenly matched.

ALEX TREBEK: WATSON!

WATSON: Who is Jude?

ALEX TREBEK: Yes!

WATSON: What is Last Judgment?

ALEX TREBEK: Right.

Brad.

BRAD RUTTER: What is the nineteen-aughts or the 1900s?

ALEX TREBEK: Yes! Ken.

KEN JENNINGS: What is Greece?

NARRATOR: At the end of the first round, it's anyone's game.

ALEX TREBEK: Sauron is right. That puts you into a tie for the lead.

DAVID GONDEK: The first round, five thousand-five thousand. They could beat us. We have seen it happen in sparring matches.

DAVE FERRUCCI: Certainly, if you wanted a cliffhanger after the first game, it is going well. Me? Not so much.

NARRATOR: But as the match continues, WATSON dominates, consistently beating the humans to the buzz.

ALEX TREBEK: WATSON.

WATSON: Who is Isaac Newton?

ALEX TREBEK: You are right!

WATSON: Who is C.S. Lewis?

ALEX TREBEK: Yes!

WATSON: What is guitar?

ALEX TREBEK: Right.

NARRATOR: It starts to look like a blowout...

ALEX TREBEK: And with that you move to 36,681.

NARRATOR: ...when, in the last round, Ken Jennings stages a comeback.

ALEX TREBEK: This national newspaper raised its newsstand price by 25 cents, to a dollar. Ken?

KEN JENNINGS: What is USA Today?

ALEX TREBEK: Right.

Ken?

KEN JENNINGS: What are national borders?

ALEX TREBEK: Good.

KEN JENNINGS: Martin Luther King?

ALEX TREBEK: Yes, you're right!

NARRATOR: Going into "Final Jeopardy," anything can happen.

DAVE FERRUCCI: We just want to avoid a stupid answer at this point.

ALEX TREBEK: Here is the clue: William Wilkinson's An Account of the Principalities of Wallachia and Moldavia inspired this author's most famous novel.

"Who is Bram Stoker?" You are correct; the author of Dracula.

Over to Ken Jennings now, and we find, "Who is Stoker?"

I, for one, welcome our new computer overlords.

NARRATOR: If WATSON gets this one, he will win.

DAVE FERRUCCI: If we get "Final Jeopardy" right, I'm going to kiss Eric.

ALEX TREBEK: Now we come to WATSON. Looking for Bram Stoker, and we find, "Who is Bram Stoker?"

DAVE GONDEK: It's kind of overwhelming. We won. And I think people will say, "Yup, WATSON was the best player there."

NARRATOR: No question, WATSON's victory is a major milestone for A.I., but perhaps it's an even greater achievement for the human intelligence that created him.