A computer has duped humans into thinking it is a 13-year-old boy from Ukraine, becoming the first machine to pass the iconic Turing test. Russian-born Vladimir Veselov, the U.S.-based co-creator of the winning chatbot -- short for chatter robot -- hopes the milestone will help raise interest in artificial intelligence.
RFE/RL: Your computer program made history earlier this month when it passed the famous Turing test, which assesses a machine's ability to engage in human-like conversation. It was the first time a computer successfully passed the test, set by celebrated British computer scientist and mathematician Alan Turing in 1950. The program created by your team was able to convince a panel of judges, with a 33-percent success rate, that he was a 13-year-old boy from Ukraine named Eugene Goostman. Does this mean Eugene can be described as artificial intelligence?
Veselov: Our robot can imitate a five-minute dialogue with a human. A lot of people are now voicing doubts that our robot truly passed the test. The test, however, was really quite objective. There were 30 judges and during every round they had two windows open -- one with a real person, another with a robot. They didn't know which was which. They each held two five-minute dialogues.
RFE/RL: What kind of questions was asked?
Veselov: Mostly along the lines of: "Hi, how are you? Who are you? Where are you from?" But some interrogators also asked tricky questions like "Paul is playing football. What is Paul doing?" or "What color is the red car?"
RFE/RL: When did you start working on this program?
Veselov: In 2001, 13 years ago. So Goostman is now as old as his incarnation.
RFE/RL: Was this program created specifically to compete in the Turing test, or did it initially have other purposes?
Veselov: We wanted to create a universal tool which was not merely a chatbot, which could be used to process texts, to calculate the costs of complex products like insurance policies, for example. Then we started thinking about how to demonstrate this technology. One of the simplest, most accessible ways turned out to be the Loebner prize (Eds: An annual competition using the Turing test), which is open to everyone.
I'm amazed at the number of critical and at times downright offensive comments. Some people claim the Turing test was not conducted properly. It's true the interrogators were not all experts, like at the Loebner prize. Many didn't know how to ask tricky questions. But Alan Turing himself said the judges, the interlocutors, should be ordinary people, "average interrogators."
We can't really talk about a historic step in the development of artificial intelligence, however. In a way, it's the robot as a literary and psychological creation that passed the test.
RFE/RL: So you are saying the Turing test investigates a computer's ability to imitate humans rather than the actual presence of artificial intelligence?
Veselov: Yes, although I do think creating artificial intelligence is possible. When something can be clearly defined, it can often be automated. We are learning to automate a growing number of actions that we once thought were restricted to humans, for example text recognition and voice recognition. Now Google is starting to produce cars that will be driven by computers. This is a complex algorithm, you could call it artificial intelligence.
RFE/RL: Nonetheless, does Eugene bring us a step closer to artificial intelligence, or are we talking about entertainment technology?
Veselov: I would say it's more of an entertainment technology. On the upside, I hope it will help draw attention to this topic. It would be great if more schoolchildren and students stopped playing computer games and instead tried to develop better robots, to create genuine artificial intelligence.
RFE/RL: What made you choose Eugene Goostman, a 13-year-old boy from Odesa in Ukraine, as your program's incarnation? Is it because a teenager is expected to answer questions less skilfully than an adult?
Veselov: That may have been part of the reasoning. But ultimately, anyone can say they don't know the answer to a question. Programs are not required to imitate professors. It's [co-creator] Yevgeny Demchenko who came up with this character, he grew up in Ukraine himself. There are a number of topics that can be discussed [with Eugene Goostman], including his father, his mother. We thought it was a rather original idea.
RFE/RL: Did you prepare Eugene for the test? Judges were likely to ask about the current political crisis in Ukraine. But Eugene, it turns out, still believes Leonid Kuchma is the president of Ukraine.
Veselov: No, we didn't prepare at all. Modifying the knowledge base is a risky procedure. When you add new information, new patterns can overlap with older ones and the overall quality can suffer.
RFE/RL: What's the difference between the Turing test and the Loebner prize?
Veselov: The rules for the Loebner prize have changed, now there is a preselection. Fifteen questions are asked, the same for all robots. The four robots that give the best answers make it to the final round. So what is assessed is less the ability to hold a conversation than to answer specific questions. In the end, the winning robots don't really resemble humans.
I would describe the Loebner prize as a kind of beauty contest for robots. It focuses mostly on measuring the intellectuality of robots, their ability to answer tricky questions. The test we passed had a slightly different goal. But it's hard to say which contest is more important for science.
RFE/RL: Your computer program made history earlier this month when it passed the famous Turing test, which assesses a machine's ability to engage in human-like conversation. It was the first time a computer successfully passed the test, set by celebrated British computer scientist and mathematician Alan Turing in 1950. The program created by your team was able to convince a panel of judges, with a 33-percent success rate, that he was a 13-year-old boy from Ukraine named Eugene Goostman. Does this mean Eugene can be described as artificial intelligence?
Veselov: Our robot can imitate a five-minute dialogue with a human. A lot of people are now voicing doubts that our robot truly passed the test. The test, however, was really quite objective. There were 30 judges and during every round they had two windows open -- one with a real person, another with a robot. They didn't know which was which. They each held two five-minute dialogues.
RFE/RL: What kind of questions was asked?
Veselov: Mostly along the lines of: "Hi, how are you? Who are you? Where are you from?" But some interrogators also asked tricky questions like "Paul is playing football. What is Paul doing?" or "What color is the red car?"
RFE/RL: When did you start working on this program?
Veselov: In 2001, 13 years ago. So Goostman is now as old as his incarnation.
RFE/RL: Was this program created specifically to compete in the Turing test, or did it initially have other purposes?
Veselov: We wanted to create a universal tool which was not merely a chatbot, which could be used to process texts, to calculate the costs of complex products like insurance policies, for example. Then we started thinking about how to demonstrate this technology. One of the simplest, most accessible ways turned out to be the Loebner prize (Eds: An annual competition using the Turing test), which is open to everyone.
I'm amazed at the number of critical and at times downright offensive comments. Some people claim the Turing test was not conducted properly. It's true the interrogators were not all experts, like at the Loebner prize. Many didn't know how to ask tricky questions. But Alan Turing himself said the judges, the interlocutors, should be ordinary people, "average interrogators."
We can't really talk about a historic step in the development of artificial intelligence, however. In a way, it's the robot as a literary and psychological creation that passed the test.
RFE/RL: So you are saying the Turing test investigates a computer's ability to imitate humans rather than the actual presence of artificial intelligence?
Veselov: Yes, although I do think creating artificial intelligence is possible. When something can be clearly defined, it can often be automated. We are learning to automate a growing number of actions that we once thought were restricted to humans, for example text recognition and voice recognition. Now Google is starting to produce cars that will be driven by computers. This is a complex algorithm, you could call it artificial intelligence.
RFE/RL: Nonetheless, does Eugene bring us a step closer to artificial intelligence, or are we talking about entertainment technology?
Veselov: I would say it's more of an entertainment technology. On the upside, I hope it will help draw attention to this topic. It would be great if more schoolchildren and students stopped playing computer games and instead tried to develop better robots, to create genuine artificial intelligence.
RFE/RL: What made you choose Eugene Goostman, a 13-year-old boy from Odesa in Ukraine, as your program's incarnation? Is it because a teenager is expected to answer questions less skilfully than an adult?
Veselov: That may have been part of the reasoning. But ultimately, anyone can say they don't know the answer to a question. Programs are not required to imitate professors. It's [co-creator] Yevgeny Demchenko who came up with this character, he grew up in Ukraine himself. There are a number of topics that can be discussed [with Eugene Goostman], including his father, his mother. We thought it was a rather original idea.
RFE/RL: Did you prepare Eugene for the test? Judges were likely to ask about the current political crisis in Ukraine. But Eugene, it turns out, still believes Leonid Kuchma is the president of Ukraine.
Veselov: No, we didn't prepare at all. Modifying the knowledge base is a risky procedure. When you add new information, new patterns can overlap with older ones and the overall quality can suffer.
RFE/RL: What's the difference between the Turing test and the Loebner prize?
Veselov: The rules for the Loebner prize have changed, now there is a preselection. Fifteen questions are asked, the same for all robots. The four robots that give the best answers make it to the final round. So what is assessed is less the ability to hold a conversation than to answer specific questions. In the end, the winning robots don't really resemble humans.
I would describe the Loebner prize as a kind of beauty contest for robots. It focuses mostly on measuring the intellectuality of robots, their ability to answer tricky questions. The test we passed had a slightly different goal. But it's hard to say which contest is more important for science.
