Can a computer program compete with humans (and maybe beat them) in playing a quiz game that requires language understanding and wide knowledge?
IBM already answered this question with a big YES when Watson beated the top champions of Jeopardy! in 2011.
They emplyed a team of 30 researchers for 4 years in order to reach that historical goal, but can I do it too?
I approached a similar, but probably easier problem, playing "Who Wants to Be a Millionaire?", a quiz game where the player just has to choose among 4 possible answers. Nonetheless the problem was challenging enough to try to solve it.
I decided to rely only on high quality knowledge sources, Wikipedia and DBpedia, rather than leveraging the whole web, and to search in them for evidence that supported each of the four possible answers and rank them according to this evidence. I came up with 1200 different criteria including several distributioanl semantic models and combined them with a machine learning algorithm (Random Forests with regression trees used as a pointwise learning to rank algorithm).
I than asked about 100 university students, researchers and professors to answer a set of questions copied from the "Who Wants to Be a Millionaire?" uk and italian boardgames, so that I could have a baseline to compare to.
The results were beyond expectation: humans had an accuracy of about 50% while the program has almost 80% of the answers correct. It was morehover, a lot better in playing the game too, winning an average of € 115000 compared to the € 6000 average won of the humans.
This plot shows the performances of the program and the humans (and a baseline using Google results) for all the different levels of the game. At each level the questions are supposed to get harder to answer.
I published a conference and a journal paper about this work, the latter, titled "Playing with Knowledge: A Virtual Player for “Who Wants to Be a Millionaire?” that Leverages Question Answering Techniques", is way more detailed and interesting to read. You may find it in the Publication page or click here.