Communications of the ACM
AIs Are Being Tested to See How Well They nderstand Our Thoughts
Researchers at the University of California, Berkeley, have created a set of tests that probe the progress of artificial intelligences in understanding the world; none have passed the tests yet,
Credit: Kosei Saito/Getty
University of California, Berkeley (UC Berkeley) researchers have created a set of tests to explore the progress of artificial intelligence (AI) in understanding the world.
Each test is comprised of a short paragraph describing a scenario involving people and a few questions for the AI to answer about it; these queries revolve around identifying first and second order beliefs.
UC Berkeley's Aida Nematzadeh produced 10,000 scenarios and associated questions that tested theory of mind via first and second order beliefs, and her team fed them to four AIs, none of which successfully passed (the pass level was 95%). The highest-scoring AI achieved a score of 94.3%.
The team found inserting a sentence with no relation to the tested scenarios could fool the AIs, whose scores declined by 5% to 20%. The implication was that the AIs were failing to properly grasp the meaning of the text.
From New Scientist
View Full Article
Abstracts Copyright © 2018 Information Inc., Bethesda, Maryland, USA
No entries found