Communications of the ACM
Understanding and Improving Crowdsourcing Models
Credit: Courtesy of Growbusiness.com
In recent years, crowdsourcing has gained more adherents as governments, businesses, and others aim to tap into the collective wisdom of crowds. However, the technique has limitations and shortfalls, particularly when it comes to gauging the value and accuracy of contributions.
Now, a network of researchers from Applied Research Associates (ARA) in Raleigh, North Carolina, and seven U.S. universities are attempting to create more efficient and useful crowdsourcing models that could be used to predict everything from terrorist attacks to changes in societal eating habits.
The project, called Aggregative Contingent Estimation System (ACES), was initiated in July with funding from the U.S. agency Intelligence Advanced Research Projects Activity (IARPA). "There is a lot of evidence that crowdsourcing is very successful," says Dirk Warnaar, principal investigator at ARA. "The goal of this project is to further refine methods and make forecasting more accurate."
The team has created a Web site, Forecastingace.com, and has invited the public to register and participate for the duration of the four-year project. All participation is anonymous so that volunteers can provide honest and uncensored thoughts and opinions, Warnaar explains. So far, more than 1,600 people have signed up.
Researchers gather all the predictions from the site and drop them into a database, where they're analyzed for accuracy. The team uses both statistical and cognitive modeling methods to develop probabilistic models and "best understand what the aggregated judgment should be," says Mark Stevers, a University of California, Irvine professor who manages data and content aggregation for the ACES project.
The program also provides feedback to participants about how accurate their predictions are. "People tend to be overconfident in their forecasts and part of the goal is to help them 'calibrate' and understand how they can improve," notes Eric Stone, an associate professor of psychology at Wake Forest University. He says that researchers are exploring ways to keep participants engaged, including competitions, game metaphors, and interfaces.
The end goal is to improve the current forecasting system, which typically relies on a few acknowledged analysts and experts to make predictions. Not only do natural biases enter the picture, some people have their own axe to grind or are paid by companies they're researching, Warnaar says. The crowdsourcing algorithms the ACES group is developing determine who's best at making predictions, or certain types of predictions, and it's able to weight these individuals more heavily when building a forecast.
"What's exciting about this project is we're using crowdsourcing procedures that have been tested in the lab and specific environments, but we're now applying them on a large scale," Stone says. "Improving the quality of forecasting has real world benefits."
Sidebar: A Timeline of Bad Forecasts
1943: "I think there is a world market for maybe five computers." –Thomas Watson, chairman of IBM.
1946: Television won't be able to hold on to any market it captures after the first six months. People will soon get tired of staring at a plywood box every night." –20th Century Fox mogul Darryl Zanuck. Number of hours per day that TV is on in an average U.S. home: 6 hours, 47 minutes.
1962: "We don't like their sound, and guitar music is on the way out." –Decca Recording Co. rejecting the Beatles. To date, The Beatles have sold approximately 1.2 billion records.
1981: "No one will ever need more than 637 KB of memory for a personal computer. 640 KB of memory should be enough for anybody." –Bill Gates, co-founder and CEO of Microsoft. In 2011 the average personal computer has 1 Gigabyte of memory. There are 1 million KB in 1GB.
2005: "Next Christmas the iPod will be dead, finished, gone, kaput." –Sir Alan Sugar. Apple has reportedly sold 220 million iPods since 2001.
Source: ARA
Samuel Greengard is an author and journalist based in West Linn, Oregon.
No entries found