Communications of the ACM

Indeed, there may be as much as $2 trillion worth of computer-related information assets that are important in the U.S. economy but are largely unmeasured, says Erik Brynjolfsson, director of the MIT Center for Digital Business.

As of 2012, about 2.5 exabytes of personal data are created each day–and that number is doubling every 40 months or so.

Meanwhile, more data cross the Internet every second than were stored in the entire Internet just 20 years ago. This gives companies an opportunity to work with many petabyes of data in a single data set—and not just from the Internet. For instance, it is estimated that Walmart collects more than 2.5 petabytes of data every hour from its customer transactions.

[A petabyte is one quadrillion bytes, or the equivalent of about 20 million filing cabinets’ worth of text. An exabyte is 1,000 times that amount, or one billion gigabytes.]

"We have all become walking data generators," says Brynjolfsson. "Every time we move around with our mobile phones, we are spewing enormous amounts of bits that tell where we are, what we’re buying, who we’re talking to, and what our preferences are. That information can be captured, aggregated, and targeted to help companies understand consumer demand better and ultimately develop products and promote products tailored to that consumer demand."

It’s not just that more streams of data are being generated but entirely new streams of data, says Justin Grimmer, assistant professor in Stanford University’s Political Science Dept.

Previously, Grimmer says, if companies wanted to develop a business strategy, they might send a survey to customers to get generalized information. "But today," he says, "social media enables companies to tailor what they do to very specific characteristics of their customers and to be responsive to what they are telling them by monitoring Twitter and Facebook posts. Companies can then customize their advertising and their new product development accordingly."

Decisions are also increasingly based on data and analysis rather than experience and gut feelings, Grimmer adds.

MIT’s Brynjolfsson did a study of several hundred large companies to understand whether they were becoming more data-driven in their decision-making. While a third had not, another third relied extensively on big data and analytics to make decisions.

"We took the results and compared it to their business performance," Brynjolfsson recalls. "What we found was the third that was more data-driven was significantly more productive–about 5-6% more productive–than their competitors that weren’t as data-driven."

Researchers observe big data has spawned considerable opportunities for computer scientists whose skillsets are becoming more in demand as companies pay a premium for people who can organize and manage very large datasets and structure them in a way that one can get meaningful information from them.

And, in the academic world, big data is affecting the rapid growth of interdisciplinary collaboration.

"You see people trained in the social sciences who are taking CS courses and very frequently collaborating with computer scientists who are attending social science conferences and new conferences formed to address both fields," says Gary King, director of Harvard’s Institute for Quantitative Social Science. "That may be the next big academic discipline to emerge. Not that we need another silo; it just reflects the connections between these two big areas that hadn’t been connected as much before."
 

Video: Erik Brynjolfsson on "Going Beyond Big Data To Nanodata"