Communications of the ACM
Code Is Key to Successful Economic Study Replication, Researcher Says
Credit: The Brookings Institution
Colleen Carey, an assistant professor of policy analysis and management at Cornell University, is co-author of a study that was successfully replicated by the Federal Reserve team that looked at 67 papers from reputable economics journals and found fewer than half of the papers could be replicated. She explains why her paper was successfully replicated and the importance of posting code, which makes it possible for other researchers to catch errors, or identify data-analysis choices that change the results.
"From the Peaks to the Valleys: Cross-State Evidence on Income Volatility Over the Business Cycle" is published in Review of Economics and Statistics.
"What the Federal Reserve authors did was test whether the 'replication policies' that journals have instituted in the past decade are facilitating replication," Carey says. "My co-author and I complied with the replication policy fully, as Chang and Li can confirm. But many authors do not comply or do not comply fully enough to allow replication.
"After my paper with Stephen Shore was published, I sent the journal all the code that we had used to upload our original raw data, massage it into the analytical dataset on which we would estimate our results, and then run the key regressions and make the results into tables. What Chang and Li did was download it and run the code. This sounds simple, but it isn't. It can be very difficult to make sense of someone else's code; in addition, I was a novice at coding – this was my first big project – so I often coded in a very roundabout way. But I made sure the code worked when I uploaded it, so in a way I'm not that surprised that it worked for them."
"Better compliance with journal policies would help economists ensure that our policy-relevant findings are actually correct. Posting code makes it possible for other researchers to catch errors, or identify data-analysis choices that seem innocuous but actually change the results. Most importantly, it makes it much easier to experiment with alternative empirical strategies that can provide deeper insight into the underlying economic phenomena," Carey says.
No entries found