Communications of the ACM
The Real Software Crisis: Repeatability as a Core Value
Where is the software in programming language research? In our field, software artifacts play a central role: they are the embodiments of our ideas and contributions. Yet when we publish, we are evaluated on our ability to describe informally those artifacts in prose. Often, such prose gives only a partial, and sometimes overly rosy, view of the work. Especially so when the object of discourse is made up of tens of thousands of lines of code that interact in subtle ways with different parts of the software and hardware stack on which it is deployed. Over the years, our community's culture has evolved to value originality above everything else, and our main conferences and journalsa deny software its rightful place.
Science advances faster when we can build on existing results, and when new ideas can easily be measured against the state of the art. This is exceedingly difficult in an environment that does not reward the production of reusable software artifacts. Our goal is to get to the point where any published idea that has been evaluated, measured, or benchmarked is accompanied by the artifact that embodies it. Just as formal results are increasingly expected to come with mechanized proofs, empirical results should come with code.
Comments
Sven Apel
Hi Shriram & Jan,
Thanks for your insights, its a great article!
Interestingly, I noticed a connection to the problem of encouraging,
conducting, and evaluating replication studies in software-engineering research.
Based on a survey involving 79 PC and editorial-board members of major
software-engineering venues, we found that there is a deep confusion and
disagreement in the community regarding the value and guidelines of
empirical studies, in general, and replication, in particular.
http://www.infosun.fim.uni-passau.de/cl/publications/docs/SiSiAp15.pdf
The situation is not unlike the situation you outline...
Best,
Sven
Displaying 1 comment