Communications of the ACM
Incentivizing Quality and Impact in Computing Research
Over the past few years, the computing-research community has been conducting a public conversation on its publication culture. Much of that conversation has taken place in the pages of Communications. (See http://cra.org/scholarlypub/.) The underlying issue is that while computing research has been widely successful in developing fundamental results and insights, having a deep impact on life and society, and influencing almost all scholarly fields, its publication culture has developed certain anomalies that are not conducive to the future success of the field. A major anomaly is the reliance of the fields on conferences as the chief vehicle for scholarly publications.
While the discussion of the computing-research publication culture has led to general recognition that the "system is suboptimal," developing consensus on how the system should be changed has proven to be exceedingly hard. A key reason for this difficulty is the fact the publication culture does not only establish norms for how research results should be published, it also creates expectations on how researchers should be evaluated. These publication norms and research-evaluation expectations are complementary and mutually enforcing. It is difficult to tell junior researchers to change their publication habits, if these habits have been optimized to improve their prospects of being hired and promoted.
Comments
Rizos Sakellariou
The question is: after all this inflationary pressure, are we in a good position to judge quality? Or, for as long as quantity and metric-based assessment are favored by high-level management (let alone the buzz, big data and the like), eventually there will be allies who win the debate? Isn't this something similar to "bad currency drives out good currency"?
Ben Shneiderman
While we are all in favor of high quality research publications, the dilemma is that hiring, tenure and promotion committee consist of senior faculty whose definitions of quality may not be sympathetic to the topics and methods used by younger researchers, thereby slowing innovation in our fast moving field. By balancing subjective impressions with objective data on download counts, social media discussion, and eventually citation counts, young researchers can demonstrate the value of their work to skeptical senior faculty.
Roberto Sebastiani
"We should not be impressed by large research grants, but ask what the actual yield of the funded projects has been." Two thumbs up for this sentence (among others). In an era in which -- at least in Europe -- careers are evaluated mostly for the amount of money one brings in, it was time some very-authoritative scientist raised this issue.
My two cents on the issue "A major anomaly is the reliance of the fields on conferences as the chief vehicle for scholarly publications."
IMHO the main problem is, most CS journals -- some with notable exception -- have reviewing times which are geological eras wrt. the evolution speed of CS (e.g., I've personally experienced 18-months for having the first review) so that journals are often no more considered by a computer scientist as a vehicle to spread the visibility of his/her results, but only as a cumbersome and incredibly slow way of improving his/her own CV.
Joshua Grochow
Although I agree with the opinion, I have often wondered how to reconcile the desire to recognize true quality, rather than to count beans, with scale? My understanding is that the number of applications per faculty position is often in the hundreds, and the Malthusian problem in academia is not getting any better (even with the recent increase in CS tenure lines, following increasing enrollments in CS undergraduate majors).
Displaying all 4 comments