Communications of the ACM
We Need to Automate the Declaration of Conflicts of Interest
Credit: Alexander Supertramp
Over the last 70 years of computer science research, our handling of conflicts of interest has changed very little. Each paper's corresponding author must still manually declare all their co-authors' conflicts of interest, even though they probably know little about their most senior coauthors' recent activities. As top-tier conference program committees increase past 500 members, many with common, easily confusable names, PC chairs with thousands of reviews to assign cannot possibly double-check corresponding authors' manual declarations against their paper's assigned reviewers. Nor can reviewers reliably catch unreported conflicts. Audits at recent top-tier venues across several areas of computer science each uncovered more than 100 instances where, at the first venue, a pair of recent coauthors failed to declare their conflict of interest; at the second venue, someone was assigned to review a recent co-author's submission; and at the third venue, someone reviewed a submission written by a prior co-author from any year. Even the concept of a conflict deserves closer scrutiny: an audit at yet another recent top-tier venue edition found more than 100 cases in which prior co-authors from any year reviewed each other's submissions.
These are issues of scale. Seventy years of exponential growth have turned our little village into a metropolis, and our handling of conflicts of interest (conflicts for short) has not kept pace with our community's growth. But we computer scientists are experts at scaling up! We have already addressed issues of scale in many other aspects of our review processes, including enhancements such as double-blind review, multiple submission deadlines, opportunities for revision and rebuttal, and online submission and review management systems.
Comments
Sushil Jajodia
I wholeheartedly agree - it's high time that this is done. Perhaps NSF will see fit to make this happen.
Sushil Jajodia
Gunnar Wolf
Thank you very much for this viewpoint, I much enjoyed reading it. And most important it got my brain working on a related issue.
"Conflict of interest" is most often defined in terms of human-to-human relations. I cannot be trusted not to be biased for or against somebody I know personally. But my feeling is that a much larger conflict of interest should be declared on topics (or even viewpoints on given topics) where each of us has a strong professional or emotional investment.
After all, when working as a reviewer for a conference or a journal, papers are most often anonymized. As you point out, I might have worked with a given author several years back, and my conflict of interest with her should have dilluted; I won't even know what topic she is working on, and should be reasonably shielded from bias when reviewing her newest paper.
But conflicts of interest on topics or viewpoints are much harder and carry a much stronger bias; in all but the closest inter-personal relationships, we should be trusted to behave professionally and not bias our ratings of a person's work on a non-objective basis. But it is quite hard for me to properly judge without bias an article that supports or contradicts my core beliefs!
As an example, I recently was asked to review an article on an implementation of e-voting (won't discuss its details further). I have worked over a decade on trying to explain to the public the dangers of digitizing the voting process, and it's very hard for me to judge based only on the facts stated on the article, but I think I did a decent job. We were four reviewers; three of us gave our opinions and suggested "major revisions" based on missing facts and views on the process outlined. The fourth reviewer made a very short review, consisting basically of "there is no way this can be made to work reliably". This fourth reviewer matched my gut feeling but I think that was not a professional, academic, article-merit-based answer. No conflict of interest was present in the sense the reviewer did not seem to be biased against the author, but they were clearly biased against the _topic_.
Of course, declaring a conflict on such an issue... should not come as automatic. We should always try and set our personal viewpoints aside, judging articles on their real merits based on the scientific method. And if I have invested a decade of my life in a topic... Well, I should be regarded as an expert on it, and thus am a good reviewer. But I should work _really_ hard on showing a biased behavior.
Displaying all 2 comments