Communications of the ACM
Technical opinion
Assessing the Value of Is Journals
What is the value of a journal? Over the years, journals have come to fulfill many different goals for authors, editors, and reviewers alike. However, most would agree that the real value of a journal is in the dissemination of knowledge for use by others. Typically, we would expect that the more superior the journal, the better the papers, the greater the dissemination of knowledge, and the higher the prestige. If papers are often cited by others, then the contribution to the discipline is obvious and is expounded by the dissemination of knowledge.
Recently, in the information systems area, there has been a proliferation of new journals. More than 100 journals are currently being published and more are on the way every year. As an academic, it is becoming increasingly difficult to recognize the quality of outlets for research. Given this situation, it is perhaps not surprising that the branding of established and traditional journals has carried through in the perceptions of many as the key outlets for our research. However, what is surprising is that few ever actually question the value of traditional outlets, which may be seen by some as `untouchable'.
None would argue that the top-ranked journals in recent surveys have high quality and rigorous review processes. To have papers accepted for publication in those journals, to present them to a worldwide audience via traditionally prestigious outlets, is an achievement and a career goal. Notwithstanding, once those papers have been published, what then? Will people recognize the value of research and begin to use it? Will they pull the information and cite it?
With this in mind, we set about establishing whether perceptions of IS journals matched with the reality of dissemination. The information most readily available on dissemination is that of citation, and, for example, this has recently been applied to computer science journals [3]. In this case, we used the journal citation reports published by ISI, both from the sciences (the Science Citation Index 2002) [2] and social sciences (the Social Sciences Citation Index 2000) [1]. Against these, we examined the results of one of the most recent and most cited surveys of global perceptions of IS journals, published in Communications in September 2001 [4]. Within the IS discipline there are various communities, such as those that cut across geographic and thematic boundaries [4]. However, such an analysis is beyond the scope of this columnas is typical in these discussions, we focus on the global IS community as the unit of analysis. Table 1 shows the global ranking of the top 50 journals in the survey along with the impact factor of the journal. Non-specialist journalsjournals that are more oriented to business, management, or operations researchare highlighted in the table.
Sorting the journals by impact factor reveals some interesting results and these are shown in Tables 2 and 3. We have distinguished between IS outlets and general business, management, or operations research. It is worth noting that 12 of the journals did not have impact factors according to the data we had available (see Table 1) and these have been omitted from Table 2. However, none of these were in the top rankings, the first starting at rank 18. In addition, IEEE and ACM Transactions had highly varying levels of impact and thus were also omitted.
The data suggests there is consistency in several of the top-rated journals, but wide variety in others. At the top of the rankings is Communications, which clearly has the highest impact of all journals assessed. Close behind is MIS Quarterly, again with an impact factor above two. This is followed by a more technical journal, Human-Computer Interaction, and Information and Management, both of which have been promoted by the ranking. IS Research falls to fifth in the ranking. Overall, the biggest gainers are the Journal of the ACM, Behaviour and IT, and Human-Computer Interaction. The biggest demotions are for Journal of MIS, Data Base, and the Information Systems Journal.
In the business-oriented journals, there is also some change in the ranking. It is worth noting that four of the journals had impacts above those of the IS journals, but that these are more general outlets and thus not easily comparable. The Academy of Management publications rated in the first four, along with Administrative Science Quarterly and Harvard Business Review. All had impact factors above two. Overall, the biggest gainers were Organisational Behaviour and Human Decision-making, Operations Research, and the Academy of Management Review. The publications experiencing the biggest declines were Decision Sciences and Management Science. It is worth bearing in mind that operations research journals tend to be more quantitative and specialized, and consequently tend to have lower readerships. Thus, this is likely to lead to lower impact scores for this subset of journals, highlighted in Table 3.
The data suggests there is consistency in several of the top-rated journals, but wide variety in others.
Taken at face value, these results imply a difference in the perceptions of and use of material from different journals publishing IS research. There are some obvious limitations to the analysis, particularly the absence of impact scores for lesser-ranked or newer journals. Such journals typically have a time lag in building a citation impact score. Nevertheless, the remaining data does paint an indicative picture. Some IS journals appear to have rankings that contradict their `true' value to other researchers; others have inflated perceptions and little impact on the IS discipline. For some of the best journals, the ranking is relatively consistent, which is comforting, but for some of the mid-ranked journals the picture appears very muddy indeed.
We all have perceptions of journals that have been developed, for example, by branding and our own experiences in submitting, editing, and reviewing papers. Whether perceptions meet with the reality of research dissemination is a question that is open for debate. The real value of a journal is in what we get out of it, not its history, process, or what we put in.
References
1. ISI. Science Citation IndexJournal Citation Report. Thomson ISI, Philadelphia, 2002.
2. ISI. Social Science Citation IndexJournal Citation Report. Thomson ISI, Philadelphia, 2000.
3. Katerattanakul, P., Han, B. and Hong, S. Objective quality ranking of computing journals. Commun. ACM 46, 10 (Oct. 2003), 111114.
4. Mylonopoulos, N.A. and Theoharakis, V. Global perceptions of IS journals. Commun. ACM 44, 9 (Sept. 2001), 2933.
Tables
Table 1. Perception and impact of IS journals.
Table 2. Ranking of journals by impactinformation systems.
Table 3. Ranking of journals by impactbusiness, management, and OR.
©2005 ACM 0001-0782/05/0100 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2005 ACM, Inc.
No entries found