Communications of the ACM
Home/Magazine Archive/September 2001 (Vol. 44, No. 9)/On Site: Global Perceptions of Is Journals/Full Text
Communications of the ACM
On Site: Global Perceptions of Is Journals
As the pressure for scientifically rigorous and relevant research mounts [10], authors need to identify those outlets with the highest visibility for their work and those publications in which readers seek the best sources for informed IS research.
Increasingly, academics and institutions all over the world place significant importance to journal rankings for promotion, tenure, and assessment purposes. In particular, U.S. college and university faculty promotion and tenure decisions are decided on the basis of academic research output in top-tier journals. Furthermore, the Research Assessment Exercise in the U.K. ranks university departments for the purpose of distributing government research funds. This process measures research excellence by assessing where faculty publishes, taking into account the respective journal standing. There is evidence to suggest that universities on both sides of the Atlantic increasingly use journal lists for internal assessment and promotion purposes.
We report survey results that contribute to the general interest in journal ranking. Prior studies of IS journal rankings have been limited to the North American (Canada and the U.S.) IS community. However, there is substantial research evidence that academics from different regions of the world have different research approaches [1]. Similarly, IS practitioners from different countries assign different priorities to IS management issues [8]. Taking this perspective into consideration, we seek to identify if IS journals are perceived differently across regions of the world. We also provide three measures of journal ranking in order to generate a richer picture of how quality may actually be interpreted. Moreover, our questionnaire design addresses an important issue raised in prior research. Hardgrave and Walstrom [3] rightly argue their data cannot make the common distinction between A-level and B-level journals. We address this issue head-on by asking respondents to make the classification themselves. Finally, in an effort to be as representative as possible we were able to attract nearly 1,000 responses, by far the largest sample in this type of research.
Data collection was carried out through an online questionnaire. We targeted members of the ISWorld mailing list and the IS Faculty Directory on www.isworld.org. These sources represent the most complete and authoritative community of "information management scholars and practitioners."1 Of the 3,855 email invitations that we sent out, 1,094 bounced (2,761 email recipients) and 1,010 responses were collected of which 979 were usable. This represents a 35.45% usable response rate. By region, we received 605 responses from North America, 224 from Europe and 150 from Australasia (Asia, New Zealand, and Australia). The questionnaire asked respondents to rank up to 10 journals they consider to be in the top tier in terms of their contribution to the IS field. Next, respondents were asked to rank another 10 journals as second tier.2 Eighty-seven journals were listed to facilitate respondent recall and selection. The list of journals was compiled by drawing on previous studies [25, 9]. Further academic and demographic information was sought on the second page of the questionnaire.
Table 1 summarizes respondent profiles. Full professors represent 27% of the sample, associate professors (or equivalent) represent 29%, and assistant professors (or equivalent) represent 30%. Another 10% of the sample are Ph.D. students who have joined the ISWorld mailing list. The vast majority of respondents possess a doctorate (82%). While most of the respondents have had their highest degree in IS (63%), a sizeable 13% have a background in economics and management and another 9% in computer science.
North America, Europe, and Australasia: Are They Worlds Apart?
A recurrent argument in the IS research debate refers to the diverse approaches adopted by researchers in different parts of the world [7]. The present survey offers an excellent opportunity to revisit this issue in the context of journal perceptions. Table 2 presents three measures of journal ranking by region. The data is presented in a fashion similar to [6], which studied the perceptions of academics toward marketing journals. The two leftmost columns list journals in order of popularity in the whole (world) sample. Popularity corresponds to votes, in the sense that it counts the number of times respondents selected to rank the particular journal. For each region, alongside the popularity count we indicate the respective rank in parentheses. Average position is the average placement of a journal by respondents that chose to rank it. "Percent Top 10" is the percentage of occurrences of the journal in the top tier. For example, 850 respondents from the world sample chose to rank the MIS Quarterly (MISQ) and 95% of them placed it in the top tier, giving it an average position of 3.1.
There are certain notable similarities and differences in the ways respondents from North America, Europe, and Australasia perceive their journals, with the latter positioned somewhere in between the other two. With respect to the use of journal lists for promotion purposes, 42% North American respondents reported the use of such lists, which is twice that for Europe (Table 1). Of the 10 most popular journals in North America, six are also in the European top 10 and eight in the Australasian top 10. In terms of "Popularity," MISQ is first in North America, shares the first position with Communications in Europe, while the latter has a slight lead in Australasia Journal of MIS (JMIS) ranks 4th in popularity in North America whereas it makes it only to the 14th most popular position in Europe and 8th in Australasia. While the European Journal of IS (EJIS) appears to be fairly popular in Europe and Australasia (3d and 4th, respectively) it is nowhere near as popular in North America (13th). Data Base ranks 13th and 14th in terms of popularity in North America and Australasia, respectively, but 30th most popular in Europe. In contrast, while Europeans think fairly highly of IS Journal (ISJ) (7th most popular), it lags in popularity in Australasia and North America (16th and 21st, respectively). Decision Sciences is a journal favored in North America (6th) but less highly regarded in Europe (29th) and in Australasia (the 14th place is almost in the middle between the other two regions). A predictable discrepancy is observed with the German-speaking Wirtschaftsinformatik, which ranks 20th in Europe but was selected only a few times in North America and Australasia.
In terms of "average position," Information Systems Research (ISR) leads by a narrow margin in North America and is consistently second in Europe and Australasia after MISQ. While Management Science (MS) is on average placed slightly above 5th position in North America and Australasia, it occupies the 8th position in Europe. A key observation here is that popularity as a measure of journal quality is distinct from the average position received. In this study respondents were forced to make trade-offs between journals and to place them individually in ranking positions, in contrast to previous studies where respondents could rate more than one journal equally. Therefore, we could argue that our measures of average position received and percent top 10 might be considered as more precise surrogates for the perceived quality of scholarly work. Having said that, popularity sets apart those journals respondents have as "top of mind." For example, in North America, ISR receives a slightly higher average position than MISQ while the latter is more popular. Such discrepancies are more pronounced in Europe. Notably, EJIS receives an average position of 8.5 but comes 3d in popularity after MISQ and Communications. ISR receives the second-best average position but comes 4th in popularity.
Percent top 10 as a measure offers yet another perspective. North Americans and Australasians alike, appear to have a well-formed opinion regarding the journals that should occupy the top three positions. MISQ, ISR, and MS have over 90% placements. Particularly in North America, the 4th journal in terms of percent top 10 receives less than 80% in the top tier. Overall, the journals with most placements in the top tier in the world are MISQ, ISR, Communications, JMIS, MS and IEEE Transactions (IEEET). Decisions Sciences presents a notable case in that North Americans place it in the top tier 73% of the time while the respective figure for Europe is only 39%. Conversely, EJIS receives 59% of its placements in the top tier by Europeans, while only 28% by North Americans. This suggests certain journals demonstrate a distinct regional identity.
Our study has generated a wealth of data offering a finer detail of analysis. In order to arrive at a judgement regarding the perceived quality of journals, we invite readers to jointly consider the measures presented. Indeed, by taking both "popularity" and "average position" into account, our study is consistent with the last Communications survey (when comparing our North American sample). Our measure of popularity tracks the Hardgrave and Walstrom [3] ranking closely and any differences tend to be corrected by taking into account the average position received.
Another important aspect of journal impact is its readership. A journal may rank highly in terms of tier position but it may not be accessible to a wide audience. We asked respondents to list the five journals they read most. Table 3 presents the ranking of journals with respect to readership by region. Clearly, actual readership is highly correlated with perceptions, particularly popularity (Table 2). A notable difference with Table 2 is that Organization Science has entered the 10 most-read journals in North America, reflecting its affinity to the issues facing IS academics although it is not a mainstream IS journal. Table 2 demonstrates a certain divergence between Europe and the rest of the world. In particular, Europeans do not read ISR and Management Science as much. On the contrary, they peruse EJIS, ISJ, and the IEEEC a lot more. North Americans differ from both the Europeans and the Australasians in that they read Decision Sciences a lot more. German respondents in the European sample have made a strong statement with Wirtshaftsinformatik.
Conclusion
This study contributes to the ongoing concern about the quality of scholarly publications by introducing a global perspective and by presenting multiple measures of journal quality and impact. Differences between the three regions have been identified in terms of characteristic discrepancies in popularity, position, percent top 10 and readership. The undisputed leading journals in the world are MISQ, Communications, and ISR. Journals such as EJIS, Decision Sciences and ISJ have an identifiable regional impact. The study's key finding is that perceived journal quality is a multifaceted concept, the assessment of which may vary considerably depending on the measurement approach and the subjective regional stance one chooses to adopt.
References
1. Collin S.O, Johansson U., Svensson K. and Ulvenblad P.O. Market segmentation in scientific publications: Research patterns in American vs. European management journals. British J. Manag. 7, (1996), 141154.
2. Gillenson M. and Stutz J. Academic issues in MIS: Journals and books. MIS Quart. 15, 4, (1991), 447452.
3. Hardgrave B. and Walstrom K. Forums for MIS scholars. Commun. ACM 40, 11, (Nov. 1997), 119124.
4. Hardgrave B., Walstrom K. and Wilson R. Forums for MIS Scholars. Commun. ACM 38, 3, (Mar. 1995), 93102.
5. Holsapple C., Johnson L., Manakyan H. and Tanner J. Business computing research journals: A normalized citation analysis. J. Manag. Info. Syst. 11, 1 (1994), 131140.
6. Hult G.T., Neese W.T. and Bashaw R.R. Faculty perceptions of marketing journals. J. Marketing Education (Spring 1997), 3752.
7. Robey, D. Research commentary: Diversity in information systems research: Threat, promise and responsibility. Info. Syst. Res. 7, 4 (1996), 400409.
8. Watson, R.T., Kelly G.G., Galliers R.D. and Brancheau J.C. Key issues in information systems management: An international perspective. J. Info. Syst. Manag. 13, 4, (1997), pp. 91115.
9. Whitman M.E, Hendrickson A. and Townsend A.M. Academic rewards for teaching, research and service: Data and discourse. Info. Syst. Res. 10, 2 (1999), 99109.
10. Zmud, R. Editor's comments. MIS Quart. 20, 2 (1996), xxixxiv.
Footnotes
1ISWorld Mission Statement; www.isworld.org/isworld/mission.html.
2 Readers can find the questionnaire as well as the full results at www.alba.edu.gr/survey.
©2001 ACM 0002-0782/01/0900 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2001 ACM, Inc.
No entries found