Communications of the ACM
Home/Magazine Archive/October 2003 (Vol. 46, No. 10)/Objective Quality Ranking of Computing Journals/Full Text
Communications of the ACM
Objective Quality Ranking of Computing Journals
Quality rankings of journals are of particular interest to academicians since publication in prestigious journals has significant influence on peer recognition, tenure and promotion decisions, and compensation increases. Thus, many previous studies [2, 4, 6, 911] have attempted to assess journals, and compare their quality relative to others. Most journal quality evaluations use surveys of industry practitioners and members of academia, and then rank journals based on survey results. Unfortunately, ranking journals using the survey methodology, which is primarily a reflection of respondents' perceptions, may be affected by the subjective stances respondents choose to adopt [9], and some potential measurement biases [2]. Hence, it is not surprising that study results based on surveys have shown some inconsistency in journal quality evaluations and their final rankings [2]. Such studies have also lacked comprehensiveness. As seen in Table 1, a total of 111 journals were evaluated by five previous studies [4, 6, 911]; among them only 11 journals were considered by all five studies, whereas 77 journals (approximately 70%) were ranked by only one or two studies.
In contrast to earlier research, this study employs citation analysis to assess journal quality and ranking. Citation analysis is considered the best objective methodology for assessing the impact of a journal and its articles [1]. The value of citation analysis is based on the view that journal influence rankings should be derived from the actual use of journals and their articles [2], since research is usually based on the foundation of one or more past scientific achievements [8]. Moreover, this study provides an evaluation and ranking for journals with a primary focus on IS and computing areas. Journals with multidisciplinary articles, such as Management Science, Operations Research, and Decision Sciences are not considered in this analysis. Since citation analysis is used herein, those journals lacking citation data, or with insufficient data, are not included, such as the Journal of Management Information Systems, and Communications of the AIS. After an exhaustive examination of all journals addressed in the aforementioned five previous studies, 27 academic IS/CS journals or related practitioner publications were identified and included in this study (see Table 2).
After compiling a list of 5,868 target articles published in the 27 selected journals from 1995 to 1998, we recorded and summarized the numbers of references, or citing counts, provided in each article. We also compiled a list of the citations, or cited counts, made to each target article. To ensure a reasonable citation history for analysis, we allowed for a two-year lag time between IS article publication and citation, since the modal elapsed time was found to be approximately two years [5]. That is, all citation data was collected between 1997 and 2000 from the Social Science Citation Index (SSCI) and the Science Citation Index (SCI), which cover a broad range of publications. After finding a total of 15,632 citations made to the 5,868 target articles, we further classified two groups of the target articles as receiving either no citation or at least 20 citations. Also, we recorded the number of self-citations coded for each target article.
Since journal quality is a multifaceted concept that can be reflected by different measurements [9, 12], we derived seven citation-based indices of journal quality from the collected citation data (see Table 2). Assuming all indices are equally important, the 27 selected journals were finally ranked based on their arithmetic mean scores of all rankings in all seven indices, which are described as follows:
Citations per article. This index is the average number of citations received per target article published in a particular year. It is considered an appropriate index in measuring a journal's influence since it eliminates the size effect of the number of articles published in a journal [3]. Results indicate that MISQ, ISR, and JACM are the top three journals with a high number of citations per article. CACM is ranked sixth. Interestingly, many technically oriented journals such as AI, IEEESE, ACMDB, HCI, HCS, JCSS, and AIMag are ranked in the top 12. The finding that these technically oriented journals ranked relatively higher than several other well-recognized journals differs from previous study findings.
Un-cited ratio. The percentage of journal articles not cited in a particular year is measured by the un-cited ratio. The study results show that MISQ and ISR are the only two journals with single digit percentages in this index, meaning their articles were generally extremely popular for citations. Additionally, CACM and several technically oriented journals, such as HCI, AI, IEEESE, ACMDB, IBM, HCS, and JCSS ranked in the top 12, with the lowest un-cited ratios.
20+ citations ratio. This index refers to the percentage of target articles published in a particular year with 20 or more citations. Numerical findings indicate that ISR has the highest 20+ citations ratio, followed by MISQ, JACM, AI, and CACM, respectively. Note also that the top 12 scorers on this index include several technically oriented journals, such as IEEESE, AIMag, HCS, IBM, HCI, and JCSS.
The ranking results of this study reveal an important finding supported by most of our quality indices: on average, journals with a technical or a specialty focus attain high rankings.
Annual mean citations per article. This index provides a normalized quality measurement of the target article by considering the number of years elapsed since its publication. The index is derived by dividing the citations per article by the number of years elapsed since publication of the target articles; thus, the time effect on the number of citations received is eliminated. On average, top-ranked MISQ receives 1.96 citations annually for each article it publishes. technically oriented journals, such as AI, IEEESE, ACMDB, HCI, AIMag, HCS, and JCSS, also received high rankings on this index.
Cited-to-citing ratio. Knowledge flows among journals are considered a fundamental measurement of a journal's scientific status and prestige in the social exchange model [7]. Journals that emphasize basic research are likely to be cited more often than they cite others, and they are valued as "knowledge sources" that contribute knowledge to the community. In contrast, applied journals are more likely to cite others than they are to be cited, and they are perceived as "knowledge storers" that collect knowledge produced by others [12]. Given this connotation, the cited-to-citing ratio is a good indicator of journal quality. A relatively high ratio indicates the journal is a knowledge source, while a relatively low ratio indicates the journal is a knowledge storer. The analysis results show that CACM ranked first with the highest ratio, and IEEESE, IBM, AIMag, AI, JCSS, HCI, ACMDB, and HCS all ranked in the top twelve.
Self-citations. Self-citation occurs when any of the authors of the citing article are also listed as authors of the cited article (the target article). This index for a particular year is the ratio of the number of self-citations to the total number of citations made to the target articles published in that year. CACM was found to have the lowest percentage of self-citations, followed by ISM, ACMCS, MISQ, and IBM.
Current article impact. This index measures the average number of citations that target articles published during the most recent two-year period received during that same time period [3]. The index for each reference year is derived by dividing the number of citations made to the target articles published during two years prior to the reference year by the number of target articles published during the same time period. If a journal receives a relatively high current article impact score, this indicates researchers direct their attention to the journal for important ideas to produce new knowledge [12]. The top five high scorers in the current article impact index are MISQ, JACM, AI, CACM, and IEEES, followed by technically oriented journals such as HCI, AIMag, IBM, ACMDB, JCSS, and HCS.
Conclusion
Based on the average ranking scores derived for each journal across all seven citation-based indices, the top five computing journals are MISQ, ISR, CACM, JACM, and IEEESE. Our findings closely resemble results reported in several previous studies [6, 910, 11]; that is, this study confirms that MISQ, ISR, and CACM are the top three journals of IS and computing areas. In addition, the ranking results of this study reveal an important finding supported by most of our quality indices: on average, journals with a technical or a specialty focus attain high rankings. As verified, technically oriented journals such as JACM, IEEESE, AI, HCI, IBM, AIMag, ACMDB, HCS, and JCSS all achieved high rankings based on their final average ranking scores. While these journals may not receive much recognition by general audiences, their publications are frequently used and cited by researchers whose interests are in the journals' niche areas.
While our research tends to be more objective, readers should be aware that our study is confined to journals with a strong focus in the IS and computing areas. Some top-quality multidisciplinary journals, such as Management Science, in which seminal IS articles are published, are not included in this study, and neither are those journals with insufficient citation data, such as the Journal of Management Information Systems. This exclusion might underrate the contribution of those journals in the IS field. Another potential limitation in citation-based ranking is that its quality indices could be skewed by excessive citation of controversial articles that may not make true contributions in the IS or computing fields. Thus, readers should cautiously exercise their judgment in interpreting results produced in this study, with no exclusion of other non-IS or computing journals that publish high-quality IS articles.
References
1. Brown, L. and Gardner, J. Using citation analysis to assess the impact of journals and articles on contemporary accounting research (CAR). Journal of Accounting Research 23, 1 (1985), 84109.
2. Cooper, R., Blair D., and Pao, M. Communicating MIS research: A citation study of journal influence. Information Processing & Management 29, 1 (1993), 113127.
3. Garfield, E. Citation Indexing: Its Theory and Application in Science, Technology, and Humanities. Wiley, New York, NY, 1979.
4. Gillenson, M. and Stutz, J. Academic issues in MIS: journals and books. MIS Q 15, 4 (1991), 447452.
5. Hamilton, S. and Ives, B. Knowledge utilization among MIS researchers. MIS Quarterly, 6, 4 (1982), 6177.
6. Hardgrave, B. and Walstrom, K. Forums for MIS scholars. Commun. ACM 40, 11 (Nov. 1997), 119124.
7. Johnson, J. and Podsakoff, P. Journal influence on the field of management: an analysis using Salancik's Index in a dependency network. Academy of Management Journal 37, 5 (1994), 13921407.
8. Kuhn, T. The Structure of Scientific Revolutions, 3E. The University of Chicago Press, Chicago, IL, 1996.
9. Mylonopoulos, N. and Theoharakis, V. Global perceptions of IS journals. Commun. ACM 44, 9 (Sept. 2001), 2933.
10. Walstrom, K., Hardgrave, B., and Wilson R. Forums for management information systems scholars. Commun. ACM 38, 3 (Mar.1995), 93107.
11. Whitman, M., Hendrickson, A., and Townsend, A. Academic rewards for teaching, research, and service: Data and discourse. Information Systems Research 10, 2 (1999), 99109.
12. Zinkhan, G. and Leigh, T. Assessing the quality ranking of the Journal of Advertising, 19861997. Journal of Advertising 28, 2 (1999), 5170.
©2003 ACM 0002-0782/03/1000 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2003 ACM, Inc.
No entries found