Communications of the ACM
Home/Magazine Archive/November 2005 (Vol. 48, No. 11)/Assessing the Relative Influence of Journals in a.../Full Text
Communications of the ACM
Assessing the Relative Influence of Journals in a Citation Network
Scholarly journals are actively involved in the creation and diffusion of knowledge. Their formal communications are documented through citations, a process that results in a citation network of related journals [6, 7]. A citation occurs between journals A and B when an article in A references an article that was published in B. Journal A is called the citing journal while B is referred to as the cited journal. In this example, B is perceived to be a source of knowledge and A is viewed as the storer of knowledge [12]. The influence that a journal wields on another, as well as insights into how communicative and productive a journal is within its network, may be discerned by examining the intricate citing and cited patterns between journals [9]. Unlike several studies that focused on ranking journals in a discipline, the purpose of this article is to assess the relative influence of a journal by determining the key sources and storers of knowledge in a citation network. In addition, the study also shows the cohesiveness of journals based on the reciprocity (that is, mutual influence) of their citation flows [7].
The ranking and influence of a journal in a social network engendered by the interchange of citations is of interest to academic institutions, journal editors, and aspiring scholars. This has a direct bearing on the recognition of scholars and the prestige they are accorded in a field [12]. Several articles have used objective (for example, citation analysis) or subjective means (for example, perceptions of scholars) to determine the ranking of IS and computing journals [25, 11]. The use of citations can overcome biases that arise from the measurement of subjective responses [3].
However, there are some limitations of using only raw citations to determine the importance of a journal. In particular, they do not account for biases that stem from high self-citations and/or from the failure to simultaneously consider both the citing and cited patterns of a journal [7]. This study uses a log-multiplicative model to control for some of these biases [6, 7].
Data Collection and Results
The log-multiplicative model has proved to be useful for assessing the relative importance or influence of a journal in a network of journals [7]. The model includes parameters that account for self-citations as well as for the variations in the quantity of citations being received (that is, being cited by other journals) or being sent (citing other journals). Thus, it can be applied to a citation matrix whose rows represent the citing journals (called sending journals) and the columns stand for the journals being cited (called receiving journals) [6, 7].
The citation network considered in this study comprises 27 journals that were identified in a recent article in CACM [3]. For each of these journals, a list of all the articles referenced between 1998 and 2002 was obtained from the Science Citation Index (SCI) and the Social Science Citation Index (SSCI) using ISI's Web of Knowledge database. For each journal, the frequency of citations received from and sent to every other journal in the network was then computed. A part of the resulting 27 X 27 citation matrix is shown in Figure 1. The diagonals represent self-citations.
In order to select the model to provide the best fit for the citation data, a log-linear model as well as log-multiplicative models allowing for more than one dimension of association between sending (S) and receiving (R) citations were evaluated using a software called lEM.1 The Bayesian Information Criterion (BIC) was used as an index of fit [1]. The lower the value of this index the better the fit of the model. A log-multiplicative model with five dimensions of association between R and S was found to be the most appropriate fit. The software also estimates the log-linear parameters that show the differences between journals in both their volume of citing as well in the number of citations they receive [7]. These estimates are suggestive of the relative influence of a journal both as a receiver (or a knowledge source) and as a sender (or a knowledge storer) of citations. Ranked in order of influence, the accompanying table summarizes these results. Positive values in the third and fifth columns of the table imply an above average influence on the network [6].
Discussion
The third column in the table is the one of interest in determining the relative importance of a journal in the network considered in the study. This column shows the estimate of the log-linear parameter for receiving citations (R), which is an adjusted influence of the journal as a source of knowledge. It is an indication of the volumes of citations a journal receives, controlling for self-citations and citations sent to other journals in the network [7]. Journals frequently cited are highly likely to be valuable sources of knowledge, while those that often cite others are storers or borrowers of knowledge. The importance of a journal is a function of how useful it is as a source of knowledge. In general, such journals are deemed to be more prestigious than those that serve as storers/users/borrowers of knowledge [12]. The rankings shown in the fourth column of the table reflect the relative influence of the journals used in this study.
Consistent with the findings of a previous study [3], CACM appears to be the most influential source of knowledge for the other journals in the network. In other words, after controlling for self-citations and the number of citations being sent by CACM to others in the network, CACM appears to be most cited by other journals.
The fifth column shows the adjusted influence of a journal as a storer of knowledge. This measure is tied to the citations a journal sends to others in the network. These estimates tend to be low for journals that have restrictions on the number of references that can be included in an article. This is also the reason why cited to citing ratios can be misleading if the study does not account for restrictions imposed on the number of references.
Cohesiveness is a measure of reciprocal citing behaviors between journals. That is, journals that mutually influence each other are likely to be more cohesive [6]. From Figure 1, we can discern the reciprocal citing behaviors of ISR and MISQ (as they cite each other heavily) but neither cites or is cited significantly by either JACM or JCSS. The same is true of the JACM/JCSS pair in relation to ISR/MISQ. Following the technique employed by [6], the cohesiveness of the journals in the network was determined using the scores of each of the journals along the five dimensions. The Ward's method, one of the more popular of the agglomerative hierarchical clustering procedures [8], was used to identify inherent clusters in the journals based on the five dimensions of citation flows. Journals clustering together (for example, ISR and MISQ) are deemed to be cohesive. The hierarchical tree obtained through Ward's method is shown in Figure 2, which also shows the rankings based on the relative influence of these journals as sources of knowledge. The linkage distance is a measure of proximity, with shorter distances representing a higher degree of closeness between the journals.
The figure indicates two broad clusters, labeled Socio-technical and Technical. Journals grouped under the socio-technical category focus primarily on information systems, organizational factors in computing, and socio-technical issues. The technical journals are geared toward engineering with an emphasis on computer science topics. Smaller clusters may be identified within these two categories. For example, there is a clear separation of the European and North American journals related to information systems. Journals that deal with AI and/or Expert Systems as well as those that span the boundary between human computing and AI cluster together.
Computing professionals are the main audience for journals such as CACM, IBM Journal, IEEE Computer, IEEE Transactions on Software Engineering (IEEESE), and the Journal of Systems and Software (JSS). While the first three provide valuable insights to technically aware readers on diverse topics related to technology, IEEESE and JSS publish theoretical and empirical studies on topics related to software engineering of interest to the practitioner community. The last cluster, labeled Computer Science, consists of journals that are very technical in nature. These deal primarily with computer science topics such as database design, issues in the design and implementation of programming languages, mathematical theories that are relevant to the field of computing, algorithms, and the like.
Conclusion
Scientific progress occurs through the creation and dissemination of knowledge. This process is facilitated by the interchange of ideas between journals, as evidenced by the citation flows between them. This article examined the citing and cited patterns of 27 journals in IS and other areas of computing in order to assess their relative influence. Following a technique previously employed [6], we used a log-multiplicative model with multiple dimensions of association to determine the primary sources and storers of knowledge in a network comprising these journals. The cohesion between journals was shown by applying a hierarchical clustering procedure to the five dimensions that were obtained.
It must be noted that the results are specific to the network of journals included in the study. A notable omission from this list is the Journal of Management Information Systems (JMIS). JMIS was omitted because the previous study [3] from which the pool of journals was drawn did not include it. Moreover, JMIS was not indexed in the Web of Knowledge database in the initial year of interest for this study.
Interesting insights may be discerned by examining inter- and intracluster citation flows. For example, intercluster citation flows may reveal which clusters are the most important sources of knowledge, while an intracluster analysis using the same approach outlined in this study can show which journals are most influential within their cohesive group.
References
1. Clogg, C. and Shihadeh, E. Statistical Models for Ordinal Variables. Sage Publications, Thousand Oaks, CA, 1994.
2. Hardgrave, B.C. and Walstrom, K.A. Forums for MIS scholars. Commun. ACM 40, 11 (Nov. 1997), 119124.
3. Katerattanakul, P., Han, B. and Hong, S.G. Objective quality ranking of computing journals. Commun. ACM 46, 10 (Oct. 2003), 111114.
4. Mylanopoulos, N. and Theoharakis, V. Global perception of IS journals. Commun. ACM 44, 9 (Sept. 2001), 2933.
5. Peffers, K. and Ya, T. Evaluating the universe of outlets for information systems research: Ranking the journals. J. of IT Theory and Application 5, 1 (2003), 6384.
6. Pieters, R. and Baumgartner, H. Who talks to whom? Intra- and interdisciplinary communication of economics journals. J. Economic Literature 40, 2 (June 2002), 483509.
7. Pieters, R., Baumgartner, H., Vermunt, J. and Bijmolt, T. Importance and similarity in the evolving citation network of the International Journal of Research in Marketing. Intern. J. Research in Marketing, 16 2 (June 1999), 113127.
8. Punj, G. and Stewart, D.W. Cluster analysis in marketing researchReview and suggestions for application. J. Marketing Research 20, 2 (May 1983), 134148.
9. Shaw, W.M. Information theory and scientific communication. Sociometrics, 3, 3 (1981), 235249.
10. Vermunt, J. lEM: A General Program for the Analysis of Categorical Data. Tilburg University, Netherlands, 1997.
11. Walstrom, K.A. and Hardgrave, B.C. Forums for management information systems scholars: III. Info & Mgmt 39, 2 (2001), 117124.
12. Zinkhan, G. and Leigh, T. Assessing the quality ranking of the journal of advertising, 19861997. J. Advertising 28, 2 (Summer 1999), 5170.
Footnotes
1 The software was developed by Jeroen K. Vermunt of Tilburg University in the Netherlands. See [10] for details.
Figures
Figure 1. Sample matrix of citation data.
Figure 2. Cohesiveness of IS and computing journals based on citation flows and their relative influence rankings.
©2005 ACM 0001-0782/05/1100 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2005 ACM, Inc.
No entries found