Communications of the ACM

Communications of the ACM

Demographic Changes in Is Research Productivity and Impact

By Mohamed Khalifa, Kathy Ning
Communications of the ACM, April 2008, Vol. 51 No. 4, Pages 89-94
10.1145/1330311.1330329
Comments

Compared to research in other business disciplines, information systems (IS) research is relatively in its infancy. In the last decade, an increasing number of academic institutions have recognized IS as a discipline and have created IS departments/groups. These developments introduced important changes to the demographics of IS researchers. The regional differences and top performers have changed considerably.

In this research, we analyze the development of IS research in the last decade with an emphasis on demographic changes. More specifically, we examine IS research productivity and impact, investigating changes in regional and institutional contributions and highlighting the top performers for both academic and non-academic institutions. This research should be of interest to academics and professionals alike. The reported results will allow firms and academic institutions to benchmark their research performance and to identify top performers for potential collaboration.

---

This research should be of interest to academics and professionals alike. The reported results will allow firms and academic institutions to benchmark their research performance and to identify top performers for potential collaboration.

---

This study will also provide researchers with important indicators of IS research, for example, overall productivity and impact, evolution over the last 10 years, internationalization, concentration, and the level of industry involvement.

Productivity refers to the total IS research publications output. Consistent with prior studies we measure it with the adjusted count (fraction based on the number of co-authors) of research articles published by IS researchers in top journals in IS and referent disciplines. In addition, we also account for the impact of the research output, that is, the level of dissemination, which is measured with the adjusted impact scores (yearly impact ratios of the journal where a research article gets published). The impact ratios are reported by the Science Citation Index and the Social Sciences Citation Index.

Although more "objective" than perceptions, adjusted counts and impact ratios do not fully account for the rigor and prestige of the journals. Additional objective measures (for example, acceptance ratios), however, are not readily available and subjective measures (for example, journal rankings based on perceptions) are usually controversial. A quick Web search reveals that except for very few top journals, institutional rankings of IS journals differ significantly. We therefore opted not to mix objective and subjective measures, while acknowledging the limitations of our approach.

The journal selection is based on the most recent citation-based ranking [1], with minor differences. We chose a cut-off of 0.1 for the impact ratio, removing Journal of Computer Information Systems (only 0.034 when listed). We also removed IEEE Computer, as it was not clear which journal/magazine the authors meant. We could identify several journals/ magazines with such a name, but all having impact ratios different from the one reported in [1]. Furthermore, most previous rankings did not include such a journal, but listed instead the IEEE Transactions [3, 4]. We therefore consistently included IEEE Transactions on Engineering Management, IEEE Transactions on Systems, Man, and Cybernetics, and IEEE Transactions on Software Engineering [3, 4]. We also added the Journal of the Association of Information System (JAIS). Although this journal has no impact ratio due to its relatively short history, it is generally regarded as a rising top-ranked journal. Indeed, several recent studies have included JAIS as one of the important IS journals, for example, [2-4]. Without an impact ratio, this journal counts for productivity calculation only. We ended up with 25 IS journals. For referent disciplines, we included the top 11 journals from the original ranking.

The collection of information about all articles published in the selected 36 journals during the last decade (1995–2004) took 40 person-months. To prevent errors, we incorporated several validity checks within the data entry system. We also assigned three individuals to check all entries and reconcile discrepancies. The resulting database consists of 18,711 research articles written by 24,517 authors from 4,111 institutions. The identification of research articles is based on the ISI classification. An article is included in the analysis if it has at least one IS co-author.

Given the multi-disciplinary nature of IS research, we adopted a rather broad view of IS affiliation, defining IS authors as those that satisfy one of the following criteria: published in an IS journal; listed in the AIS directory; or published in a non-IS journal but are clearly affiliated with an IS department. We could identify 8,362 articles published by 6,760 IS authors from 1,901 institutions.

To examine changes in productivity and research impact in the last decade, we compared the first half period (1995–1999) to the second one (2000–2004). The overall productivity of IS research had a moderate increase of 14%, from 3,639.07 adjusted article counts in the first period to 4,132.46 in the second period. The impact of IS research, however, experienced a dramatic boost with the adjusted impact scores increasing from 2,260.18 to 4,573.51. This important growth rate of 102% provides a strong indication for the enhanced recognition and influence of IS research. While the overall productivity of IS research has increased slightly, its impact has doubled.

Back to Top

Regional Analysis

We limited our regional analysis to North America, Europe, and Asia, as they account for over 95% of the publications. As depicted in Figure 1, North America institutions continuously dominated IS research in the last decade with an annual productivity level ranging from 423.25 to 572.89 and accounting for 58.2% to 69.4% of the global productivity. It is worth noting that such results are somehow expected given that the selected journals are predominantly U.S.-based. Although North American productivity dropped during 1995–1997 by 19.4%, it picked up again gradually in 1998–2002 and in 2003 it experienced a sharp increase from 460.38 to 572.89. The annual productivity level of Europe remained relatively stable, ranging from 136.83 to 184.95 and accounting for 18% to 24% of the IS research publications. The biggest change occured in Asia with its annual productivity increasing from 50.71 (6% share) in 1995 to 149.67 (16.5% share) in 2004. Asia is bridging its productivity gap with Europe with a growth rate of 65% over the last decade compared to 8% for both North America and Europe.

As for the overall impact, North America experienced a decrease in the first period (from 374.54 in 1995 to 218.76 in 1999) and a boost in the second (from 402.32 in 2000 to 742.97 in 2004). The impacts of Asia and Europe, on the other hand, increased steadily. Consequently, the gap between North America and the other two regions decreased in the first period and sharply widened in the second. The overall impact of Europe increased from 68.5 in 1995 to 242.42 in 2004, while that of Asia increased from 34.16 to 176.11. Interestingly, while Asia was able to bridge its productivity gap with Europe, it could not reduce the impact gap, which actually increased from 34.34 in 1995 to 66.31 in 2004. The changes in overall impact are largely due to changes in productivity.

To control for the productivity effects, we also examined the annual average impact ratios (average adjusted impact score for a single publication). Figure 2 shows a small decrease in the first period (1995–1999) but a clear upward trend in the second (2000–2004). The average impact of Europe improved the most (from 0.50 in 1995 to 1.39 in 2004) with an average annual growth rate of 13.9%, followed by North America with 11% (from 0.71 in 1995 to 1.38 in 2004), and Asia with 10% (from 0.67 in 1995 to 1.18 in 2004). In 2004 Europe's impact (1.39) exceeded that of North America (1.38). Europe bridged its impact gap with North America.

To examine the extent to which IS research productivity is evenly spread among academic institutions in different regions, we examined the regional concentration ratios (percentage of the output of top 20 productive institutions). As indicated in Figure 3, North America and Europe have more or less similar concentration ranging from 29% to 44%, sharply contrasting with that of Asia. Although the concentration ratios are slightly declining from 75% in 1995 to 60% in 2004, research in Asia remains highly concentrated with 20 institutions contributing over 70% of the publications in the last decade.

Back to Top

Institutional Analysis

Table 1 presents the changes in the top 20 academic institutions from the first period (1995–1999) to the second (2000–2004) based on total productivity and impact. It is important to keep in mind the size factor in interpreting these results, as institutions with large IS faculty are more likely to have better scores. It is interesting to notice that the productivity rankings are different from the impact rankings and that three institutions in the top 20 productivity list are not in the impact list in both periods. These results emphasize that productivity does not necessarily lead to impact and that both indicators must be considered in evaluating an institution's research performance.

The dominance of Massachusetts Institute of Technology is obvious. It is consistently ranked first, widening its lead in the second period for both productivity (over 50% higher than the second in line) and impact (almost 100% higher than number 2). Another interesting observation is the absence of European institutions in the top performers. Asian institutions, however, are becoming more competitive. The number of Asian universities listed in the top 20 increased from three in the first period to five in the second for productivity and from one to two for impact. These results are consistent with the high concentration ratios of Asia. Although the overall productivity of Asia is similar to that of Europe, fewer institutions are driving it. Asian institutions such as National University of Singapore and City University of Hong Kong are now among the top 10 in both productivity and impact.

---

While still dominated by North American institutions, there are signs of internationalization with Asia increasing its productivity and Europe enhancing its overall impact.

---

The emergence of Asian leaders is contributing further to the performance dynamism. A comparison of the two periods reveals important changes in rankings and in the composition of the top performers with seven new entrants for both productivity and impact. Although the composition of the top 20 academic performers has considerably changed from the first period to the second one, it is still characterized by the absence of European institutions and the dominance of MIT.

Several firms are actively conducting IS research. However, their relative contribution in the last decade was minimal, ranging from 4.1% to 5.8% of the total productivity and from 4.6% to 6.5% of the total impact. Table 2 presents the changes for both productivity and impact in the top 10 industry performers from the first period to the second one. An analysis of the industry performers reveals similar patterns to the ones reported for academic institutions in terms of dominance, dynamism, and regional representation. One firm, IBM, is consistently dominating IS research. During the period of 2000–2004, its productivity is almost three times that of the second productivity performer (Accenture) and its impact is more than double that of the second impact performer (Microsoft). A comparison of the two periods also reveals a very high level of dynamism with six new entrants in the top 10 for productivity and seven for impact.

As for regional representation, only one Asian firm and one European firm enter the current top 10 firms. An interesting observation about research done by the industry is that although the productivity of the top 10 firms is much smaller than that of the top 10 academic institutions, their average impact is higher (1.64 vs. 1.25). Industry top performers publish fewer but higher-impact articles than their academic counterparts.

Back to Top

Conclusion

In conclusion, we would like to highlight the modest growth of IS research productivity and the impressive improvement of its impact. While still dominated by North American institutions, there are signs of internationalization with Asia increasing its productivity and Europe enhancing its overall impact. The composition of the top performers is dynamic, but with consistent academic and industry leaders. The dynamism and internationalization trends should contribute further to the enhancement of the IS research diversification and recognition.

Back to Top

Back to Top

Back to Top

Back to Top

Back to Top

Back to top

The Digital Library is published by the Association for Computing Machinery. Copyright © 2008 ACM, Inc.

---

No entries found