Communications of the ACM

Communications of the ACM

Grasping the Complexity of Is Development Projects

By Weidong Xia, Gwanhoo Lee
Communications of the ACM, May 2004, Vol. 47 No. 5, Pages 68-74
10.1145/986213.986215
Comments

The complexity of IS development projects (ISDPs) can be understood and measured along two dimensions: organizational/technological and structural/dynamic. The results of our 2002 study of 541 ISDPs in North American organizations suggest that when it comes to the complexity of ISDPs, the technological aspects are more apparent, but the organizational aspects have more significant effects on ISDP performance and outcomes. To improve project performance, ISDP managers must learn to manage the organizational, as well as the technological, aspects of their ISDPs.

---

ISDPs are inherently complex because they deal not only with technological issues but with organizational factors largely beyond the project team's control.

---

Projects are the basic unit organizations use to manage their IS development efforts, but, historically, IS projects have been characterized by a notoriously high failure rate [5]. For example, the Standish Group reported in 1994 that U.S. companies had spent more than $250 billion each year in the early 1990s on IS projects, with only 16.2% ultimately considered successful by their IS executives [10]. A Standish Group 2001 report [9] found that, although U.S. companies invested four times more money in IS projects on an annual basis in 2000 than they did in the 1990s, only 28% were considered successful. These results suggest that the success rate of IS projects, though improving, was and continues to be low. As a result, an enormous amount of money is wasted by U.S. companies on failed IS projects. Thus, they have a strong economic incentive to improve their IS project performance.

IS development involves the analysis, design, and implementation of applications and systems to support business operations in an organizational context. ISDPs involve work on new applications and systems yet to be installed, as well as on the enhancement of existing applications and systems, including those developed in-house and those developed through off-the-shelf packaged software.

One reason IS projects fail so often is that they are generally more complex than project teams expect them to be [7]. ISDPs are inherently complex because they deal not only with technological issues but with organizational factors largely beyond the project team's control. Moreover, both information technology and the business environment are constantly changing, making any given ISDP's business requirements and technical specifications difficult to estimate and manage. IS organizations are under increasing budget and performance pressure to do more with less resources and deliver systems through shorter development life cycles [12]. Rapid technological changes and tougher business demands make ISDPs even more complex, exacerbating the problems with poor ISDP performance [3]. To improve their ISDP success rate and return on investment, organizations must develop strategies to manage their ISDP complexity.

ISDP managers must understand the project characteristics constituting that complexity. Moreover, they need to understand the significant effects of ISDP complexity on project performance to be able to convince top management why managing it is so important. Since both ISDP complexity and project performance are multidimensional, ISDP managers must also understand how the various ISDP complexity components affect the various project performance measures. Such an understanding will help ISDP managers focus on the complexity components with the greatest potential effect on the project success measures most important to their organizations.

Here, we propose a taxonomy describing the key dimensions and components of ISDP complexity. Based on this taxonomy, we collected survey data about 541 ISDPs from North American organizations to empirically develop a measurement tool for assessing ISDP complexity. We then analyze the 541 ISDPs to provide additional insights about the effect of ISDP complexity on project performance, addressing three key questions:

• What are the key components of ISDP complexity?;
• How does ISDP complexity, if it can be measured, affect project performance?; and
• Do the various components of ISDP complexity affect project performance equally, and if they don't, how do they differ?

Back to Top

Taxonomy of ISDP Complexity

Researchers and practitioners alike use the term "ISDP complexity" in their writings and practices [1, 7, 9, 12]. However, to our knowledge, there are no well-defined frameworks in the literature that can be used to systematically describe the key dimensions and characteristics of ISDP complexity. Also lacking are commonly accepted measurements that can be used to assess ISDP complexity. By building on the literature and incorporating insights from ISDP managers, what we state here represents a first step toward developing such a framework and measurement tool while providing initial empirical evidence of the negative effects of ISDP complexity on project performance. Two literature sources—concerning general project management and software project risk factors—are particularly relevant; we used both to develop our taxonomy and measures.

The project management literature identifies a number of project dimensions and characteristics as constituents of project complexity. For example, [1] defines it in terms of the number of these elements and their interdependency. Applying this concept, it defines two types of project complexity: organizational (types of and number of relationships among hierarchical levels, formal organizational units, and specialization) and technological (types of and number of relationships among inputs, outputs, tasks, and technologies). Another dimension of complexity, described in [11], is uncertainty in both project goals and in the means used to achieve them. Uncertainty in this case refers to the extent to which project goals and means are ill-defined and thus subject to future changes; uncertainty in systems requirements/scope and in new information technologies are examples of goal and mean uncertainties. By integrating the dimensions proposed in [1, 11], [12] defines two distinct aspects of project complexity: structural (a project's underlying structure) and uncertain (a project's uncertain or changing nature). Uncertainty, according to [12], adds to a project's structural complexity and can be viewed as yet another constituent dimension of project complexity.

The literature on software project risk factors helped us develop our framework and measures. Software project risk has been defined as an uncertainty condition that can threaten the successful completion of an ISDP [8]. In essence, the components of ISDP complexity can be viewed as drivers of many types of project risk. The importance of software project risk can be understood in terms of "some combination of risk frequency (that is, how likely it is that the risk will occur) and risk impact (such as how serious a threat the risk represents if it does occur)" [6]. Most software project risk studies have sought to identify these factors [4] without developing a deeper understanding of their dimensions and effects. Nevertheless, the literature on software project risk factors provides an important basis for addressing the dynamic, or uncertainty-based, aspects of ISDP complexity. In addition, it provides a rich source for developing measures of ISDP complexity. In general, ISDP managers can classify the common risk factors either as organizational or as technological; for example, among the top 11 risk factors identified in [6], one was related to IT and the rest to the organizational aspects of software project risk.

As part of our study, we interviewed and ran focus groups with 74 ISDP managers to refine the dimensions and characteristics of ISDP complexity (see the sidebar "How the Study Was Done"). We now propose a taxonomy of ISDP complexity consisting of two dimensions: organizational/technological and structural/dynamic (see Figure 1). Consistent with [1, 6], ISDP managers consider ISDP complexity in terms of either its organizational or technological aspects. Drawing on [11, 12], ISDP managers can assess both the organizational and the technological aspects of ISDP complexity in terms of either the structural complexity among the current project components or the dynamic/uncertain nature resulting from the potential changes that may occur. In this taxonomy, each dimension suggests two distinct aspects of ISDP complexity rather than a continuum-based variable.

The taxonomy defines ISDP complexity as consisting of four components:

• Structural organizational (Structural_Org). Reflects the nature and strength of the relationships among project elements in the organizational environment, including project resources, support from top management and users, project staffing, and skill proficiency levels of project personnel;
• Structural IT (Structural_IT). Captures the complexity of the relationships among the IT elements, reflecting the diversity of user units, software environments, nature of data processing, variety of technology platforms, need for integration, and diversity of external vendors and contractors;
• Dynamic organizational (Dynamic_Org). Captures the pattern and rate of change in ISDP organizational environments, including changes in user information needs, business processes, and organizational structures; it also reflects the dynamic nature of the project's effect on the organizational environment; and
• Dynamic IT (Dynamic_IT). Measures the pattern and rate of changes in the ISDP's IT environment, including changes in IT infrastructure, architecture, and software development tools.

We created and used a rigorous four-phase instrument-development process to measure ISDP complexity in the study. We collected the empirical data through a Web-based survey, as outlined in the sidebar. The table here lists the measurements for the four ISDP complexity components, as well as the mean scores corresponding to each measure from the sample of 541 ISDPs. We assessed all measures of ISDP complexity using a seven-point scale in which higher numbers indicate greater complexity.

Figure 2 outlines the mean scores of the four components of ISDP complexity as assessed by the project managers in the sample. The Structural_IT component had the highest complexity score, followed by Dynamic_Org, Structural_Org, and Dynamic_IT. When it comes to technological complexity, ISDP managers in the sample perceived structural complexity to be greater than dynamic complexity. In contrast, with respect to organizational complexity, these same managers ranked dynamic complexity higher than structural complexity.

As shown in the table, the scores for the measures for Structural_IT were higher than those for the other complexity components, indicating that Structural_IT complexity was more apparent to the ISDP managers than were the other components.

Back to Top

Project Performance

Though widely recognized both in the literature and among practitioners that ISDP complexity causes poor project performance, little empirical evidence supports this contention. Using the measurement of ISDP complexity, we first examined the relationship between overall ISDP complexity and project performance. We then analyzed the various effects of the four ISDP components on project performance. We captured ISDP performance through four widely used measures: project delivery time; cost; system functionality; and end-user satisfaction.

The correlation analysis we performed on the 541 ISDPs suggests that overall ISDP complexity is negatively associated with all four measures of project performance. Greater ISDP complexity is associated with delayed project delivery, cost overruns, reduced system functionality, and lower end-user satisfaction. To explore the ways through which the four ISDP complexity components affect project performance, we used regression analysis to compute the effects of the four complexity components on the four measures of project performance. Figure 3 suggests that indeed the four ISDP complexity components have different effects on project performance. Structural_Org seemed to be the most critical complexity component directly affecting all four measures of project performance. Among the four, Structural_Org had the greatest effect on end-user satisfaction, followed by project delivery time, system functionality, and cost.

Dynamic_Org negatively affected project cost performance, indicating that changes in organizational structure and business processes might cause changes in project scope, which in turn leads to cost overruns. Dynamic_IT complexity influenced only system functionality. One plausible explanation for this limited correlation is that changes in IT infrastructure, architecture, and software development tools might cause interruptions in the technical foundations and tools critical to the ISDP, making it difficult to meet initial system functionality requirements, then hinder delivery of required system functionality. However, Structural_IT did not have a significant direct effect on any of the project performance measures.

From the perspective of project performance, end-user satisfaction and project delivery time are primarily influenced by Structural_Org complexity. Project cost performance is most affected by Structural_Org and Dynamic_Org complexity. And system functionality is most affected by Structural_Org and Dynamic_IT complexity.

Back to Top

Conclusion

Our extensive literature review, interviews, and focus group discussions with ISDP managers and a survey of 541 North American ISDPs helped us develop a taxonomy and measurement of ISDP complexity. We then tested the relationships between ISDP complexity and project performance. The taxonomy consists of two dimensions of ISDP complexity—organizational/technological and structural/dynamic—that in turn define four components of ISDP complexity.

The taxonomy highlights the multidimensional nature of ISDP complexity and can be used as a conceptual tool for defining and communicating ISDP complexity. The measurement we developed in the study can be used as another tool to begin assessing and managing the various aspects of ISDP complexity. Without such measurement, organizations find it difficult to collect baseline data about ISDP complexity, identify areas of concern, and develop effective strategies for specific problems. A measurement tool also allows them to track project complexity over time, along with the organizational techniques they need to deal with the components of ISDP complexity and project performance. Such data enables organizations to establish knowledge bases that can be used for planning and controlling future ISDPs.

Our North American ISDP study provided empirical evidence of the negative effects of ISDP complexity on project performance. The results suggest that greater ISDP complexity is associated with project delays, cost overruns, reduced system functionality, and decreased user satisfaction. In addition, they suggest that the four components of ISDP complexity have different effects on project performance.

One of our more interesting findings is that Structural_Org complexity has a relatively stronger effect on all four measures of project performance than the other three components of complexity. To improve their ISDP performance, organizations need to pay close attention to the organizational aspects, along with the technological aspects, of their ISDPs. Project managers must develop strong relationships with top management and end users alike. And organizations must assign qualified personnel with the required knowledge and skills to ISDPs.

Another interesting finding is that although the ISDP managers in the study sample perceived Structural_IT complexity as having the highest complexity scores, it did not significantly affect ISDP performance. This suggests that although the technological aspects of ISDP complexity are most apparent to and problematic for project managers, they might not have the greatest effect on actual project performance, largely because of the organization's technological maturity. Though technology itself poses complexity and risk issues to ISDPs, as suggested in [6], they may not significantly affect project performance because technological complexity and risk are largely within a project's control. Understanding the effects of the four ISDP complexity components allows project managers to improve their use of limited resources by focusing on the most influential ISDP complexity components.

The study also yielded insights on how a particular measure of project performance is affected by the various components of ISDP complexity. If an organization's focus is on-time delivery and user satisfaction, project managers must pay more attention to structural/organizational factors. If the focus is project cost, project managers must emphasize not only the structural/organizational factors but changes in the organizational environment as well. If the focus is system functionality, project managers must attend to both the structural/organizational factors and changes in the technology environment. This insight could help them focus on the most critical complexity components based on the specific performance measures that need improvement.

Our work represents a first step toward developing frameworks and measures to enable ISDP managers, as well as their top management counterparts, to understand and manage their ISDPs. Future research ought to investigate the organizational and technological factors, processes, and mechanisms influencing ISDP complexity. Organizations can then control the complexity of their ISDPs, thus enhancing their ISDP success rate.

Back to Top

Back to Top

Back to Top

Back to Top

Back to Top

The Digital Library is published by the Association for Computing Machinery. Copyright © 2004 ACM, Inc.

---

No entries found