Communications of the ACM

Communications of the ACM

How to Avoid Disappointment By Design

By Dorit Nevo, Michael R. Wade
Communications of the ACM, April 2007, Vol. 50 No. 4, Pages 43-48
10.1145/1232743.1232746
Comments

In a paradox experienced by many IT managers, organizational information systems (OISs) are implemented and function exactly as designed—a success by all objective measures—yet users are still disappointed and consider them unsuccessful. Exploring this paradox here, we recommend ways to reduce the root causes of such disappointment.

Figure.

Researchers and practitioners alike have long focused on the success of IT projects [9]. Measures of success include time and budget considerations, financial indicators, and usage rates. However, according to the Standish Group's 2004 annual report of IT project resolution history (www.standishgroup. com), 29% of projects failed, and another 53% were "challenged." Only 18% were deemed by respondents to have been successful. A failure rate as high as this or indeed a success rate as low as this would not be tolerated in any other industry. Yet, according to the same study, the rates for failed, challenged, and successful IT projects have been virtually unchanged since the mid-1990s. Could it be that IT professionals simply deliver bad products and poor services? Of course not, but what accounts for the widespread disappointment with so many systems?

The answer depends to a great extent on expectations. When OIS performance does not match OIS expectations, the system is perceived as unsuccessful. Thus, even if it is elegantly designed, well built, and functions exactly to specifications, the system can still disappoint users if it falls short of (often lofty) expectations (see the sidebar "Discounting Expectations").

Consider the example of a customer relationship management system. If the purchaser expects it to help produce better customer service, then disappointment is sure to follow, as much more than technology is needed to achieve this goal. Similar examples can be drawn for knowledge management systems, enterprise systems, and other forms of OIS. Many are so overloaded with expectations, they are almost doomed to fail in the eyes of their users. How can organizations avoid this problem?

A key to satisfying customers is to match their expectations with performance or overdeliver. However, the antecedents of expectations for OISs are numerous and—unlike simple consumer goods—often not well understood. OISs are complex, multi-faceted product/service offerings requiring coordinated efforts by many different groups, each with its own set of expectations.

Here, we reveal the related pitfalls and provide insight on managing expectations to support more successful OIS adoption and use. We combine insight from past work on expectations with data obtained from focus groups and identify several major issues organizations need to focus on to manage stakeholder expectations related to OISs. The importance of meeting or exceeding expectations has long been known to affect user satisfaction. A leading schema, called expectations-confirmation theory, or ECT, from the marketing field, is often used to model the relationships among expectations, performance, and user satisfaction (see Figure 1) [7, 10]. ECT consists of four main variables: expectations, performance, disconfirmation, and satisfaction. Expectations are predictive, indicating expected product or service attributes at some point in the future [10]. Expectations serve as the comparison standard in ECT—how consumers evaluate performance [6]. If performance deviates from this expectation, the consumer's expectation is disconfirmed. If a product outperforms expectations (positive disconfirmation), the result is post-purchase satisfaction. If a product falls short of expectations (negative disconfirmation), the consumer is likely to be dissatisfied [7, 10], provided initial expectations were positive [8]. Disconfirmation is most often measured by asking respondents whether something is "better than/worse than expected."

ECT is most commonly applied to simple consumer goods and services (such as video cameras, Internet service providers, and Web sites). More complicated is understanding expectations from large-scale OISs (such as ERP modules, CRM systems, and Voice over IP solutions). As noted earlier, these systems are complex, producing multiple forms and levels of expectations. Here, we explore insights obtained through a study we conducted to investigate the role of expectations in a large-scale VoIP solution. The findings suggest that the traditional ECT approach, outlined in Figure 1, may need to be revised to account for OIS complexity. They offer useful guidelines to organizations designing and implementing IT solutions that cater to the expectations of their stakeholders.

In the study, four focus groups were conducted over two days during the summer of 2004 in Toronto. The OIS was a hosted VoIP solution¹ only recently introduced to the market by a large telecommunications vendor. It was an appropriate vehicle for the study since it was a complex OIS with both product and service components. Furthermore, there was a widespread lack of understanding of the features and benefits of organizational VoIP solutions and thus a high potential to create expectations. Despite the promise of reduced costs, increased functionality, and improved organizational performance, the VoIP market in 2004 had still not lived up to industry expectations [4]. The disappointing uptake of VoIP solutions by organizations and consumers alike can be traced to a number of factors, including lower-than-expected cost savings, less-than-perfect reliability and quality of service, and insufficient bandwidth [5]. Many of these factors relate more to expectations generated by the proponents of the technology than to the technology itself.

Participants in the focus groups were senior-level IT, systems, or telecommunications managers in mid- and large-size organizations. They were the primary decision makers regarding telecommunications solutions in these organizations. The sampling frame for the research was drawn from the Dun & Bradstreet directory (www.dnb.com), and participants were pre-screened for appropriateness for the study. Four groups of eight or nine participants were drafted, for a total of 33 participants. The groups were screened to ensure that none of the participating firms were currently using a VoIP solution but that each was familiar with the concept.

---

There appears to be a greater downside risk to underdelivering on expectations than an upside reward for overdelivering.

---

The discussion guide was organized as follows: A professional facilitator first introduced participants to the focal topic through a general discussion of existing OISs (such as CRM and ERP) within their organizations. This phase was designed to get the participants to think in terms of expectations and performance measures. The second phase aimed to generate a list of expectations specifically related to the hosted VoIP solution; the facilitator asked participants to review marketing material on the VoIP solution. The telecommunications partner provided two product brochures; the first presented a marketing view of the solution and the second fact-based FAQs. The participants then discussed the expectations that had been generated by this material, organizing them into higher-level categories (such as quality, reliability, and cost). Finally, the facilitator asked participants to consider hypothetical scenarios of negative and positive disconfirmation outcomes and discuss the effect of disconfirmation on satisfaction with the system.

We then used the insights obtained from the focus groups to augment the basic ECT model, adapting it to the special characteristics of large-scale OISs. We organized our insights into three main issues—categories of expectations, relations among stakeholder groups, and effect on satisfaction—organizations must emphasize to manage OIS expectations.

Back to Top

Categories of Expectations

I really feel this is a marketing piece. They paint a very good picture. It seems too flawless. Reading between the lines, you are at their mercy. You have no control. From reading this, I would expect the install to be flawless. It's too good to be true. My expectation would be it would be beneficial in the long run, but the install would be rocky"—Information administrator, large professional services firm.

Members of each focus group collectively identified VoIP expectations, then organized them into higher-order categories (see the table here). Theoretical validation for these categories also existed, as they mapped well to the constructs identified by the model of IS success in [2, 3]. The table also includes examples of expectations generated within each category, many derived directly from the marketing material provided by the telecommunications vendor.

As most ECT research focuses on discrete consumer goods, product performance has been the predominant factor on which to base expectations. However, research on expectations for information systems indicates that they involve more than just product performance [11]. Our findings support this research, indicating that in the case of the hosted VoIP solution, expectations reflected a number of factors, ranging from technical details to user-oriented features, to high-level organizational needs. This is significant, as it ties together research on information systems and ECT to highlight the fact that disappointment may be linked to multiple factors, not just to product performance.

According to ECT, these categories affect not only expectations but also attitudes about performance, disconfirmation, and satisfaction. IT managers and system designers should identify and examine each of the three categories separately in order to pinpoint the source of potential disappointment. For example, a VoIP solution might meet expectations concerning the quality of the system itself but fall short for business sustainability and competitive advantage. The practical implication of such a result could involve a reexamination of the need for the VoIP solution in the organization. If that solution performs below expectations (such as less functionality or less efficiency than expected), the organization should reexamine the specific solution it purchased rather than the use of VoIP in general. Understanding the source of disappointment itself is valuable knowledge for an organization.

An additional implication of the expectation categories is their differential effect on overall satisfaction [11]. Since satisfaction from the OIS is the sum total of the satisfaction from the individual expectation categories, IT professionals must understand the importance of each category in generating overall satisfaction. Identifying the most important categories, perhaps by polling users in advance, may save organizations valuable resources by enabling them to focus on the most important single contributing factor to satisfaction and subsequently lead to more successful systems.

Back to Top

Multiple Stakeholders

"If my non-IT people knew how to use it and were excited about it, I would be extremely happy"—Director of networking, large apparel retailer.

"We didn't want to build up too much expectation. We were concerned in selling the expectations to our internal customers as to whether or not the product could deliver what the vendor said it could."—Managing director of communications, large accounting firm.

Building on the previous section, it is relevant to ask to whom is the OIS important. Expectations not only can vary by type but by stakeholder as well [9]. Most ECT research has assumed a single stakeholder—the end user, or person who chooses, purchases, and uses the product. However, we found that with complex OISs (such as VoIP systems) the end user is not likely to be the same person who manages the system or is authorized purchasing it in the first place. Each stakeholder is likely to have a different set of expectations about the system, with some potential overlap. Figure 2 makes this point, using the traditional classification of IS support for strategic, managerial, and operational decision making.

Moreover, evidence uncovered in the study supports interdependence of the satisfaction among the various stakeholders. We found participants' satisfaction was driven by their own disconfirmation, as well as by how well the technology solution worked for other stakeholders in the organization. For example, IT managers made it clear that until other internal stakeholders were satisfied, they themselves would not be satisfied.

The fact that OIS satisfaction depends on confirming the expectations of multiple stakeholders has important implications for IT professionals. It is thus critical that they devise a strategy for managing expectations and ensuring that an OIS's capabilities and benefits are well understood by multiple stakeholders prior to its introduction into the organization. Understanding the expectations and interdependencies of stakeholder groups is also key in devising measurements for OIS success [9]. Thus, IT managers and system designers must finely balance the expectations of each group.

An additional benefit of studying stakeholder satisfaction lies in the organization's ability to explain (and predict) OIS failure. IT managers can be dissatisfied with a new system that operates according to their expectations, just because the new system does not meet end-user expectations. Understanding this alternate cause of dissatisfaction can assist organizations in achieving satisfaction among all stakeholder groups and improving the success rates of IS projects.

Back to Top

Effect on Satisfaction: The Zone of Variation

"There is functionality that can be excessive. I doubt it would actually increase productivity. Some of them are difficult to put a benefit to. We need our phones to be the thing that works regardless of anything. If they go down, we might as well go home."—Director of IT, large beverage retailer.

During the focus-group sessions respondents were asked to consider hypothetical scenarios of met, unmet, or exceeded expectations and rate their level of satisfaction for each of the expectation categories in each scenario considered. For all eight categories of expectations, satisfaction in the case of simple confirmation, or met expectations, ranged from 3.6 to 4.1; on a scale of 1 to 5, 1 was extremely dissatisfied, and 5 was extremely satisfied. This result is in line with the ECT literature, suggesting that simple confirmation of expectations leads to satisfaction but not to delight; thus scores were lower than 5.

When asked to indicate (on a scale of 210 to 110, with 0 being "met expectations") how their satisfaction might change for negative or positive disconfirmation, respondents indicated a greater shift toward the negative than the positive (see Figure 3). There appears to be a greater downside risk to underdelivering on expectations than an upside reward for overdelivering. For example, Figure 3 suggests that if the system greatly exceeds expectations on quality, it would receive a rating of 15.6. But beware if these expectations are greatly unmet, as the rating drops to 28.6. This asymmetry was consistent for all eight categories of expectations studied.

We now introduce a general indicator measuring the power of the effect of disconfirmation on satisfaction, computed from the distance between the extreme negative and extreme positive values for each of the expectations categories. We term this difference the "zone of variation," as it represents the extent of variation in stakeholders' satisfaction evaluations for negative and positive disconfirmation.² To illustrate, our findings reveal that cost has a higher zone of variation (28.2 to 16.8) than revenue (27 to 16.1) or efficiency (26.2 to 15.3). This difference implies that satisfaction with respect to the cost of the system is more sensitive to negative or positive disconfirmation than satisfaction resulting from revenue gains or efficiency.

The zone of variation represents the range of how expectations influence stakeholder satisfaction and provide a useful tool for organizations to assess the best- and worst-case scenarios for a new OIS. As with identifying the importance of expectations categories, the main contribution of identifying the zone of variation is in saving valuable organizational resources. For example, Figure 3 outlines how an organization can learn that overpromising and underdelivering on the reliability of the VoIP solution is more costly than overpromising on the system's efficiency, since negative disconfirmation for reliability yields lower levels of satisfaction than does efficiency. By the same logic, the organization would gain more from overdelivering system reliability than by exceeding expectations for the system's efficiency.

The zone of variation highlights where most gains can be made (and most pain avoided) by managing expectations, thus providing organizations with a powerful cost-benefit analysis tool for managing expectations. While the specific ranges in Figure 3 are applicable to our example—the hosted VoIP solution—they may not hold for other solutions. IT managers may have to assess the zone of variation for each type of project. Nevertheless, the concept of the zone of variation and its implications are readily generalized.

Back to Top

Conclusion

"[The project] took a really long time, over three years. There had been really high expectations. We expected it to work really well, make life easier. But it ended up making things worse."—Telecom manager, 500-employee firm.

Our aim here has been to address why so many OISs are deemed failures by their users, exploring the paradox that comes from the difference between success and satisfaction; that is, systems may function as designed but still disappoint stakeholders and thus be regarded as failures. Disappointment is inextricably linked to the formation and confirmation of expectations. Thus, we've drawn on ECT, using focus-group data to enhance the model and increase its fit with large-scale OISs.

It is not sufficient to design and produce a functioning OIS. Relevant stakeholders must have an accurate perception of the functionality of the system. System designers should focus not only on users in the analysis and design stages [1] but collaborate with project managers and marketers to ensure that expectations generated by OISs match the system's capabilities. While some embellishment may be necessary to "make the sale," it behooves organizations to consider the process by which expectations affect stakeholder satisfaction perception and actual project success.

This work contributes to the general understanding and management of OIS expectations. We've highlighted the importance of using different categories of expectations within ECT to capture the special nature of large-scale OISs. We've proposed that the importance of expectations moderates the strength of their effect on satisfaction. We've shown how to relate expectations and satisfaction of all stakeholders involved in the adoption and use of an OIS. By adding the satisfaction levels of other stakeholders to the model, we've provided yet another explanation for dissatisfaction, enabling organizations to better identify problems and focus on their solution. Finally, we've also proposed the notion of the zone of variation, identifying the range of satisfaction gain (or loss) to be expected from managing expectations. This notion and its application enables organizations to focus resources where they have the most to gain. In a world of overhyped solutions and concepts, the IT professional must be able to determine a system's capabilities and match stakeholder expectations with those capabilities.

Back to Top

Back to Top

Back to Top

Back to Top

Back to Top

Back to Top

The Digital Library is published by the Association for Computing Machinery. Copyright © 2007 ACM, Inc.

---

No entries found