Communications of the ACM

Virtual extension

Do SAP Successes Outperform Themselves and Their Competitors?

By Richard J. Goeke, Robert H. Faley
Communications of the ACM, October 2009, Vol. 52 No. 10, Pages 113-117
10.1145/1562764.1562793
Comments

It's been over 10 years since corporate America embraced ERP systems, but hard evidence on the financial benefits that ERP systems have provided has been elusive.⁹ This debate has spilled into the mainstream media, as America's two largest ERP vendors regularly advertise that their customers have benefited financially by using their products; some of these ads even cite studies that discredit the other's claims of financial superiority.

Investments in IT are typically justified by the productivity and profitability improvements that follow their implementation. It seems intuitive that IT will help streamline existing business processes, which should lead to a more efficient and ultimately more profitable company. Organizations of all types and sizes have invested heavily in IT based on this simple rationale, but the associated financial benefits have been difficult to nail down. While managers struggled to value their firms' IT investments, researchers tried to better understand the factors that made it so difficult to value corporate investments in IT. Among the factors that have been suggested, three may be especially useful. They are:

• Firm-specific resources and capabilities that meaningfully impact the success of IT implementations;²
• External forces that exert themselves on the firm;⁶ and
• The nature of the financial indices used to value IT investments.³

Much of the early research on the value of IT investments was based on industry-level data that masked the effects of important firm-specific resources and capabilities. When researchers finally examined firm-level data, they realized that differences in, for example, IT expertise and management, the quality of a firm's leadership and other human resources, and the uniqueness of its operations affected the success of IT implementations. Companies with firm-specific advantages generally out-produced their competitors.^1,2

The success of IT implementations also depends on the unique set of unwieldy external forces that exert themselves on the firm. For example, Melville etal.⁶ suggest a host of external forces that may affect the impact of IT implementations on organizational performance, including the degree of competition within an industry, the impact of the firm's trading partners, and country characteristics. Unfortunately, studies of the relationship between IT and organizational performance are plagued by disagreements about the external constructs that should be examined, how these constructs are operationalized, and the nature of their interrelationships. There have also been concerns expressed about the financial indices used to measure the affects of IT implementations on corporate performance. Foremost among these indices are measures of corporate productivity and profitability.³

Productivity is associated with how efficiently a firm manages its business processes to produce a dollar of sales. For example, employee productivity is often calculated as the dollar level of sales generated per dollar paid to employees (net sales/employee cost). Firms usually have substantial control over their business processes, and this makes them potentially easier to measure and value financially.³ For example, firms often use proprietary processes to more efficiently manage their inventories in order to generate a higher level of sales. Therefore, measuring how efficiently a firm manages its inventory can be calculated using inventory turnover (net sales/inventory).

On the other hand, profitability is an organizational performance measure that can be affected by factors unrelated to the IT investment.³ These factors include, for example, the number and quality of the firm's competitors and trading partners, intra-firm shifts in spending, and macro-economic changes in interest rates, exchange rates and inflation; many pundits would argue that even the formal recognition as a success by their trading partner should provide a competitive advantage that eventually shows up in measures of organizational performance. However, the ability to measure the impact of an IT implementation on organizational performance depends on the extent the impact both ripples through to the firm's bottom line and can be accurately differentiated from a host of confounding business factors.

Thus, IT implementations that enhance a firm's ability to better manage business processes are more likely to have an impact that can be both accurately measured and valued credibly. Although IT implementations should also have an impact on organizational performance, accurately measuring their impact has proven to be difficult.³ The very mixed results regarding the post-adoption profitability of firms implementing ERPs may be an example of this.^4,10 We suspect that the impact of ERP and other IT implementations on organizational performance will be easy to value financially if we can ever measure them accurately (that is, absent the distortions created primarily by hard to control business factors).

Based on what we've described above, we would expect to see significant improvements in both productivity and profitability after the SAP implementation period for SAP successes themselves (SAPPROD and SAPPROF). And we would expect SAP successes to improve versus their competitors (DIFPROD and DIFPROF). However, as also noted above, we believe that the inability to accurately measure the component(s) of profitability directly related to the ERP implementation is likely to compromise the profitability results.

Back to Top

Our Study

We collected productivity and profitability data for SAP successes and their competitors for the three years prior to as well as the three years following the year in which the successful SAP implementation took place. We compared the financial performance of SAP successes three years before and three years after the SAP implementation period. We also compared the financial performance of SAP successes versus their competitors over the same time period.

We focused our analysis on the two reasons most frequently given by companies (as well as by SAP and its customers) for implementing ERP systems: inventory reduction and profitability improvement.⁸ We used net sales/inventory to measure inventory turnover (our measure of productivity) and operating earnings/net sales to measure operating profit (our measure of profitability). Operating earnings is also known as earnings before interest, taxes, depreciation and amortization are deducted (EBITDA).

We selected firms explicitly deemed "SAP Customer Successes" at the SAP Web site (www.sap.com).¹⁰ The data needed to compute the productivity and profitability measures were obtained from the Compustat database for all publicly traded, U.S.-based SAP successes. These two criteria assured that the publicly available financial data on all sample firms were subject to similar accounting rules (not the case for most foreign and privately-held firms or for government agencies or divisions of larger companies). A total of 57 SAP successes met these criteria; 34 of these firms had complete inventory and 36 had complete profitability data. Over 30 unique Standard Industrial Classification (SIC) codes are represented among the 36 SAP successes.

While other researchers have compared each SAP implementer with a single competitor matched on size and SIC code (at the two-digit level), we compare each SAP success to the average across all of its publicly traded, U.S. competitors at the four-digit SIC code level. We believe that matching each SAP success with its business competitors during the time frame of the study produces more meaningful comparisons (for example, by controlling for industry influences) and more realistic comparisons (that is, firms compete with all other firms in their industry, and not just those of similar size). Of those competitors on which data were available, 94% (843/890) had complete inventory data and 94 percent (1337/1417) had complete profitability data.

Finally, each success's SAP implementation period had to be determined. Unfortunately, most firms do not report the dates of their SAP implementation period⁸ and SAP considers these data proprietary. Instead, we used the procedure developed by Hunton et al.⁵ that relies on keyword searching the Lexis-Nexis Business News database for announcements that an agreement to implement an ERP system had been signed between SAP and each success. As did Hunton et al.,⁵ we set one year from that date as the end of the SAP success's implementation period. The names of the SAP successes in the study sample as well as the number of their direct competitors during the time frame of the study are presented in Table 1.

We analyzed the data using regression discontinuity. Although it is not a new technique, regression discontinuity is especially useful when an event takes place that changes the intercept and/or slope of a regression line.⁷ This technique is widely used in economics to examine the effects of new policies on important outcome variables. For example, a tightening of welfare eligibility rules is an event that may have an impact on the employment rate. To determine whether this has occurred, the slope and intercept of the employment-rate regression line before and after the rule change are compared.⁷ If no significant difference exists in either the intercept or slope of the regression line after the implementation period, the implementation has not had the expected effect. This is illustrated in Figure 1.

We believe that the regression discontinuity design is ideal for examining the impact of an ERP implementation. For example, the significance of the intercept and slope differences illustrated in Figure 2 can be easily analyzed to determine whether β₃ = 0 and whether β₂ = 0, respectively.⁷ The computations are relatively simple as well as easy to interpret. This permits, for example, a simple computer program to compile and analyze the data on an ongoing basis until a predetermined power level for the statistical test is achieved.

Back to Top

Results

The mean profitability and inventory turnover for SAP successes and their competitors for each of the three years before and after the SAP implementation period are reported in Table 2.

Regression discontinuity equations were created for each expected outcome using the data reported in Table 2. The regression discontinuity results for the four expected outcomes are presented in Table 3.

Outcome 1. The first outcome (SAPPROD) posited that the inventory turnover of SAP successes would be significantly greater after the SAP implementation period. The regression discontinuity results for Outcome 1 are illustrated in Figure 2.

Although the slopes of the two regression lines in Figure 2 are not significantly different, the increase in the intercept (β₃) is significant. Therefore, Outcome 1 is supported.

Outcome 2. The second outcome (SAPPROF) posited that the average after-implementation profitability of SAP successes would be significantly greater than their average before-implementation profitability. The regression discontinuity results did not support Outcome 2: there was neither a significant change in the slope (β₂) nor intercept (β₃) of the regression line after the SAP implementation period.

Outcome 3. The third outcome (DIFPROD) posited that the inventory turnover of SAP successes would be significantly greater than the average inventory turnover across their direct competitors after the implementation period. The regression discontinuity results supported Outcome 3: the intercepts for SAP successes and the average across their direct competitors before and after the implementation period are significantly different. Interestingly, the regression equation explained 99.3% of the variance in inventory turnover between SAP successes and their direct competitors.

Outcome 4. The fourth outcome (DIFPROF) posited that the profitability of SAP successes would be significantly greater than the average profitability across their direct competitors after the implementation period. The regression discontinuity results did not support Outcome 4: the intercepts and slopes for SAP successes and the average across their direct competitors before and after the implementation period are not significantly different. Interestingly, the regression equation explained over 80% of the variance in profitability between SAP successes and their direct competitors.

Back to Top

Discussion

The correct way to determine the business value of IT investments has been debated by managers and researchers for many years; the value of ERP investments is the latest installment of this debate, and already, research on the business value of SAP implementations has produced mixed results. For example, SAP successes in the chemical and pharmaceutical industries did not significantly improve inventory management or operating income in the 3 1/2 years after SAP implementation.¹⁰ On the other hand, it has been reported that SAP implementers significantly improved their inventory management and pre-tax income versus their competitors.⁴ Moreover, SAP implementers significantly improved their inventory turnover and profitability during the SAP implementation period.⁴ Our results fall somewhere in the middle: we found significant improvement in inventory management, but not in profitability. While our results won't end this debate, our findings are worth noting for a number of reasons.

First, we've added some fuel to the fire that rages between proponents and detractors of ERP systems. On the one hand, our results indicate that SAP successes did not significantly improve their profitability or their profitability versus their competitors. This is a compelling story, because the "best" firms had no real profitability improvement over their competitors. On the other hand, we found that SAP successes did significantly improve their inventory management, both against themselves (where they act as their own control group) and relative to their competition. While these results may cast doubt on what SAP means by "Success" (especially where profitability is concerned), we would not leap to such a conclusion until all of the factors that influence profitability are better isolated and accurately measured.

A second reason that our results are worth noting is that the regression discontinuity design proved to be a useful tool for examining changes in important financial outcomes before and after an IT implementation. The regression discontinuity design facilitates an assessment of whether an IT implementation significantly affects the post-implementation regression line. Moreover, even when accounting for something as complex as a SAP implementation, the regression discontinuity equations explained nearly all of the variance in inventory turnover and inventory turnover differential, plus a substantial portion of the variance in profitability differential.

Finally, we believe that our comparative results are based on a more compelling methodology that compares SAP successes to the average across all their similarly situated competitors rather than to a single matched-competitor. One of the advantages of our approach is that it is easier to implement. On the other hand, using a single-competitor matching methodology requires the identification and understanding of the parameters that describe the optimum single competitor. Although a number of different parameters have been suggested (for example, return on sales, total assets, and total revenues), there is no consensus either about what are the parameters or how they interact. We also believe that our approach results in a fairer, more accurate, and much more realistic comparison. After all, competition is imperfect and very few companies have only one competitor.

What can be learned from our study:

• SAP successes significantly improve their inventory management after the SAP implementation period; SAP successes' inventory management is also significantly better than the average across their actual competitors during the same time period. This is extremely important, because SAP successes improved their inventory management relative to all of their competitors. While a number of competitors may have implemented an ERP system, this shows that being designated by SAP as a success is meaningful.
• The regression discontinuity design is an easy-to-use methodology for assessing the relationship between changes in important business-related outcomes that occur after an event, such as an IT implementation.
• While managers should expect their IT investments to more than pay for themselves, our research suggests ERP investments will payoff to the extent that firms force business-process improvements to ripple through to the bottom line. Otherwise, we must do a much better job of understanding profitability and other important constructs and their interrelationships.

Back to Top

Back to Top

Back to Top

Back to Top

Back to Top

Back to top

The Digital Library is published by the Association for Computing Machinery. Copyright © 2009 ACM, Inc.

---

No entries found