Communications of the ACM

Contributed articles

The Productivity Paradox in Health Information Technology

By Quang "Neo" Bui, Sean Hansen, Manlu Liu, Qiang (John) Tu
Communications of the ACM, October 2018, Vol. 61 No. 10, Pages 78-85
10.1145/3183583
Comments

"Health information technology connects doctors and patients to more complete and accurate health records ... This technology is critical to improving patient care, enabling coordination between providers and patients, reducing the risk of dangerous drug interactions, and helping patients access prevention and disease management services."— President Barack Obama, Presidential Proclamation on National Health Information Technology Week, September 12, 2011

Health information technology (HIT)—the application of information technologies to enable and enhance the delivery of healthcare services—has been a central point of focus for U.S. healthcare policy since 2007. Both Presidents George W. Bush and Barack Obama outlined bold goals for HIT adoption as a key facet of each of their healthcare reform efforts, promising significant benefits for healthcare providers and patients alike.²⁰ Clinical HIT systems, including electronic health records (EHRs), health information exchanges (HIEs), computerized provider order entry (CPOE), and telemedicine technologies, are seen as critical remedies to the complexity and inefficiency that have long plagued the U.S. healthcare industry.^a

Back to Top

Key Insights

In 2009, the U.S. allocated more than $30 billion, aiming to reduce healthcare costs and increase quality of care through adoption and use of HIT systems.¹ In that same year, the Office of the National Coordinator for Health Information Technology (ONC) was established as part of the Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009 to drive HIT adoption and coordinate development of critical HIT infrastructure. The ONC oversees a range of programs (such as regional extension centers, HIEs, privacy and security policies, workforce development, and curriculum development). The HITECH Act introduced the principle of "meaningful use" of HIT, a set of guidelines for the substantive adoption and application of HIT, including corresponding incentives and penalties to motivate increased use.³

Despite aggressive investment and governmental support, evidence of HIT's contribution to health outcomes remains mixed.⁷ A 2014 report from the U.S. Government Accountability Office (GAO) suggested that meaningful use requirements have had a modest effect, and a comprehensive strategy is needed to achieve better quality of care through HIT.¹⁴ In addition, while several studies highlight perceived benefits of HIT use (such as better clinical decision making and improved communications), other research suggests the observable effects are limited or even negative, marked by the risk of disrupted workflows, degradation of physician-patient relationships, and reduced clinical insight.²⁵ In light of these findings, many researchers and public-policy observers have called for additional studies to provide credible evidence of improved health outcomes through expanded use of HIT.²⁶

Back to Top

Evidence from New York

To explore the effect of HIT adoption on health outcomes, we consider the evidence from the State of New York. As the country's fourth most populous state and a national leader in HIT investment and adoption, New York offers a valuable context for assessing the effect of growing use of clinical HIT. Since 2007, New York has invested more than $840 million^b in health information infrastructure. In that time, a variety of initiatives within the state have sought to foster information exchange, improve quality and outcomes of care, reduce healthcare costs, and engage constituents in their care.²² Specifically, the state has focused on establishing governance and policies that increase participation in regional HIEs and encourage EHR system adoption by hospitals and individual providers. These efforts align with federal HIT meaningful-use initiatives aimed at creating better management of medical records and seamless coordination among healthcare providers across boundaries.^c

To understand HIT effects among New York healthcare providers, we conducted a mixed-methods study using both quantitative and qualitative approaches. Our quantitative analyses used publicly available data from New York HIEs, New York State websites, and databases made available by the not-for-profit American Hospital Association and the U.S. Centers for Medicare and Medicaid Services. The dataset covered the period 2014–2015 for more than 180 hospitals across the state. We tested a structural model in which higher HIT investments would lead to increased adoption and use of EHR systems and HIEs that in turn would result in better health outcomes.^d We tested the model using partial least squares software; for details, see the online appendix "Research Methodology"; dl.acm.org/citation.cfm?doid=3183583&picked=formats. In addition to our quantitative analyses, we conducted a series of semi-structured interviews with more than 20 healthcare professionals from 2013 to 2016 to explore their experience around adoption and use of HIT systems. Respondents included multiple classes of clinicians (such as private practitioners, hospital physicians, and nurse-practitioners), managers, and IT professionals. The interviews were transcribed and coded in NVivo software to identify common patterns and themes.⁴

In general, we observed that in New York State, 2014–2015, substantial HIT investments led to the widespread acquisition and use of EHR systems, implementation of clinical decision-support functionality, and significant participation in HIEs. Specifically, New York healthcare providers implemented most EHR functionalities classified as "basic" (see Figure 1). On average, New York hospitals implemented 5.48 out of six basic EHR functions (such as electronic document viewing, results viewing, CPOE, and decision support); and hospitals differ only by the degree of implementation around other advanced EHR functionalities (such as barcode identification, tele-health, mobile device connections). Additionally, the number of new hospitals joining local HIEs corresponds to the surge in the state's public funding for HIT investment in 2008, significantly augmented in 2015 and 2016 (see Figure 2).^e As of 2018, over 80% of New York healthcare-provider organizations—162 out of 197—had joined HIEs and regularly exchange medical records data electronically.

Figure 1. Adoption of EHR functionalities by hospitals in New York State.

Figure 2. HIE new participation rate and HIT investments from state grants in New York State, 2007–2017.

While the majority of New York hospitals have implemented and used EHR and HIEs in their practice, the evidence is inconclusive with respect to how these initiatives have affected quality of care and broad health outcomes across the state. We found no evidence of a relationship between HIT use and such critical health outcomes as improved interpersonal care, customer satisfaction, customer loyalty, patient mortality, and reduced ER waiting times (see Figure 3). These results are in line with previous studies suggesting unclear evidence of HIT effects.¹⁵

Figure 3. Effects of HIT investment on hospital performance.

While HIE participation and EHR use levels reveal no significant relationships with most outcome measures, we were surprised to find EHR use also does have a significant adverse relationship with patient readmission rates and complication rates. To further explore this counterintuitive result, we looked at the social-capital index in each county where the hospitals operate. The social-capital index²⁷ reflects the socio-economic growth of a community.^f The post-analyses suggest areas with low social capital often see higher readmission rates and complication rates. This low score is due to such factors as rural market, low social support, and low educational rate. One possible explanation for our counterintuitive finding is that hospitals in areas with low social capital encounter inherent difficulties that in turn increase patient readmission and complication rates regardless of their use of HIT. We encourage future research into this relationship.

Augmenting our quantitative analysis, our conversations with healthcare providers suggest mixed feelings and skepticism toward the expected values of HIT. In particular, many clinicians were concerned that HIT initiatives were too often not motivated by patient-oriented objectives and might undermine rather than enhance the quality of care providers render. Prominent concerns include the perception that HIT adoption results in extra workload, ineffective communication, poor information quality, and ineffectiveness addressing operational needs. The following illustrative statements highlight the concerns shared by our respondents:

"This whole business about electronic medical records helping with communication I think is a total fallacy. I think it really hinders communication, unless you freehand-type or you dictate, which defeats the main purpose of electronic medical records." — Physician, Pediatrics

"I hear complaints from patients saying, 'They're looking at the computer and not at me.'" — Physician, Pediatrics

"This is my issue with all electronic medical records: The notes that are generated have an awful lot of words but communicate very little." — Physician, Family Practice

Figure. Explaining the IT productivity paradox in HIT contexts.

"The highlighted efficiency from reducing duplicate lab tests and cutting costs is just not there yet. I am not really sure that an EHR will provide the savings that are talked about." — Physician, Internal Medicine

"I have charting at home. I ended up having to get a laptop through my work budget to bring home so that I wasn't sitting at the office until ... I would see my last person around 4:20, and I would be there until 6:30 doing charting because of being slow with the system and be more attentive to the patient than I was to the computer." — Nurse Practitioner

In summary, our mixed-methods analyses suggest strong evidence of increased adoption and use of EHR and HIE among New York healthcare providers but cast doubt on the claim of substantial HIT effects on health outcomes.

Back to Top

Assessing HIT

The challenge of finding evidence of practical benefits accruing from IT investment is not unique to healthcare. Indeed, the IT productivity paradox,⁵ an apparent disconnect between investment in IT resources and discernible impact on organizational performance, has been widely observed with earlier waves of IT adoption in manufacturing and other industrial sectors. In a seminal disposition on the phenomenon, Erik Brynjolfsson⁵ summarized a number of concerns that emerged in the 1980s and early 1990s around a lack of productivity gains corresponding to rapid adoption of IT resources. Several analysts had noted significant growth in technological investment and innovation across developed economies had coincided with disappointing gains—or even declines—in productivity.^11,23 It appears that just as in the manufacturing sector, HIT is struggling to produce credible improvements in key measures of performance. The IT productivity paradox has once again surfaced in the healthcare industry.

In his exploration of the phenomenon, Brynjolfsson⁵ suggested four possible explanations: mismeasurement, temporal lags, redistribution, and mismanagement. Mismeasurement refers to the idea that we lack appropriate measures for productivity in a service-based economy, with most traditional, manufacturing-oriented measures of productivity failing to account for indirect benefits (such as quality and customer satisfaction). The issue of a temporal lag centers on the possibility that gains from IT investment could take years to develop as organizations change their ways of working and the skills of their personnel. Less optimistically, redistribution suggests the dearth of productivity improvements could be the result of new IT resources merely shifting productivity gains (or losses) from some market participants to others. That is, IT may indeed create productivity gains for some players, but such gains are counterbalanced by losses for other individuals or organizations. Finally, Brynjolfsson⁵ said the productivity paradox could derive from the fact that "IT really is not productive at the firm level" or that managers have not been able to apply IT resources effectively.

"One of the health plans locally made an attempt at doing reporting [on provider efficiency]. They based it totally on cost. So [when they looked at the report] one of the physicians that was in the top had died six months before. He looked very efficient from a cost perspective. He hadn't generated any cost to the system." — Director, Medical Society

In the years since the initial explorations of the productivity paradox, the apparent disconnect between IT investment and organizational outcomes has been largely resolved; that is, researchers have concluded that the first two explanations—mismeasurement and lagged effects—were the primary drivers of the paradoxical observations⁶ and that IT investment is indeed correlated with significant improvement in various measures of value at firm, industry, and country levels, but such gains might take years to materialize.^12,13,28 However, the idiosyncratic characteristics of the healthcare sector (such as institutional heterogeneity, combination of public and private influences, and comparatively late adoption of IT innovations) underscore important differences with the sectors explored previously. Consequently, a thorough consideration of diverse possible factors is warranted.^17,19 Indeed, the four proposed explanations associated with the IT productivity paradox suggest critical clues for considering the inconclusive effects of contemporary HIT investment (see the table here).

As our analysis highlights, the idiosyncratic nature of the healthcare domain introduces a range of relatively novel outcome measures for HIT investment, including quality of care, readmission rates, complication rates, and diagnostic accuracy. While these are well-established measures of effectiveness for health services, their appropriateness for evaluation of the efficiency and effectiveness of HIT remains to be seen. Interestingly, the concept of "meaningful use" that has driven adoption of much HIT since the passage of the HITECH Act focuses almost exclusively on measures of input, or use of a certified system for records capture, reporting, and data exchange. Despite incentivizing inputs, the ultimate objective of the meaningful-use guidelines is substantive improvement in health outcomes (such as quality of care and fewer medical errors). This disconnect suggests we may need better measures to capture the contribution of HIT investment to those ultimate objectives.²⁹

With respect to the question of a temporal lag, a number of studies have suggested this is a critical issue in the healthcare context. For example, Menon et al.²¹ found it takes, on average, from two to four years for HIT systems to improve health outcomes in a given healthcare-provider organization. Many providers lack the necessary IT skills to quickly get acquainted with new HIT tools and procedures, making implementation more challenging. Given the fact that the uptick of HIT investment commenced only in 2009, it may take many more years for HIT influence to ripple across healthcare providers.

The possibility of redistributive effects also warrants consideration in the HIT context. As the comments of our study respondents underscore, many healthcare providers fear the efficiency in reporting and data analysis HIT engenders for insurance firms and regulators comes at the expense of decreased efficiency for clinicians who actually deliver clinical care. Indeed, this shifting of efficiencies and burdens can be seen in one of the most common organizational responses to HIT adoption: dedicated "scribes" to capture data during a clinical encounter. The question of whether efficiency gains in one facet of the healthcare system are partially outweighed by efficiency or process losses elsewhere in the system thus requires additional analysis.

Finally, the mismanagement of IT resources may well play a role in the mixed results of HIT adoption. Concerns expressed to us by healthcare providers regarding the usefulness of HIT resources suggest the possibility of missteps in the design, implementation, and/or ongoing use of these systems. These concerns lead to negative perceptions of HIT that likely result in misuse and jeopardize overall performance. Yet such concerns from multiple stakeholders are hardly captured in HIT development, and IT staff is inexperienced in helping and adjusting the new systems to local needs. In our interviews, several healthcare providers expressed their struggles in managing new systems due to their limited time and personal technology anxiety.

While each of the proposed mechanisms for paradoxical outcomes has some applicability in the healthcare context, the rich vein of research that grew out of the productivity paradox also offers some critical caveats for assessing the practical effect of IT investment and use.^12,13,28 First, significant variation exists across firms and industries with respect to the effect of IT investment on organization performance.⁹ Second, this variation and the existence of temporal lags are tied to the fact that performance gains are often associated not merely with the adoption of new IT resources but with the concomitant redesign of business processes and investment in complementary assets and skills.^6,28 Finally, the healthcare literature reveals that measures of productivity or business value remain ambiguous and highly contingent on firm or industry conditions. Applying these lessons in the context of HIT, the evidence points to the need for more research to understand the complex nature of the healthcare industry and its business processes, along with interdependence among healthcare stakeholders in HIT development, adoption, and use.

Back to Top

Beyond the Paradox

Based on our analyses of the effects of clinical HIT adoption, we find that a number of viable mechanisms are available for achieving enhanced health outcomes as a result of expanded HIT use, moving from meaningful use to meaningful results. The U.S. healthcare sector is an interdependent system. Leveraging and extending past insights from research on the productivity paradox and IT business value in general, we find it would benefit from a collective approach that brings together such diverse entities as hospitals, insurance companies, regulators, and HIT vendors to seek systemic improvements.

---

We found no evidence of a relationship between HIT use and such critical health outcomes as improved interpersonal care, customer satisfaction, customer loyalty, patient mortality, and reduced ER waiting times.

---

Efforts by the healthcare community. Resolution of the apparent HIT productivity paradox will require more than the isolated efforts of healthcare providers, calling for a community effort. To this end, we suggest a stronger leadership role for HIE-facilitating entities, including regional health information organizations (RHIOs). As the ONC acknowledges, RHIOs are central to data exchange across healthcare institutions.³⁰ Given the challenges in the healthcare industry, we propose that RHIOs should be more than mere data clearinghouses but formalized institutions that significantly improve HIT use, especially in two major roles:

Encourage learning and adaptation mechanisms in HIT practices. As with many enterprise IT systems, HIT platforms are frequently complex and rigid, requiring significant resources and enterprise-level effort to implement effectively. For such complex projects to yield tangible results, it takes time for users to adapt to new routines and practices, patients to get accustomed to new processes and functionality, and in-house IT staff to discern what system modifications would make the new system better fit with local needs. RHIOs can serve as a platform through which different parties can share resources, help others learn, and contribute back to the broader community. In addition to creating a mechanism for the development and exchange of a shared knowledgebase, these organizations represent a bridge between different types of hospitals: large/small, public/private, urban/rural. Managers can consider practices proposed in RHIO-based discourses to foster learning and adaptation in HIT adoption (such as using collaborative teams to explore HIT functionalities, rewards to enforce positive behaviors, and centers of excellence around HIT best practices).

Put users at the center of the HIT experience. Commonly found in our interviews and in the HIT literature is the concern that HIT policies have pushed healthcare providers toward a techno-centric perspective in which HIT is pursued "for IT's sake" and HIT systems are designed without substantive input from prospective users.¹⁰ It is critical not to lose sight of the most important HIT stakeholders—the patients whose wellness is directly affected by HIT use and the healthcare providers who guide the patients through the treatment process. Healthcare providers thus need to encourage both policymakers and technology developers to emphasize inclusion of patients and healthcare providers in the design processes of HIT systems. To enable a coherent and seamless experiences across HIT systems, RHIOs can act as a forum for users' experiences to be heard, providers' suggestions to be noted, and community members' opinions to be constructively formed. Such a collaborative approach is essential to HIT success, because, despite the existence of competitive forces among healthcare providers, patient wellness should be regarded as the ultimate goal for all parties.

Academic research. Although the literature on HIT evaluation is expanding rapidly, there has not been a parallel increase in academic understanding of how HIT contributes to patient outcomes or how it can be used to improve health and healthcare. The related research should be adapted to meet the needs of clinicians, healthcare administrators, and health policymakers. We thus suggest the following actions for academic researchers:

Develop enhanced measurements for clinical HIT impact. As noted, the healthcare system today lacks adequate outcome-oriented measurements of the efficiency and effectiveness of HIT. For example, the U.S. Department of Health and Human Services released final criteria of "meaningful use" in 2010, aiming to improve quality and efficiency of care by encouraging clinicians and hospitals to use EHRs. However, as of 2017, the existing measurement of "meaningful use" focused exclusively on input metrics. Accordingly, researchers are well positioned to develop appropriate means of outcome measurement to connect HIT investment with productivity and clinical relevance. One important improvement that can be made in HIT evaluation is increased measurement of context, implementation, and context-sensitivity of effectiveness.¹⁸ Exploring contextual and/or organizational factors would help address lingering questions about potential mismeasurement in assessing the long-term impact of HIT. As we have noted in reference to the observed adverse effect of EHR use on patient readmission and complication rates in the New York State context, a range of factors (such as urban/rural setting, social capital within a region, and academic vs. non-academic hospital adoption) can influence the contribution of HIT use on health outcomes. Clarifying the most relevant factors would thus aid the healthcare field in untangling the causal dynamics around HIT adoption and use. In addition, another important improvement regarding HIT evaluation would be increased use of evidence-based and clinical HIT research.²⁴ Using rich data generated through clinical HIT systems, future studies could examine how HIT as "informatic intervention" can significantly improve patients' health outcomes. Other initiatives (such as the Precision Medicine Initiative launched in 2016 by the U.S. National Institutes of Health) also underscore the need for more evidence-based HIT research in the future.^g

---

Many clinicians were concerned that HIT initiatives were too often not motivated by patient-oriented objectives and might undermine rather than enhance the quality of care providers render.

---

Learn how to realize value from HIT. Early studies of HIT adoption and use focused largely on determining whether a particular HIT functionality created value and to what extent. With increasing adoption of EHRs and other forms of HIT, it is no longer sufficient for researchers to ask whether HIT creates value in terms of health outcomes.¹⁶ As researchers, we need to help healthcare providers and policymakers learn how to realize value from HIT. That is, while HIT is being adopted, researchers should focus on exploring the causal mechanisms underlying its use to deliver health value to patients. Such theory-building research could help clarify the antecedents of the productive application of HIT resources. In particular, such research could leverage recent research shifting from consideration of simple IT use to effective, enhanced, or idiosyncratic use of IT resources.^2,8 Insight from the research could inform the aforementioned efforts among healthcare system participants to identify and disseminate best practices and foster more productive use patterns.

Efforts by policymakers. Policymakers play a significant role in each of the measures we have proposed, as in community building through RHIOs and advancing outcome-oriented measures of HIT use. While they should work with academic researchers and the industry to identify more relevant metrics for healthcare providers, it is equally important they maintain a holistic view of the healthcare value chain. Instead of focusing on policies that incentivize only EHR adoption or HIE participation, policymakers should also consider how to promote experimentation both within and across geographic boundaries. This might include more flexible use-style incentive programs that reward not only hospital-by-hospital efforts but also cross-hospital, cross-state, and cross-boundary initiatives. It is difficult today to promote technologies that provide value across geographical locations (such as telemedicine) or across institutional boundaries (such as healthcare supply-chain systems). In order to promote innovation and collaboration, policymakers might thus want to consider measures that target multiple parties in a healthcare value chain rather than a limited number of dominant players. This would include support for public-private partnerships that bring together healthcare providers, payer organizations, and HIT providers or initiatives that include large-scale participation groups (such as the Precision Medicine Initiative). Such efforts could leverage emergent technologies (such as big data analytics platforms, mobile health apps, and social media) to quickly assess the efficacy of a diverse set of HIT projects and channel resources toward the ones that show the greatest promise for bridging the gap between HIT use and health outcomes across populations.

Back to Top

Conclusion

IT use in the healthcare industry has experienced tremendous growth and attention since 2007. Yet concrete and credible evidence that HIT improves health outcomes remains inconclusive. Our investigation of New York State healthcare providers further indicates the healthcare industry may be experiencing an ongoing HIT productivity paradox, mirroring earlier patterns in manufacturing and other industrial sectors. While potential HIT contribution to health outcomes remains an open question, we suggest a collective approach is needed to address the many issues raised by the HIT productivity paradox and hope our research invites further inquiry into this important issue.

Back to Top

Back to Top

Back to Top

The Digital Library is published by the Association for Computing Machinery. Copyright © 2018 ACM, Inc.

---

No entries found