Communications of the ACM
The Profession of IT
A Clash of Civilizations
Credit: IR Stone
We are living at a unique moment in history where long-held views of how the world works and what it takes to be successful at innovation are falling short. Once upon a time we "knew" that research is the first stage in a pipeline that leads to marketable products. We "knew" that if we invested sufficiently in basic research that we would generate applied research and attain a competitive lead. We "knew" that we could maximize those investments if we applied sufficient controls on each stage of the pipeline. We "knew" this organized pipeline is historically sound because we saw how successful modern factories had become at turning out high-quality, low-cost products at scale. We "knew" these truths applied for businesses, organizations, and governments.
The pipeline idea traces back to the development of 19th century factories when innovation meant transforming inventions into products for the masses. We "knew" pipelines would work more generally than in factories because every change that emerged in the world seemed to unfold in distinct sequential stages. We saw pipelines all around us. We "knew" the education pipeline from K-12 through college to postgraduate school produced the educated workforce both industry and government need to compete successfully. We "knew" the military pipeline transformed raw recruits into military professionals and allowed the success of the all-volunteer forces. We "knew" the political pipeline transformed local leaders into state and national leaders. We "knew" the career pipeline transformed fresh graduates into professionals who could support a company's offers. We "knew" that, in all these domains, the innovation pipeline would systematically stage by stage transform raw ideas into useful outcomes. Over the years, the pipeline idea has been codified into business theory and government policy. It has survived so long because pipeline processes delivered good and often repeatable results that sustained our confidence in what "we knew."
Today, we are starting to question what we knew. Many people sense with a vague unease that pipeline thinking is no longer sufficient. The education pipeline is not delivering the trained personnel to meet the demands of industry, the military, or civil service. The military pipeline no longer meets military targets for recruitment, retention, or talent development. The career pipeline no longer supports company goals for internal professional development and stable market position, as people jump from company to company in search of new experiences, higher salaries, and higher positions. The innovation pipeline seems too slow, expensive, and rigid for the modern world. The U.S. Department of Defense expects to allocate 17% of its 2024 budget for research and development, and yet in the new geopolitical climate, many political and military leaders have lost faith that the investment in the research pipeline will generate innovations at a sufficient pace and scale to remain competitive. Many are looking for new approaches to generating innovation faster and more reliably.
The Eureka Story
One approach to resolving this tension has been to augment the pipeline with the Eureka story. The Greek word Eureka means "I have found it." The Eureka story idolizes the inventors who made the significant discoveries and seeded the innovation pipeline. From this perspective, to get more innovation we need to stimulate and reward more inventive creativity. There are multiple problems with this story. It ignores the work of the many others who take the first risk in adopting the practices of the invention or who work in the community to sustain adoption. It ignores the large number of innovations of practice that have no identifiable inventor. It ignores the waste of effort and subsequent disillusionment among the many whose ideas are filtered out.
The Eureka story has inspired new practices for generating ideas, such as contests or hackathons, which harness the crowd rather than individual inventors. Leaders in all fields often report they have more ideas to consider than resources to pursue all of them. Merely adding more ideas at the beginnings of the pipelines does not generate more or higher quality output. The Eureka story adds complexity to the pipeline model but does not increase the overall rate of adoption.
The Valley of Death
The Valley of Death is another augmentation of the pipeline story. It emerged to explain the failures of the pipeline model. According to this story, when an artifact exits the pipeline it does not readily transition into adoption because of bureaucratic, social, management, or cultural barriers. The barriers are so tall and strong that getting past them all is next to impossible. The Valley of Death now appears in pipeline diagrams as a new stage that must be managed but is poorly understood and does not improve adoption rates. While this story gives some insight into why transition from invention to adoption did not happen, it gives no insight into how to make that transition more successful.
Adoption Thinking
So, if pipeline thinking and its augmentations are not reliable paths to adoption, what is? The answer is right under our nose: Foster the conversations in which adoption happens.
Many are looking for new approaches to generating innovation faster and more reliably.
Consider an example. Generative AI in the form of ChatGPT was offered for public access at the end of November 2022. It touched off enormous waves of interest. Within two weeks, 100 million users were trying out the new technology. Within two months, there were a dozen how-to-use-GPT books available from Amazon. Tech companies quickly announced plans to incorporate the technology into their browsers, office packages, online services, and other apps. Hundreds of proposals for startups appeared and venture capitalists found funding to back them. The speed of adoption of generative AI has been astonishing. All this happened in conversations. The conversations spread rapidly through the Internet and media. Many people started experimenting to see whether generative AI would be a good advisor, artist, or writer, and they shared their findings widely in social media and as preprints of research papers. Moods of awe and surprise and intense enthusiasm blossomed. They were later joined by moods of distrust and fear that the generative AI technology might get out of control and cause great damage to societies. The ideas and moods spread rapidly round the world.
Innovation leaders were everywhere. OpenAI was first, offering its new technology for public experimentation. Other tech companies quickly followed with their own versions. Many experimenters and tinkerers generated greater understanding. The AI experts who started having doubts about the technology led a conversation about the role of government regulation to manage the dangers.
All this has happened in the space of swirling conversations among many people. The collective conversations took on a life and momentum of their own, beyond the control of any one person. Two broad themes emerged. One focuses on how we might harness the power of different forms of AI to address concerns in our community, such as democratizing knowledge work and relieving humans of dull, dirty, and dangerous work. The other focuses on breakdowns that have appeared around trustworthiness, unpredictable "hallucinations," worker displacement, and intellectual property. Calls to subject generative AI to "the discipline of the pipeline" are intended to slow the pace of adoption and maybe put the genie back in the bottle.
This is an example of an adoption conversation. Adoption is unruly and chaotic. The process empowers people to address longstanding concerns as well as iterate new practices as AI is adopted. It shows that innovation is emergence of new practices in a community. Adoption of new practice happens as people reach agreements and make commitments in their many conversations. New practices are emerging from the collective work of leaders at every level as they use the technology and then make new offers enabled by the technology.
This interpretation of innovation has been around for a long time. It appears in the stories of leaders nurturing their communities to commit to new practices. Leaders do this by fostering conversations that expose concerns and offering new practices to take care of them. The common stories of innovation leaders emphasize qualities such as charisma, creativity, genius, brilliance, connections, extroversion, and good luck. However, there are numerous examples of successful innovation leaders who lacked these qualities. These qualities are not needed to be successful.
Successful innovation leaders rely on a skill set—competence in the conversations that lead to adoption. The skill set includes eight essential practices—sensing, envisioning, offering, adopting, sustaining, executing, embodying, and mobilizing.a They are essential because omitting any one of them is likely to block the innovation. The good news is that anyone can learn them.
Adoption always requires leadership, but not the charismatic type. In most innovations, we see many people exercising leadership skills. The leadership is often distributed by necessity because most innovations involve many members of the community. Leadership is present and essential, but distributed among the many, not concentrated in the few.
The Clash
These two interpretations of innovation are established civilizations whose inhabitants have lived in them for a long time and have identified their professional offers with them. Reconciling them is a challenge. Bob Metcalfe captured the challenge with his famous saying, "Invention is a flower, innovation is a weed." The work of invention and the work of adoption seem to be, paradoxically, simultaneously complementary and contradictory. Pipeline thinking aims to control the process of moving inventions into marketable products, with its focus on professional management, order, and cost containment in the production of new artifacts and technologies. But by itself production does not produce adoption. Adoption thinking aims to bring order into the conversations of unruly, unpredictable, and often chaotic human communities as they strive to collectively embody a new practice. But by itself, adoption does not produce the artifacts that become the tools enabling the new practice. Production is a linear-ordered process; adoption is a chaotic process.
The clash has become more apparent in recent years as the concern in business and government for more and faster adoption has increased. Pipeline advocates argue that the discipline and control of the pipeline should be extended deeper into organizations, touching every project. In their view, failures arise from too many unpredictable and chaotic things happening in the raw, undisciplined social spaces of communities. Adoption advocates argue that success is often compromised by trying to impose a linear order on the nonlinear process of emergence because it limits the very conversations necessary for adoption. In their view, adoption cannot be achieved by imitating a production process.
Success at both adoption and production depends on personnel and managers being competent in these essential conversation skills.
Whereas the pipeline focuses on control, orderliness, and predictability at producing technology, adoption focuses on the embodiment of the new practice in the using community. How do we choose between these apparently conflicting worlds?
Resolving the Clash
We regard this as a false choice. The logic of the pipeline is, as it was in the beginning, to organize the production of artifacts. It is not useful to organize social and human processes. The logic of adoption conversations is to bring about community agreement on new practices. They do not implement the production lines that supply tools to support new practices. We need both logics.
The reconciliation comes when you realize that both production and adoption happen in social space. Social space is a churning sea of conversations in a community that maintains relationships, commitments, coordination, customs, and stability. Innovation is a change of practice in this space.
Many innovations depend on production of new technology at a scale sufficient to provide everyone in the community with the tools they need to carry out the new practice. The conversations that drive production of tools to support a new practice are not the same as those to generate adoption of a new practice. As mentioned earlier, the conversations that drive adoption are sensing, envisioning, offering, adopting, sustaining, executing, embodying, and mobilizing. The conversations that drive production are:
- Conversations for action: loops whereby two parties bring about a condition of satisfaction. Every stage in a pipeline depends on these conversations for its execution. If the personnel working on that stage do not complete their action loops, their stage will be a source of bottlenecks and complaints that interfere with overall production.
- Conversations for possibilities are needed whenever contingencies arise in production.
- Conversations for context and relationship are needed so that workers see the importance and relevance of their work and enjoy camaraderie with their fellows.
Success at both adoption and production depends on personnel and managers being competent in these essential conversation skills. Unfortunately, education in these skills is uncommon in technical fields, which favor "hard" over "soft" skills. Many find that the soft skills needed to get their work adopted are harder than the hard skills of their work. The good news is that learning the soft skills of leading adoption is not that hard once you know what they are.
The skills of managing conversations in social space dissolve the Valley of Death and make it navigable.
Notice the potential harmony: Results of production are brought into adoption by the adoption conversations; new production lines are brought into operation by previous adoption conversations. The pipeline model is embedded in social space and social space is embedded in pipelines. But they are not integrated. That is hurting our ability to achieve the much-sought end result: adoption.
Footnotes
a. P. Denning and R. Dunham. The Innovator's Way. MIT Press, 2010.
We are grateful to Nick Dew, Dan Finkenstadt, Garth Jensen, and Dave Newborn for insights as we were preparing this column.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2023 ACM, Inc.
No entries found