Communications of the ACM
The profession of IT
Mastering the Mess
Have you ever found yourself in a messa difficult situation from which you would like to extricate, but there is no obvious way out? I will discuss here some recent findings from our innovation project that shed light on effective strategies for finding paths out of messes. Because there is a strong link between big messes and major innovations, mastering the mess is likely to help you create your next innovation.
Let's start with examples. Most of the great innovations of computing were responses to messes that once existed. The digital electronic computer, for example, was motivated by two messes that arose during World War IIoverwhelming stress on hand calculator teams to rapidly produce accurate ballistic tables for a growing arsenal of munitions, and devastating surprise attacks that the Allies could not block because the Enigma cipher hid German communications. Table 1 lists a few computing messes and the innovations that resolved them. All these innovations were social revolutions, supported to varying degrees by new technologies: large communities of people had to buy into new thinking and adopt new practices.
Messes are worse than problems. The word "problem" often refers to a difficult situation, but usually with the expectation that solving the problem will end the difficulty. Messes are complex and inscrutable. There is no agreement on whether the mess manifests a single problem or the convergence of many. With a mess, the problem is that we don't know to characterize it as a problem.
The Familiar Mess
Our world today is awash in messes. My students have no trouble listing dozens of them. They are not hard to spotTable 2 lists the most common signs. A few notable contemporary examples of messes are: user interfaces (the "GUI mess"); identity theft; securing networks against attack; spam; information overload; getting dependable, reliable, useful, safe, and secure software; getting large hastily formed networks for disaster relief to work effectively; inspiring young people to take up careers in computer science ("enrollment crisis"); global warming; and discovering terrorist plots beforehand. If history is a guide, each of these will engender a major innovation. Entrepreneurs know this well: they dive into messes they care about, knowing they have a ready market when they find ways out.
Long-term messes become so familiar they look normal to most people living in them. In the absence of a means to fundamentally change the mess, people change themselves. They accept the mess as part of reality and build their worlds around it. This is why a disruptive innovation that promises an end to the mess is so threatening: it challenges everything connected with the mess, including social power structures and deep beliefs.
Are All Messes Resolved by Disruptions?
There are two broad classes of innovations. Sustaining innovations are continuous improvements; because they achieve lower costs and greater efficiency, they are likely to be socially acceptable and valued. Disruptive innovations shift to new paradigms (new belief systems and practices); because they change who has power, they are likely to be resisted and not win immediate social acceptance. In fact, the resistance to disruptive innovations helps perpetuate the mess.
The distinction is widely understood and frequently discussed. In The Innovator's Dilemma, Clayton Christenson demonstrates that many disruptions begin with a competitor going after the established companies' low-end business [1]. In Creative Destruction, Richard Foster and Sarah Kaplan discuss how winning companies allow themselves to be disrupted in order to maintain strong market positions [3]. In Warfighting and Disruptive Technologies, Terry Pierce concludes that successful disruptive military innovations occurred because their leadership initially disguised them as (socially acceptable) sustaining innovations [5].
In the military, continuous aim gunnery (circa 1908) illustrates a sustaining innovation and aerial bombing (circa 1922) a disruptive innovation [5]. In the computing business sector, the continuing evolution of chips according to Moore's Law (circa 1965) illustrates a sustaining innovation, and Amazon.com (circa 1995) and iTunes (circa 1999) illustrate disruptive innovations. By allowing authors and artists to sell titles directly without going through a publishing house, Amazon.com and iTunes are disrupting traditional publication processes. Both were initially presented as sustaining innovations (online stores) and have now accumulated sufficient market power that the disruption cannot be blocked.
Sometimes it's not immediately clear whether an innovation is sustaining or disruptive. Distance learning is an example. To many educators, distance learning looks like a natural extension of existing teaching practices. And yet many people react to it as if it were a disruption. The current state of affairs with distance learning bears many of the characteristics of a mess. This means that the people working to advance distance learning would be better off applying the strategies discussed in this column than offering more analyses and lists of benefits.
Messes seldom persist indefinitely. The beginning of a mess's resolution is marked by a transformational event, such as introduction of a new technology or a new social practice (see the figure here). In most of the examples listed in Table 1, the transformational events were new technologies. We are today enjoying their settlements; the messes are gone, and the technologies are so well integrated into everyday practice that we seldom notice them.
Now, the key observation is that transformational events usually turn out to be disruptive innovations. The reason is that the paradigm (belief system) in which people are living cannot resolve the current situation. All attempts to resolve the issue with sustaining innovations have failed. The people cannot see beyond their mental framework to a big enough picture that would enable a resolution.
While most messes resolve after a disruptive innovation, a few are resolved by ongoing sustaining innovations. For example, those who believed that expensive computing was creating a mess of the computing-haves and computing-have-nots saw the resolution in the relentless chain of sustaining innovations known as Moore's Law.
Nonetheless, when confronted with a mess, the odds are that it's an opportunity for disruptive innovation. Turning this opportunity into action is easier said than done. The historical observer ("A" in the figure) can easily see the mess, the transformation, and the settlement. But what about observer "B"? All that person can see is the mess itself. The transformational event may already have occurred, but, lacking the historian's perspective, this observer cannot recognize it.
B's predicament is much like that of a person thrown into a labyrinth. B cannot see the labyrinth as a whole and has no idea which way to move at a junction or how long it might take to exit. A, who can see the labyrinth as a whole, has no such difficulty. The best B can do is devise a good strategy for traversing the labyrinth so he can exit in the least possible time without retracing any paths.
Are All Disruptions Preceded by Messes?
Although the transformation pattern is common, the mess is not a necessary precondition for disruptive innovation. Many innovations arise from other conditions and motives. For example: Someone serendipitously stumbles onto something wonderful (penicillin). Someone envisions a whole new potential (Alan Kay's Dynabook led to laptop computers). Someone creates a new social entity that unleashes creativity and new values (Mothers Against Drunk Driving). Someone seeks an inexpensive way to give to the many what only the well-to-do can afford (Unix, Internet telephony, low-cost inkjet printers); many disruptive innovations arise from this source [1].
Many innovators whose work could be explained in this way actually saw themselves as dealing with a mess. The perception of a mess can be a powerful motivator. The strategies discussed here can be quite effective in helping them attain their innovations.
Categories of Mess
Messes come in four categories (see Table 3) [4]. Unfortunately, only the historical observer can see exactly what kind of mess it was. You need adaptive strategies that don't depend on which type of mess you're in. For example, you can set out to learn everything known about the mess. If your mess is Category I or II, you'll sooner or later discover the solution knowledge and bring in an expert to help you deploy it for your mess. If your mess is Category III, you won't find an expert, but you will find that the confounding "system" responds in the same way every time you probe it with the same experiment. In Category IV, the confounding "system" will adapt to your probes and will not respond the same way; it will appear chaotic.
One thing is for sure: you will need higher levels of skill to work with the higher categories of mess. You can recognize Category III because repeating an experiment repeats the same outcome. You can then formulate a strategy based on the repeating patterns you observe in the experiments. This is an empirical process of discovery, the scientific method at work. Sometimes the patterns needed to resolve the mess cannot be observed with current methods; only with finer instruments will it become possible to make finer distinctions. Throughout most of the 1800s, for example, physicists were at odds over whether light traveled in a medium called ether. They could not answer because they lacked the instruments to measure ether. In 1887, the Michelson-Morley experiment provided the instrument. That the instrument could not detect the ether became part of Einstein's inspiration for relativity, in which light travels at the same speed in all frames and there is no ether. Category III messes demand a level of skill comparable to a scientist at the cutting edge who is capable of inventing new ways to interpret the data.
Category IV demands the highest level of skill. The messy system adapts and changes as it is probed. It tends to thwart attempts to change the mess. The innovator must look for ways to bring about local, mess-free organization that can be propagated to the whole. This requires astute networking and political skills. Nancy Roberts refers to Category IV messes as wicked problems [6, 7]. Wicked problems defy repeated attempts to solve them. They demand skilled political and social leaders who can bring about new social agreements and new organization within the system. In 1980, Candace Lightner founded Mothers Against Drunk Driving to confront the mess in the legal system that tolerated drunk drivers and their carnage. Lightner and her colleagues showed great skill in attracting media attention and in gaining the support of politicians for new laws (see www.madd.org.)
Strategies
To find a way out of a mess, you'll need an innovation. I've already discussed how innovators rely on seven foundational personal practices [2]. However, because messes are complex and inscrutable, we need higher-level strategies to deal with them; six are described here.
Declare. You must begin by declaring that you see a mess and intend to do something about it. Your declaration is needed because many people find the mess to be completely normal and see no point in fighting it. Your declaration will mobilize others who may be willing to join you in the struggle.
Learn all about. Make yourself a student of the mess. Learn everything you can about it. Read what's been written, talk to people about what they know, gather data, perform experiments. Eventually, you'll either come across an expert who can solve it for you, or you'll figure out it's a Category III or IV mess and need a team with the right expertise. When you become an expert on the mess, you'll see patterns that no one else has seen, which you can exploit to find a way out.
Question the paradigm. The "paradigm" is the belief system in which everyone is operating. The existence of a mess is strong evidence that the paradigm is not able to resolve the problem and in fact may be the cause of the mess. Therefore, try to identify all the assumptions in the belief system and see if any of them is questionable in the current situation. Pay special attention to anomalies; they reveal the limitations of the paradigm.
Think together. You need to look outside the current paradigm to find a solution to the mess. But you don't know what "outside" looks like; you are blind to what's outside and lack the language to discuss it or even think about it. So bring together people who look at the world in different ways and are willing to think it through with you. Chances are that the group will see something together that no individual saw alone. Collaborate with them on a solution.
Lead. All the declarations, learning, questioning, and thinking will come to naught unless someone steps up to lead the change. If that's not you, you'd better convince someone else to do it.
Disguise. This is Terry Pierce's advice. Assuming you do find a path out of the mess, you are quite likely to encounter resistance to your proposed disruptive innovation. So find a way to make it blend in with what people find familiar. The more it blends in the less likely it will inspire resistance. You can get enough of a critical mass of people buying in to sustain the innovation when other people start to recognize the disruptive nature of your plan. (Think of Amazon.com and iTunes as examples.)
Conclusion
I have discussed messes, which are difficult, intransigent situations that people want to exit but feel stuck in. While some messes may be irresolvable, we can often find ways out of messes through six basic strategies.
Human adaptation to the mess explains the hostility that often greets the disruptive innovator. Their innovations challenge people's identities and social power structures. The hostility comes as a great and sad surprise to many innovators, especially those who focus on the technical problem and not on the people. This social characteristic of reaction to mess disruptors explains why Pierce proposed the disguise strategy.
References
1. Christenson, C. The Innovator's Dilemma. Harvard Business School Press, 1997.
2. Denning, P.J., and Dunham, R. Innovation as language action. Commun. ACM 49, 5 (May 2006), 4752.
3. Foster, R., and Kaplan, S. Creative Destruction. Currency, 2001.
4. Kurtz, C.F. and Snowden, D.J. The new dynamics of strategy: Sense-making in a complex and complicated world. IBM Systems J. 43, 3 (Mar. 2003), 462483.
5. Pierce, T. Warfighting and Disruptive Technologies: Disguising Innovation. Frank Cass, 2004.
6. Roberts, N.C. Wicked problems and network approaches to resolution. The International Public Management Review 1, 1 (Jan. 2000).
7. Roberts, N.C. Coping with wicked problems. In L. Jones, J. Guthrie, and P. Steane, Eds. International Public Management Reform: Lessons From Experience. Elsevier, London, 2001.
©2007 ACM 0001-0782/07/0400 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2007 ACM, Inc.
No entries found