Communications of the ACM
Home/Magazine Archive/November 2006 (Vol. 49, No. 11)/Decision Making in Very Large Networks/Full Text
The profession of IT
Decision Making in Very Large Networks
On August 29, 2005, Hurricane Katrina's 35-foot storm surge and strong winds damaged over 90,000 square miles of Louisiana and Mississippi. The damaged area lost most basic infrastructures: power, communications, water, transportation, and law-and-order. The Federal Emergency Management Agency (FEMA) was charged to coordinate numerous federal, state, and local agencies in a massive relief effort. Although there was no official timetable, early statements by local officials and by wishful commentators led many people to believe the FEMA-coordinated network could complete search and rescue operations within a week, restore power and communications within two weeks, get basic food and necessities distribution working within a month, and make substantial progress toward repair and rebuilding within a year. The network achieved none of these objectives. Instead, it was plagued by severe interorganization coordination problems, jurisdictional disputes, information overload, and fraud and waste of relief funds.
Although leadership and planning were rightly criticized, few have asked whether leadership was really the problem. There are good reasons to believe that no hierarchical, centrally directed network, such as the federal agencies attempted, could have met those expectations. We'll explore why this may be so.
Hastily Formed Networks
In an earlier column, we defined a Hastily Formed Network (HFN) to be an organizational structure that is (a) put together quickly in response to an emergency, crisis, or urgent situation, (b) from a collection of entities who have expertise or local responsibility to help but have not worked together before, (c) and who accept no higher decision-making authority [1]. Everything is challenged. As noted previously for Katrina, basic infrastructure may be seriously impaired or completely broken. The participating organizations' local practices, terminology, data formats, and culture make communication and information sharing among them very difficult. Overwhelmed with the enormity of the problem relative to their resources and training, they fall into defensive positions and do not coordinate well. There is no common hierarchy to define an authority to set overall direction and resolve differences.
The network everyone hoped for from FEMA was of this kind. It was big and complex and had little time to adapt or learn. FEMA was not alone. Hastily formed networks for tsunami relief along the Indian Ocean in 2004 and several earthquakes in 2005 faced similar problems. They had extreme difficulty fulfilling their expectations.
The high-bandwidth and high-connectivity of modern networks makes it seem that HFNs ought to be easy. But they are clearly not. We have much to learn before we can make them regularly successful. What fundamental limitations of HFNs might be getting in the way?
Hyper-Networks
It's helpful to step back and examine what we mean by "network" and what network science is beginning to tell us about the limitations of different levels of network. This will shed considerable light on the HFN conundrum.
Abstractly, a network is a collection of nodes and links. Network engineers see the nodes and links as routers and channels governed by protocols such as IP and TCP. Organizational specialists see the network as a representation of social relationships such as who sends email to whom, or who gets advice from whom. These two ideas are coming together. The "network" now includes the people as well as the telecommunications. We judge networks by how well they solve problems, not how well they move bits.
Altogether there are four levels of meaning to the word network (see Table 1) [4]. The levels represent a progression of organizational complexity. The first (lowest) level is the physical communication components that make connections and transfer packets and signals. The second level is protocols that manage information flows within the network. The third level brings humans explicitly into the network as players in complex "enterprises" that integrate people using telecommunications. The Internet is the "nervous system" of new virtual organizations, such as eBay (for auctions) and iTunes (for music), that are no longer contained by corporate or agency boundaries. The fourth level is ecosystems of interacting multiple organizations, all seeking to collaborate toward objectives extending beyond any of their own normal scopes. The emerging field of network science deals with fundamental principles and limitations at all these levels, applying them to analyze and design networks.
We use the term hyper-network for the entities at level 4. The HFN is the newest kind of hyper-network, having the special characteristic that they have little time to learn and adapt before producing results.
These levels reflect the evolution of networks over time. In the 1960s, the focus was on the physical structure of networks; in the 1970s it was on information flows; in the 1980s and 1990s it was on new organizational forms; and in the 2000s it moved to multiple interacting organizations. Thus levels 3 and 4 were not part of the original ISO models of the 1970s or the middleware models of the 1980s. At level 3 individual organizations use new kinds of coordination to achieve their missions faster, cheaper, and more conveniently. At level 4, federated organizations cooperate toward a single purpose, but retain their separate identities and may compete vigorously on matters outside that purpose.
What Kind of Organization?
Perhaps the most common organizational form is the hierarchical structure that recursively combines subunits into larger units. The hierarchy defines chains of command and many information flows within the organization. Tom Malone says that hierarchical organizations are good at managing decision making and information flows when communications are slow or expensive [5].
By making communications fast and inexpensive, the Internet enabled new kinds of organizations to form and flourish. Amazon.com was one of the first Internet-based commercial organizations. It provides a virtual storefront to a database of millions of books, and it dispatches messages to a network of participating suppliers who ship books promptly. Another successful Internet-only organization is eBay. It allows buyers and sellers to make offers and close deals. Its reputation service lets buyers and sellers rate each other's timeliness and integrity without a centralized credit rating service; those who do not keep their promises are quickly ostracized.
Viewed as product-selling virtual storefronts, Amazon and eBay are single organizations at level 3 in the model. But both entities spawned bigger organizations that wholly contain them. Both companies cultivated virtual communities, which are ecosystems at level 4, comprising many customers and companies as components. For example, Amazon.com has a large network of book suppliers, including regular commercial publishers and on-demand self-publishers; it also has a partner network of organizations that promote books for their own purposes and provide click-through to Amazon. Such collaborative business networks help multiple companies move products and services rapidly through their networks and increase the volume and pace of transactions for everyone. Other examples include business-to-business (B2B) exchanges, music distribution centered on Apple's iPod and iTunes, and the eBay community of participating businesses.
One of the first entities at level 4 was the open consortium, a federation of organizations cooperating on a common purpose [6]. The open software movement uses consortia to coordinate contributions to systems such as Linux. The World Wide Web Consortium (W3C) has about 400 member organizations that participate in consensus processes to advance Web technology. The Internet Engineering Task Force (IETF) coordinates advancement of Internet technology. There are many other consortia.
The Hastily Formed Network is the newest form to appear at level 4. Most of the participating entities have hierarchical internal structures (such as local police, fire, and city governments, military services), but some may not (for example, the W3C might participate). The HFN is neither a complete hierarchy nor a completely flat organization. Such federations represent a new kind of organizational challenge because no top level exists (or can be expected) to coordinate the components effectively.
Scaling Up the Hierarchy
Our intuition, based on years of experience and familiarity with hierarchical organizations, is that it should be possible to form hierarchical organizations at level 4. But we don't see hierarchical organizations at level 4, and the most visible attempt to force itthe hierarchical HFNdoes not work. Where is our intuition misguiding us?
The word hierarchy stands for two distinct conceptsaggregation and decision making. They need to be separated. Hierarchical aggregation means the grouping of organizational units into levelsfor example, the national network contains regional networks that contain local networks. Hierarchical decision making means structuring decisions through a chain of command. Decisions in level 4 organizations are distributed and collaborative rather than hierarchical. Aggregation scales up from level 3; decision making does not.
So the problem with a hierarchical FEMA HFN is the expectation of a chain of command. Many people believe that modern high-bandwidth, high-connectivity networking provides the tools by which FEMA could overlay a chain-of-command structure onto national, regional, and local relief organizations. But the lesson of hyper-networks is that this is unlikely to work.
Could FEMA have succeeded by imposing a sort of martial law? Military organizations are among the largest hierarchical decision makers in the world. But they are not hyper-networks. They are internally homogeneous, they have common standards for communicating across organizational boundaries, and all their members agree to the chain of command. In the Katrina relief effort, most civilian authorities accepted military help grudgingly, and some banned the military from significant operations in their territory. Transfer of data and situational awareness across the "civil-military boundary" was a huge problem.
The component organizations in hyper-networks guard their separate identities, cultures, and practices. They respect no higher authority. They have adopted decentralized decision making, because it enables them to work for the common purpose without giving up their separate identities.
The Central Limit
The preceding discussion establishes only that any attempt to make an HFN operate hierarchically is likely to fail because the players are unlikely to follow the rules or play that game. But there is a deeper reason, which at least partially explains the players' reluctance to try to make a hierarchical HFN work.
The reason is that hyper networks are actually economic systems in which many people engage in transactions involving resource transfers that contribute toward the purpose around which they cooperate. As early as 1937, F.A. Hayek, who in 1974 received a Nobel Prize for his work, argued that centrally directed economic systems cannot succeed [3]. In 1945, he said: "The problem is precisely how to ... dispense with the need of conscious control, and how to provide inducements which will make the individuals do the desirable things without anyone having to tell them what to do" [4].
Thus he argued it is impossible for the central director to aggregate all the fragments of information from the many individuals into a "total state" from which an optimal allocation of resources could be calculated.
Hayek argued that economic systems need to leave the decision making to individuals acting in what they perceive as their own best interests, while setting up information mechanisms that allow individuals to choose decisions that are good for the system as a whole.
The HFN encounters the same conditions as Hayek says exist in an economic system:
- There is too much data to aggregateso a central director could not construct or process the total state.
- The data evolves continuously; the channels are always full and there is always a backlog of unprocessed data getting stale.
- No single person (or small group) has enough perspectives to solve the many complementary and interacting management problems visible in the data.
- The data streaming from a central director would make information glut everyone's problem and limit everyone's ability to act wisely.
What are the Alternatives?
Clearly, decentralized decision making is the only way an HFN can go. What might decentralized decision making look like in this context? In Table 2 we have listed four main functions of an organization, the standard centralized approaches to implementing them, and decentralized approaches that have emerged at level 4 to implement them. Conclusion: there are plenty of alternatives to central decision making.
Value-Based Operations
To accelerate development of effective modern organizations we will need a change of perspective about "network"from a focus on moving bits (levels 1 and 2) to a value-based distribution system (levels 3 and 4).
Traditional organizations often have years to learn what information is valuable and how to make it flow. Hyper-networks must learn what is valuable in much shorter times, and HFNs must learn in a matter of days. This is why hyper-networks embody fast-responding means such as market price and supplier reputations for members to learn what is valuable to other members.
No one has yet experimented with market-based mechanisms for optimizing flow by value in HFNs. Here is an example of how a market mechanism could significantly improve effectiveness of rescue and recovery operations. Suppose FEMA created a "market board" to post offers of help and of need-of-help. Anyone in the region, from an individual to one of the relief organizations, could post either requests for particular kinds of help or offers of expertise and equipment. Someone in need of help can find an organization offering to help and close a deal with it. Someone with something to offer can approach someone needing help and close a deal. Such a Web-based service was used successfully by the military in Afghanistan to exchange information on situational awareness. The Web site c4imarkets.com is a prototype of such a service for future disaster relief efforts.
Conclusion
HFNs, like other hyper-networks, must employ organizational forms compatible with the nature of the organizational challenges at that level. Decentralized decision making and focused sharing of high-value situation information should be base principles.
These comments apply to how HFNs execute once established. We have not commented on complex issues of preplanning, which include contingency plans, exercises to develop familiarity, wargaming, system designs, evacuations, and advance positioning of assets. Suffice it to note that governments are finding it difficult to do preplanning that actually helps the crisis when it happens. Perhaps developing skill at decentralized decision making and federated execution would be a more fruitful way to prepare.
Many large-scale government operationsnow enabled by the existence of a high-bandwidth, high-connectivity networkneed to learn how to share situation information and to make decisions locally through market-like mechanisms that focus resources on high-value activities. Because central control virtually guarantees failure at these scales, governments will need to adopt the best practices of hyper-networks, including federated, decentralized control, and value-based information management.
References
1. Denning, P. Hastily formed networks. Commun. ACM 49, 4 (Apr. 2006), 1520.
2. Hayek, F. Economics and knowledge. Economia 4 (1937), 3354.
3. Hayek, F. The use of knowledge in society. American Economic Review XXXV, 4 (Sept. 1945), 519530.
4. Hayes-Roth, R. Hyper-Beings. Booklocker.com, 2006.
5. Malone, T. The Future of Work. Harvard Business School Press, 2004.
©2006 ACM 0001-0782/06/1100 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2006 ACM, Inc.
No entries found