Communications of the ACM
What Storytelling Can Do For Information Visualization
For as long as people have been around, they have used stories to convey information, cultural values, and experiences. Since the invention of writing and the printing press until today, technology and culture have constantly provided new and increasingly sophisticated means to tell stories. More recently, technology, entertainment, and art have converged in the computer. The ancient art of storytelling and its adaptation in film and video can now be used to efficiently convey information in our increasingly computerized world.
A well-told story conveys great quantities of information in relatively few words in a format that is easily assimilated by the listener or viewer. People usually find it easier to understand information integrated into stories than information spelled out in serial lists (such as bulleted items in an overhead slide). Stories are also just more compelling. For example, despite its sketchiness, the story fragment in Figure 1 is loaded with information, following an analysis similar to that of John Thomas of IBM Research [5]. We find that Jim uses technology (a pager and the Internet) and is dedicated to his job. Many other pieces of information can be deduced about Jim and his work, as well as about his relationships with his coworkers, as noted in the right side of the figure. The story does not express all this information explicitly; some is only implied; for example, we can surmise that Jim is probably not at the gym and his attendance at the meeting is important to his boss and coworkers, as well as to his company's business performance.
As in most stories, this one involves uncertainties, including, say, the number of colleagues Jim has tried to contact about his situation. Authors regularly and purposely include such uncertainties; if they are used skillfully, readers or viewers clear them up through their own imaginations, supplying their own experiences and expectations, and as a result may feel intellectually and emotionally attached to the story. Moreover, the story about Jim is a written narrative; the same story could be presented through other modalities, such as images alone or as a combination of images and words.
A Story Is Worth a Thousand Pictures
Images, too, hold a considerable amount of information a viewer might grasp quickly. But images are susceptible to uncertainties and might require some declarative statements to clear them up.
If one wanted to present Jim's story visually, transforming the words into a visual representation would not be straightforward. For example, how can a visual presentation show that Jim feels sick or that the time is early in the morning? How could it show the passage of three hours? Could a big clock on the wall be used in some way to count down the time? Trying to represent the story in a single static image is especially problematic. The whole picture is more than a single image. Since the events in the story are time-dependent, animation might be useful. But as in many historic silent films, relying solely on visual media might be inadequate for delivering the film's intended message. To truly represent the author's intention, perhaps the integration of animation and words would work best.
Transforming the representation from text narrative to visual domain requires adding more information to the presentation. For example, in a purely narrative representation, Jim's whereabouts are unclear and may, in any case, be irrelevant for telling the story. In a visual representation, Jim would have to be placed in some visual environment, possibly his bedroom. More information would then have to be added about the bedroom, such as the color of the walls, the pictures on the walls, and the sheets, pillows, and furniture.
These details cannot be left out without the overall picture looking odd. Film director Alfred Hitchcock and screenwriter John Michael Hayes addressed this situation in transforming a text-based narrative into the 1954 movie Rear Window. The original novella by Cornell Woolrich left out most of the neighbors in the apartment buildings overlooking the rear courtyard and included mainly the killer (Raymond Burr), his victim wife (Irene Winston), and the protagonist (Jimmy Stewart). But for the sake of telling a compelling story visually, Hitchcock needed to include (and show) neighbors in the various windows of the buildings interacting with one another while allowing their relationships to evolve gradually.
Please note that for Jim's particular story, the narrative is much more economical than its ultimate visual representation; this is not generally true. However, the visual representation might be more compelling and memorable than the script. After all, we all began our lives getting most of our information visually. To represent information in as compelling a way as possible, we need to choose and exploit an appropriate medium and genre to impart it, support our mission, and communicate with our intended audience.
Visualization and Presentation
Information visualization is a process that transforms data, information, and knowledge into a form that relies on the human visual system to perceive its embedded information. Its goal is to enable the user/viewer to observe, understand, and make sense of the information. Effective visualization is far more than pretty pictures alone as a communication genre (see the sidebar "Discovering Visual Metaphors").
What makes storytelling such a valuable component of information visualization? In scientific visualization, visual means (such as a single image or an animation loop) are typically used to solve a problem or convey specific scientific information. Information visualization is often employed for quite different communication purposes. The environments in which information visualization functions involve massive streams of information and data sources arriving in real time or from existing data and information sources (see Eick's "Visualizing Online Activity" and Keim's "Visual Exploration of Large Data Sets" in this issue).
The user/viewer of the visualization needs to integrate the information streams, thoroughly understand them, and make decisions based on their information in a timely fashion. Examples of the environments in which visualizations are most likely to be used include command centers, such as those in the power, transportation, and telecommunications industries, the executive offices of global corporations, and military installations. The problem in these environments is how to structure and present the information, so it is displayed efficiently, coherently, and economically, as well as what to include and what to leave out (the audience fills in the gaps). Moreover, presenting the information in a compelling and appealing way that enables it to be understood quickly is highly desirable.
All this thinking, planning, decision-making, and data transformation and formatting means the resulting information visualization is more than a single image or animation clip; it's a kind of show business. Producers and directors of film, video, theater, and television commercials face similar problems of how to show an optimal amount of information in a way that keeps the audience in its seats while delivering a message. The difference between visualizations and traditional entertainment media is the information and story conveyed in information visualization environments are usually much more complicated than those typically shown in films or the theater or on television programs and commercials.
Story-like Visual Presentation
The flood of complex information moving into industrial and military command centers needs to be analyzed, then communicated by commanders to their colleagues, as well as to higher and lower echelons. Since such incoming information is not naturally organized in any consistent way, it is difficult for the audience to understand what's going on without further processing. However, sorting it according to the geographical locations it refers to could improve understandingat least imparting the facts already known about each object in the geographical area of concern. Time-dependence further complicates matters.
Figure 2 is a shooting script in a hypothetical command-and-control situation (adapted from an example presented last year by Brigadier General Keith Holcomb, U.S. Marine Corps, Ret., as part of a DARPA Command Post of the Future exercise). It describes a situation in which a number of enemy positions surround a friendly school with children trapped inside as de facto hostages as the crossfire fills the air overhead and both sides move toward confrontation. To represent the information visually in a story-like fashion, we might divide the script into two parts, as shown in the figure:
Building the picture. Starting from an overview map, this part of the script and the visualization in the figure describe the actions and whereabouts of the different objects shown on the map.
Animating the events. Presenting these events versus time as they occur, animation is used to reinforce the information while making it clear.
The reasons for choosing the various elements of the script are based on the designer's experience understanding how stories are told in literature and visual media. The script has to establish a number of basic aspects of the story being told:
Setting mood and place in time. Included is narration explaining the display and the changes as they occur over time. Here, the voiceover starts with "It is now early in the morning; the time is H+8 [eight hours after the time H]"
Continuity. In creating a better and more appealing representation (at least for a Western audience accustomed to a continuous storytelling style), the designer who created the visualizations in the figure tried to make the transition between disparate pieces of information appear more continuous. For example, the map is one of the story's unifying devices. To enhance continuity, the visualization shows an overview map; it can then zoom into a particular building. The zooming action needs to be gradual enough that the audience won't lose touch with the context. The visualization might then zoom out to get back to the overview.
The "camera," or the visualization's adjustable point of view, can continuously pan or zoom into the map. Similar techniques are used frequently in commercial film makingeven in some classic silent movies, including the 1919 expressionist Das Cabinet des Dr. Caligari by German director Robert Wiene, and later in Hitchcock's Rear Window and even the Rhapsody in Blue sequence in Walt Disney Co.'s Fantasia 2000.
Some films employ discontinuous transitions between consecutive scenes, as in Sergei Eisenstein's films about revolutionary Russia The Battleship Potempkin in 1925 and 10 Days That Shook the World in 1927. Charlie Chaplin's 1936 Modern Times includes images of herded sheep, followed immediately by images of a crowd of people going to work. This contrast conveys the message that working people are treated like sheep by their company's management.
Filling gaps (a term coined by Tom Armour, a former program manager of DARPA's Genoa program). In Figure 2, instead of stating explicitly that there are children in the friendly school building, children might be shown superimposed on the image of the schoolas in Casablanca, directed by Michael Curtiz in 1942, whose opening shows a map of Europe and North Africa with superimposed images of refugees. The audience fills the meaning gap between the two images being shown simultaneously, thus understanding the message. Presenting the children in this scenario also increases the audience's awareness of what the friendly forces need to protect, as well as the emotional content of the information being conveyed.
Conflict and ambiguity resolution. Zooming in and out also helps resolve conflicts and ambiguities. When audience members see the layout of the area of conflict, they ask themselves what they know about the objects on the screen. Zooming and panning helps them see the objects in more detail, as in Ron Fricke's 1992 film Baraka about the surprising connections between various peoples and the spaces they inhabit and which was shot in 24 countries on six continents and didn't include a single word of dialogue; the voiceover in the figure explains what is being shown on the screen.
The story of the military effort to rescue the children stuck in the school building ends with a voiceover explaining how the commander views the situation, resolving unanswered questions, including: How strong is the enemy? and Why is the enemy quiet now? The commander speaks in a way that reminds the audience of the voiceover at the beginning of the story ("It is now early in the morning"). Using similar statements at the beginning and end of a presentation helps communicate the sense that the pattern of the presentation is complete; in this example, however, only a partial completion of the story's pattern is possible. The commander cites the early hour in the morning to explain why the enemy is not active at present (cause and effect).
Increasing attention. To encourage the audience to pay more attention to the lines of fire from the enemy toward the school, the system adds them to the picture when the fire is active; the lines blink whenever they are mentioned by the voiceover.
Effective redundancy. After going over the objects on the map and their activities during the last eight hours (Part 1 in Figure 2), the commander then switches to an animation loop (Part 2 in Figure 2) showing the events as they have have occurred over time. In Part 1, the time dependence was explained by the voiceover and required the audience's logical mind to put the information in context. The animation communicates the time-dependence in a more straightforward way.
These storytelling techniques have to support the story and convey its informationwhile disappearing from the mind of the audience. However, even when using all of them, a visualization designer cannot in most cases transform any particular set of facts into a story in a traditional sense, with a beginning, middle, and end, or give it the weight and sensibility of a mythological tale.
The Comics Metaphor
Shown a visualization presentation that relies solely on animation to show a particular course of events, viewers may be unable to recall what they saw on the screen only moments before. Unlike stories told in film and the theater, their visualization counterparts shown in command centers and other information visualization environments might include many visual objects moving around on the screen, even in and out of the screen area (see Figure 3). The limited capacity of human short-term memory determines how much and for how long any viewer is able to remember information. To overcome this limitation, information visualization designers might also illustrate their target information through a comic-book-style metaphor. At any given point in timeone for every two-hour periodthe user/viewer would see a separate picture, as if it were an individual panel on a comic book page. A large display is especially useful for this type of presentation because it allows inclusion of many windows.
Unlike paper, however, the frame-by-frame information visualization genre is dynamic. Thus, each individual comic-book-like frame might include a dynamic image whose orientation can be changed either manually or automatically. For example, the user/viewer could change the point of view of the scene shown in each frame while rotating it dynamically to offer a variety of perspectives on the screen. The pictures in each frame are built by the visualization designer using the familiar visual vocabulary used in comic books for the past 100 years. For example, to denote an object's physical activity, it could be shown with short spikes emanating from its sides, as in the depiction of the G-shaped building in the figure.
Information can be presented in many ways, or genres. The choice of genre, as well as the presentation medium, affects content, as well as what the audience gets from the process, following the Canadian literary and media scholar (and professor at the University of Toronto) Marshal McLuhan's well-known pre-PC-era insight: "The medium is the message," and American philosopher and educator John Dewey's comment: "We learn what we do." Getting information from a bulleted list taps the logical mode of the human mind; getting it from stories taps the human mind's creative and artistic mode. This is why author Stephen Denning in his book The Springboard advises people making presentations to start with stories rather than with slides [1]. The choice of the particular genre of presentationwhether narrative, written narrative, static, or dynamic visualcan have a positive, as well as negative, effect on the learning process for the audience.
Challenges
These examples of how to develop and use information visualization hint at the difficulty of having to present complex and massive amounts of information, especially in real time, even when using visual media. Effective presentations using the storytelling approach require skills like those familiar to movie directors, beyond a technical expert's knowledge of computer engineering and science. Creating a presentation is not just a matter of being literate in visual media and storytelling but depends on a frame of mind that caters to other modes of human information processing and thinking. Moreover, even someone deeply knowledgeable about computers and graphics cannot expect to become a storyteller overnight.
Information visualization is a creative process difficult to formalize. Technology in the form of computers has given us a dynamic new visual medium beyond paper. It unites film, television, radio, and the Internet, simplifies the production process, and is beginning to bring it vast quantities of data transformed into easily understood images to a mass commercial audience. These developments are extremely valuable to people interested in visual storytelling.
What else can technology do for storytelling? For one thing, some researchers are seeking to develop machines that write simple stories (see Roger Schank's Tell Me a Story [4]); only time will determine the benefits of this approach. Meanwhile, other recent advances have made it easier than ever to manually construct and display a story visually (once you know what you are doing). The various principles discussed earlier could be integrated into a system to assist users in converting facts into story-like presentations.
Implementation of the storytelling elements, such as the zooming options, could also be made available semiautomatically. This would be useful in command centers where preparation time is often constrained. Such a system could also monitor the creation of the story and suggest the use of appropriate elements (such as continuity). It could also check the presentation time allocated to each part of the story and comment if some parts occur too slowly or quickly (thus distracting and even confusing the audience).
Conclusion
Storytelling is an ancient art rooted in our common human culture, as well as in our physiology and psychology. (Bran Ferren, cochairman and chief creative officer of Applied Minds, Inc., describes storytelling as the world's second-oldest profession.) Technology provides us with new media and genres that can now be used to convey information in a story-like fashion. Still, we need to further understand the characteristic interactions of each genre with each particular audience, its advantages and disadvantages, and how it might affect content and learning. (John Seely Brown, chief scientist of Xerox Corp., describes the effort to set up the social conventions for a new genre as a slow process of negotiation between the developer and the public.) Anyone who needs to make someone else understand something can then choose the most appropriate genre and medium for the information, problem, and audience.
References
1. Denning, S. The Springboard: How Storytelling Ignites Action in Knowledge-Era Organizations. Butterworth-Heinemann, Boston, 2000.
2. Gershon, N. and Eick, S. Visualization's new tack: Making sense of information. IEEE Spect. 32, 11 (Nov. 1995), 3856.
3. Robertson, G., Card, S., and Mackinlay, J. Information visualization using 3D interactive animation. Commun. ACM 36, 4 (Apr. 1993), 5771.
4. Schank, R. Tell Me A Story. Northwestern University Press, Evanston, IL, 1990.
5. Thomas, J. IBM's Knowledge Socialization Project; see www.research. ibm.com/knowsoc/project_index.html
Footnotes
This work, which was inspired by discussions with Bran Ferren, is supported by the Command Post of the Future and Genoa programs at the U.S. Defense Advanced Research Projects Agency, Arlington, VA.
Figures
Figure 1. Jim's story fragment (left) and some of the information embedded in it (right).
Figure 2. Two-part script for the visual representation of information (sorted by location); visual operations are in red.
Figure 3. A comic-like representation (left) of the two-part script in
©2001 ACM 0002-0782/01/0800 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2001 ACM, Inc.