Communications of the ACM
The Digitization of Cultural Practices
Artist's View iPad app for exhibition "Americans in Florence. Sargent and the American Impressionists" 2012. Netribe, Bologna, Italy.
Credit: palazzostrozzi.org
Over the past few decades, computing researchers have developed a wealth of new systems to preserve and share cultural resources. Computing platforms give everyone access to high-resolution images of ancient cave paintings and Google artwork; 3D scanning systems digitize Michelangelo's sculptures and Pompeii's remains;7 and haptic devices allow users to touch and feel ancient sculptural forms.1 More recently, researchers have employed collaborative and crowd-sourced software in the restoration of specific artifacts and environments (such as paintings and archeological sites).9 These approaches tend to fall within one of three categories: digitally reconstructing objects and landscapes from the past; broadening access to cultural resources through remote distribution platforms; and digitally representing and archiving cultural artifacts and media (see Figure 1). In each, researchers position digitization techniques to support cultural preservation.
Key Insights
However, among these technological developments, researchers often miss a key element—cultural practices. Even as our cultural artifacts and media carry on digitally for future generations, our reasons for reading books, exploring contexts for understanding artwork, and developing ways to share and celebrate everyday practices remain relatively rare. And for good reason, as computing researchers find it difficult to identify, codify, and digitally record cultural practices. Cultural practices do not figure as squarely in the language of computation as many abstract and theoretical models. They emerge as forms of memory, in which the public remembers and forgets the past as a lived and constantly mutating collective experience. In this regard, researchers do not conserve cultural practices in the sense they freeze them in time by making them explicit. On the contrary, they preserve cultural practices by "enacting" tradition and experience. Digitization can thus enable and extend the work of preservation. In making this argument, we look at practices, after Schatzki,15 as arrays of human activity, temporally situated events involving rehearsed, materially mediated actions and embodied social relations. The United Nations Educational, Scientific, and Cultural Organization (http://en.unesco.org/) identifies practices as "very fragile by their very nature." While they bring to life some of our most celebrated cultural forms, they are easily overlooked by the public.2
Here, we consider the role computing techniques play in sustaining cultural practices. By tracing them in relation to developments in computing research, we see a need to revise our widely accepted digital preservation techniques. Inspired by a large body of literature from the digital humanities, we urge understanding the dynamic nature of cultural forms as an important strategy for preservation. However, this view does not fit within the dominant conceptual framework defined by computing researchers, a rubric grounded in the assumption that cultural histories are stabilized by the material forms on which they are encoded, stored, and replicated.
As an example of how culture might be preserved, consider the Artist's View iPad application,11 designed by Netribe of the Palazzo Strozzi in Florence, Italy, to familiarize museum visitors with the paintings in the 2012 exhibition "Americans in Florence: Sargent and the American Impressionists." Its designers took advantage of the museum's close proximity to many of the Florentine and Tuscan landscapes portrayed in the paintings by guiding visitors through the locations in which exhibiting artists painted or found their inspiration. When the visitors would reach any of the locations, the application would let them juxtapose the painting with the scene the painting represents. The application does more than reconstruct objects from the past by leveraging geolocation and augmented reality techniques so users imagine the spark behind a given artist's inspiration. Such applications give people a rich context with which to interpret and engage historic artifacts, providing access to aesthetic resources and uncanny moments hidden from view on a traditional museum floor. By extending the practice of visiting, paintings, books, and other historical artifacts could become part of a visitor's deeper museum experience.
The Palazzo Strozzi thus put in place an intriguing though limited interaction with cultural history. We observed the same ecological environments that once inspired the creators of Renaissance paintings, yet our view was shaped by the modern contexts in which they develop. Visitors today can still discover Frank Duveneck's tranquil rendering of the Ponte Vecchio bridge as they walk along contemporary streets, full of bustling commerce and international tourism. The painterly depictions of landscapes are at once drawn near and given depth. Situated in the same space at another time, they feel physically familiar but socially out of reach. Sometimes evocative, other times pedagogical, the software renderings share a specific (and partial) view of an artist's brushwork and composition.
Combining modern computer-vision techniques with historical objects, applications like Artist's View prompt questions like: What can computing do to facilitate our experience with the past? What are the limits of cultural engagement through such modern technological developments as gesture recognition, networking, and miniaturized displays? And how can computing techniques inform what it means to preserve cultural practices alongside artifacts? To answer, consider two projects that begin to extend and disrupt today's work of computational preservation.
Exploring Cultural Practices with Computation
One author, Roccetti, discovered his allergy to gluten when he was in his 40s, resulting in disconnection with the culinary aspect of his Italian heritage, traditionally linked to such food. Though not particularly unusual, Roccetti's allergy introduced a challenge, as he could no longer consume pasta, a staple of Italian culinary life. As a lifelong resident of Bologna, the gastronomically (and politically) vibrant city south of Venice, Roccetti sought to connect with a celebrated aspect of his past. Bolognese tortellini, the small belly-button-shaped dumplings traditionally prepared in chicken broth, had pervaded his childhood and that of generations of family members before him. Working with another author, Marfia, as well as with others at the University of Bologna, Roccetti viewed exploring tortellini as not only a delicacy to consume but as a cultural pastime worth preserving—the technique of producing pasta. Out of this interest came Tortellino X-perience, a game designed to teach people how to make the Northern Italian tortellini (see Figure 2).13 Designed for the 2010 International Expo of Shanghai, the system uses a webcam and simple gesture-recognition techniques to enable Expo visitors, as well as users today, to learn and repeat the basic methods required to make tortellini. Projecting an interactive video above an empty countertop, the system prompts them to repeat a set of prespecified actions in preparation of a tortellini recipe, parsing the actions in the form of a game. Particular actions are required during each preparation phase and shown to the player. When the player reproduces the actions in the video, a set of vision algorithms compares the player's gestures against a set of encoded demonstrated actions. Finally, the system presents the subsequent step in the tortellini-making process.
In practice, the system leads users through a step-by-step process to perform all actions necessary to prepare tortellini. In the first step, users pour flour, eggs, and water onto the center of the countertop. In the second, they mix together all the ingredients, making circular gestures, until a ball of dough takes shape. They perform the third step with a rolling pin, rolling it forward and backward until a thin layer of dough takes shape. In the fourth step, they cut the thin foil of dough into squares with a knife. The fifth step involves stuffing the resulting squares, and the sixth and seventh folding the stuffed squares of dough, finally ready to be cooked. Et voilà, virtual tortellini are ready for virtual consumption.
Thousands of miles west of Bologna, the third author, Rosner, had been designing interactive media for the Adler Planetarium and Astronomy Museum in Chicago. In collaboration with educators and astronomers, she used Adobe Macromedia software to develop interactive games for exploring astrophysical data. The software was made available to museum visitors through kiosks throughout the museum's exhibit halls. Producing and maintaining the software—and resulting science narratives—involved installing frequent software updates, fixing broken hardware, and monitoring networking errors. This design work was unlike the creative activities (such as knitting) she had practiced since childhood. Knitting, as a process of interlocking loops of yarn with two needles, presented tangible histories of production by indexing her handiwork in the stitches. While Rosner worked at the Adler, a growing number of people was rediscovering or newly embracing the activity of knitting in which they discovered disruptive politico-spatial practices (such as "yarn bombing"), or applying old knitted garments like scarves and sweaters to public infrastructure (such as lamp poles, fire hydrants, and trees). She also discovered how to add digital enhancements to knitting; knitters regularly use online videos and how-to instructions to learn new techniques, connecting with other knitters on craft-related social-networking sites and posting updates on knitting blogs. Rosner was struck by the qualitative difference between the vibrant social relationships emerging around tools for knitting and the interactive digital displays she had designed for the museum environment. Knitting seemed to embody cultural practice that contrasted with conventional digital content presentation and codification tools. Knitting emphasized the means of production alongside the artifacts being produced.
With knitting as a focus, Rosner began work with Kimiko Ryokai of the University of California, Berkeley, to develop Spyn14 a system that enables association of digital records with locations on hand-knit fabric (see Figure 3). The system evolved over three years (2009–2011) from an Adobe Flex application running on a Mobile PC to an Android application running on a T-Mobile G1 smartphone. As they knit, knitters could mark a position on fabric and link digital media (such as images, text, and video) to the same location on the garment. By pointing a smartphone's camera at a desired position, they could later retrieve the associated information. The original system used infrared ink patterns printed on yarn to associate locations on fabric with the collected media. The pattern could be recognized through a simple visual edge-detection algorithm and used as an index, or virtual link, to retrieve the digital content (such as a homemade video or musical sounds) linked to the position. The system still allows knitters to store associations between digital media and fabric through their smartphones.
Tortellino X-perience and Spyn enable people to enact traditional experiences by creating new cultural resources. Tortellino X-perience engages tortellini making by teaching embodied techniques, tacit knowledge, and presuppositions that could, in turn, impart and spur interest in Bolognese culture. Spyn extends opportunities for social annotation and storytelling around the highly feminized handicraft of knitting by allowing particular social relations to emerge.
Taking a step back, both projects inform how we think about the digitization of cultural practices. The kinds of information that become most relevant, useful, and evocative when preparing tortellini or knitting sweaters differed among the people using these tools. Some visitors to the Tortellino X-perience exhibit found the sequence of actions depicted on screen a compelling activity in learning to cook. Others narrowed their focus on the resemblance between their own gestures and those shown on the screen, gaining social value from performing similar gestures. With Spyn, the volume of recorded media associated with the knitted fabric being created overwhelmed some recipients. The pressure for reciprocity could intensify to the point of discomfort, as when one user referred to the Spyn knit as "emotional blackmail." Other times the digital records presented a sense of timelessness, or, as another user called it, a "4D object." The system shifted what knitting could accomplish, enabling the recipient of the garment to access stories of its making, sometimes changing the meaning of the knitted item (such as by turning a pair of gloves into a travel journal, a hat into a mix tape, or a vest into a puzzle). Such varied responses illustrate the fickle nature of the digital, or how social environments in which cultural practices are situated inform and transform how digital information is interpreted and disclosed over time.
These projects also highlight new opportunities for sustaining the kinds of data we might want to preserve. Tortellini apprentices may not care about the precision of gesture replication, as gestures are often readjusted and reoriented while working. Knitters may not care how many stitches it took to make a particular hat or what they were thinking when making it, since the hat may already embody some shared value or ideal. In this sense, the data collected through cameras or other sensors has little value in isolation. Rather, the patterns of social activity that unfold as part of engaging the devices—rhythms of interaction expressed through annotated stitches or moments of reflection viewing video sequences—may reveal more about the history and collective memory of a craft practice than would a gesture alone. In enabling people to extend the work of their hands through digital media, the system simultaneously invites the conservation and symbolic transformation of a given practice.
Technological Paradox
We thus reach a paradox of preservation: Things change in order to remain the same. As we aim to sustain cultural practices, enabling the "replication" of techniques through digitization, we simultaneously change the replicated forms. Roccetti and Marfia designed Tortellino X-perience to emphasize the technical process, guiding the actions leading to traditional Italian dumplings, yet often neglected the cultural resources and communities of practice in which tortellini making developed. Further, the system separated the means of production from the embodied and tactile engagement particular to its tools—rolling pin, knife, and spoon—and ingredients—flour, eggs, salt, and water. Rosner designed Spyn to focus on the social relations tightly bound with fragments of fabric, perhaps a knitter associating the music listened to while knitting with stitches on her mother's scarf. However, Spyn is not designed to track the pattern or gestures with which the stitches were made. Each system was built to sustain critical modes of cultural engagement but also left some out. Practices of preservation inevitably come with transformation.
Beyond limitations of codification, cultural practices rely on digital infrastructures consisting of hardware and software that could ultimately wear out. As a result, cultural practices not only change but deteriorate as well; for example, with Spyn, Android phones wear out, as the software used to store the data becomes incompatible with subsequent versions of the Android operating system, and users are no longer able to access the data. Preservation yields disruption.
Knitters using Spyn choose the kinds of stories they want to tell through digital media collected while knitting and control the degree to which the fabric is legible to vision algorithms.
From cave painting to cataloging, humans have always built cultural records of ritual activity and embodied knowledge. By giving their practices material form, they transform performance into imagery, text, and audio to hold onto some aspects of their tacit experience while discarding others. A viewer may not fully grasp the oration reflected in an heirloom painting of a sermon, and the painting may not exist for the next generation if not stored properly. In this sense, the question of what to preserve (and how) is nothing new. Yet with digitization comes something else. Educational institutions can embrace ways of learning and interacting not previously available. As with Tortellino X-perience and Spyn, users perform an activity not only by following the gestures shown in a video but by integrating what they learn from the video with additional practices (such as digital annotation through computer vision). Linking digital forms and patterns, these added resources in turn shift the very purpose of the practice itself; for example, with Spyn, a knitted hat becomes a mixtape. Preservation is not the work of encoding our lived experience but rather enabling memory practices, the modes of performance through which users reimagine the past.
What Next?
A wealth of digitization projects aim to support the capture, storage, and representation of material resources, including books, archeological sites, and works of art;5 for example, Levoy et al.'s Google Books project6 designed techniques for scanning fragile texts, making it possible to capture, parse, and store pages from antiquarian books without having to open the books completely. Opening and flattening antiquarian volumes, as with conventional scanners, endangers bookbindings, or glued or sewn bands holding together a book's pages. Injuring pages, in turn, disrupts the process of digitization, as it could degrade what is conventionally viewed as the original artifact, the physical book. However, when establishing a book's provenance depends on the edition of a volume or to whom it belonged, how should scanning be applied? Is the scanner able to capture the mottling on the page margins or evidence of dog-eared pages? Google Books offers a digitization method that applies advanced imaging techniques to digital archiving even as it prompts questions as to what exactly researchers aim to preserve.
In a second project, Proudfoot and Levoy12 developed what they call a "3D computer archive" of Renaissance artist Michelangelo's sculptures by scanning the original forms as they found them in museums. They sought to create a lasting archive of the artworks by stitching together data from multiple sources, including a planar light field scanner, handheld lightfield scanner, and low-res models for planning purposes. In the process, they accommodated variable ambient light, filling holes through space-carving techniques and aligning scans from multiple gantry angles and positions. This archive increased access to the statues, as well as public availability of the artifacts, even as they displaced their cultural context.
Computing researchers view digitization in these settings as computer science and engineering. Engineers built the scanners and use them to scan and archive physical artifacts. Museum attendees and others using the scans have little influence on what books are available to them and how they were preserved in digital form. While an engineer could decide not to scan the first edition of Miguel de Cervantes Saavedra's Don Quixote, believing the fifth and sixth editions were sufficient, a scholar of Latin literature might require the first edition to contextualize 17th century interpretations. In Tortellino X-perience and Spyn, not only engineers but also craftspeople and others—Italian chefs, pasta makers, and knitters—interested in producing the actual items take up the work of digitization. Users of Tortellino X-perience shape the vision algorithm's understanding of events and subsequent sequence of actions shown on a video screen. Knitters using Spyn choose the kinds of stories they want to tell through digital media collected while knitting and control the degree to which the fabric is legible to vision algorithms. By creating, using, and revisiting digitized files, contemporary pasta makers and knitters take part in the digitization and, in turn, the preservation of cultural practices.
Conclusion
Our study of Tortellino X-perience and Spyn find people use computing resources to connect their current experience to past events, comparing their own unique gestures to the delicate pinch of the tortellini dough rendered by a digital display to remember a given technique. Likewise, they use media associated with knitted fabric and garments to recall lost stitches at particular moments. Even though the revisited pinching and knitting techniques will never be exactly as originally performed, the physical dough pinching and needle manipulation help users construct memories and recognize particular practices as part of a cultural tradition. Preserving cultural practices involves moving beyond digital historiography and its institutionalized requirements to explicit evidence, documentation, authenticity, and provenance. Viewing cultural practices through the lens of memory practices, researchers and practitioners find them "preserved" through histories in constant flux, intertwined with collective memory.
Exploring computing technologies through the lens of cultural practices reveals the complex nature of digitization and degradation, enabling us to comment on the status of the ephemeral, how when we trace the ephemeral, the ephemeral changes the behavior we hope to trace. Modern technologies are able to register and reify the practices central to our cultural heritage but also reconfigure them.4 We thus face the challenge of understanding the kinds of transformations made possible through digitization and the physical craft-making patterns due to cultural preservation.
Acknowledgment
The authors would like to thank the Italian ALTruism Enabling Resource Network for supporting this work.
References
1. Bergamasco, M., Avizzano, C., Di Pietro, G., Barbagli, F., and Frisoli, A. The museum of pure form: System architecture, robot and human interactive communication. In Proceedings of the 10th IEEE International Workshop on Robot and Human Interactive Communication (Paris, Sept. 18–21). IEEE Press, Piscataway, NJ, 112–117.
2. Bouchenaki, M. The interdependency of the tangible and intangible cultural heritage. In Proceedings of the 14th ICOMOS General Assembly and International Symposium (Victoria Falls, Zimbabwe, Oct. 27–31). ICOMOS, Paris, 2003, 1–5.
3. Brown, J.S. and Duguid P. Borderline issues: Social and material aspects of design. Human-Computer Interaction 9, 1 (1994), 3–36.
4. Ferretti, S., Furini, M., Palazzi, C.E., Roccetti, M., and Salomoni, P. WWW recycling for a better world. Commun. ACM 53, 4 (Apr. 2010), 139–143.
5. Kallinikos, J., Aaltonen, A., and Marton, A. A theory of digital objects. First Monday 15, 6 (2010); http://pear.accc.uic.edu/ojs/index.php/fm/article/view/3033/2564
6. Levoy, M., O'Sullivan, J.K., and Uhlik, C.R. Acquiring and Using Three-Dimensional Information in a Document Scanning System. U.S. Patent No. 7,586,655. U.S. Patent and Trademark Office, Washington, D.C., 2009; http://assignments.uspto.gov/assignments/q?db=pat&pat=7586655
7. Levoy, M., Pulli, K., Curless, B., Rusinkiewicz, S., Koller, D., Pereira, L., Ginzton, M., Anderson, S., Davis, J., Ginsberg, J., Shade, J. and Fulk, D. The digital Michelangelo project: 3D scanning of large statues. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (New Orleans, July 23–28). ACM Press, New York, 2000, 131–144.
8. Mayer-Schonberger, V. Delete: The Virtue of Forgetting in the Digital Age. Princeton University Press, Princeton, NJ, 2011.
9. Mazzeo, R., Palazzi, C.E., Roccetti, M., and Sciutto, G. Computer-assisted pigment identification in artworks. In Proceedings of the Third European Conference on Internet and Multimedia Systems and Applications (Chamonix, France, Mar. 14–16). ACTA Press, Anaheim, CA, 2007, 266–271.
10. McIntosh, A. and Prentice, R.C. Affirming authenticity: Consuming cultural heritage. Annals of Tourism Research 26, 3 (July 1999), 589–612.
11. Netribe. Artist's View iPad app. Bologna, Italy; https://itunes.apple.com/mg/app/artists-view/id504132267?mt=8
12. Proudfoot, R.A. and Levoy, M. Systems and Methods for Glare Removal Using Polarized Filtering in Document Scanning. U.S. Patent No. 8,174,739. U.S. Patent and Trademark Office, Washington, D.C., May 8, 2012; http://assignments.uspto.gov/assignments/q?db=pat&pat=7561312
13. Roccetti, M., Marfia, G., and Zanichelli, M. The art and craft of making the tortellino: Playing with a digital gesture recognizer for preparing pasta culinary recipes. Computers In Entertainment 8, 4 (Dec. 2010), 28.
14. Rosner, D.K. and Ryokai, K. Spyn: Augmenting knitting to support storytelling and reflection. In Proceedings of the 10th International Conference on Ubiquitous Computing (Seoul, Sept. 21–24). ACM Press, New York, 2008, 340–349.
15. Schatzki, T.R. A primer on practices: Theory and research. Chapter in Practice-Based Education: Perspective and Strategies, J. Higgs, R. Barnett, S. Billett, and M. Hutchings, Eds. Sense Publishers, Rotterdam, 2012, 13–26.
Figures
Figure 1. Systems designed to sustain cultural resources: (a) digitally reconstructing objects and landscapes from the past; (b) broadening access to cultural resources through remote distribution platforms; and (c) digitally representing and archiving cultural artifacts and media.
Figure 2. Tortellino X-perience; storyboard steps.
Figure 3. Spyn mobile phone software associates digital records in the creative process—captured through audiovisual media, text, and geographic data—with physical locations on handmade fabric; credit: James Jordan, California Magazine.
Figure. From Artist's View iPad app for exhibition "Americans in Florence. Sargent and the American Impressionists" 2012. Netribe, Bologna, Italy.
©2014 ACM 0001-0782/14/06
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and full citation on the first page. Copyright for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or fee. Request permission to publish from [email protected] or fax (212) 869-0481.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2014 ACM, Inc.