Communications of the ACM
Cerf's up
Multisensory Adventures
In this column, I want to draw your attention to two books. One has been published to great acclaim and the other is still in process. They resonate with a visceral intensity for which I was honestly unprepared and surprised. The first, Multisensory Experiences, Where the Senses Meet Technology, by Carlos Velasco and Marianna Obrist, is to be published by Oxford University Press. The authors explore concepts we experience every day but don't necessarily understand fully. We are familiar with the five senses (sight, sound, touch, taste, smell). Our brains transduce these physical phenomena into neural pulses that flood along many pathways and interact in many ways. Interestingly, all these senses are translated into essentially similar neural signals but they are processed in a complex and interconnected neural web producing what we call experience.
Sensory memories are powerful. When I smell cigar smoke, I am transported back to my early childhood with my grandfather and mother during World War II. The olfactory memory recalls sights, sounds, scenes, and other sensory phenomena. Our senses are linked, not only in real time but also in recall.
We are learning more about our senses as our ability to measure phenomena improves. Taste is more than sweet, sour, bitter, salty, umami, and metallic since it is also heavily combined by our brains with smell. Most of the taste of wine is olfactory, for example. If reincarnation is real, I hope I am reincarnated as a wine-tasting dog! We are also learning there are neural circuits in the brain that literally function as spatial maps allowing us to navigate to places we have been before. It is remarkable how quickly these maps can be constructed.
This book posits that it is possible to deliberately create multisensory experiences in the form of art and to explore how experience can be altered with subtle changes in sensory ambience. A scene can go from cheerful and uplifting to scary and threatening depending on the choices made for background sound. For me, this illustrates viscerally, how experience is influenced by multimedia inputs.
Understanding and appreciating multisensory experience can come in many forms. The famed chef José Andrés invented new restaurants where especially talented chefs produce tiny, bite-sized tapas that produce unexpected multisensory effects. One dish is made with liquid nitrogen (I kid you not!) and after you pop this into your mouth, your outgoing nasal breath makes you look like a dragon. Another looks like a little palm frond beach hut, but it is actually a deconstructed Caesar salad. Your eyes tell you one thing but you mouth and nose tell you otherwise. You will find many examples of these kinds of artificially generated multi-sensory experiences accounted for in the book. I think you will find it an eye-opening experience to read what these authors have to say about the conflation of our senses produced by high-dimensional synthetic effects. There is nothing simple about our senses and this book explains why.
We are learning there are neural circuits in the brain that literally function as spatial maps allowing us to navigate to places we have been before.
The second book I call to your attention is the 2015 Pulitzer-prize winning All the Light We Cannot See, by Anthony Doerr. By good fortune, I read this book after reading Multisensory Experiences. Marie-Laure is blind and her father is teaching her to navigate the town they live in. It is difficult, but eventually success comes. Now experience the multisensory ensemble she uses to navigate home:
"... one snowy Tuesday in March she squats on her heels on the sidewalk. The faintly metallic smell of the falling snow surrounds her. Listen. Cars splash along streets, and snowmelt drums through runnels; she can hear snowflakes tick and patter through the trees. She can smell the cedars in the Jardin des Plantes a quarter mile away. Here the Metro hurtles beneath the sidewalk: that's the Quai Saint-Bernard. Here the sky opens up, and she hears the clacking of branches: that's the narrow stripe of gardens behind the Gallery of Paleontology... They walk up their street now, she is sure of it. They are outside their building. Marie-Laure finds the trunk of the chestnut tree that grows past her fourth-floor window, its bark beneath her fingers. Old friend..."1
I strongly recommend both books. May your senses combine in collaborative celebration!
The Digital Library is published by the Association for Computing Machinery. Copyright © 2020 ACM, Inc.
No entries found