Communications of the ACM
Researchers Add Emotional Context to Electronic Communication
iStockPhoto.com
Information and communications technology (ICT) is all about communication but it has never provided the semantic richness offered by the non-verbal cues that typically pepper face-to-face conversation. That is about to change. European researchers believe they have developed a system that could finally add reliable context and mood information to voice and text communication.
The EU-funded project is developing Psychologically Augmented Social Interaction Over Networks, or PASION for short. The team boasts among its partners the best and the brightest in high-tech approaches to non-verbal communication. It also includes telecoms heavyweight, Telecom Italia, which may be able to incorporate some of the innovations of the PASION research into real-world projects.
Communicating with PASION
Modern information and communication technology provides a vast array of channels for all types of communication: Web pages, SMS, email, twitter, and the list goes on and on.
But despite the breadth of our communication channels, what we say still lacks the depth of face-to-face communication. That is because, so far, ICT has failed to provide the non-verbal cues and context information that is so important to the way we communicate.
When we talk we use non-verbal cues like facial expressions or physical movements. We know if the people we are talking to are excited, busy, relaxed, tired, bored, etc. In electronic communication this sort of information is missing.
Surfers have tried to fill the gap with emoticons, smiley faces or other cues that hint at the intent of the writer around the words themselves. These cues explain such subtle qualities as sarcasm, teasing, anger or irony – vital contextual information for successful communication.
It is a workaround, but it hardly approaches the depth of communication we have when face to face with someone.
Researchers working on the PASION project hope to change that. They are using a large number of approaches and have tested a range of technologies. One partner looked at cues for proximity and localization, while another looked at social feedback and a third examined physiology and emotion among online poker players.
Other research looked at different ways to assess mood or context, investigating facial expression analysis as well as physiological sensors that provide real-time feedback on users’ physiology.
Telecom Italia, the coordinating partner, built the service-oriented architecture needed to support real-life applications.
Another partner, the Helsinki School of Economics, built tools to augment voice, text and instant messaging with non-verbal information for use by knowledge workers. While the University of Lincoln developed Familiars, an online social game that incorporates facial expression analysis, psycho-physiological data and social indicators based on user interactions.
All this work was topped by PASION field trials, involving 200 users in four European countries. Users were very interested in augmentation and were particularly keen on tools that provided information on other users’ mood, activity and availability. The project is now planning a new series of trials based on the user feedback.
How it works
All these tools are combined in different ways, for different scenarios, to approach a new language for non-verbal communication. "There are really two areas we are dealing with here: work and friends," explains Richard Walker, chief of dissemination at the PASION project.
"And both of those can be divided again into two broad categories, synchronous communication like a telephone call, and asynchronous communication through text, email or instant messaging," notes Walker.
PASION has developed approaches to deal with all these different kinds of communication. Some of the non-verbal cues are provided by traditional Internet practices like the use of emoticons and the smiley faces. But unlike traditional icons, the PASION system uses an independent system to identify mood.
"For example, we could have biosensors monitoring heart rate, pulse, body temperature and so on to establish the state of arousal of you correspondent: are they bored or interested? PASION can tell you, objectively," Walker reveals.
Popular with younger surfers
Other tools can be used to identify the facial expression of a person, or to track the dynamic of a group conference. "Are there some people dominating, while others give no input? Tracking tools could help conference moderators to invite comment from under-represented participants," says Walker.
Still other tools can map the relationships in a group, showing who has contact with all the others, and who is relatively isolated.
There are a vast range of potential applications for PASION, in both mass and niche markets, and in vertical and horizontal applications, and the team has just begun to explore the technology's potential.
"How this will be used we can't say just yet, but we can say there are many applications, and while older people are perhaps a little bit hesitant about revealing their state of mind, young people and people who use social networking sites are very keen," Walker relates.
Nonetheless, all the systems and tools developed by PASION have incorporated robust privacy systems to ensure participants control what is revealed about them.
The PASION project received funding from the Future and Emerging Technologies (FET) strand of the Sixth Framework Program for research.
This is the first of a two-part special feature on the PASION project.
From ICT Results
No entries found