Communications of the ACM
Software Annotates Images For Semantic Search
The IMAGINATION application adds 'imagenotions' or annotations to images automatically to prepare them for a semantic search.
Credit: ImageNotion
Innovative software developed in Europe that makes it easier to organize, search and navigate collections of digital images will soon be available to media agencies, photographers and, potentially, anyone trying to keep up with photo-happy Facebook or Flickr friends. The ImageNotion software, which is expected to go on sale next year, takes a user-friendly approach to semantic image annotation and search, a technology that links the content of photos to concepts so as to make the images understandable by computers.
Such systems have typically required end users to use a manually developed ontology — a lexicon of predefined concepts used to assign machine-readable semantic meaning to information — and then train the software to correctly annotate different images. For example, an apple would need to be defined in an ontology for fruit and then photos of apple trees could be tagged as such.
The ImageNotion system strips away much of that complexity for the end user, combining semantic annotation with a variety of other technologies, from text mining and object recognition to face detection and face identification, in order to permit many more images to be accurately annotated with little or no user intervention.
"When you mention ontologies to most people they just switch off. A photographer, an image agency employee or a Web user doesn't want and shouldn't have to learn how the technology works, they just want to be able to use it," explains Gabor Nagypal, who oversaw development of the ImageNotion software as technical and scientific coordinator of the EU-funded IMAGINATION project. "Because of that, our goal has been to make the technology transparent and intuitive to use," he adds.
The incorporation of other technologies, such as face detection and recognition, means that considerably less has to be done by the user to ensure photos are tagged correctly. The face detection system, which identifies the presence of people in a photograph, functions with around 80 percent accuracy, Nagypal says.
Meanwhile, face recognition, which seeks to identify who those people are — a trickier technology to perfect — starts to produce consistent results after five to ten images of a person have been tagged.
Saving Time and Money
"In a real-world environment, the accuracy will differ depending on a variety of factors, although for a professional image agency or photographer dealing with thousands of photos, any reduction in the amount of manual annotation needed makes a huge difference in terms of time and cost savings," Nagypal, a developer at German software company disy, explains.
Another key innovation is that the creation of ontology elements, which the IMAGINATION team refers to as "imagenotions," is an integral part of the software, instead of being dealt with as a separate process. Imagenotions can, therefore, easily be changed and updated by end users at any time.
"If they realize that a notion does not exist for a particular person or object, they can easily add it to the system. For example, the system would automatically identify the first photo it comes across of Angela Merkel as a person and a woman. The user could then define her profession as German chancellor and state that the photo was taken at a certain EU meeting. Those terms could then, with a few clicks, be used to pull up a list of other photos from the meeting and connect to information about other people who were present," Nagypal explains.
In the case of some photos, such as already archived images, text mining software extracts much of that information from the caption and draws connections with the output of the face detection and recognition systems automatically.
Navigating between images is simplified by a system of embedded links, so that clicking on the face of Chancellor Merkel will take the user directly to a display of other images of her.
The software is currently being tested by the French photo agency Photo12 and the Italian State Library of Modern and Contemporary History, both IMAGINATION project partners, as well as by several German photographers. A Web-based beta version is also available for testing on the Web site of ImageNotion Ltd., a spinoff company set up by some project members to start commercializing the software.
"We are currently refining the software, but we hope to be ready to start selling it as a stand-alone or Web-based product next year. First, we will target professional users, primarily photo agencies, archives and libraries who need a quicker and easier way to annotate and find images, then we will look to provide it as a software service for individual photographers and, finally, as an embedded application, which could be used on other Web sites such as Facebook or Flickr," Nagypal explains.
"We are looking for investors and partners to help us get it off the ground," he adds.
IMAGINATION received funding from the ICT strand of the EU's Sixth Framework Program for research.
View video tutorials of the ImageNotion application in action.
From ICT Results
No entries found