Communications of the ACM
AI Deepnets Throw Light on Ancient History
Charred scrolls from Herculaneum can’t be opened easily, but X-ray scanning can reveal their contents, and a machine-learning algorithm has detected Greek letters on several lines of this rolled-up papyrus.
The pace of innovation in artificial intelligence (AI) often seems so frenetic that it feels like we are being thrown headlong into the future almost on a daily basis, with ever-smarter AI models arriving to enhance drug discovery, medical diagnostics, speech recognition, and generative capabilities of all kinds.
But here's the thing: it turns out that the future is far from the whole story for AI, as the deep learning technology at the heart of it is now also throwing a surprisingly bright spotlight on many aspects of the past. The deep past, in some cases.
Specifically, historians of many stripes are now collaborating with computer scientists on a number of AI-based projects designed to extract lost or hidden knowledge from a swathe of ancient artifacts or artworks. These include the following history projects (which we'll then look at in greater detail) in which deep learning is being used:
- to tease out previously unknown facts about aspects of art history and the paintings, drawings, and artists of the past;
- to restore the lost text of inscriptions from a host of ancient Greek artifacts, and attribute source location and time of creation to each, and
- to read the text buried deep inside scorched, unopenable scrolls burned in the volcanic eruption that destroyed Pompeii.
Such efforts are being made possible thanks to the ability of biologically inspired deep neural networks to perform intricate feats of pattern recognition, on scales beyond any human ability. By training models on vast volumes of historical artifacts, such as inscribed stone and metalwork, ancient documents, and paintings and drawings, computer-assisted historians are able to make new discoveries.
Deep learning's art attack
Such historians include David Stork, an adjunct professor at Stanford University and an author who has pioneered the application of machine learning and computer vision to art history analysis. Writing in the journal Nature in October 2023, Stork said he believes applying AI methods at scale—that is, training models on many thousands of pieces of artwork for algorithms to draw conclusions from—will do for art scholars what the microscope has done for biologists, and the telescope for astronomers.
As an example, in a paper presented at last January's Electronic Imaging Symposium in San Francisco, Stork, working with Stanford student Jean-Peïc Chou, trained a deep learning model on 11,000 digital images of portrait paintings dating from the early Renaissance to the Modern era.
They then tracked the head pose of the subjects in those portraits, using computer vision algorithms, and discovered that painters of the "primitive" and "naïve" schools favored "a highly restricted range of pose angles, primarily frontal," while other types of artists, such as the expressionists, "employed a far greater range of angles." Specialist findings, yes, but this kind of information is meat and potatoes to art scholars.
An unexpected result emerged, too, Stork says: "Our method was so sensitive, we could find left-handed self-portraitists within our large database of portraits. Just by the nature of an artist's handedness, and the optics of mirrors, right-handed and left-handed artists differ in the orientation they appear within self portraits. We weren't looking for this in our research, but found it."
And in a paper to be published in the May 2024 edition of Communications, Stork will reveal findings on "using multi-modal deep networks" to determine the meaning conveyed by artworks. "This requires a level of analysis much deeper than that currently dominating A.I. research," he says.
Stork says that being able to train models on huge tranches of artworks humans could never cope with is the key here. "Our current work performs the kind of tasks traditional art scholars perform, but more accurately, and at a scale thoroughly impractical 'by eye'." He adds that his latest research involves training models with more than 20,000 paintings.
What else are they pointing their arty deepnets at? Says Stork, "We have worked on trends in color palettes throughout certain art periods, and through the careers of individual artists. And one student, now working at DeepMind in London, did a nice analysis of the trends in the colors of van Gogh."
Ancient inscription resurrection
As luck would have it, it is in fact DeepMind that developed the next historical machine intelligence we'll look at: Ithaca. Available to historians through an interactive Web interface, Ithaca is a deep neural network whose aim is to restore missing text in damaged or worn fragmentary ancient Greek inscriptions on stone, pottery, and metal artifacts. On top of that, it can also make a good stab at "attribution"; that is, working out where it came from and when it was written.
Launched in March 2022, Ithaca is a collaboration of scholars at the Università Ca' Foscari of Venice, the University of Oxford, and the Athens University of Economics and Business. The idea was hatched in 2017 after Yannis Assael, a Google DeepMind staff research scientist, and historian Thea Sommerschield of Ca' Foscari, brainstormed a need for it. "We delved into the most daunting challenges faced by historians and it was through these discussions that we recognized the immense potential for cooperation between artificial intelligence and historical research. The collaboration presented a unique opportunity to cast new light on our past," Assael says.
"Specifically, our work focuses on three critical tasks: the textual restoration of ancient documents, the accurate geographical attribution of historical texts, and the precise chronological attribution of these texts."
To do all this, DeepMind set out to train Ithaca using 178,551 known ancient Greek inscriptions known as the PHI dataset, after the Packard Humanities Institute which compiled it. Crucially, for attribution, each PHI inscription had to have metadata related to locations (from 84 ancient regions) and a reliable way of determining the time of its writing. After filtering to ensure it was only using robust, actionable data, Ithaca was trained on 78,608 inscriptions.
Remarkably, perhaps, Ithaca has been a runaway success. "The expert historians we worked with achieved 25% accuracy when working alone to restore ancient texts. But, when using Ithaca, their performance increases to 72%," the DeepMind development team wrote on their launch blog in March 2022.
Fast-forwarding to today, and it has gone from strength to strength: the freely available interface attracts over 300 weekly queries from researchers, Assael told Communications in December 2023, and research papers are now appearing based on its findings. On top of that, he says, educational institutions are integrating the model in their work, with, for example, colleges in Ghent, Belgium, developing an A.I.-assisted history curriculum based on Ithaca.
While DeepMind's task is tough, at least the users of Ithaca can see the text that needs repairing. The next team we'll hear about cannot even open their target documents, thanks to a volcano.
A purple patch for papyrus.
The last of our deepnet-based forays into history first hit paydirt in October 2023, when computer scientists Brent Seales and Stephen Parsons at the University of Kentucky, in Lexington, revealed the first word ever read from an unopened papyrus scroll that was seemingly unreadable after being scorched when Mount Vesuvius erupted in AD 79.
That infamous eruption buried not only Pompeii but also Herculaneum, another Roman town on the Bay of Naples, in 20 meters of searingly-hot volcanic ash. In the 18th century, site excavators found 800 charred, carbonized papyrus scrolls there, in a library. Most have remained unopened, due to their extreme fragility, and they are stored at the Institut de France in Paris.
However, because attempts had been made to open some of the scrolls, leaving ink on some fragments, Seales and Parsons had some recovered two-dimensional surface text (which is most visible in the infrared) to use as initial ground truth to help train a machine learning model. In other words, they knew what kind of text they were looking for.
To find out what was written in ink in the hidden depths of two still-rolled-up scrolls, the Kentucky team had them both scanned using computed tomography, or CT (that is, X-rayed in slices along their length – see video) at 4-micrometre resolution at the Diamond Light Source, a synchrotron in Oxford, UK.
"The neural network for ink detection is a convolutional neural network, but as far as neural networks go, especially these days, it has a relatively simple architecture. Our main contribution is not the architecture itself, but the alignment of infrared photos [of ink on detached fragments] with CT images that creates the training dataset," says Parsons.
To encourage computer scientists to improve the text search routine by modifying and perfecting the model, to cope with the different resolutions of CT scans and images from scroll fragments, Seales established a $1m (total) cash-prize competition called the Vesuvius Challenge, backed by Silicon Valley prize donors.
And it began coming up trumps in October 2023: working separately, competitors Casey Handmer, a former JPL engineer, Luke Farritor, a SpaceX intern from Nebraska, and Yousef Nader, a Berlin-based bioroboticist, worked towards finding text in the scrolls, with Farritor narrowly coming first, and winning $40,000, and Nader and Handmer winning $10,000 each. The word Farritor found was 'porphyras', which is ancient Greek for purple.
This landmark first finding showed the value of a tech competition, says Parsons. "The winning competitors used machine learning models built using the same fundamental framework as we did, but they found modifications that enabled the models to work inside the rolled scrolls, which was very exciting."
"Primarily, this came in the form of creating training data from within the rolled scrolls. One way to do this, one competitor found, was to inspect the scroll data very closely by eye, finding a handful of places where he could actually see ink on the surface himself from within the scroll. So he labeled those regions, and used them as training data. The model then was able to detect more ink, revealing writing."
As competitors search for more words, and the Kentucky team focuses on improving their scanning, where else does Parsons think the technique could be used? "There are many other collections these methods could work for. But for now, Herculaneum remains my North Star. It's just the perfect combination of historically significant and technically challenging work."
Paul Marks is a technology journalist, writer, and editor based in London, U.K.
No entries found