Communications of the ACM
When Computers Are The Hackers
David Brumley and other members of winning team ForAllSecure, preparing for the U.S. Defense Advanced Research Projects Agency's Cyber Grand Challenge.
Credti: Tom Geller
With neon-lit showcraft and nearly $4 million dollars in cash prizes, the first Cyber Grand Challenge opened a new frontier in cyber defense as intelligent systems, not humans, competed in a hack-and-defend, capture the flag-style contest. Seven teams from a field of over 100 qualified to participate, each preparing a cyber reasoning system (CRS) for two years before the final battle on Aug. 4.
The event was the latest in a series of "Grand Challenges" issued occasionally since 2004 by the U.S. Defense Advanced Research Projects Agency (DARPA). Previous Challenges involved physical robots and driverless cars; the next will apply machine intelligence to radio spectrum allocation. With a budget of $55 million, this iteration of the Cyber Grand Challenge was the first focused on computer security.
The project came to a head on a Paris Las Vegas conference center stage in front of thousands of attendees and online viewers. The all-day tournament opened to the public at 3:30 p.m., attracting mostly attendees of DEF CON, the annual hacker conference that opened the same day. Astrophysicist Hakeem Oluseyi and hacker "Visi" provided running commentary for the 96-round game from 5 p.m. until its conclusion at 8 p.m., with each team's brightly-lit, DARPA-supplied supercomputer running on a stage within an "air gap" to prevent tampering. (A Blu-ray disc-changing robot was the only data link from within the gap to the outside world.) Their human trainers bit their nails and watched the results in a team lounge next to the stage, helpless to intervene.
Enter the Coliseum
Team size ranged from two to over 20 people, and came from both industry and academia. They were:
- CSDS, a two-person team whose CRS "Jima" was sponsored by the Center for Secure and Dependable Systems (CSDS) at the University of Idaho.
- Deep Red, comprised of engineers from defense contractor Raytheon, with a CRS named "Rubeus."
- disekt, from the University of Georgia, running "CRSPY."
- ForAllSecure, a small, Pittsburgh-based company composed of former students of Carnegie Mellon University and led by professor David Brumley, running "Mayhem."
- Shellphish, a crowd favorite led by students at the University of California, Santa Barbara's Computer Security Group (Seclab), running "Mechanical Phish."
- TECHx, a collaboration between cybersecurity company GrammaTech and University of Virginia personnel, running "Xandra."
- CodeJitsu, an international team with members from the University of California, Berkeley, Syracuse University, and startup Cyberhaven, running "Galactica."
The CRSs attempted to discover vulnerabilities in a new, Linux-based operating system, DECREE, provided by DARPA. They would then try to "prove" these vulnerabilities by attacking their opponents, while automatically creating software patches to protect themselves. Visualization software showed multiple views of the action, with scoring based on system availability, security (defense), and evaluation (offense).
The game uncovered some surprises, including the discovery of (and defense against) vulnerabilities that DECREE's designers didn't deliberately include. Most notable to DARPA program manager Michael Walker was the proof of a vulnerability that commentator Visi called "the Everest of program analysis;" a customized version of the Sendmail crackaddr bug. "This bug was so hard that it was recreated and studied; papers have been published [asking], 'why can't we automatically find this?'" said Walker. "And one of these machines found it, proved its existence, navigated the path to it and said, 'I proved this bug.' To me, [that's] a huge thing."
In the end, the CRSs authored a total of 421 new pieces of security software.
There Can Be Only One
Competition was fierce, with the highest scorer leading the lowest by less than 11%. Availability proved crucial, as it was treated as a multiplier in the scoring system. Nevertheless, the ultimate winner was the ForAllSecure team's "Mayhem," scoring 270,042 points and taking home the $2-million grand prize, despite experiencing an availability crisis mid-game.
TECHx's "Xandra" reaped the second-place $1-million prize with 262,036 points, with Shellphish's "Mechanical Phish" taking third place (and $750,000) with 254,452 points.
Mayhem went on to compete against the best human hackers in DEF CON's capture-the-flag tournament -- where it finished last in a field of 15. That did not surprise or bother DARPA's Walker, who acknowledged this was only the first step toward fully automated defense against cyber attacks.
"We wanted to follow in the tradition of reasoning machines, machines like Deep Blue and Watson ... with the goal of someday, some descendent of this technology, of these prototypes, being able to compete with the world's best. [We started] by giving these machines a league of their own, in the same way that in 1970 the ACM created the world's first all-machine chess tournament. That all-machine tournament gave rise to top competition machines."
Tom Geller is an Oberlin, Ohio-based writer and documentary producer.
No entries found