Communications of the ACM
Telling Stories Using Computer Science
"The whole point is still to tell a story, but you're using computer science and math to do that," says MIT senior Shannon Kao.
Credit: Allegra Boverman / MIT
For MIT senior Shannon Kao, expert storytelling is essential, even — if not especially — when it comes to coding. The computer science major relies on narrative everywhere from her science fiction writing to her research on educational computer games at the MIT Media Lab — and it all stems from a childhood replete with books.
Kao grew up in Michigan and then China, where her mother, who was her school's librarian, exposed her and her two younger brothers, from an early age, to everything from picture books to hefty novels.
"Instead of hanging out, we would all just grab a book and sit in our living room and read," Kao says with a laugh.
After two semesters of organic chemistry doused her interest in medical school, Kao stumbled onto a course, Introduction to Electrical Engineering and Computer Science, and immediately saw computer science as a way to use her love of math to build something tangible and interactive. The summer after her freshman year, she took on a research position with the Affective Computing Group at the Media Lab, where she worked on a free app called StoryScape.
The program, geared toward families with developmentally challenged children, lets users drag and drop animated characters and illustrations, from a gallery Kao helped build, onto a page where users can write original stories, and can then share those stories with others. The animated characters can react to stimuli in users' environments, such as loud noises, or the shaking of the phone or tablet on which the app is installed.
Working on StoryScape's graphic gallery was Kao's first experience in a computer science lab — her first research position at MIT was in a biology lab — but even though she had very little expertise in the field at that point, the new research setting immediately felt natural.
"It's important that there's a strong story underneath something, and the rest will follow," she says.
Modeling for Beginners
Kao has since expanded her research to MIT's Scheller Teacher Education Program (STEP), which develops learning technology. For the past 2 1/2 years, she has worked with software developer Daniel Wendel, a research associate in MIT's Department of Urban Studies and Planning, on a project called StarLogo. It teaches students with no computer science background how to build a program for modeling decentralized systems, like traffic jams.
StarLogo's accessibility to inexperienced coders hinges on a system of blocks-based programming, which works like virtual LEGO bricks: To build a program, rather than writing out lines of code using individual symbols and numbers, users drag and drop ready-made blocks of text that code for 3-D graphics; only certain combinations of text blocks create a functioning program. The process of building a modeling program this way is a lot like building a story — it needs a coherent beginning, middle, and end, or else it won't function. Kao's role has been to develop the 3-D graphics that the blocks code for.
"[Blocks] make programming more intuitive for people who don't necessarily have the background," she says.
Kao has helped run several workshops to make ongoing improvements to StarLogo. STEP invites in parents and children with no programming background to complete a series of challenges; the researchers then ask for feedback on usability. Some of the biggest issues the team has encountered are with interfaces that control zooming and scrolling. After each workshop, it's back to the lab, where Kao and her colleagues whittle away at a list of tweaks in preparation for the next workshop and set of feedback.
Doodles to Animation
Kao's interest in telling stories through design and graphics started as mindless doodles in class, but soon grew into full-on illustrations that she later learned to turn into animations, using her computer science skills to bring her art to life. Looking at the illustrations, it's easy to see that some of her inspiration comes from Japanese animator Hayao Miyazaki, but Kao cites the 2007 Disney film "Ratatouille" as her favorite animated movie.
"I feel like part of my interest in art is that I was just in this constant stream of picture books and young adult books that I would read regardless of what age I was," she says. "I still really enjoy some picture books."
Kao also works with written narrative: As the literature editor of Rune, MIT's literary magazine, during her sophomore year, she was responsible for vetting incoming submissions. For the past two years, she has focused more on her own writing, winning MIT's Ilona Karmel Prize for Science Fiction in 2013 and 2014.
Despite her passions in art and literature, Kao's occupational focus remains with computer science, but always with her hobbies and upbringing as her guideposts.
"Computer graphics is that in-between space," she says. "You need to have some kind of aesthetic sensibility, since the whole point is still to tell a story, but you're using computer science and math to do that."
No entries found