Communications of the ACM
Nanophotonics Flattens Camera Lenses
Nine meta-photonic lenses (upper left and right) integrated on top of a CMOS imager chip make use of graphics processor units to learn a generative diffusion model that reconstructs an indoor (bottom left) and outdoor (bottom right) image.
Credit: Princeton University
Modern smartphone cameras have all but made obsolete the once-ubiquitous single-lens reflex (SLR) camera. Even so, today's camera lenses acquire images in the same way they did in the 19th century--by focusing light on a nanoscale sensing plane using a 100-times-or-higher stack of glass lenses that correct the optics defined by Gauss in 1843 (Dioptrische Untersuchungen).
Now, finally, camera lens technology has caught up to the 21st century with nanoscale meta-optics.
"Meta-optics is becoming a powerful tool for imaging—allowing the optics for a variety of imaging applications to be significantly miniaturized. The additional degrees of design freedom afforded by meta-optics, along with their unique properties in terms of polarization control, chromatic dispersion, and multiplexing, have been shown to have significant potential impact in many applications, especially when integrated with the right computational algorithms for reconstruction," said Ashok Veeraraghavan, a professor of electrical and computer engineering and of computer science at Rice University and principle investigator of the university's Computational Imaging Group (who was not involved with the meta-optics prototype described here).
Until now, even the cameras in 21st-century high-end smartphones required a stack of seven or more 19th-century-style lenses atop their complementary metal oxide semiconductor (CMOS) imager chip, causing the lens to protrude from the phone's body. In fact, in all high-end smartphones, the camera lens length is the limiting factor on how thin the phone's body can be made. Now researchers at Princeton University, in collaboration with colleagues at the University of Washington, have used meta-optics to successfully flatten the camera lens to just 700 nanometers (billionths of a meter), rivaling the pin-hole "camera obscura."
"Our aim was to design computational nano-photonic meta-lenses that make very thin cameras possible, more than two orders of magnitude thinner than today," said Felix Heide, an assistant professor of computer science at Princeton University and leader of the Princeton Computational Imaging Lab.
The seminal pin-hole camera obscura was lens-less—in fact, it was invented before the glass lens itself; the phenomenon underlying it is noted in Chinese texts as old as 1046 BC, when it was used to view solar eclipses in a darkened room (to prevent damage to the eye). The blossoming of the photographic camera had to wait for the adaptation of optical lenses from telescopes in the 19th century (and the invention of photographic plates/film). Since then, the number of methods for magnifying images while correcting for lens distortions (aberrations) has steadily added bulk to the lens stack—feet long for some SLR telephoto lenses, and approximately 7mm (little more than a quarter of an inch) for high-end smartphones.
"Flat optical systems promise to enable a host of break-through imaging capabilities, from SLR-quality lenses on the back of a cellphone to cameras so thin they could be embedded under the skull to monitor brain activity. Accordingly, designing effective flat optical systems is one of the most important open problems in computational imaging," said Christopher Metzler, an assistant professor in the department of computer science at the University of Maryland and former member of Stanford's Computational Imaging Lab (who was not involved with the meta-optics prototype described here).
The importance of reinventing the lens became a high priority at the turn of the 21st century, when serious research progress into flattening the lens stack was achieved. The first step was micro-miniaturizing the pin-hole concept into what was called coded apertures using flat masks (FlatCam, 2016). The light from various sized pinhole-like rectangles in the mask, when computationally merged, was designed to overcome the long exposure time required by a single pinhole. Unfortunately, this technique trades off resolution for distance between the mask and the imager chip, so when flattened, could not match the resolution of the multiple-stack glass-lens cameras we see today.
The next step to a flat lens moved beyond the restrictions of pinhole masks to diffusors (DiffuserCam, 2020). When used with monochrome spectral-filter arrays, they accomplished lensless hyper-spectral imaging used in agriculture and medicine, but not for full-color imaging.
For full-color imaging, now Heide et. al. have "demonstrated that a flat nano-photonic meta-lens array when used with machine learning-based reconstruction algorithms allows you to realize compact, wide field-of-view color imaging. Particularly impressive is the quality of the reconstruction results—in terms of both resolution and color fidelity,' said Veeraraghavan, inventor of the FlatCam.
Heide et. al.'s flat quasi-periodic arrays of nano-photonic meta-lenses (sub-wavelength light scatterers) were engineered to manipulate the full spectrum of visible light (broadband) with corrections performed by specialized inference engines trained by graphics processing units (GPUs), resulting in lensless high-resolution full-color spectrum visible-light images rivaling those made by conventional glass lens stacks.
"Ours is the first demonstration of high-quality broadband nano-photonic imaging outside the lab," said Heide about their group's Thin On-Sensor Nanophotonic Array Cameras in a paper in ACM's 2023 Transactions on Graphics. The researchers also publish their results open-source, including all code, optical design files, and datasets at GitHub.
According to Heide, high-resolution full-color images are reconstructed by the group's flat nano-photonic meta-lenses, composed of circular arrays of 700-nanometer high transparency (silicon nitride) pillars with a pitch of 350 nanometers and widths that vary between 100 and 300 nanometers. Since visible light's wavelength is between 400 and 700 nanometers, these sub-wavelength-wide pillars exhibit meta-material properties.
In the prototype, nine side-by-side meta-lenses, in a three-by-three configuration, together feed image data to high-speed reconstruction algorithms using inverse filtering, diffusion, and merging of the results from the nine separate meta-lenses. The parameters for the reconstruction algorithms were learned by a deep neural network (DNN). Their results, the researchers say, outperform other "flat lens" approaches, and unlike previous monochrome images from hyper-spectral flat lenses, Heide et. al.'s ultra-thin camera meta-lens is capable of imaging scenes with accurate full-color reproduction.
"Arguably, their [Heide et. al.] most important insight is that one can create a high-resolution wide field-of-view flat optical imaging system by designing an array of lower-resolution narrow field-of-view optical elements, whose measurements can be computationally integrated. This scalable framework massively simplifies the design process and allows them to demonstrate high-quality wide field-of-view mega-pixel-scale imaging with thin nano-photonic-based lenses," said Metzler.
The DNN learning process used training data from the group's own high-resolution images, the MIT 5K high-resolution SLR image database, and a sampling of relatively low-resolution images from the ImageNet database. After training, the parameters were fixed for the camera-resident inference engine that corrects colors, distortions, and aberrations. The prototype combined the output from nine planar nano-photonic lenses affixed atop the imager chip to achieve a 100-degree field of view—equivalent to about an 18-mm conventional wide-angle lens on a 35-mm camera.
Commercial camera lenses using this approach are expected to eventually use hundreds of flat meta-lenses to cover the entire surface of large CMOS imager chips with hundreds of mega-pixels. To boot, telephoto to ultra-wide-angle images can be computationally reconstructed from a single meta-lens shot.
One limitation is the need for a high-speed application-specific integrated circuit (ASIC) to process the output of the meta-lenses arrayed atop the imager chip, since conventional GPUs would generate too much heat to be feasible for portable cameras. "Despite this limitation today, a modern smartphone ASIC can efficiently implement our reconstruction process, potentially enabling fast inference-on-the-edge devices in the future," said Heide.
Also for the future, the researchers will focus on packing their meta-lenses tighter (using light baffles to separate them instead of dead space, as in the prototype). They also hope their nano-photonic meta-lens arrays will stimulate the computational photography community to reexamine light-field arrays that capture 4D images (by including the direction from which each three-dimensional light ray came).
R. Colin Johnson is a Kyoto Prize Fellow who has worked as a technology journalist for two decades.
No entries found