Zongfu Yu, Ang Chen and Efram Khoram of the University of Wisconsin spearheaded a research team that developed the concept for a “smart” piece of clear glass that would use nanophotonic structures to distinguish among images of handwritten numerals, without external power or electronic circuits. [Image: Photo by Sam Million-Weaver]
From time to time, OPN has reported on so-called smart glass—window material that dynamically adjusts color and shading in response to external temperature, for example, and that even channels incoming sunlight to photovoltaic cells at the edges to generate electricity (see “A Smart View of the Future,” OPN, November 2018). Scientists from three U.S. universities have now proposed a scheme for a clear glass that would sport a very different kind of intelligence: the ability to perform simple image recognition, with no external power or sensors required (Phot. Res., doi: 10.1364/PRJ.7.000823).
The team’s concept for embedding artificial intelligence (AI) into glass—which thus far has been demonstrated only on a computer—relies on tiny nanophotonic structures dispersed in the glass. The resulting hybrid nanophotonic material takes on the role of the nonlinear activation function in an artificial neural net (ANN). The setup allows the glass, like an ANN, to be “trained” until it has the capability to distinguish among different input images (handwritten numerals, in the case of the team’s numerical proof-of-concept).
The system has some hurdles for practical use—in particular, a potentially laborious and computationally demanding training process to develop glasses for individual uses. Nonetheless, the researchers believe that, as the system is refined and perfected, it could offer an interesting platform for a variety of specific applications, such as biometric security locks requiring zero power input.
Training a neural net
In the training of a typical deep-learning ANN architecture, an input is passed into a layer of computational nodes—analogous to neurons in the human brain—which weight the input variables with a set of coefficients. The weighted sum then is acted upon by a so-called nonlinear activation function, analogous to the function that encodes the action potential (the threshold for firing) of neurons in a biological brain.
In so-called supervised learning, the output of each training step is compared with the input, and the errors between output and input are parameterized in a cost function; those errors are then used to re-wire the ANN to move closer to the correct answer. Eventually, after many iterations, the ensemble of artificial neurons in the network is optimized to “fire” in a specific way for a given input, and network is trained to spit out an output that matches that input—and, thus, to accomplish tasks like pattern and even facial recognition.
Toward photonic neural-net computing
In practice, both the training of complex electronic neural networks and their operation on actual data once trained require repeated, formidable matrix multiplications that gobble up large amounts of electrical power. That has spurred a quest in recent years for neural-net architectures that replace some or all of the network’s electronic components with photonic ones, to potentially slash the power requirements (see “Neuromorphic Photonics,” OPN, January 2018).
Some notable recent prototypes of photonic neural networks have involved multilayer networks that have used arrays of elements such as programmable Mach-Zehnder interferometers. The researchers behind the new work—led by OSA member Zongfu Yu at the University of Wisconsin, Madison, USA—wanted to see if artificial neural inference could be built into a system using the tools of nanophotonics instead.
A nanophotonic neural medium
Specifically, the team’s numerical experiments envision a nanophotonic neural medium (NNM) consisting thin sheet of glass, with dimensions of 80 and 20 times the wavelength of the light, that’s impregnated with subwavelength-scale air bubbles, initially in a random pattern. An image of a numeral, converted to a vector and encoded as light intensity variations, passes into one side of the glass; interference and scattering on the embedded nanostructures causes the light to bounce around in the model glass and eventually to be guided toward one or more of ten output ports on its other side.
In the team’s numerical experiments, a piece of model glass including nanophotonic structures is “trained” to route light from images of handwritten numerals to a specific output port corresponding to that number. [Image: Zongfu Yu]
Drawing on an approach previously published in Optica by the research team of OSA Fellow Shanhui Fan at Stanford University, Yu’s team showed numerically that such a glass NNM could be trained to accurately route input data from an image of a handwritten numeral to a specific output port corresponding to that number. The training consists of iteratively solving Maxwell’s equations for the electric field in the modeled nonlinear medium. Then, the resulting gradients of the cost function are used to update the structural parameters of the model material, and the updated material is used for the next training iteration.
The team showed that, after 66 training steps (using batches of sample images), the medium could be tuned to recognize a handwritten numeral (such as “8”) and route it to the correct output port with 77.3% accuracy. And the team says that accuracy could be improved still further by using a glass of larger size.
Toward a menu of inexpensive “smart things”
In an e-mail to OPN, Yu noted that, while it’s possible that the training of smart glasses could happen in situ rather than in a computer—for example, via refractive-index modulation using phase-change materials—at present it’s likely that the training will take place in a computer, to create designs for nanophotonically AI-enabled glasses that could then be fabricated for specific applications. In the team’s vision, the materials will “serve as inexpensive and even disposable ‘smart things’ that work under extremely tight energy, bandwidth and dollar budgets,” said Yu. “This is where these task-specific smart glasses fit in.”
For example, in a press release accompanying the research, Yu envisioned a piece of glass that could act as a biometric lock, trained to recognize only one person’s face. “Once built, it would last forever without needing power or internet,” Yu said, “meaning it could keep something safe for you even after thousands of years.”
At this point, Yu told OPN, the team is focusing on “trying to move the concept toward practical applications,” such as object recognition in a practical setting. This, he noted, will require a better understanding of the nonlinear optical materials that might be used, as well as work to make the glass more tolerant of diffusive background light. “Our next milestone,” Yu said, “is to recognize a face, which is computationally more challenging than the handwritten digits.”
In addition to researchers at the University of Wisconsin, the paper included authors from the Massachusetts Institute of Technology and Columbia University.