On-chip photonic-neural-network processors have potential benefits in both speed and energy efficiency but have not yet reached the scale to compete with electronic processors. The dominant paradigm is to build integrated-photonic processors using relatively bulky discrete components connected by single-mode waveguides. A far more compact alternative is to avoid explicitly defining any components and instead sculpt the continuous substrate of the photonic processor to directly perform the computation using waves freely propagating in two dimensions. In this talk, I will present our recent work [1] on experimentally realizing this approach with a device whose refractive index as a function of space, n(x,z), can be rapidly reprogrammed. This device combines photoconductive gain with the electro-optic effect in a lithium niobate slab waveguide. Using this device, we performed neural-network inference with up to 49-dimensional input vectors in a single pass.
[1]: T. Onodera*, M.M. Stein*, et al. arXiv:2402.17750 (2024)
On-chip photonic-neural-network processors promise benefits in both speed and energy efficiency but have not yet reached the scale to compete with electronic processors. The dominant paradigm is to build integrated-photonic processors using discrete components connected by single-mode waveguides. A far more compact alternative is to avoid discrete components and instead sculpt a complex and continuous microphotonic medium in which computations are performed by multimode waves controllably propagating in two dimensions. We show our realization of this approach with a device whose refractive index as a function of space can be rapidly reprogrammed. We demonstrate optical computations much larger and more error-resilient than previous photonic chips relying on discrete components. We argue that beyond photonic-neural-network processors, devices with such arbitrarily programmable index distributions enable the realization of a wide range of photonic functionality.
Deep learning acceleration with integrated photonics has typically employed a circuit-centric approach with Mach-Zehnder interferometers. This requires a large spatial footprint, which has motivated the direct training of spatial refractive index distribution within a slab waveguide. Here, we demonstrate through simulations that nonlinear optical material platforms with large electro-optic coefficients can capitalize on this approach. We show that a linear device with realistic device parameters can perform 50 by 50 unitary matrix multiplications. We also performed MNIST digit classification, achieving 90.5% classification accuracy with minimal digital preprocessing. Finally, we comment on device implementation with Lithium Niobate or Barium Titanate slab waveguides.
We report on the realization of an on-chip waveguide platform capable of creating arbitrary two-dimensional refractive index profiles in situ and in real-time. The device exhibits complex multimode dynamics which we train to perform machine learning. We tune the refractive index profile in situ using a backpropagation algorithm to perform audio and image classification with up to 50-dimensional inputs. The two-dimensional programmability is realized by sandwiching a photoconductive film and a lithium niobate slab waveguide between two flat electrodes. While applying voltage between the electrodes, we program the effective index of the waveguide by projecting different light patterns onto the photoconductive film. The effective index increases by 10^-3 in illuminated regions via the electro-optic effect, free from any measurable memory effects or cyclic degradation. In conclusion, we developed a photonics platform with versatile spatial programmability that opens new avenues for optical computing and photonic inverse-design.
Numerous applications in science and technology nowadays utilize deep learning to tackle challenging computational tasks. With the increasing demand for deep learning, high-speed and energy-efficient accelerators are urgently needed. Although electronic accelerators are flexible, optical computers holds great promise due to their potential for massive parallelism and low power consumption. However, optical computing platforms demonstrated so far have mostly been limited to relatively small-scale computing tasks, despite the potential for scalability. Here, we propose and demonstrate a hardware-efficient design that allows deployment of a reconfigurable deep neural network (DNN) architecture without a direct isomorphism to standard DNN designs. Our proposed system is scalable and supports larger-scale computing. Our system realizes an optical neural network (ONN) using a digital micromirror device (DMD) for encoding data and trainable parameters, a complex medium for random complex weight mixing, and a camera for nonlinear activation and optical readout. A straight-through estimator enables backpropagation, even with a DMD as a binary encoding device. With this ONN as an elementary building block and automating the search for neural architectures, we can build complex and deep ONNs for a range of large-scale computing tasks, such as 3D medical image classification. The architecture-optimized deep ONNs are deployed by time-multiplexing data streams in one system. Our system enables large-scale training and inference in situ. Furthermore, we demonstrate that our system is capable of achieving task accuracies close to that of state-of-the-art benchmarks with more complex architectures implemented in silico.
I will overview our work on analog neural networks based on photonics and other controllable physical systems. I will show how backpropagation can efficiently train physical neural networks (PNNs), and how to design physical network architectures for physics-based machine learning. I will review our work showing how nonlinear photonic neural networks may enhance computational sensing and how photonic neural networks may be operated robustly deep into low-energy regimes where quantum noise would ordinarily be a limiting factor. Finally, I will show that PNNs offer fundamental advantages for scaling AI models such as Transformers.
Photonic neural networks have been developed as a hardware platform to accelerate machine-learning inference. Digital micromirror devices (DMDs) have been playing a critical role in developing a variety of photonic neural networks for their ability to manipulate millions of optical spatial modes in a 2D plane at >10 kHz frame rate. DMDs have not only enabled high-throughput machine-learning inference but also made hardware-in-the-loop training possible with photonic neural networks. In this talk, we will review the functions of DMDs in a plethora of photonic-neural-network architectures and discuss how MEMS-based technologies can enable novel photonic neural networks in future.
Utilizing the input-output transformation of ultrafast nonlinear pulse propagation in quadratic media, we experimentally construct a multilayer physical neural network to perform both audio and handwritten image classification. We introduce a hybrid in-situ in-silico backpropagation algorithm, physics-aware training, that is resilient to the simulation-reality gap, to train physical neural networks. The methodology for constructing and training physical neural networks applies to generic complex physical systems. To demonstrate its generality, we also built and trained physical neural networks out of analog electronic circuits and multimode mechanical oscillators to perform image classification.
Linear optics has been long applied to image compression. However, it is widely known that nonlinear neural networks outperform linear models in terms of feature extraction and image compression. Here, we show a nonlinear multilayer optical neural network using a commercially available image intensifier as a scalable optical-to-optical nonlinear activation function. We experimentally demonstrated that nonlinear ONNs outperform linear optical linear encoders in a variety of non-trivial machine vision tasks at a high image compression ratio (up to 800:1). We have shown that nonlinear ONNs can directly process optical inputs from physical objects under natural illumination, which provides a new pathway towards high-volume, high-throughput, and low-latency machine vision processing.
We developed and implemented a deep optical neural network (ONN) design capable of performing large-scale training and inference in situ. For each elementary building block in the ONN, we introduce trainable parameters in a programmable device, weight mixing with a diffuser, and nonlinear detection on the camera for activation and optical readout. With automated reconfigurable neural architecture search, we optimized the architecture of deep ONNs that can perform multiple tasks at high speed and at large scale. The task accuracies achieved by our experiments are close to state-of-the-art benchmarks with conventional multilayer neural networks.
In conventional approaches to computer-vision tasks such as object recognition, a camera digitally records a high-resolution image and an algorithm is run to extract information from the image. Alternative image-sensing schemes have been proposed that extract high-level features from a scene using optical components, filtering out irrelevant information ahead of conversion from the optical to electronic domains by an array of detectors (e.g., a CMOS image sensor). In this way, images are compressed into a low-dimensional latent space, allowing computer-vision applications to be realized with fewer detectors, fewer photons, and reduced digital post-processing, which enables low-latency processing. Optical neural networks (ONNs) offer a powerful platform for such image compression/feature extraction in the analog, optical domain. While ONNs have been successfully implemented using only linear operations, which can still be leveraged for computer-vision applications, it is well known that adding nonlinearity (a prerequisite for depth) enables neural networks to perform more complex processing. Our work realizes a multilayer ONN preprocessor for image sensing, using a commercial image intensifier as an optoelectronic, optical-to-optical nonlinear activation function. The nonlinear ONN preprocessor achieves compression ratios up to 800:1. At high compression ratios, the nonlinear ONN outperforms any linear preprocessor in terms of classification accuracy on a variety of tasks. Our experiments demonstrate ONN image sensors with incoherent light, but emerging technologies such as metasurfaces, large-scale laser arrays, and novel optoelectronic materials, will provide the means to realize a variety of multilayer ONN preprocessors that act on coherent and/or quantum light.
Coherent Ising machines (CIMs) are an experimentally promising class of physics-based computational architectures that embed hard combinatorial optimization problems into systems of coupled nonlinear optical oscillators. The solution-finding mechanisms employed by CIMs feature complicated dynamical bifurcations occurring on a network scale, posing significant challenges to the development of theory and models for their underlying principles of operation. These difficulties are especially pronounced in the ultra-low-power or quantum regimes where the benefits in computational efficiency over conventional optimization algorithms are expected to be largest. We discuss some of our recent approaches and results at this intersection of dynamical systems theory and quantum model reduction, which have highlighted some potentially useful architectures and applications on the horizon for CIMs.
Photons generally weakly interact with each other. While this feature is often useful, for instance, in communication, controllably inducing strong photon-photon interactions is necessary for deterministic, all-optical quantum information processing. Consequently, the weak nonlinearity in optics has historically motivated the use of measurement-based feedback schemes for photonic quantum information processing. Here, we theoretically investigate if coherent photonic quantum information processing could be realized by enhancing weak nonlinearities via Gaussian operations. We find that a Kerr medium with weak material nonlinearity can be engineered to implement a coherent cubic phase gate. Recent progress in developing photonic platforms supporting strong spatial and temporal confinement of pulsed light in low loss and low dispersion nonlinear waveguides could enable an experimental demonstration in the near future.
The advent of dispersion-engineered and highly nonlinear nanophotonics is expected to open up an all-optical path towards the strong-interaction regime of quantum optics by combining high transverse field confinement with ultra-short-pulse operation. Obtaining a full understanding of photon dynamics in such broadband devices, however, poses major challenges in the modeling and simulation of multimode non-Gaussian quantum physics, highlighting the need for sophisticated reduced models that facilitate efficient numerical study while providing useful physical insight. In this manuscript, we review our recent efforts in modeling broadband optical systems at varying levels of abstraction and generality, ranging from multimode extensions of quantum input-output theory for sync-pumped oscillators to the development of numerical methods based on a field-theoretic description of nonlinear waveguides. We expect our work not only to guide ongoing theoretical and experimental efforts towards next-generation quantum devices but also to uncover essential physics of broadband quantum photonics.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.