Model-free optical processors using in situ reinforcement learning with proximal policy optimization

Yuhang Li; Shiqi Chen; Tingyu Gong; Aydogan Ozcan

doi:10.1038/s41377-025-02148-7

Your Location：

Home >

Browse articles >

Model-free optical processors using in situ reinforcement learning with proximal policy optimization

Original Articles

- Model-free optical processors using in situ reinforcement learning with proximal policy optimization
- Light: Science & Applications Vol. 15, Issue 1, Pages: 263-276(2026)
- Affiliations：
  
  1.Electrical and Computer Engineering Department, University of California, Los Angeles, CA, USA
  2.California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, USA
  3.Computer Science Department, University of California, Los Angeles, CA, USA
  4.Bioengineering Department, University of California, Los Angeles, CA, USA
- Author bio：
  
  Aydogan Ozcan (ozcan@ucla.edu)
- Funds：
- DOI：10.1038/s41377-025-02148-7
  CLC：
- Received：07 July 2025，
  
  Revised：2025-11-17，
  
  Accepted：27 November 2025，
  
  Published Online：01 January 2026，
  
  Published：31 January 2026
- Accepted：
Scan QR Code
Li, Y. H. et al. Model-free optical processors using in situ reinforcement learning with proximal policy optimization. Light: Science & Applications, 15, 263-276 (2026).
DOI：

Li, Y. H. et al. Model-free optical processors using in situ reinforcement learning with proximal policy optimization. Light: Science & Applications, 15, 263-276 (2026). DOI： 10.1038/s41377-025-02148-7.

摘要

Abstract

Optical computing holds promise for high-speed

energy-efficient information processing

with diffractive optical networks emerging as a flexible platform for implementing task-specific transformations. A challenge

however

is the effective optimization and alignment of the diffractive layers

which is hindered by the difficulty of accurately modeling physical systems with their inherent hardware imperfections

noise

and misalignments. While existing in situ optimization methods offer the advantage of direct training on the physical system without explicit system modeling

they are often limited by slow convergence and unstable performance due to inefficient use of limited measurement data. Here

we introduce a model-free reinforcement learning approach utilizing Proximal Policy Optimization (PPO) for the in situ training of diffractive optical processors. PPO efficiently reuses in situ measurement data and constrains policy updates to ensure more stable and faster convergence. We validated our method through both simulations and experiments across a range of in situ learning tasks

including targeted energy focusing through a random diffuser

image generation

aberration correction

and optical image classification

demonstrating in each task better convergence and performance. Our strategy operates directly on the physical system and naturally accounts for unknown real-world imperfections

eliminating the need for prior system knowledge or modeling. By enabling faster and more accurate training under realistic experimental constraints

this in situ reinforcement learning approach could offer a scalable framework for various optical and physical systems governed by complex

feedback-driven dynamics.

关键词

Keywords

references

Wright, L. G. et al. Deep physical neural networks trained with backpropagation. Nature 601 , 549–555 (2022)..

Momeni, A. et al. Backpropagation-free training of deep physical neural networks. Science 382 , 1297–1303 (2023)..

Momeni, A. et al. Physics-inspired neuroacoustic computing based on tunable nonlinear multiple-scattering. Print at https://doi.org/10.48550/arXiv.2304.08380 https://doi.org/10.48550/arXiv.2304.08380 (2023).

Momeni, A. et al. Training of physical neural networks. Nature 645 , 53–61 (2024)..

Brunner, D. et al. Roadmap on neuromorphic photonics. Print at https://doi.org/10.48550/arXiv.2501.07917 https://doi.org/10.48550/arXiv.2501.07917 (2025).

Lin, X. et al. All-optical machine learning using diffractive deep neural networks. Science 361 , 1004–1008 (2018)..

Yildirim, M. et al. Nonlinear processing with linear optics. Nat. Photonics 18 , 1076–1082 (2024)..

Chang, J. L. et al. Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification. Sci. Rep. 8 , 12324 (2018)..

Chen, S. Q. et al. Optical generative models. Nature 644 , 903–911 (2025)..

Shen, Y. C. et al. Deep learning with coherent nanophotonic circuits. Nat. Photonics 11 , 441–446 (2017)..

Wetzstein, G. et al. Inference in artificial intelligence with deep optics and photonics. Nature 588 , 39–47 (2020)..

Momeni, A. & Fleury, R. Electromagnetic wave-based extreme deep learning with nonlinear time- Floquet entanglement. Nat. Commun. 13 , 2651 (2022)..

Hughes, T. W. et al. Wave physics as an analog recurrent neural network. Sci. Adv. 5 , eaay6946 (2019)..

Vandoorne, K. et al. Experimental demonstration of reservoir computing on a silicon photonics chip. Nat. Commun. 5 , 3541 (2014)..

Xia, F. et al. Deep learning with passive optical nonlinear mapping. Print at https://doi.org/10.48550/arXiv.2307.08558 https://doi.org/10.48550/arXiv.2307.08558 (2024).

Duport, F. et al. All-optical reservoir computing. Opt. Express 20 , 22783–22795 (2012)..

Bu, T. et al. Efficient optical reservoir computing for parallel data processing. Opt. Lett. 47 , 3784–3787 (2022)..

Chen, Z. J. et al. Deep learning with coherent VCSEL neural networks. Nat. Photonics 17 , 723–730 (2023)..

Furuhata, G., Niiyama, T. & Sunada, S. Physical deep lear ning based on optimal control of dynamical systems. Phys. Rev. Appl. 15 , 034092 (2021)..

Liu, C. et al. A programmable diffractive deep neural network based on a digital-coding metasurface array. Nat. Electron. 5 , 113–122 (2022)..

Lillicrap, T. P. et al. Backpropagation and the brain. Nat. Rev. Neurosci. 21 , 335–346 (2020)..

Ruiz Euler, H. C. et al. A deep-learning approach to realizing functionality in nanoelectronic devices. Nat. Nanotechnol. 15 , 992–998 (2020)..

Scellier, B. & Bengio, Y. Equilibrium propagation: bridging the gap between energy-based models and backpropagation. Front. Comput. Neurosci. 11 , 24 (2017)..

Mengu, D. et al. Misalignment resilient diffractive optical networks. Nanophotonics 9 , 4207–4219 (2020)..

Deng, R., Zhou, J. & Zhang, W. Y. Bandlimited communication with one-bit quantization and oversampling: transceiver design and performance evaluation. IEEE Trans. Commun. 69 , 845–862 (2021)..

Shi, L. et al. Towards real-time photorealistic 3D holography with deep neural networks. Nature 591 , 234–239 (2021)..

Spall, J., Guo, X. X. & Lvovsky, A. I. Hybrid training of optical neural networks. Optica. 9 , 803–811 (2022)..

Zhao, G. Y.&Zhou, R. J. Model-free computer-generated holography. Proceedings of 2022 IEEE Region 10 Conference. 1-3 (IEEE, 2022).

Hughes, T. W. et al. Training of photonic neural networks through in situ backpropagation and gradient measurement. Optica 5 , 864–871 (2018)..

López-Pastor, V. & Marquardt, F. Self-learning machines based on Hamiltonian echo backpropagation. Phys. Rev. X 13 , 031020 (2023)..

Hermans, M. et al. Trainable hardware for dynamical computing using error backpropagation through physical media. Nat. Commun. 6 , 6729 (2015)..

Gu, J. Q. et al. L 2 ight: enabling on-chip learning for optical neural networks via efficient in-situ subspace optimization. Proceedings of the 35th International Conferen ce on Neural Information Processing Systems. Curran Associates Inc., 662. Print at https://proceedings.neurips.cc/paper/2021/hash/48aedb8880cab8c45637abc7493ecddd-Abstract.html https://proceedings.neurips.cc/paper/2021/hash/48aedb8880cab8c45637abc7493ecddd-Abstract.html (2021)..

Hinton, G. The forward-forward algorithm: some preliminary investigations. Print at https://doi.org/10.48550/arXiv.2212.13345 https://doi.org/10.48550/arXiv.2212.13345 (2022).

Pai, S. et al. Experimentally realized in situ backpropagation for deep learning in photonic neural networks. Science 380 , 398–404, https://doi.org/10.1126/science.ade8450 (2023)..

Nøkland, A. Direct feedback alignment provides learning in deep neural networks. Proceedings of the 30th International Conference on Neural Information Processing Systems. (Curran Associates Inc., 2016) 1045-1053.

Launay, J. et al. Hardware beyond backpropagation: a photonic co-processor for direct feedback alignment. Print at https://doi.org/10.48550/arXiv.2012.06373 https://doi.org/10.48550/arXiv.2012.06373 (2020).

Bueno, J. et al. Reinforcement learning in a large-scale photonic recurrent neural network. Optica. 5 , 756–760 (2018)..

Nakajima, M. et al. Physical deep learning with biologically inspired training method: gradient-free approach for physical hardware. Nat. Commun. 13 , 7847 (2022)..

Wang, C. H. et al. Memristor-based neural networks with weight simultaneous perturbation training. Nonlinear Dyn. 95 , 2893–2906 (2019)..

McCaughan, A. N. et al. Multiplexed gradient descent: fast online training of modern datasets on hardware neural networks without backpropagation. APL Mach. Learn. 1 , 026118 (2023)..

Zhang, H. et al. Efficient on-chip training of optical neural networks using genetic algorithm. ACS Photonics 8 , 1662–1672 (2021)..

Yan, Q. Q. et al. Multi-wavelength optical information processing with deep reinforcement learning. Light Sci. Appl. 14 , 160 (2025)..

Li, S. Y. et al. OpticGAI: generative AI-aided deep reinforcement learning for optical networks optimization. Proceedings of the 1st SIGCOMM Workshop on Hot Topics in Optical Technologies and Applications in Networking. (ACM, 2024) 1-6.

Natalino, C.&Monti, P. The optical RL-Gym: an open-source toolkit for applying reinforcement learning in optical networks. Proceedings of the 2020 22nd International Conference on Transparent Optical Networks. (IEEE, 2020) 1-5.

Kanno, K., Naruse, M. & Uchida, A. Adaptive model selection in photonic reservoir computing by reinforcement learning. Sci. Rep. 10 , 10062 (2020)..

Silver, D. et al. Deterministic policy gradient algorithms. Proceedings of the 31st International Conference on Machine Learning. (JMLR. org, 2014) I-387-I-395.

Song, Y.&Ermon, S. Generative modeling by estimating gradients of the data distribution. Proceedings of the 33rd Conference on Neural Information Processing Systems. (NeurIPS, 2019).

Zhao, Y. Y. et al. Gradient-assisted focusing light through scattering media. Opt. Lett. 46 , 1518–1521 (2021)..

Zhao, G. Y., Shu, X. & Zhou, R. J. High-performance real-world optical computing trained by in situ gradient-based model-free optimization. IEEE Trans. Pattern Anal. Mach. Intell. 47 , 7194–7205 (2025)..

Yang, J. M. et al. Anti-scattering light focusing by fast wavefront shaping based on multi-pixel encoded digital-micromirror device. Light Sci. Appl. 10 , 149 (2021)..

Skalli, A. et al. Model-free front-to-end training of a large high performance laser neural network. Print at https://doi.org/10.48550/arXiv.2503.16943 https://doi.org/10.48550/arXiv.2503.16943 (2025).

Schulman, J. et al. Proximal policy optimization algorithms. Print at https://doi.org/10.48550/arXiv.1707.06347 https://doi.org/10.48550/arXiv.1707.06347 (2017).

Mohamed, S. et al. Monte Carlo gradient estimation in machine learning. J. Mach. Learn. Res. 21 , 132 (2020)..

Deng, L. The MNIST database of handwritten digit images for machine learning research [best of the web ] . IEEE Signal Process. Mag. 29 , 141–142 (2012)..

Rombach, R. et al. High-resolution image synthesis with latent diffusion models. Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. (IEEE, 2022) 10674-10685.

Altman, N. & Krzywinski, M. The curse(s) of dimensionality. Nat. Methods 15 , 399–400 (2018)..

Ulyanov, D., Vedaldi, A. & Lempitsky, V. Deep image prior. Int. J. Computer Vis. 128 , 1867–1888 (2020)..

Niknam, F., Qazvini, H. & Latifi, H. Holographic optical field recovery using a regularized untrained deep decoder network. Sci. Rep. 11 , 10903 (2021)..

Li, R. J. et al. Physics-enhanced neural network for phase retrieval from two diffraction patterns. Opt. Express 30 , 32680–32692 (2022)..

Peng, Y. F. et al. Neural holography with camera-in-the-loop training. ACM Trans. Graph. 39 , 185 (2020)..

Chakravarthula, P. et al. Learned hardware-in-the-loop phase retrieval for holographic near-eye displays. ACM Trans. Graph. 39 , 186 (2020)..

de A. Belbute-Peres, F. et al. End-to-end differentiable physics for learning and control. Proceedings of the 32nd International Conference on Neural Information Processing Systems. (Curran Associates Inc., 2018) 7178-7189.

Goodman, J. W. Introduction to Fourier Optics. (Roberts&Co., 2005).

Chen, S., Li, Y., Wang, Y., Chen, H.&Ozcan, A. Optical generative models. Nature, 644, 903-911 (2025).

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰