Metasurface-enabled on-chip multiplexed diffractive neural networks in the visible

Xuhao Luo; Yueqiang Hu; Xiangnian Ou; Xin Li; Jiajie Lai; Na Liu; Xinbin Cheng; Anlian Pan; Huigao Duan

doi:10.1038/s41377-022-00844-2

Go to Nature Website

Your Location：

Home >

Browse articles >

Metasurface-enabled on-chip multiplexed diffractive neural networks in the visible

Articles

- Metasurface-enabled on-chip multiplexed diffractive neural networks in the visible
- Light: Science & Applications Vol. 11, Issue 7, Pages: 1443-1453(2022)
- Affiliations：
  
  1.National Research Center for High-Efficiency Grinding, College of Mechanical and Vehicle Engineering, Hunan University, Changsha 410082, China
  2.Institute of Precision Optical Engineering, School of Physics Science and Engineering, Tongji University, Shanghai 200092, China
  3.Advanced Manufacturing Laboratory of Micro-Nano Optical Devices, Shenzhen Research Institute, Hunan University, Shenzhen 518000, China
  4.2nd Physics Institute, University of Stuttgart, Pfaffenwaldring 57, 70569 Stuttgart, Germany
  5.Max Planck Institute for Solid State Research, Heisenbergstrasse 1, 70569 Stuttgart, Germany
  6.Greater Bay Area Institute for Innovation, Hunan University, Guangzhou 511300, China
- Author bio：
  
  Yueqiang Hu (huyq@hnu.edu.cn)
  Huigao Duan (duanhg@hnu.edu.cn)
- Funds：
- DOI：10.1038/s41377-022-00844-2
  CLC：
Scan for full text
Luo, X. H. et al. Metasurface-enabled on-chip multiplexed diffractive neural networks in the visible. Light: Science & Applications, 11, 1443-1453 (2022).
DOI：

Luo, X. H. et al. Metasurface-enabled on-chip multiplexed diffractive neural networks in the visible. Light: Science & Applications, 11, 1443-1453 (2022). DOI： 10.1038/s41377-022-00844-2.

摘要

Abstract

Replacing electrons with photons is a compelling route toward high-speed, massively parallel, and low-power artificial intelligence computing. Recently, diffractive networks composed of phase surfaces were trained to perform machine learning tasks through linear optical transformations. However, the existing architectures often comprise bulky components and, most critically, they cannot mimic the human brain for multitasking. Here, we demonstrate a multi-skilled diffractive neural network based on a metasurface device, which can perform on-chip multi-channel sensing and multitasking in the visible. The polarization multiplexing scheme of the subwavelength nanostructures is applied to construct a multi-channel classifier framework for simultaneous recognition of digital and fashionable items. The areal density of the artificial neurons can reach up to 6.25 × 10,6, mm,−2, multiplied by the number of channels. The metasurface is integrated with the mature complementary metal-oxide semiconductor imaging sensor, providing a chip-scale architecture to process information directly at physical layers for energy-efficient and ultra-fast image processing in machine vision, autonomous driving, and precision medicine.

关键词

Keywords

references

Y. LeCun, Y. Bengio , G. Hinton . Deep learning . Nature , 2015 . 521 436 -444. .

J. Schmidhuber . Deep learning in neural networks: an overview . Neural Netw. , 2015 . 61 85 -117 . DOI:10.1016/j.neunet.2014.09.003http://doi.org/10.1016/j.neunet.2014.09.003 .

P. A. Merolla , 等 . A million spiking-neuron integrated circuit with a scalable communication network and interface . Science , 2014 . 345 668 -673 . DOI:10.1126/science.1254642http://doi.org/10.1126/science.1254642 .

A. K. Jain , J. C. Mao , K. M. Mohiuddin . Artificial neural networks: a tutorial . Computer , 1996 . 29 31 -44 . DOI:10.1109/2.485891http://doi.org/10.1109/2.485891 .

He, K. M. et al. Deep residual learning for image recognition. Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. 770–778 (Las Vegas, IEEE, 2016).

Graves, A. et al. Speech recognition with deep recurrent neural networks. Proceedings of 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. 6645–6649 (Vancouver, IEEE, 2013).

T. Young , 等 . Recent trends in deep learning based natural language processing [Review Article] . IEEE Computational Intell. Mag. , 2018 . 13 55 -75 . DOI:10.1109/MCI.2018.2840738http://doi.org/10.1109/MCI.2018.2840738 .

D. Silver , 等 . Mastering the game of Go with deep neural networks and tree search . Nature , 2016 . 529 484 -489 . DOI:10.1038/nature16961http://doi.org/10.1038/nature16961 .

K. H. Jin , 等 . Deep convolutional neural network for inverse problems in imaging . IEEE Trans. Image Process. , 2017 . 26 4509 -4522 . DOI:10.1109/TIP.2017.2713099http://doi.org/10.1109/TIP.2017.2713099 .

Y. Rivenson , 等 . Deep learning microscopy . Optica , 2017 . 4 1437 -1443 . DOI:10.1364/OPTICA.4.001437http://doi.org/10.1364/OPTICA.4.001437 .

A. Sinha , 等 . Lensless computational imaging through deep learning . Optica , 2017 . 4 1117 -1125 . DOI:10.1364/OPTICA.4.001117http://doi.org/10.1364/OPTICA.4.001117 .

W. Ma , 等 . Deep learning for the design of photonic structures . Nat. Photonics , 2021 . 15 77 -90 . DOI:10.1038/s41566-020-0685-yhttp://doi.org/10.1038/s41566-020-0685-y .

A. W. Rix , A. Bourret , M. P. Hollier . Models of human perception . BT Technol. J. , 1999 . 17 24 -34 . DOI:10.1023/A:1009662506355http://doi.org/10.1023/A:1009662506355 .

G. Wetzstein , 等 . Inference in artificial intelligence with deep optics and photonics . Nature , 2020 . 588 39 -47 . DOI:10.1038/s41586-020-2973-6http://doi.org/10.1038/s41586-020-2973-6 .

H. J. Caulfield , S. Dolev . Why future supercomputing requires optics . Nat. Photonics , 2010 . 4 261 -263 . DOI:10.1038/nphoton.2010.94http://doi.org/10.1038/nphoton.2010.94 .

G. Genty , 等 . Machine learning and applications in ultrafast photonics . Nat. Photonics , 2021 . 15 91 -101 . DOI:10.1038/s41566-020-00716-4http://doi.org/10.1038/s41566-020-00716-4 .

B. J. Shastri , 等 . Photonics for artificial intelligence and neuromorphic computing . Nat. Photonics , 2021 . 15 102 -114 . DOI:10.1038/s41566-020-00754-yhttp://doi.org/10.1038/s41566-020-00754-y .

Y. C. Shen , 等 . Deep learning with coherent nanophotonic circuits . Nat. Photonics , 2017 . 11 441 -446 . DOI:10.1038/nphoton.2017.93http://doi.org/10.1038/nphoton.2017.93 .

H. L. Zhou , 等 . Photonic matrix multiplication lights up photonic accelerator and beyond . Light.: Sci. Appl. , 2022 . 11 30 DOI:10.1038/s41377-022-00717-8http://doi.org/10.1038/s41377-022-00717-8 .

J. Liu , 等 . Research progress in optical neural networks: theory, applications and developments . PhotoniX , 2021 . 2 5 DOI:10.1186/s43074-021-00026-0http://doi.org/10.1186/s43074-021-00026-0 .

X. Lin , 等 . All-optical machine learning using diffractive deep neural networks . Science , 2018 . 361 1004 -1008 . DOI:10.1126/science.aat8084http://doi.org/10.1126/science.aat8084 .

T. K. Zhou , 等 . Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit . Nat. Photonics , 2021 . 15 367 -373 . DOI:10.1038/s41566-021-00796-whttp://doi.org/10.1038/s41566-021-00796-w .

J. Li , 等 . Spectrally encoded single-pixel machine vision using diffractive networks . Sci. Adv. , 2021 . 7 eabd7690 DOI:10.1126/sciadv.abd7690http://doi.org/10.1126/sciadv.abd7690 .

C. Qian , 等 . Performing optical logic operations by a diffractive neural network . Light.: Sci. Appl. , 2020 . 9 59 DOI:10.1038/s41377-020-0303-2http://doi.org/10.1038/s41377-020-0303-2 .

T. Yan , 等 . Fourier-space diffractive deep neural network . Phys. Rev. Lett. , 2019 . 123 023901 DOI:10.1103/PhysRevLett.123.023901http://doi.org/10.1103/PhysRevLett.123.023901 .

E. Goi , 等 . Nanoprinted high-neuron-density optical linear perceptrons performing near-infrared inference on a CMOS chip . Light.: Sci. Appl. , 2021 . 10 40 DOI:10.1038/s41377-021-00483-zhttp://doi.org/10.1038/s41377-021-00483-z .

O. Kulce , 等 . All-optical information-processing capacity of diffractive surfaces . Light.: Sci. Appl. , 2021 . 10 25 DOI:10.1038/s41377-020-00439-9http://doi.org/10.1038/s41377-020-00439-9 .

J. K. Weng , 等 . Meta-neural-network for real-time and passive deep-learning-based object recognition . Nat. Commun. , 2020 . 11 6309 DOI:10.1038/s41467-020-19693-xhttp://doi.org/10.1038/s41467-020-19693-x .

Y. Luo , 等 . Design of task-specific optical systems using broadband diffractive neural networks . Light.: Sci. Appl. , 2019 . 8 112 DOI:10.1038/s41377-019-0223-1http://doi.org/10.1038/s41377-019-0223-1 .

M. S. S. Rahman , 等 . Ensemble learning of diffractive optical networks . Light.: Sci. Appl. , 2021 . 10 14 DOI:10.1038/s41377-020-00446-whttp://doi.org/10.1038/s41377-020-00446-w .

M. Veli , 等 . Terahertz pulse shaping using diffractive surfaces . Nat. Commun. , 2021 . 12 37 DOI:10.1038/s41467-020-20268-zhttp://doi.org/10.1038/s41467-020-20268-z .

N. F. Yu , 等 . Light propagation with phase discontinuities: generalized laws of reflection and refraction . Science , 2011 . 334 333 -337 . DOI:10.1126/science.1210713http://doi.org/10.1126/science.1210713 .

S. L. Sun , 等 . High-efficiency broadband anomalous reflection by gradient meta-surfaces . Nano Lett. , 2012 . 12 6223 -6229 . DOI:10.1021/nl3032668http://doi.org/10.1021/nl3032668 .

A. Silva , 等 . Performing mathematical operations with metamaterials . Science , 2014 . 343 160 -163 . DOI:10.1126/science.1242818http://doi.org/10.1126/science.1242818 .

J. Sol , D. R. Smith , P. del Hougne . Meta-programmable analog differentiator . Nat. Commun. , 2022 . 13 1713 DOI:10.1038/s41467-022-29354-whttp://doi.org/10.1038/s41467-022-29354-w .

H. Babashah , 等 . Integration in analog optical computing using metasurfaces revisited: toward ideal optical integration . J. Optical Soc. Am. B , 2017 . 34 1270 -1279 . DOI:10.1364/JOSAB.34.001270http://doi.org/10.1364/JOSAB.34.001270 .

W. W. Fu , 等 . Ultracompact meta-imagers for arbitrary all-optical convolution . Light.: Sci. Appl. , 2022 . 11 62 DOI:10.1038/s41377-022-00752-5http://doi.org/10.1038/s41377-022-00752-5 .

N. M. Estakhri , B. Edwards , N. Engheta . Inverse-designed metastructures that solve equations . Science , 2019 . 363 1333 -1338 . DOI:10.1126/science.aaw2498http://doi.org/10.1126/science.aaw2498 .

A. Babaee , 等 . Parallel analog computing based on a 2×2 multiple-input multiple-output metasurface processor with asymmetric response . Phys. Rev. Appl. , 2021 . 15 044015 DOI:10.1103/PhysRevApplied.15.044015http://doi.org/10.1103/PhysRevApplied.15.044015 .

M. Camacho , B. Edwards , N. Engheta . A single inverse-designed photonic structure that performs parallel computing . Nat. Commun. , 2021 . 12 1466 DOI:10.1038/s41467-021-21664-9http://doi.org/10.1038/s41467-021-21664-9 .

A. Momeni , 等 . Generalized optical signal processing based on multioperator metasurfaces synthesized by susceptibility tensors . Phys. Rev. Appl. , 2019 . 11 064042 DOI:10.1103/PhysRevApplied.11.064042http://doi.org/10.1103/PhysRevApplied.11.064042 .

D. D. Wen , 等 . Helicity multiplexed broadband metasurface holograms . Nat. Commun. , 2015 . 6 8241 DOI:10.1038/ncomms9241http://doi.org/10.1038/ncomms9241 .

J. X. Li , 等 . Addressable metasurfaces for dynamic holography and optical information encryption . Sci. Adv. , 2018 . 4 eaar6768 DOI:10.1126/sciadv.aar6768http://doi.org/10.1126/sciadv.aar6768 .

Y. Q. Hu , 等 . All-dielectric metasurfaces for polarization manipulation: principles and emerging applications . Nanophotonics , 2020 . 9 3755 -3780 . DOI:10.1515/nanoph-2020-0220http://doi.org/10.1515/nanoph-2020-0220 .

A. Arbabi , 等 . Dielectric metasurfaces for complete control of phase and polarization with subwavelength spatial resolution and high transmission . Nat. Nanotechnol. , 2015 . 10 937 -943 . DOI:10.1038/nnano.2015.186http://doi.org/10.1038/nnano.2015.186 .

Y. Q. Hu , 等 . Trichromatic and tripolarization-channel holography with noninterleaved dielectric metasurface . Nano Lett. , 2020 . 20 994 -1002 . DOI:10.1021/acs.nanolett.9b04107http://doi.org/10.1021/acs.nanolett.9b04107 .

L. Mandel , E. Wolf . Some properties of coherent light . J. Optical Soc. Am. , 1961 . 51 815 -819 . DOI:10.1364/JOSA.51.000815http://doi.org/10.1364/JOSA.51.000815 .

E. Wolf , E. W. Marchand . Comparison of the kirchhoff and the Rayleigh-Sommerfeld theories of diffraction at an aperture . J. Optical Soc. Am. , 1964 . 54 587 -594 . DOI:10.1364/JOSA.54.000587http://doi.org/10.1364/JOSA.54.000587 .

N. A. Rubin , 等 . Matrix Fourier optics enables a compact full-Stokes polarization camera . Science , 2019 . 365 eaax1839 DOI:10.1126/science.aax1839http://doi.org/10.1126/science.aax1839 .

Y. Lecun , 等 . Gradient-based learning applied to document recognition . Proc. IEEE , 1998 . 86 2278 -2324 . DOI:10.1109/5.726791http://doi.org/10.1109/5.726791 .

H. Xiao , K. Rasul , R. Vollgraf . Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms . arXiv , 2017 . 1708 07747 .

Z. Wang , 等 . Integrated photonic metasystem for image classifications at telecommunication wavelength . Nat. Commun. , 2022 . 13 2131 DOI:10.1038/s41467-022-29856-7http://doi.org/10.1038/s41467-022-29856-7 .

B. Wang , 等 . Visible-frequency dielectric metasurfaces for multiwavelength achromatic and highly dispersive holograms . Nano Lett. , 2016 . 16 5235 -5240 . DOI:10.1021/acs.nanolett.6b02326http://doi.org/10.1021/acs.nanolett.6b02326 .

J. Deng , 等 . Spatial frequency multiplexed meta-holography and meta-nanoprinting . ACS Nano , 2019 . 13 9237 -9246 . DOI:10.1021/acsnano.9b03738http://doi.org/10.1021/acsnano.9b03738 .

Y. J. Bao , J. C. Ni , C. W. Qiu . A minimalist single-layer metasurface for arbitrary and full control of vector vortex beams . Adv. Mater. , 2020 . 32 1905659 DOI:10.1002/adma.201905659http://doi.org/10.1002/adma.201905659 .

Y. Q. Hu , 等 . 3D-integrated metasurfaces for full-colour holography . Light.: Sci. Appl. , 2019 . 8 86 DOI:10.1038/s41377-019-0198-yhttp://doi.org/10.1038/s41377-019-0198-y .

X. H. Luo , 等 . Integrated metasurfaces with microprints and helicity-multiplexed holograms for real-time optical encryption . Adv. Optical Mater. , 2020 . 8 1902020 DOI:10.1002/adom.201902020http://doi.org/10.1002/adom.201902020 .

Y. Q. Hu , 等 . Electrically tunable multifunctional polarization-dependent metasurfaces integrated with liquid crystals in the visible region . Nano Lett. , 2021 . 21 4554 -4562 . DOI:10.1021/acs.nanolett.1c00104http://doi.org/10.1021/acs.nanolett.1c00104 .

S. Q. Li , 等 . Phase-only transmissive spatial light modulator based on tunable dielectric metasurface . Science , 2019 . 364 1087 -1090 . DOI:10.1126/science.aaw6747http://doi.org/10.1126/science.aaw6747 .

J. K. George , 等 . Neuromorphic photonics with electro-absorption modulators . Opt. Express , 2019 . 27 5181 -5191 . DOI:10.1364/OE.27.005181http://doi.org/10.1364/OE.27.005181 .

M. Akie , 等 . GeSn/SiGeSn multiple-quantum-well electroabsorption modulator with taper coupler for mid-infrared Ge-on-Si platform . IEEE J. Sel. Top. Quantum Electron. , 2018 . 24 3400208 DOI:10.1109/JSTQE.2018.2827673http://doi.org/10.1109/JSTQE.2018.2827673 .

Y. Z. Xiao , H. L. Qian , Z. W. Liu . Nonlinear metasurface based on giant optical Kerr response of gold quantum wells . ACS Photonics , 2018 . 5 1654 -1659 . DOI:10.1021/acsphotonics.7b01140http://doi.org/10.1021/acsphotonics.7b01140 .

D. N. Christodoulides , 等 . Theory of incoherent self-focusing in biased photorefractive media . Phys. Rev. Lett. , 1997 . 78 646 DOI:10.1103/PhysRevLett.78.646http://doi.org/10.1103/PhysRevLett.78.646 .

P. del Hougne , 等 . Learned integrated sensing pipeline: reconfigurable metasurface transceivers as trainable physical layer in an artificial neural network . Adv. Sci. , 2020 . 7 1901913 DOI:10.1002/advs.201901913http://doi.org/10.1002/advs.201901913 .

H. Y. Li , 等 . Intelligent electromagnetic sensing with learnable data acquisition and processing . Patterns , 2020 . 1 100006 DOI:10.1016/j.patter.2020.100006http://doi.org/10.1016/j.patter.2020.100006 .

C. Saigre-Tardif , 等 . Intelligent meta-imagers: from compressed to learned sensing . Appl. Phys. Rev. , 2022 . 9 011314 DOI:10.1063/5.0076022http://doi.org/10.1063/5.0076022 .

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰