LOEN: Lensless opto-electronic neural network empowered machine vision

Wanxin Shi; Zheng Huang; Honghao Huang; Chengyang Hu; Minghua Chen; Sigang Yang; Hongwei Chen

doi:10.1038/s41377-022-00809-5

Go to Nature Website

Your Location：

Home >

Browse articles >

LOEN: Lensless opto-electronic neural network empowered machine vision

Articles

- LOEN: Lensless opto-electronic neural network empowered machine vision
- Light: Science & Applications Vol. 11, Issue 6, Pages: 1118-1129(2022)
- Affiliations：
  
  Beijing National Research Center for Information Science and Technology (BNRist), Department of Electronic Engineering, Tsinghua University, Beijing, 100084 China
- Author bio：
  
  Hongwei Chen (chenhw@tsinghua.edu.cn)
- Funds：
- DOI：10.1038/s41377-022-00809-5
  CLC：
- Published：30 June 2022，
  
  Published Online：04 May 2022，
  
  Received：30 October 2021，
  
  Revised：15 April 2022，
  
  Accepted：20 April 2022
- Accepted：
Scan QR Code
Shi, W. X. et al. LOEN: Lensless opto-electronic neural network empowered machine vision. Light: Science & Applications, 11, 1118-1129 (2022).
DOI：

Shi, W. X. et al. LOEN: Lensless opto-electronic neural network empowered machine vision. Light: Science & Applications, 11, 1118-1129 (2022). DOI： 10.1038/s41377-022-00809-5.

摘要

Abstract

Machine vision faces bottlenecks in computing power consumption and large amounts of data. Although opto-electronic hybrid neural networks can provide assistance

they usually have complex structures and are highly dependent on a coherent light source; therefore

they are not suitable for natural lighting environment applications. In this paper

we propose a novel lensless opto-electronic neural network architecture for machine vision applications. The architecture optimizes a passive optical mask by means of a task-oriented neural network design

performs the optical convolution calculation operation using the lensless architecture

and reduces the device size and amount of calculation required. We demonstrate the performance of handwritten digit classification tasks with a multiple-kernel mask in which accuracies of as much as 97.21% were achieved. Furthermore

we optimize a large-kernel mask to perform optical encryption for privacy-protecting face recognition

thereby obtaining the same recognition accuracy performance as no-encryption methods. Compared with the random MLS pattern

the recognition accuracy is improved by more than 6%.

关键词

Keywords

references

LeCun, Y., Bengio, Y.&Hinton, G. Deep learning.Nature521, 436–444 (2015)..

Krizhevsky, A., Sutskever, I.&Hinton, G. E. Imagenet classification with deep convolutional neural networks. InProceedings of the 25th International Conference on Neural Information Processing System. 1097–1105 (Lake Tahoe, Nevada: Curran Associates Inc, 2012).

He, K. M. et al. Deep residual learning for image recognition. InProceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 770–778 (Las Vegas, NV, USA: IEEE, 2016).

Zhu, X. X. et al. Deep learning in remote sensing: a comprehensive review and list of resources.IEEE Geosci. Remote Sens. Mag.5, 8–36 (2017)..

Rivenson, Y. et al. Deep learning microscopy.Optica4, 1437–1443 (2017)..

Young, T., Hazarika, D., Poria, S.&Cambria, E. Recent trends in deep learning based natural language processing.ieee Comput. Intell. Mag.13, 55–75 (2018)..

Rivenson, Y., Wu, Y. C.&Ozcan, A. Deep learning in holography and coherent imaging.Light Sci. Appl.8, 85 (2019)..

Al-Qizwini, M. et al. Deep learning algorithm for autonomous driving using GoogLeNet. InProceedings of 2017 IEEE Intelligent Vehicles Symposium (IV), 89–96 (Los Angeles, CA, USA: IEEE, 2017).

Shi, Q. F. et al. Deep learning enabled smart mats as a scalable floor monitoring system.Nat. Commun.11, 4609 (2020)..

Xiong, H. Y. et al. The human splicing code reveals new insights into the genetic determinants of disease.Science347, 144 (2015)..

Helmstaedter, M. et al. Connectomic reconstruction of the inner plexiform layer in the mouse retina.Nature500, 168–174 (2013)..

Li, J. X. et al. Class-specific differential detection in diffractive optical neural networks improves inference accuracy.Adv. Photonics1, 046001 (2019)..

Luo, Y. et al. Design of task-specific optical systems using broadband diffractive neural networks.Light Sci. Appl.8, 112 (2019)..

Mengu, D. et al. Misalignment resilient diffractive optical networks.Nanophotonics9, 4207–4219 (2020)..

Kulce, O. et al. All-optical information-processing capacity of diffractive surfaces.Light Sci. Appl.10, 25 (2021)..

Rahman, M. S. S. et al. Ensemble learning of diffractive optical networks.Light Sci. Appl.10, 14 (2021)..

Lin, X. et al. All-optical machine learning using diffractive deep neural networks.Science361, 1004–1008 (2018)..

Li, J. X. et al. Spectrally encoded single-pixel machine vision using diffractive networks.Sci. Adv.7, eabd7690 (2021)..

Goi, E. et al. Nanoprinted high-neuron-density optical linear perceptrons performing near-infrared inference on a CMOS chip.Light Sci. Appl.10, 40 (2021)..

Wetzstein, G. et al. Inference in artificial intelligence with deep optics and photonics.Nature588, 39–47 (2020)..

Yan, T. et al. Fourier-space diffractive deep neural network.Phys. Rev. Lett.123, 023901 (2019)..

Shen, Y. C. et al. Deep learning with coherent nanophotonic circuits.Nat. Photonics11, 441–446 (2017)..

Harris, N. C. et al. Linear programmable nanophotonic processors.Optica5, 1623–1631 (2018)..

Zhang, Q. M. et al. Artificial neural networks enabled by nanophotonics.Light Sci. Appl.8, 42 (2019)..

Harris, N. C. et al. Quantum transport simulations in a programmable nanophotonic processor.Nat. Photonics11, 447–452 (2017)..

Feldmann, J. et al. All-optical spiking neurosynaptic networks with self-learning capabilities.Nature569, 208–214 (2019)..

Xu, X. Y. et al. 11 TOPS photonic convolutional accelerator for optical neural networks.Nature589, 44–51 (2021)..

Peng, H. T. et al. Neuromorphic photonic integrated circuits.IEEE J. Sel. Top. Quantum Electron.24, 6101715 (2018)..

Xu, X. Y. et al. Photonic perceptron based on a Kerr Microcomb for high‐speed, scalable, optical neural networks.Laser Photonics Rev.14, 2000070 (2020)..

Chang, J. L. et al. Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification.Sci. Rep.8, 12324 (2018)..

Zhou, T. K. et al. Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit.Nat. Photonics15, 367–373 (2021)..

Chen, H. G. et al. ASP Vision: optically computing the first layer of convolutional neural networks using angle sensitive pixels. InProceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 903–912 (Las Vegas, NV, USA: IEEE, 2016).

Miscuglio, M. et al. Massively parallel amplitude-only Fourier neural network.Optica7, 1812–1819 (2020)..

Pad, P. et al. Efficient neural vision systems based on convolutional image acquisition. InProceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 12282–12291 (Seattle, WA, USA: IEEE, 2020).

LiKamWa, R. et al. Energy characterization and optimization of image sensing toward continuous mobile vision. InProceeding of the 11th Annual International Conference on Mobile Systems, Applications, and Services, 69–82 (Taipei, China: ACM, 2013).

LiKamWa, R. et al. RedEye: analog ConvNet image sensor architecture for continuous mobile vision.ACM SIGARCH Computer Architecture N.44, 255–266 (2016)..

Asif, M. S. et al. FlatCam: Thin, lensless cameras using coded aperture and computation.IEEE Trans. Comput. Imaging3, 384–397 (2017)..

Khan, S. S. et al. FlatNet: towards photorealistic scene reconstruction from lensless measurements.IEEE Trans. Pattern Anal. Mach. Intell.44, 1934–1948 (2020)..

Hinojosa, C., Niebles, J. C.&Arguello, H. Learning privacy-preserving optics for human pose estimation. InProceedings of 2021 IEEE/CVF International Conference on Computer Vision, 2553–2562 (Montreal, QC, Canada: IEEE, 2021).

Dejonckheere, A. et al. All-optical reservoir computer based on saturation of absorption.Opt. Express22, 10868–10881 (2014)..

Lim, G. K. et al. Giant broadband nonlinear optical absorption response in dispersed graphene single sheets.Nat. Photonics5, 554–560 (2011)..

Miscuglio, M. et al. All-optical nonlinear activation function for photonic neural networks [Invited].Optical Mater. Express8, 3851–3863 (2018)..

Cheng, K. T. et al. Electrically switchable and permanently stable light scattering modes by dynamic fingerprint chiral textures.ACS Appl. Mater. Interfaces8, 10483–10493 (2016)..

Ke, Y. J. et al. Smart windows: electro-, thermo-, mechano-, photochromics, and beyond.Adv Energy Mater9, 1902066 (2019)..

Van der Asdonk, P.&Kouwer, P. H. J. Liquid crystal templating as an approach to spatially and temporally organise soft matter.Chem. Soc. Rev.46, 5935–5949 (2017)..

Li, Z. L. et al. Dielectric meta-holograms enabled with dual magnetic resonances in visible light.ACS Nano11, 9382–9389 (2017)..

Kim, I. et al. Outfitting next generation displays with optical metasurfaces.ACS Photonics5, 3876–3895 (2018)..

Hu, C. Y. et al. Video object detection from one single image through opto-electronic neural network.APL Photonics6, 046104 (2021)..

Hu, C. Y. et al. FourierCam: a camera for video spectrum acquisition in a single shot.Photonics Res.9, 701–713 (2021)..

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰