LOEN: Lensless opto-electronic neural network empowered machine vision

Wanxin Shi; Zheng Huang; Honghao Huang; Chengyang Hu; Minghua Chen; Sigang Yang; Hongwei Chen

doi:10.1038/s41377-022-00809-5

Go to Nature Website

Your Location：

Home >

Browse articles >

LOEN: Lensless opto-electronic neural network empowered machine vision

Articles

- LOEN: Lensless opto-electronic neural network empowered machine vision
- LOEN: Lensless opto-electronic neural network empowered machine vision
- LSA 2022年11卷第6期页码：1118-1129
- Affiliations：
  
  Beijing National Research Center for Information Science and Technology (BNRist), Department of Electronic Engineering, Tsinghua University, Beijing, 100084 China
- Author bio：
  
  Hongwei Chen (chenhw@tsinghua.edu.cn)
- Funds：
- DOI：10.1038/s41377-022-00809-5
  中图分类号：
- 纸质出版日期：2022-06-30，
  
  网络出版日期：2022-05-04，
  
  收稿日期：2021-10-30，
  
  修回日期：2022-04-15，
  
  录用日期：2022-04-20
- Accepted：
Scan QR Code
LOEN: Lensless opto-electronic neural network empowered machine vision[J]. LSA, 2022,11(6):1118-1129.

Shi, W. X. et al. LOEN: Lensless opto-electronic neural network empowered machine vision. Light: Science & Applications, 11, 1118-1129 (2022).
LOEN: Lensless opto-electronic neural network empowered machine vision[J]. LSA, 2022,11(6):1118-1129. DOI： 10.1038/s41377-022-00809-5.

Shi, W. X. et al. LOEN: Lensless opto-electronic neural network empowered machine vision. Light: Science & Applications, 11, 1118-1129 (2022). DOI： 10.1038/s41377-022-00809-5.

摘要

Abstract

Machine vision faces bottlenecks in computing power consumption and large amounts of data. Although opto-electronic hybrid neural networks can provide assistance

they usually have complex structures and are highly dependent on a coherent light source; therefore

they are not suitable for natural lighting environment applications. In this paper

we propose a novel lensless opto-electronic neural network architecture for machine vision applications. The architecture optimizes a passive optical mask by means of a task-oriented neural network design

performs the optical convolution calculation operation using the lensless architecture

and reduces the device size and amount of calculation required. We demonstrate the performance of handwritten digit classification tasks with a multiple-kernel mask in which accuracies of as much as 97.21% were achieved. Furthermore

we optimize a large-kernel mask to perform optical encryption for privacy-protecting face recognition

thereby obtaining the same recognition accuracy performance as no-encryption methods. Compared with the random MLS pattern

the recognition accuracy is improved by more than 6%.

关键词

Keywords

references

LeCun, Y., Bengio, Y.&Hinton, G. Deep learning.Nature521, 436–444 (2015)..

Krizhevsky, A., Sutskever, I.&Hinton, G. E. Imagenet classification with deep convolutional neural networks. InProceedings of the 25th International Conference on Neural Information Processing System. 1097–1105 (Lake Tahoe, Nevada: Curran Associates Inc, 2012).

He, K. M. et al. Deep residual learning for image recognition. InProceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 770–778 (Las Vegas, NV, USA: IEEE, 2016).

Zhu, X. X. et al. Deep learning in remote sensing: a comprehensive review and list of resources.IEEE Geosci. Remote Sens. Mag.5, 8–36 (2017)..

Rivenson, Y. et al. Deep learning microscopy.Optica4, 1437–1443 (2017)..

Young, T., Hazarika, D., Poria, S.&Cambria, E. Recent trends in deep learning based natural language processing.ieee Comput. Intell. Mag.13, 55–75 (2018)..

Rivenson, Y., Wu, Y. C.&Ozcan, A. Deep learning in holography and coherent imaging.Light Sci. Appl.8, 85 (2019)..

Al-Qizwini, M. et al. Deep learning algorithm for autonomous driving using GoogLeNet. InProceedings of 2017 IEEE Intelligent Vehicles Symposium (IV), 89–96 (Los Angeles, CA, USA: IEEE, 2017).

Shi, Q. F. et al. Deep learning enabled smart mats as a scalable floor monitoring system.Nat. Commun.11, 4609 (2020)..

Xiong, H. Y. et al. The human splicing code reveals new insights into the genetic determinants of disease.Science347, 144 (2015)..

Helmstaedter, M. et al. Connectomic reconstruction of the inner plexiform layer in the mouse retina.Nature500, 168–174 (2013)..

Li, J. X. et al. Class-specific differential detection in diffractive optical neural networks improves inference accuracy.Adv. Photonics1, 046001 (2019)..

Luo, Y. et al. Design of task-specific optical systems using broadband diffractive neural networks.Light Sci. Appl.8, 112 (2019)..

Mengu, D. et al. Misalignment resilient diffractive optical networks.Nanophotonics9, 4207–4219 (2020)..

Kulce, O. et al. All-optical information-processing capacity of diffractive surfaces.Light Sci. Appl.10, 25 (2021)..

Rahman, M. S. S. et al. Ensemble learning of diffractive optical networks.Light Sci. Appl.10, 14 (2021)..

Lin, X. et al. All-optical machine learning using diffractive deep neural networks.Science361, 1004–1008 (2018)..

Li, J. X. et al. Spectrally encoded single-pixel machine vision using diffractive networks.Sci. Adv.7, eabd7690 (2021)..

Goi, E. et al. Nanoprinted high-neuron-density optical linear perceptrons performing near-infrared inference on a CMOS chip.Light Sci. Appl.10, 40 (2021)..

Wetzstein, G. et al. Inference in artificial intelligence with deep optics and photonics.Nature588, 39–47 (2020)..

Yan, T. et al. Fourier-space diffractive deep neural network.Phys. Rev. Lett.123, 023901 (2019)..

Shen, Y. C. et al. Deep learning with coherent nanophotonic circuits.Nat. Photonics11, 441–446 (2017)..

Harris, N. C. et al. Linear programmable nanophotonic processors.Optica5, 1623–1631 (2018)..

Zhang, Q. M. et al. Artificial neural networks enabled by nanophotonics.Light Sci. Appl.8, 42 (2019)..

Harris, N. C. et al. Quantum transport simulations in a programmable nanophotonic processor.Nat. Photonics11, 447–452 (2017)..

Feldmann, J. et al. All-optical spiking neurosynaptic networks with self-learning capabilities.Nature569, 208–214 (2019)..

Xu, X. Y. et al. 11 TOPS photonic convolutional accelerator for optical neural networks.Nature589, 44–51 (2021)..

Peng, H. T. et al. Neuromorphic photonic integrated circuits.IEEE J. Sel. Top. Quantum Electron.24, 6101715 (2018)..

Xu, X. Y. et al. Photonic perceptron based on a Kerr Microcomb for high‐speed, scalable, optical neural networks.Laser Photonics Rev.14, 2000070 (2020)..

Chang, J. L. et al. Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification.Sci. Rep.8, 12324 (2018)..

Zhou, T. K. et al. Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit.Nat. Photonics15, 367–373 (2021)..

Chen, H. G. et al. ASP Vision: optically computing the first layer of convolutional neural networks using angle sensitive pixels. InProceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 903–912 (Las Vegas, NV, USA: IEEE, 2016).

Miscuglio, M. et al. Massively parallel amplitude-only Fourier neural network.Optica7, 1812–1819 (2020)..

Pad, P. et al. Efficient neural vision systems based on convolutional image acquisition. InProceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 12282–12291 (Seattle, WA, USA: IEEE, 2020).

LiKamWa, R. et al. Energy characterization and optimization of image sensing toward continuous mobile vision. InProceeding of the 11th Annual International Conference on Mobile Systems, Applications, and Services, 69–82 (Taipei, China: ACM, 2013).

LiKamWa, R. et al. RedEye: analog ConvNet image sensor architecture for continuous mobile vision.ACM SIGARCH Computer Architecture N.44, 255–266 (2016)..

Asif, M. S. et al. FlatCam: Thin, lensless cameras using coded aperture and computation.IEEE Trans. Comput. Imaging3, 384–397 (2017)..

Khan, S. S. et al. FlatNet: towards photorealistic scene reconstruction from lensless measurements.IEEE Trans. Pattern Anal. Mach. Intell.44, 1934–1948 (2020)..

Hinojosa, C., Niebles, J. C.&Arguello, H. Learning privacy-preserving optics for human pose estimation. InProceedings of 2021 IEEE/CVF International Conference on Computer Vision, 2553–2562 (Montreal, QC, Canada: IEEE, 2021).

Dejonckheere, A. et al. All-optical reservoir computer based on saturation of absorption.Opt. Express22, 10868–10881 (2014)..

Lim, G. K. et al. Giant broadband nonlinear optical absorption response in dispersed graphene single sheets.Nat. Photonics5, 554–560 (2011)..

Miscuglio, M. et al. All-optical nonlinear activation function for photonic neural networks [Invited].Optical Mater. Express8, 3851–3863 (2018)..

Cheng, K. T. et al. Electrically switchable and permanently stable light scattering modes by dynamic fingerprint chiral textures.ACS Appl. Mater. Interfaces8, 10483–10493 (2016)..

Ke, Y. J. et al. Smart windows: electro-, thermo-, mechano-, photochromics, and beyond.Adv Energy Mater9, 1902066 (2019)..

Van der Asdonk, P.&Kouwer, P. H. J. Liquid crystal templating as an approach to spatially and temporally organise soft matter.Chem. Soc. Rev.46, 5935–5949 (2017)..

Li, Z. L. et al. Dielectric meta-holograms enabled with dual magnetic resonances in visible light.ACS Nano11, 9382–9389 (2017)..

Kim, I. et al. Outfitting next generation displays with optical metasurfaces.ACS Photonics5, 3876–3895 (2018)..

Hu, C. Y. et al. Video object detection from one single image through opto-electronic neural network.APL Photonics6, 046104 (2021)..

Hu, C. Y. et al. FourierCam: a camera for video spectrum acquisition in a single shot.Photonics Res.9, 701–713 (2021)..

浏览量

Downloads

CSCD

文章被引用时，请邮件提醒。

Submit

工具集

关联资源

无透镜光电神经网络：打造极简化机器视觉

暂无数据