A multichannel optical computing architecture for advanced machine vision

Zhihao Xu; Xiaoyun Yuan; Tiankuang Zhou; Lu Fang

doi:10.1038/s41377-022-00945-y

Go to Nature Website

Your Location：

Home >

Browse articles >

A multichannel optical computing architecture for advanced machine vision

Original Articles

- A multichannel optical computing architecture for advanced machine vision
- Light: Science & Applications Vol. 11, Issue 9, Pages: 2235-2247(2022)
- Affiliations：
  
  1.Sigma Laboratory, Department of Electronic Engineering, Tsinghua University, Beijing, China
  2.Beijing National Research Center for Information Science and Technology (BNRist), Beijing, China
  3.Tsinghua Shenzhen International Graduate School, Shenzhen, China
  4.Institute for Brain and Cognitive Science, Tsinghua University (THUIBCS), Beijing, China
- Author bio：
  
  Lu Fang (fanglu@tsinghua.edu.cn)
- Funds：
- DOI：10.1038/s41377-022-00945-y
  CLC：
Scan for full text
Xu, Z. H. et al. A multichannel optical computing architecture for advanced machine vision. Light: Science & Applications, 11, 2235-2247 (2022).
DOI：

Xu, Z. H. et al. A multichannel optical computing architecture for advanced machine vision. Light: Science & Applications, 11, 2235-2247 (2022). DOI： 10.1038/s41377-022-00945-y.

摘要

Abstract

Endowed with the superior computing speed and energy efficiency, optical neural networks (ONNs) have attracted ever-growing attention in recent years. Existing optical computing architectures are mainly single-channel due to the lack of advanced optical connection and interaction operators, solving simple tasks such as hand-written digit classification, saliency detection, etc. The limited computing capacity and scalability of single-channel ONNs restrict the optical implementation of advanced machine vision. Herein, we develop Monet: a multichannel optical neural network architecture for a universal multiple-input multiple-channel optical computing based on a novel projection-interference-prediction framework where the inter- and intra- channel connections are mapped to optical interference and diffraction. In our Monet, optical interference patterns are generated by projecting and interfering the multichannel inputs in a shared domain. These patterns encoding the correspondences together with feature embeddings are iteratively produced through the projection-interference process to predict the final output optically. For the first time, Monet validates that multichannel processing properties can be optically implemented with high-efficiency, enabling real-world intelligent multichannel-processing tasks solved via optical computing, including 3D/motion detections. Extensive experiments on different scenarios demonstrate the effectiveness of Monet in handling advanced machine vision tasks with comparative accuracy as the electronic counterparts yet achieving a ten-fold improvement in computing efficiency. For intelligent computing, the trends of dealing with real-world advanced tasks are irreversible. Breaking the capacity and scalability limitations of single-channel ONN and further exploring the multichannel processing potential of wave optics, we anticipate that the proposed technique will accelerate the development of more powerful optical AI as critical support for modern advanced machine vision.

关键词

Keywords

references

Y. LeCun, Y. Bengio, G. Hinton, 等. Deep learning. Nature, 2015. 521436-444. DOI:10.1038/nature14539http://doi.org/10.1038/nature14539.

D. Silver, 等. Mastering the game of Go with deep neural networks and tree search. Nature, 2016. 529484-489. DOI:10.1038/nature16961http://doi.org/10.1038/nature16961.

Beohar, D. & Rasool, A. Handwritten Digit Recognition of MNIST dataset using Deep Learning state-of-the-art Artificial Neural Network (ANN) and Convolutional Neural Network (CNN). In 2021 International Conference on Emerging Smart Computing and Informatics (ESCI). 542–548 (IEEE, 2021).

Pan, J., Sayrol, E., Giro-i-Nieto, X., McGuinness, K. & O'Connor, N. E. Shallow and deep convolutional networks for saliency prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 598–606 (2016).

W. Wang, J. Shen. Deep visual attention prediction. IEEE Trans. Image Process., 2017. 272368-2378. DOI:10.1109/TIP.2017.2787612http://doi.org/10.1109/TIP.2017.2787612.

Gu, X. et al. Cascade cost volume for high-resolution multi-view stereo and stereo matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2495–2504 (2020).

Chen, R., Han, S., Xu, J. & Su, H. Point-based multi-view stereo network. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 1538–1547 (2019).

Zbontar, J. & LeCun, Y. Computing the stereo matching cost with a convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1592–1599 (2015).

Kendall, A. et al. End-to-end learning of geometry and context for deep stereo regression. In Proceedings of the IEEE International Conference on Computer Vision. 66–75 (2017).

P. W. Patil, S. Murala. MSFgNet: A novel compact end-to-end deep network for moving object detection. IEEE Trans. Intell. Transportation Syst., 2018. 204066-4077. DOI:10.1109/TITS.2018.2880096http://doi.org/10.1109/TITS.2018.2880096.

Yang, Y., Loquercio, A., Scaramuzza, D. & Soatto, S. Unsupervised moving object detection via contextual information separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 879–888 (2019).

Zhang, C. et al. Optimizing FPGA-based accelerator design for deep convolutional neural networks. In Proceedings of the 2015 ACM/SIGDA international symposium on field-programmable gate arrays. 161–170 (2015).

Qiu, J. et al. Going deeper with embedded FPGA platform for convolutional neural network. In Proceedings of the 2016 ACM/SIGDA International Symposium on Field-Programmable Gate Arraysp. 26–35 (2016).

P. A. Merolla, 等. A million spiking-neuron integrated circuit with a scalable communication network and interface. Science, 2014. 345668-673. DOI:10.1126/science.1254642http://doi.org/10.1126/science.1254642.

J. Pei, 等. Towards artificial general intelligence with hybrid Tianjic chip architecture. Nature, 2019. 572106-111. DOI:10.1038/s41586-019-1424-8http://doi.org/10.1038/s41586-019-1424-8.

M. M. Waldrop. The chips are down for Moore's law. Nat. N., 2016. 530144DOI:10.1038/530144ahttp://doi.org/10.1038/530144a.

X. Lin, 等. All-optical machine learning using diffractive deep neural networks. Science, 2018. 3611004-1008. DOI:10.1126/science.aat8084http://doi.org/10.1126/science.aat8084.

J. Feldmann, N. Youngblood, C. D. Wright, H. Bhaskaran, W. H. Pernice. All-optical spiking neurosynaptic networks with self-learning capabilities. Nature, 2019. 569208-214. DOI:10.1038/s41586-019-1157-8http://doi.org/10.1038/s41586-019-1157-8.

Y. Shen, 等. Deep learning with coherent nanophotonic circuits. Nat. Photonics, 2017. 11441-446. DOI:10.1038/nphoton.2017.93http://doi.org/10.1038/nphoton.2017.93.

T. Zhou, 等. Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit. Nat. Photonics, 2021. 15367-373. DOI:10.1038/s41566-021-00796-whttp://doi.org/10.1038/s41566-021-00796-w.

T. Yan, 等. Fourier-space diffractive deep neural network. Phys. Rev. Lett., 2019. 123023901DOI:10.1103/PhysRevLett.123.023901http://doi.org/10.1103/PhysRevLett.123.023901.

M. Miscuglio, 等. Massively parallel amplitude-only Fourier neural network. Optica, 2020. 71812-1819. DOI:10.1364/OPTICA.408659http://doi.org/10.1364/OPTICA.408659.

J. Chang, V. Sitzmann, X. Dun, W. Heidrich, G. Wetzstein. Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification. Sci. Rep., 2018. 81-10. .

X. Xu, 等. 11 TOPS photonic convolutional accelerator for optical neural networks. Nature, 2021. 58944-51. DOI:10.1038/s41586-020-03063-0http://doi.org/10.1038/s41586-020-03063-0.

Prucnal, P. R., Shastri, B. J. & Teich, M. C. Neuromorphic photonics. (CRC Press, 2017).

Q. Zhang, H. Yu, M. Barbiero, B. Wang, M. Gu. Artificial neural networks enabled by nanophononics. Light Sci. Appl., 2019. 81-14. DOI:10.1038/s41377-018-0109-7http://doi.org/10.1038/s41377-018-0109-7.

J. Li, 等. Spectrally encoded single-pixel machine vision using diffractive networks. Sci. Adv., 2021. 7eabd7690DOI:10.1126/sciadv.abd7690http://doi.org/10.1126/sciadv.abd7690.

Metzler, C. A., Ikoma, H., Peng, Y. & Wetzstein, G. Deep optics for single-shot high-dynamic-range imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1375–1385 (2020).

Chang, J. & Wetzstein, G. Deep optics for monocular depth estimation and 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 10193–10202 (2019).

P. Antonik, N. Marsal, D. Brunner, D. Rontani. Human action recognition with a large-scale brain-inspired photonic computer. Nat. Mach. Intell., 2019. 1530-537. DOI:10.1038/s42256-019-0110-8http://doi.org/10.1038/s42256-019-0110-8.

Y. Luo, 等. Design of task-specific optical systems using broadband diffractive neural networks. Light Sci. Appl., 2019. 81-14. DOI:10.1038/s41377-019-0223-1http://doi.org/10.1038/s41377-019-0223-1.

Yao, Y., Luo, Z., Li, S., Fang, T. & Quan, L. Mvsnet: Depth inference for unstructured multi-view stereo. In Proceedings of the European Conference on Computer Vision (ECCV). 767–783 (2018).

Quan, D. et al. AFD-Net: Aggregated feature difference learning for cross-spectral image patch matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 3017–3026 (2019).

Luo, K. et al. Attention-aware multi-view stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1590–1599 (2020).

Daudt, R. C., Le Saux, B. & Boulch, A. Fully convolutional siamese networks for change detection. In 2018 25th IEEE International Conference on Image Processing (ICIP)p. 4063–4067 (IEEE, 2018).

Y. LeCun, Y. Bengio. Convolutional networks for images, speech, and time series. Handb. brain theory neural Netw., 1995. 33611995.

LeCun, Y., Kavukcuoglu, K. & Farabet, C. Convolutional networks and applications in vision. In Proceedings of 2010 IEEE international symposium on circuits and systems. 253–256 (IEEE, 2010).

Long, J., Shelhamer, E. & Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3431–3440 (2015).

Bertinetto, L., Valmadre, J., Henriques, J. F., Vedaldi, A. & Torr, P. H. Fully-convolutional siamese networks for object tracking. In European Conference on Computer Vision. 850–865 (Springer, Cham, 2016).

Nguyen, T., Raghu, M. & Kornblith, S. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. arXiv preprint arXiv: 2010.15327 (2020).

Lu, Z., Pu, H., Wang, F., Hu, Z. & Wang, L. The expressive power of neural networks: A view from the width. Advances in Neural Information Processing Systems, 30 (2017).

O'Mahony, N. et al. Computer vision for 3D perception. In Proceedings of SAI Intelligent Systems Conference (pp. 788–804). (Springer, Cham, 2018).

Q. An, Y. Wang, Y. Shen. Sensor deployment for visual 3D perception: A perspective of information gains. IEEE Sens. J., 2021. 218464-8478. DOI:10.1109/JSEN.2021.3050325http://doi.org/10.1109/JSEN.2021.3050325.

Liu, J. & Ji, S. A novel recurrent encoder-decoder structure for large-scale multi-view stereo reconstruction from an open aerial dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6050–6059) (2020).

Isola, P., Zhu, J. Y., Zhou, T. & Efros, A. A. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1125–1134) (2017).

Barron, J. T. & Poole, B. The fast bilateral solver. In European Conference on Computer Vision (pp. 617–632). (Springer, Cham, 2016).

Chauhan, A. K. & Kumar, D. Study of moving object detection and tracking for video surveillance. International Journal of Advanced Research in Computer Science and Software Engineering, 3 (2013).

Kulchandani, J. S. & Dangarwala, K. J. Moving object detection: Review of recent research trends. In 2015 International conference on pervasive computing (ICPC) (pp. 1–5). (IEEE 2015).

K. A. Joshi, D. G. Thakore. A survey on moving object detection and tracking in video surveillance system. Int. J. Soft Comput. Eng., 2012. 244-48. .

Zhu, P. et al. Vision meets drones: Past, present and future (2020).

Mandal, M., Kumar, L. K. & Saran, M. S. MotionRec: A unified deep framework for moving object recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2734–2743 (2020).

Vertens, J., Valada, A. & Burgard, W. Smsnet: Semantic motion segmentation using deep convolutional neural networks. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 582–589 (IEEE, 2017).

Stentz, A. J. & Boyd, R. W. Nonlinear optics. The Handbook of Photonics, 5–1 (1997).

D. N. Christodoulides, T. H. Coskun, M. Mitchell, M. Segev. Theory of incoherent self-focusing in biased photorefractive media. Phys. Rev. Lett., 1997. 78646DOI:10.1103/PhysRevLett.78.646http://doi.org/10.1103/PhysRevLett.78.646.

L. Waller, G. Situ, J. W. Fleischer. Phase-space measurement and coherence synthesis of optical beams. Nat. Photonics, 2012. 6474-479. DOI:10.1038/nphoton.2012.144http://doi.org/10.1038/nphoton.2012.144.

A. Telea. An image inpainting technique based on the fast marching method. J. Graph. Tools, 2004. 923-34. DOI:10.1080/10867651.2004.10487596http://doi.org/10.1080/10867651.2004.10487596.

Rahman, M. A. & Wang, Y. Optimizing intersection-over-union in deep neural networks for image segmentation. In International Symposium on Visual Computing. 234–244 (Springer, Cham, 2016).

Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980 (2014).

D. G. Lowe. Distinctive image features from scale-invariant keypoints. Int. J. Computer Vis., 2004. 6091-110. DOI:10.1023/B:VISI.0000029664.99615.94http://doi.org/10.1023/B:VISI.0000029664.99615.94.

C. Rother, V. Kolmogorov, A. Blake. "GrabCut" interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. (TOG), 2004. 23309-314. DOI:10.1145/1015706.1015720http://doi.org/10.1145/1015706.1015720.

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰