Photonic neuromorphic architecture for tens-of-task lifelong learning

Yuan Cheng; Jianing Zhang; Tiankuang Zhou; Yuyan Wang; Zhihao Xu; Xiaoyun Yuan; Lu Fang

doi:10.1038/s41377-024-01395-4

Your Location：

Home >

Browse articles >

Photonic neuromorphic architecture for tens-of-task lifelong learning

Original Articles

- Photonic neuromorphic architecture for tens-of-task lifelong learning
- Light: Science & Applications Vol. 13, Issue 3, Pages: 519-530(2024)
- Affiliations：
  
  1.Sigma Laboratory, Department of Electronic Engineering, Tsinghua University, Beijing 100084, China
  2.Beijing National Research Center for Information Science and Technology (BNRist), Beijing 100084, China
  3.Institute for Brain and Cognitive Science, Tsinghua University (THUIBCS), Beijing 100084, China
- Author bio：
  
  Lu Fang (fanglu@tsinghua.edu.cn)
- Funds：
- DOI：10.1038/s41377-024-01395-4
  CLC：
- Received：05 September 2023，
  
  Revised：08 January 2024，
  
  Accepted：2024-01-24，
  
  Published Online：26 February 2024，
  
  Published：31 March 2024
- Accepted：
Scan QR Code
Cheng, Y. et al. Photonic neuromorphic architecture for tens-of-task lifelong learning. Light: Science & Applications, 13, 519-530 (2024).
DOI：

Cheng, Y. et al. Photonic neuromorphic architecture for tens-of-task lifelong learning. Light: Science & Applications, 13, 519-530 (2024). DOI： 10.1038/s41377-024-01395-4.

摘要

Abstract

Scalable

high-capacity

and low-power computing architecture is the primary assurance for increasingly manifold and large-scale machine learning tasks. Traditional electronic artificial agents by conventional power-hungry processors have faced the issues of energy and scaling walls

hindering them from the sustainable performance improvement and iterative multi-task learning. Referring to another modality of light

photonic computing has been progressively applied in high-efficient neuromorphic systems. Here

we innovate a reconfigurable lifelong-learning optical neural network (L

ONN)

for highly-integrated tens-of-task machine intelligence with elaborated algorithm-hardware co-design. Benefiting from the inherent sparsi

ty and parallelism in massive photonic connections

ONN learns each single task by adaptively activating sparse photonic neuron connections in the coherent light field

while incrementally acquiring expertise on various tasks by gradually enlarging the activation. The multi-task optical features are parallelly processed by multi-spectrum representations allocated with different wavelengths. Extensive evaluations on free-space and on-chip architectures confirm that for the first time

ONN avoided the catastrophic forgetting issue of photonic computing

owning versatile skills on challenging tens-of-tasks (vision classification

voice recognition

medical diagnosis

etc.) with a single model. Particularly

ONN achieves more than an order of magnitude higher efficiency than the representative electronic artificial neural networks

and 14× larger capacity than existing optical neural networks while maintaining competitive performance on each individual task. The proposed photonic neuromorphic architecture points out a new form of lifelong learning scheme

permitting terminal/edge AI systems with light-speed efficiency and unprecedented scalability.

关键词

Keywords

references

Geiger, A. et al. Vision meets robotics: the kitti dataset. Int. J. Robot. Res. 32 , 1231–1237 (2013)..

Wang, X. Y. et al. Panda: a gigapixel-level human-centric video dataset. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle, WA, USA: IEEE, 3265–3275, (2020).

Cordts, M. et al. The cityscapes dataset for semantic urban scene understanding. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas, NV, USA: IEEE, 3213-3223, (2016).

Chang, X., Bian, L. & Zhang, J. Large-scale phase retrieval. eLight 1 , 1–12 (2021)..

Sarker, I. H. Machine learning: algorithms, real-world applications and research directions. SN Comput. Sci. 2 , 160 (2021)..

Schuman, C. D. et al. Opportunities for neuromorphic computing algorithms and applications. Nat. Comput. Sci. 2 , 10–19 (2022)..

Li, C. et al. Analogue signal and image processing with large memristor crossbars. Nat. Electron. 1 , 52–59 (2018)..

Weng, T. W. et al. Evaluating the robustness of neural networks: An extreme value theory approach. 6th International Conference on Learning Representations . Vancouver, BC, Canada: OpenReview. net, (2018).

Waldrop, M. M. The chips are down for Moore's law. Nature 530 , 144–147 (2016)..

Cheng, Y. et al. S3-Net: a fast scene understanding network by single-shot segmentation for autonomous driving. ACM Trans. Intell. Syst. Technol. 12 , 58 (2021)..

Zhen, P. et al. Fast video facial expression recognition by a deeply tensor-compressed LSTM neural network for mobile devices. ACM Trans. Internet Things 2 , 4 (2021)..

Cheng, Y. et al. DEEPEYE: A deeply tensor-compressed neural network for video comprehension on terminal devices. ACM Trans. Embedded Comput. Syst. 19 , 18 (2020)..

Yuan, X. Y. et al. A modular hierarchical array camera. Light Sci. Appl. 10 , 37 (2021)..

Cheng, Y. et al. An anomaly comprehension neural network for surveillance videos on terminal devices. 2020 Design, Automation&Test in Europe Conference&Exhibition. Grenoble, France: IEEE, 1396–1401, (2020).

Shastri, B. J. et al. Photonics for artificial intelligence and neuromorphic computing. Nat. Photonics 15 , 102–114 (2021)..

Zhang, Q. M. et al. Artificial neural networks enabled by nanophotonics. Light Sci. Appl. 8 , 42 (2019)..

Zhou, T. K. et al. Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit. Nat. Photonics 15 , 367–373 (2021)..

Yuan, X. Y. et al. Training large-scale optoelectronic neural networks with dual-neuron optical-artificial learning. Nat. Commun. 14 , 1 (2023)..

Zhu, T. F. et al. Plasmonic computing of spatial differentiation. Nat. Commun. 8 , 15391 (2017)..

Zhou, T. K. et al. Ultrafast dynamic machine vision with spatiotemporal photonic computing. Sci. Adv. 9 , 23 (2023)..

Xu, Z. H. et al. A multichannel optical computing architecture for advanced machine vision. Light Sci. Appl. 11 , 255 (2022)..

Li, J. X. et al. Spectrally encoded single-pixel machine vision using diffractive networks. Sci. Adv. 7 , eabd7690 (2021)..

Li, Y. et al. Quantitative phase imaging (QPI) through random diffusers using a diffractive optical network. Light. Adv. Manuf. 4 , 19 (2023)..

Zhu, Y. et al. Metasurfaces designed by a bidirectional deep neural network and iterative algorithm for generating quantitative field distributions. Light. Adv. Manuf. 4 , 9 (2023)..

Luo, Y. et al. Computational imaging without a computer: seeing through random diffusers at the speed of light. eLight 2 , 4 (2022)..

Lin, H. & Cheng, J. -X. Computational coherent Raman scattering imaging: breaking physical barriers by fusion of advanced instrumentation and data science. eLight 3 , 6 (2023)..

Pan, J. T. et al. Shallow and deep convolutional networks for saliency prediction. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas, NV, USA: IEEE, 598–606, (2016).

Feldmann, J. et al. All-optical spiking neurosynaptic networks with self-learning capabilities. Nature 569 , 208–214 (2019)..

Xu, X. Y. et al. 11 TOPS photonic convolutional accelerator for optical neural networks. Nature 589 , 44–51 (2021)..

Yan, T. et al. Fourier-space diffractive deep neural network. Phys. Rev. Lett. 123 , 023901 (2019)..

Miscuglio, M. et al. M assively parallel amplitude-only Fourier neural network. Optica 7 , 1812–1819 (2020)..

Chang, J. L. & Wetzstein, G. Deep optics for monocular depth estimation and 3D object detection. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision . Seoul, Korea (South): IEEE, 10192–10201, (2019).

Metzler, C. A. et al. Deep optics for single-shot high-dynamic-range imaging. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle, WA, USA: IEEE, 1372–1382, (2020).

Situ, G. H. Deep holography. Light. Adv. Manuf. 3 , 8 (2022)..

Chang, J. L. et al. Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification. Sci. Rep. 8 , 12324 (2018)..

McCloskey, M. & Cohen, N. J. Catastrophic interference in connectionist networks: the sequential learning problem. Psychol. Learn. Motiv. 24 , 109–165 (1989)..

Ratcliff, R. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psycholog. Rev. 97 , 285–308 (1990)..

McClelland, J. L., McNaughton, B. L. & O'Reilly, R. C. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psycholog. Rev. 102 , 419–457 (1995)..

Parisi, G. I. et al. Continual lifelong learning with neural networks: A review. Neural Netw. 113 , 54–71 (2019)..

Hong, X. B. et al. Lifelong machine learning: outlook and direction. Proceedings of the 2nd International Conference on Big Data Research . Weihai China: ACM, 76–79, (2018).

Valdés-Sosa, P. A. et al. Estimating brain functional connectivity with sparse multivariate autoregression. Philos. Trans. R. Soc. B: Biol. Sci. 360 , 969–981 (2005)..

Bassett, D. S. & Bullmore, E. Small-world brain networks. Neuroscientist 12 , 512–523 (2006)..

Ng, B. et al. A novel sparse graphical approach for multimodal brain connectivity inference. 15th International Conference on Medical Image Computing and Computer-Assisted Intervention . Nice, France: Springer, 707–714, (2012).

Mostafa, H., Müller, L. K. & Indiveri, G. An event-based architecture for solving constraint satisfaction problems. Nat. Commun. 6 , 8941 (2015)..

Amir, A.et al. A low power, fully event-based gesture recognition system. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu, HI, USA: IEEE, 7388–7397, (2017).

Connor, C. E., Egeth, H. E. & Yantis, S. Visual attention: bottom-up versus top-down. Curr. Biol. 14 , R850–R852 (2004)..

Schneider, W. X. Selective visual processing across competition episodes: a theory of task-driven visual attention and working memory. Philos. Trans. R. Soc. B: Biol. Sci. 368 , 20130060 (2013)..

Wang, T. Y. et al. An optical neural network using less than 1 photon per multiplication. Nat. Commun. 13 , 123 (2022)..

Zuo, Y. et al. Scalability of all-optical neural networks based on spatial light modulators. Phys. Rev. Appl. 15 , 054034 (2021)..

Yan, T. et al. All-optical graph representation learning using integrated diffractive photonic computing units. Sci. Adv. 8 , eabn7630 (2022)..

Brunner, D. et al. Parallel photonic information processing at gigabyte per second data rates using transient states. Nat. Commun. 4 , 1364 (2013)..

Lin, X. et al. All-optical machine learning using diffractive deep neural networks. Science 361 , 1004–1008 (2018)..

LeCun, Y. et al. Gradient-based learning applied to document recognition. Proc. IEEE 86 , 2278–2324 (1998)..

Zhang, Y. F. et al. Electrically reconfigurable non-volatile metasurface using low-loss optical phase-change material. Nat. Nanotechnol. 16 , 661–666 (2021)..

Li, P. N. et al. Reversible optical switching of highly confined phonon–polaritons with an ultrathin phase-change material. Nat. Mater. 15 , 870–875 (2016)..

Deng, L. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine. 29 , 141–142 (2012)..

Xiao, H., Rasul, K. & Vollgraf, R. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. Print at https://arxiv.org/abs/1708.07747 https://arxiv.org/abs/1708.07747 (2017).

Wang, M. & Deng, W. H. Oracle-MNIST: a realistic image dataset for benchmarking machine learning algorithms. Print at https://arxiv.org/abs/2205.09442 https://arxiv.org/abs/2205.09442 (2022).

Noever, D. & Noever, S. E. M. Overhead mnist: A benchmark satellite dataset . Print at https://arxiv.org/abs/2102.04266 https://arxiv.org/abs/2102.04266 (2021)..

Clanuwat, T. et al. Deep learning for classical japanese literature . Print at https://arxiv.org/abs/1812.01718 https://arxiv.org/abs/1812.01718 (2018)..

Fisher, R. A. Iris. UCI Machine Learning Repository . (1988). at https://doi.org/10.24432/C56C76 https://doi.org/10.24432/C56C76 URL..

Aeberhard, S. & Forina, M. Wine. UCI Machine Learning Repository . (1991). at https://doi.org/10.24432/C5PC7J https://doi.org/10.24432/C5PC7J URL..

Liu, W. Y. et al. Large-margin softmax loss for convolutional neural networks. Proceedings of the 33rd International Conference on International Conference on Machine Learning . New York, NY, USA: JMLR. org, (2016).

Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization. 3rd International Conference on Learning Representations (ICLR) . San Diego, CA, USA: ICLR, (2014).

Kudo, M. et al. Vowel. UCI Machine Learning Repository . (2017). at https://doi.org/10.24432/C5NS47 https://doi.org/10.24432/C5NS47 URL..

Warden, P. Speech commands: A dataset for limited-vocabulary speech recognition . Print at https://arxiv.org/abs/1804.03209 https://arxiv.org/abs/1804.03209 (2018)..

Becker, S. et al. Interpreting and explaining deep neural networks for classification of audio signals . Print at https://arxiv.org/abs/1807.03418v1 https://arxiv.org/abs/1807.03418v1 (2018)..

Salamon, J., Jacoby, C. & Bello, J. P. A dataset and taxonomy for urban sound research. Proceedings of the 22nd ACM International Conference on Multimedia . Orlando, FL, USA: ACM, 1041-1044, (2014).

Han, W. et al. An efficient MFCC extraction method in speech recognition. 2006 IEEE International Symposium on Circuits and Systems . Kos, Greece: IEEE, (2006).

Yang, J. C. et al. MedMNIST v2-A large-scale lightweight benchmark for 2D and 3D biomedical image classification. Scientific Data 10 , 41 (2023)..

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰