EventLFM: event camera integrated Fourier light field microscopy for ultrafast 3D imaging

Ruipeng Guo; Qianwan Yang; Andrew S. Chang; Guorong Hu; Joseph Greene; Christopher V. Gabel; Sixian You; Lei Tian

doi:10.1038/s41377-024-01502-5

Your Location：

Home >

Browse articles >

EventLFM: event camera integrated Fourier light field microscopy for ultrafast 3D imaging

Original Articles

- EventLFM: event camera integrated Fourier light field microscopy for ultrafast 3D imaging
- Light: Science & Applications Vol. 13, Issue 7, Pages: 1401-1415(2024)
- Affiliations：
  
  1.Department of Electrical and Computer Engineering, Boston University, Boston, MA 02215, USA
  2.Department of Physiology and Biophysics, Boston University, Boston, MA 02215, USA
  3.Neurophotonics Center, Boston University, Boston, MA 02215, USA
  4.Research Laboratory of Electronics (RLE) in the Department of Electrical Science and Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
  5.Department of Biomedical Engineering, Boston University, Boston, MA 02215, USA
- Author bio：
  
  Lei Tian (leitian@bu.edu)
- Funds：
- DOI：10.1038/s41377-024-01502-5
  CLC：
- Received：07 December 2023，
  
  Revised：27 May 2024，
  
  Accepted：2024-06-09，
  
  Published Online：26 June 2024，
  
  Published：31 July 2024
- Accepted：
Scan QR Code
Guo, R. P. et al. EventLFM: event camera integrated Fourier light field microscopy for ultrafast 3D imaging.,Light: Science & Applications, 13, 1401-1415 (2024).
DOI：

Guo, R. P. et al. EventLFM: event camera integrated Fourier light field microscopy for ultrafast 3D imaging.,Light: Science & Applications, 13, 1401-1415 (2024). DOI： 10.1038/s41377-024-01502-5.

摘要

Abstract

Ultrafast 3D imaging is indispensable for visualizing complex and dynamic biological processes. Conventional scanning-based techniques necessitate an inherent trade-off between acquisition speed and space-bandwidth product (SBP). Emerging single-shot 3D wide-field techniques offer a promising alternative but are bottlenecked by the synchronous readout constraints of conventional CMOS systems

thus restricting data throughput to maintain high SBP at limited frame rates. To address this

we introduce EventLFM

a straightforward and cost-effective system that overcomes these challenges by integrating an event camera with Fourier light field microscopy (LFM)

a state-of-the-art single-shot 3D wide-field imaging technique. The event camera ope

rates on a novel asynchronous readout architecture

thereby bypassing the frame rate limitations inherent to conventional CMOS systems. We further develop a simple and robust event-driven LFM reconstruction algorithm that can reliably reconstruct 3D dynamics from the unique spatiotemporal measurements captured by EventLFM. Experimental results demonstrate that EventLFM can robustly reconstruct fast-moving and rapidly blinking 3D fluorescent samples at kHz frame rates. Furthermore

we highlight EventLFM's capability for imaging of blinking neuronal signals in scattering mouse brain tissues and 3D tracking of GFP-labeled neurons in freely moving

C. elegans

. We believe that the combined ultrafast speed and large 3D SBP offered by EventLFM may open up new possibilities across many biomedical applications.

关键词

Keywords

references

Mertz, J. Strategies for volumetric imaging with a fluorescence microscope. Optica 6 , 1261–1268 (2019)..

Minsky, M. Memoir on inventing the confocal scanning microscope. Scanning 10 , 128–138 (1988)..

Helmchen, F. & Denk, W. Deep tissue two-photon microscopy. Nat. Methods 2 , 932–940 (2005)..

Voie, A. H., Burns, D. H. & Spelman, F. A. Orthogonal-plane fluorescence optical sectioning: three-dimensional imaging of macroscopic biological specimens. J. Microsc. 170 , 229–236 (1993)..

Levoy, M. et al. Light field microscopy. ACM Trans. Gr. 25 , 924–934 (2006)..

Guo, C. L. et al. Fourier light-field microscopy. Opt. Express 27 , 25573–25594 (2019)..

Llavador, A. et al. Resolution improvements in integral microscopy with Fourier plane recording. Opt. Express 24 , 20792–20798 (2016)..

Xue, Y. J. et al. Single-shot 3D wide-field fluorescence imaging with a computational miniature mesoscope. Sci. Adv. 6 , eabb7508 (2020)..

Xue, Y. J. et al. Deep-learning-augmented computational miniature mesoscope. Optica 9 , 1009–1021 (2022)..

Liu, F. L. et al. Fourier diffuserScope: single-shot 3D Fourier light field microscopy with a diffuser. Opt. Express 28 , 28969–28986 (2020)..

Adams, J. K. et al. Single-frame 3D fluorescence microscopy with ultraminiature lensless FlatScope. Sci. Adv. 3 , e1701548 (2017)..

Nelson, S. & Menon, R. Bijective-constrained cycle-consistent deep learning for optics-free imaging and classification. Optica 9 , 26–31 (2022)..

Nehme, E. et al. DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning. Nat. Methods 17 , 734–740 (2020)..

Pavani, S. R. P. et al. Three-dimensional, single-molecule fluorescence imaging beyond the diffraction limit by using a double-helix point spread function. Proc. Natl Acad. Sci. USA 106 , 2995–2999 (2009)..

Abdelfattah, A. S. et al. Sensitivity optimization of a rhodopsin-based fluorescent voltage indicator. Neuron 111 , 1547–1563. e9 (2023)..

Bouchard, M. B. et al. Ultra-fast multispectral optical imaging of cortical oxygenation, blood flow, and intracellular calcium dynamics. Opt. Express 17 , 15670–15678 (2009)..

Rome, L. C. & Lindstedt, S. L. The quest for speed: muscles built for high-frequency contractions. Physiology 13 , 261–268 (1998)..

Gao, L. et al. Single-shot compressed ultrafast photography at one hundred billion frames per second. Nature 516 , 74–77 (2014)..

Liang, J. Y. & Wang, L.V. Single-shot ultrafast optical imaging. Optica 5 , 1113–1127 (2018)..

Liu, X. L. et al. Fast wide-field upconversion luminescence lifetime thermometry enabled by single-shot compressed ultrahigh-speed imaging. Nat. Commun. 12 , 6401 (2021)..

Ma, Y. Y. et al. High-speed compressed-sensing fluorescence lifetime imaging microscopy of live cells. Proc. Natl Acad. Sci. USA 118 , e2004176118 (2021)..

Feng, X. H. & Gao, L. Ultrafast light field tomography for snapshot transient and non-line-of-sight imaging. Nat. Commun. 12 , 2179 (2021)..

Weber, T. D. et al. High-speed multiplane confocal microscopy for voltage imaging in densely labeled neuronal populations. Nat. Neurosci. 26 , 1642–1650 (2023)..

Wu, J. L. et al. Kilohertz two-photon fluorescence microscopy imaging of neural activity in vivo. Nat. Methods 17 , 287–290 (2020)..

Platisa, J. et al. High-speed low-light in vivo two-photon voltage imaging of large neuronal populations. Nat. Methods 20 , 1095–1103 (2023)..

Xiao, S. et al. High-throughput deep tissue two-photon microscopy at kilohertz frame rates. Optica 10 , 763–769 (2023)..

Gallego, G. et al. Event-based vision: a survey. IEEE Transa. Pattern Anal. Mach. Intell. 44 , 154–180 (2022)..

Lichtsteiner, P., Posch, C. & Delbruck, T. A 128×128 120 dB 15 μs latency asynchronous temporal contrast vision sensor. IEEE J. Solid-State Circuits 43 , 566–576 (2008)..

Willert, C. E. Event-based imaging velocimetry using pulsed illumination. Exp. Fluids 64 , 98 (2023)..

Brown, P. T. et al. Fourier synthesis optical diffraction tomography for kilohertz rate volumetric imaging. Print at https://doi.org/10.48550/arXiv.2309.16912 https://doi.org/10.48550/arXiv.2309.16912 (2023).

Chen, G. et al. Event-based neuromorphic vision for autonomous driving: a paradigm shift for bio-inspired visual sensing and perception. IEEE Signal Process. Mag. 37 , 34–49 (2020)..

Amir, A. et al. A low power, fully event-based gesture recognition system. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 7388–7397 (Honolulu, HI, USA: IEEE, 2017).

Cabriel, C. et al. Event-based vision sensor for fast and dense single-molecule localization microscopy. Nat. Photonics 17 , 1105–1113 (2023)..

Mangalwedhekar, R. et al. Achieving nanoscale precision using neuromorphic localization microscopy. Nat. Nanotechnol. 18 , 380–389 (2023)..

Lagorce, X. et al. HOTS: a hierarchy of event-based time-surfaces for pattern recognition. IEEE Trans. Pattern Anal. Mach. Intell. 39 , 1346–1359 (2017)..

Sironi, A. et al. HATS: histograms of averaged time surfaces for robust event-based object classification. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition . 1731–1740 (Salt Lake City, UT, USA: IEEE, 2018).

Ng, R. et al. Light Field Photography with A Hand-Held Plenoptic Camera (Stanford University, 2005).

Xiao, S. et al. Large-scale voltage imaging in behaving mice usingtargeted illumination. iScience 24 , 103263 (2021)..

Abdelfattah, A. S. et al. Bright and photostable chemigenetic indicators for extended in vivo voltage imaging. Science 365 , 699–704 (2019)..

Mertz, J. Optical sectioning microscopy with planar or structured illumination. Nat. Methods 8 , 811–819 (2011)..

Zhang, J. Z. et al. 3D computer-generated holography by non-convex optimization. Optica 4 , 1306–1313 (2017)..

Zhang, N., Shea, T. & Nurmikko, A. Event-driven imaging in turbid media: a confluence of optoelectronics and neuromorphic computation. Print at https://doi.org/10.48550/arXiv.2309.06652 https://doi.org/10.48550/arXiv.2309.06652 (2023).

Zhang, Z. H., Suo, J. L. & Dai, Q. H. Denoising of event-based sensors with deep neural networks. In Proceedings of SPIE 11897, Optoelectronic Imaging and Multimedia Technology VIII. 1189713 (Nantong, China: SPIE, 2021).

Hagenaars, J. J., Paredes-Vallés, F. & de Croon, G. C. H. E. Self-supervised learning of event-based optical flow with spiking neural networks. 35th Conf. Neural Inform. Process. Syst. 2021 , 7167–7179 (2021)..

Yang, Q. W. et al. Wide-field, high-resolution reconstruction in computational multi-aperture miniscope using a Fourier neural network. Optica 11 , 860–871 (2024)..

Eshraghian, J. K. et al. Training spiking neural networks using lessons from deep learning. Proc. IEEE 111 , 1016–1054 (2023)..

Wang, G. Q. et al. CED-4 CARD domain residues can modulate non-apoptotic neuronal regeneration functions independently from apoptosis. Sci. Rep. 9 , 13315 (2019)..

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰