EventLFM: event camera integrated Fourier light field microscopy for ultrafast 3D imaging

Ruipeng Guo; Qianwan Yang; Andrew S. Chang; Guorong Hu; Joseph Greene; Christopher V. Gabel; Sixian You; Lei Tian

doi:10.1038/s41377-024-01502-5

Go to Nature Website

Your Location：

Home >

Browse articles >

EventLFM: event camera integrated Fourier light field microscopy for ultrafast 3D imaging

Original Articles

- EventLFM: event camera integrated Fourier light field microscopy for ultrafast 3D imaging
- Light: Science & Applications Vol. 13, Issue 7, Pages: 1401-1415(2024)
- Affiliations：
  
  1.Department of Electrical and Computer Engineering, Boston University, Boston, MA 02215, USA
  2.Department of Physiology and Biophysics, Boston University, Boston, MA 02215, USA
  3.Neurophotonics Center, Boston University, Boston, MA 02215, USA
  4.Research Laboratory of Electronics (RLE) in the Department of Electrical Science and Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
  5.Department of Biomedical Engineering, Boston University, Boston, MA 02215, USA
- Author bio：
  
  Lei Tian (leitian@bu.edu)
- Funds：
- DOI：10.1038/s41377-024-01502-5
  CLC：
- Published：31 July 2024，
  
  Published Online：26 June 2024，
  
  Received：07 December 2023，
  
  Revised：27 May 2024，
  
  Accepted：09 June 2024
- Accepted：
Scan QR Code
Guo, R. P. et al. EventLFM: event camera integrated Fourier light field microscopy for ultrafast 3D imaging.,Light: Science & Applications, 13, 1401-1415 (2024).
DOI：

Guo, R. P. et al. EventLFM: event camera integrated Fourier light field microscopy for ultrafast 3D imaging.,Light: Science & Applications, 13, 1401-1415 (2024). DOI： 10.1038/s41377-024-01502-5.

摘要

Abstract

Ultrafast 3D imaging is indispensable for visualizing complex and dynamic biological processes. Conventional scanning-based techniques necessitate an inherent trade-off between acquisition speed and space-bandwidth product (SBP). Emerging single-shot 3D wide-field techniques offer a promising alternative but are bottlenecked by the synchronous readout constraints of conventional CMOS systems

thus restricting data throughput to maintain high SBP at limited frame rates. To address this

we introduce EventLFM

a straightforward and cost-effective system that overcomes these challenges by integrating an event camera with Fourier light field microscopy (LFM)

a state-of-the-art single-shot 3D wide-field imaging technique. The event camera ope

rates on a novel asynchronous readout architecture

thereby bypassing the frame rate limitations inherent to conventional CMOS systems. We further develop a simple and robust event-driven LFM reconstruction algorithm that can reliably reconstruct 3D dynamics from the unique spatiotemporal measurements captured by EventLFM. Experimental results demonstrate that EventLFM can robustly reconstruct fast-moving and rapidly blinking 3D fluorescent samples at kHz frame rates. Furthermore

we highlight EventLFM's capability for imaging of blinking neuronal signals in scattering mouse brain tissues and 3D tracking of GFP-labeled neurons in freely moving

C. elegans

. We believe that the combined ultrafast speed and large 3D SBP offered by EventLFM may open up new possibilities across many biomedical applications.

关键词

Keywords

references

Mertz, J. Strategies for volumetric imaging with a fluorescence microscope.Optica6, 1261–1268 (2019)..

Minsky, M. Memoir on inventing the confocal scanning microscope.Scanning10, 128–138 (1988)..

Helmchen, F.&Denk, W. Deep tissue two-photon microscopy.Nat. Methods2, 932–940 (2005)..

Voie, A. H., Burns, D. H.&Spelman, F. A. Orthogonal-plane fluorescence optical sectioning: three-dimensional imaging of macroscopic biological specimens.J. Microsc.170, 229–236 (1993)..

Levoy, M. et al. Light field microscopy.ACM Trans. Gr.25, 924–934 (2006)..

Guo, C. L. et al. Fourier light-field microscopy.Opt. Express27, 25573–25594 (2019)..

Llavador, A. et al. Resolution improvements in integral microscopy with Fourier plane recording.Opt. Express24, 20792–20798 (2016)..

Xue, Y. J. et al. Single-shot 3D wide-field fluorescence imaging with a computational miniature mesoscope.Sci. Adv.6, eabb7508 (2020)..

Xue, Y. J. et al. Deep-learning-augmented computational miniature mesoscope.Optica9, 1009–1021 (2022)..

Liu, F. L. et al. Fourier diffuserScope: single-shot 3D Fourier light field microscopy with a diffuser.Opt. Express28, 28969–28986 (2020)..

Adams, J. K. et al. Single-frame 3D fluorescence microscopy with ultraminiature lensless FlatScope.Sci. Adv.3, e1701548 (2017)..

Nelson, S.&Menon, R. Bijective-constrained cycle-consistent deep learning for optics-free imaging and classification.Optica9, 26–31 (2022)..

Nehme, E. et al. DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning.Nat. Methods17, 734–740 (2020)..

Pavani, S. R. P. et al. Three-dimensional, single-molecule fluorescence imaging beyond the diffraction limit by using a double-helix point spread function.Proc. Natl Acad. Sci. USA106, 2995–2999 (2009)..

Abdelfattah, A. S. et al. Sensitivity optimization of a rhodopsin-based fluorescent voltage indicator.Neuron111, 1547–1563. e9 (2023)..

Bouchard, M. B. et al. Ultra-fast multispectral optical imaging of cortical oxygenation, blood flow, and intracellular calcium dynamics.Opt. Express17, 15670–15678 (2009)..

Rome, L. C.&Lindstedt, S. L. The quest for speed: muscles built for high-frequency contractions.Physiology13, 261–268 (1998)..

Gao, L. et al. Single-shot compressed ultrafast photography at one hundred billion frames per second.Nature516, 74–77 (2014)..

Liang, J. Y.&Wang, L.V. Single-shot ultrafast optical imaging.Optica5, 1113–1127 (2018)..

Liu, X. L. et al. Fast wide-field upconversion luminescence lifetime thermometry enabled by single-shot compressed ultrahigh-speed imaging.Nat. Commun.12, 6401 (2021)..

Ma, Y. Y. et al. High-speed compressed-sensing fluorescence lifetime imaging microscopy of live cells.Proc. Natl Acad. Sci. USA118, e2004176118 (2021)..

Feng, X. H.&Gao, L. Ultrafast light field tomography for snapshot transient and non-line-of-sight imaging.Nat. Commun.12, 2179 (2021)..

Weber, T. D. et al. High-speed multiplane confocal microscopy for voltage imaging in densely labeled neuronal populations.Nat. Neurosci.26, 1642–1650 (2023)..

Wu, J. L. et al. Kilohertz two-photon fluorescence microscopy imaging of neural activity in vivo.Nat. Methods17, 287–290 (2020)..

Platisa, J. et al. High-speed low-light in vivo two-photon voltage imaging of large neuronal populations.Nat. Methods20, 1095–1103 (2023)..

Xiao, S. et al. High-throughput deep tissue two-photon microscopy at kilohertz frame rates.Optica10, 763–769 (2023)..

Gallego, G. et al. Event-based vision: a survey.IEEE Transa. Pattern Anal. Mach. Intell.44, 154–180 (2022)..

Lichtsteiner, P., Posch, C.&Delbruck, T. A 128×128 120 dB 15 μs latency asynchronous temporal contrast vision sensor.IEEE J. Solid-State Circuits43, 566–576 (2008)..

Willert, C. E. Event-based imaging velocimetry using pulsed illumination.Exp. Fluids64, 98 (2023)..

Brown, P. T. et al. Fourier synthesis optical diffraction tomography for kilohertz rate volumetric imaging. Print athttps://doi.org/10.48550/arXiv.2309.16912https://doi.org/10.48550/arXiv.2309.16912(2023).

Chen, G. et al. Event-based neuromorphic vision for autonomous driving: a paradigm shift for bio-inspired visual sensing and perception.IEEE Signal Process. Mag.37, 34–49 (2020)..

Amir, A. et al. A low power, fully event-based gesture recognition system. InProc. IEEE Conference on Computer Vision and Pattern Recognition7388–7397 (Honolulu, HI, USA: IEEE, 2017).

Cabriel, C. et al. Event-based vision sensor for fast and dense single-molecule localization microscopy.Nat. Photonics17, 1105–1113 (2023)..

Mangalwedhekar, R. et al. Achieving nanoscale precision using neuromorphic localization microscopy.Nat. Nanotechnol.18, 380–389 (2023)..

Lagorce, X. et al. HOTS: a hierarchy of event-based time-surfaces for pattern recognition.IEEE Trans. Pattern Anal. Mach. Intell.39, 1346–1359 (2017)..

Sironi, A. et al. HATS: histograms of averaged time surfaces for robust event-based object classification. InProc. IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1731–1740 (Salt Lake City, UT, USA: IEEE, 2018).

Ng, R. et al.Light Field Photography with A Hand-Held Plenoptic Camera(Stanford University, 2005).

Xiao, S. et al. Large-scale voltage imaging in behaving mice usingtargeted illumination.iScience24, 103263 (2021)..

Abdelfattah, A. S. et al. Bright and photostable chemigenetic indicators for extended in vivo voltage imaging.Science365, 699–704 (2019)..

Mertz, J. Optical sectioning microscopy with planar or structured illumination.Nat. Methods8, 811–819 (2011)..

Zhang, J. Z. et al. 3D computer-generated holography by non-convex optimization.Optica4, 1306–1313 (2017)..

Zhang, N., Shea, T.&Nurmikko, A. Event-driven imaging in turbid media: a confluence of optoelectronics and neuromorphic computation. Print athttps://doi.org/10.48550/arXiv.2309.06652https://doi.org/10.48550/arXiv.2309.06652(2023).

Zhang, Z. H., Suo, J. L.&Dai, Q. H. Denoising of event-based sensors with deep neural networks. InProceedings of SPIE 11897, Optoelectronic Imaging and Multimedia Technology VIII.1189713 (Nantong, China: SPIE, 2021).

Hagenaars, J. J., Paredes-Vallés, F.&de Croon, G. C. H. E. Self-supervised learning of event-based optical flow with spiking neural networks.35th Conf. Neural Inform. Process. Syst.2021, 7167–7179 (2021)..

Yang, Q. W. et al. Wide-field, high-resolution reconstruction in computational multi-aperture miniscope using a Fourier neural network.Optica11, 860–871 (2024)..

Eshraghian, J. K. et al. Training spiking neural networks using lessons from deep learning.Proc. IEEE111, 1016–1054 (2023)..

Wang, G. Q. et al. CED-4 CARD domain residues can modulate non-apoptotic neuronal regeneration functions independently from apoptosis.Sci. Rep.9, 13315 (2019)..

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰