Recurrent neural network-based volumetric fluorescence microscopy

Luzhe Huang; Hanlong Chen; Yilin Luo; Yair Rivenson; Aydogan Ozcan

doi:10.1038/s41377-021-00506-9

Go to Nature Website

Your Location：

Home >

Browse articles >

Recurrent neural network-based volumetric fluorescence microscopy

Articles

- Recurrent neural network-based volumetric fluorescence microscopy
- Light: Science & Applications Vol. 10, Issue 4, Pages: 620-635(2021)
- Affiliations：
  
  1.Electrical and Computer Engineering Department, University of California, Los Angeles, CA 90095, USA
  2.Bioengineering Department, University of California, Los Angeles, CA 90095, USA
  3.California Nano Systems Institute (CNSI), University of California, Los Angeles, CA 90095, USA
  4.David Geffen School of Medicine, University of California, Los Angeles, CA 90095, USA
- Author bio：
  
  Aydogan Ozcan (ozcan@ucla.edu)
- Funds：
- DOI：10.1038/s41377-021-00506-9
  CLC：
- Published：30 April 2021，
  
  Published Online：23 March 2021，
  
  Received：14 December 2020，
  
  Revised：24 February 2021，
  
  Accepted：02 March 2021
- Accepted：
Scan QR Code
Huang, L. Z. et al. Recurrent neural network-based volumetric fluorescence microscopy. Light: Science & Applications, 10, 620-635 (2021).
DOI：

Huang, L. Z. et al. Recurrent neural network-based volumetric fluorescence microscopy. Light: Science & Applications, 10, 620-635 (2021). DOI： 10.1038/s41377-021-00506-9.

摘要

Abstract

Volumetric imaging of samples using fluorescence microscopy plays an important role in various fields including physical

medical and life sciences. Here we report a deep learning-based volumetric image inference framework that uses 2D images that are sparsely captured by a standard wide-field fluorescence microscope at arbitrary axial positions within the sample volume. Through a recurrent convolutional neural network

which we term as Recurrent-MZ

2D fluorescence information from a few axial planes within the sample is explicitly incorporated to digitally reconstruct the sample volume over an extended depth-of-field. Using experiments on

C. elegans

and nanobead samples

Recurrent-MZ is demonstrated to significantly increase the depth-of-field of a 63×/1.4NA objective lens

also providing a 30-fold reduction in the number of axial scans required to image

the same sample volume. We further illustrated the generalization of this recurrent network for 3D imaging by showing its resilience to varying imaging conditions

including e.g.

different sequences of input images

covering various axial permutations and unknown axial positioning errors. We also demonstrated wide-field to confocal cross-modality image transformations using Recurrent-MZ framework and performed 3D image reconstruction of a sample using a few wide-field 2D fluorescence images as input

matching confocal microscopy images of the same sample volume. Recurrent-MZ demonstrates the first application of recurrent neural networks in microscopic image reconstruction and provides a flexible and rapid volumetric imaging framework

overcoming the limitations of current 3D scanning microscopy tools.

关键词

Keywords

references

Pawley, J. B.Handbook of Biological Confocal Microscopy. 3rd edn. (Springer-Verlag, New York, 2006).

Denk, W., Strickler, J. H.&Webb, W. W. Two-photon laser scanning fluorescence microscopy.Science248, 73-76 (1990)..

Horton, N. G. et al. In vivo three-photon microscopy of subcortical structures within an intact mouse brain.Nat. Photonics7, 205-209 (2013)..

Huang, D. et al. Optical coherence tomography.Science254, 1178-1181 (1991)..

Haeusler, G.&Lindner, M. W. "Coherence radar" and "spectral radar"—new tools for dermatological diagnosis.J. Biomed. Opt.3, 21-31 (1998)..

Fercher, A. F. et al. Measurement of intraocular distances by backscattering spectral interferometry.Opt. Commun.117, 43-48 (1995)..

Santi, P. A.Light sheet fluorescence microscopy: a review.J. Histochem. Cytochem.59, 129-138 (2011)..

Prabhat, P. et al. Simultaneous imaging of different focal planes in fluorescence microscopy for the study of cellular dynamics in three dimensions.IEEE Trans. NanoBiosci.3, 237-242 (2004)..

Johnson, C. et al. Continuous focal translation enhances rate of point-scan volumetric microscopy.Opt. Express27, 36241-36258 (2019)..

Abrahamsson, S. et al. Fast multicolor 3D imaging using aberration-corrected multifocus microscopy.Nat. Methods10, 60-63 (2013)..

Bouchard, M. B. et al. Swept confocally-aligned planar excitation (SCAPE) microscopy for high-speed volumetric imaging of behaving organisms.Nat. Photonics9, 113-119 (2015)..

Nakano, A. Spinning-disk confocal microscopy—a cutting-edge tool for imaging of membrane traffic.Cell Struct. Funct.27, 349-355 (2002)..

Badon, A. et al. Video-rate large-scale imaging with Multi-Z confocal microscopy.Optica6, 389-395 (2019)..

Li, H. Y. et al. Fast, volumetric live-cell imaging using high-resolution light-field microscopy.Biomed. Opt. Express10, 29-49 (2019)..

Martínez-Corral, M.&Javidi, B. Fundamentals of 3D imaging and displays: a tutorial on integral imaging, light-field, and plenoptic systems.Adv. Opt. Photonics10, 512-566 (2018)..

Song, A. et al. Volumetric two-photon imaging of neurons using stereoscopy (vTwINS).Nat. Methods14, 420-426 (2017)..

Chen, X. L. et al. Volumetric chemical imaging by stimulated Raman projection microscopy and tomography.Nat. Commun.8, 15117 (2017)..

Lu, R. W. et al. Video-rate volumetric functional imaging of the brain at synaptic resolution.Nat. Neurosci.20, 620-628 (2017)..

Pascucci, M. et al. Compressive three-dimensional super-resolution microscopy with speckle-saturated fluorescence excitation.Nat. Commun.10, 1327 (2019)..

Fang, L. Y. et al. Fast acquisition and reconstruction of optical coherence tomography images via sparse representation.IEEE Trans. Med. Imaging32, 2034-2049 (2013)..

Wen, C. Y. et al. Compressive sensing for fast 3-D and random-access two-photon microscopy.Opt. Lett.44, 4343-4346 (2019)..

Beck, A.&Teboulle, M. A fast iterative shrinkage-thresholding algorithm for linear inverse problems.SIAM J. Imaging Sci.2, 183-202 (2009)..

Boyd, S. et al. Distributed optimization and statistical learning via the alternating direction method of multipliers.Found. Trends Mach. Learn.3, 1-122 (2011)..

de Haan, K. et al. Deep-learning-based image reconstruction and enhancement in optical microscopy.Proc. IEEE108, 30-50 (2020)..

Rivenson, Y. et al. Deep learning microscopy.Optica4, 1437-1443 (2017)..

Wang, H. D. et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy.Nat. Methods16, 103-110 (2019)..

Nehme, E. et al. Deep-STORM: super-resolution single-molecule microscopy by deep learning.Optica5, 458-464 (2018)..

Rivenson, Y. et al. Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning.Nat. Biomed. Eng.3, 466-477 (2019)..

Bayramoglu, N. et al. Towards virtual H&E staining of hyperspectral lung histology images using conditional generative adversarial networks. In:Proc. 2017 IEEE International Conference on Computer Vision Workshops (ICCVW)64-71 (IEEE, Venice, Italy, 2017).

Christiansen, E. M. et al.In silicolabeling: predicting fluorescent labels in unlabeled images.Cell173, 792-803. e19 (2018)..

Ounkomol, C. et al. Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy.Nat. Methods15, 917-920 (2018)..

Rivenson, Y. et al. PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning.Light. : Sci. Appl.8, 23 (2019)..

Wu, Y. C. et al. Extended depth-of-field in holographic imaging using deep-learning-based autofocusing and phase recovery.Optica5, 704-710 (2018)..

Rivenson, Y. et al. Phase recovery and holographic image reconstruction using deep learning in neural networks.Light. : Sci. Appl.7, 17141 (2018)..

Nguyen, T. et al. Deep learning approach for Fourier ptychography microscopy.Opt. Express26, 26470-26484 (2018)..

Pinkard, H. et al. Deep learning for single-shot autofocus microscopy.Optica6, 794-797 (2019)..

Luo, Y. L. et al. Single-shot autofocusing of microscopy images using deep learning.ACS Photonics8, 625-638 (2021)..

Wu, Y. C. et al. Three-dimensional virtual refocusing of fluorescence microscopy images using deep learning.Nat. Methods16, 1323-1331 (2019)..

Barbastathis, G., Ozcan, A.&Situ, G. On the use of deep learning for computational imaging.Optica6, 921-943 (2019)..

Choy, C. B. et al. 3D-R2N2: a unified approach for single and multi-view 3D object reconstruction. In:Proc. 14th European Conference on Computer Vision (ECCV) 2016. 628-644. (Springer, Amsterdam, The Netherlands, 2016)

Kar, A., Häne, C.&Malik, J. Learning a multi-view stereo machine. In:Proc. 31st International Conference on Neural Information Processing Systems(ACM, Long Beach, CA, USA, 2017).

Petrov, P. N.&Moerner, W. E. Addressing systematic errors in axial distance measurements in single-emitter localization microscopy.Opt. Express28, 18616-18632 (2020)..

Montavon, G., Samek, W.&Müller, K. R. Methods for interpreting and understanding deep neural networks.Digital Signal Process.73, 1-15 (2018)..

Selvaraju, R. R. et al. Grad-CAM: visual explanations from deep networks via gradient-based localization. In:Proc. 2017 IEEE International Conference on Computer Vision (ICCV). (IEEE, Venice, Italy, 2017).

Çiçek, Ö. et al. 3D U-Net: learning dense volumetric segmentation from sparse annotation. In:Proc. 19th International Conference on Medical Image Computing and Computer-Assisted Intervention - MICCAI 2016. 424-432. (Springer, Athens, Greece, 2016).

Chhetri, R. K. et al. Whole-animal functional and developmental imaging with isotropic spatial resolution.Nat. Methods12, 1171-1178 (2015)..

Kumar, A. et al. Dual-view plane illumination microscopy for rapid and spatially isotropic imaging.Nat. Protoc.9, 2555-2573 (2014)..

Wu, Y. C. et al. Spatially isotropic four-dimensional imaging with dual-view plane illumination microscopy.Nat. Biotechnol.31, 1032-1038 (2013)..

Swoger, J. et al. Multi-view image fusion improves resolution in three-dimensional microscopy.Opt. Express15, 8029-8042 (2007)..

Thevenaz, P., Ruttimann, U. E.&Unser, M. A pyramid approach to subpixel registration based on intensity.IEEE Trans. Image Process.7, 27-41 (1998)..

Forster, B. et al. Complex wavelets for extended depth-of-field: a new method for the fusion of multichannel microscopy images.Microsc. Res. Tech.65, 33-42 (2004)..

Shi, X. J. et al. Convolutional LSTM network: a machine learning approach for precipitation nowcasting. In:Proc. 28th International Conference on Neural Information Processing Systems. (ACM, Montreal, Quebec, Canada, 2015).

Graves, A. et al. A novel connectionist system for unconstrained handwriting recognition.IEEE Trans. Pattern Anal. Mach. Intell.31, 855-868 (2009)..

Gregor, K. et al. DRAW: a recurrent neural network for image generation. InProc. 32nd Internnational Conference on Machine Learning 2015. 1462-1471. (PMLR, Lille, France, 2015).

Sharma, A., Grau, O.&Fritz, M. VConv-DAE: deep volumetric shape learning without object labels. InProc. 14th European Conference on Computer Vision (ECCV) 2016. 236-250. (Springer, Amsterdam, The Netherlands, 2016).

Kingma, D. P.&Welling, M. Auto-encoding variational bayes. Preprint athttp://arxiv.org/abs/1312.6114http://arxiv.org/abs/1312.6114(2014).

Wang, W. Y. et al. Shape inpainting using 3D generative adversarial network and recurrent convolutional networks. In:Proc. 2017 IEEE International Conference on Computer Vision (ICCV). 2317-2325. (IEEE, Venice, Italy, 2017).

Chen, J. X. et al. Combining fully convolutional and recurrent neural networks for 3D biomedical image segmentation. In:Proc. 30th International Conference on Neural Information Processing Systems. (ACM, Barcelona, Spain, 2016).

Tseng, K. L. et al. Joint sequence learning and cross-modality convolution for 3D biomedical segmentation. In:Proc. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3739-3746. (IEEE, Honolulu, HI, 2017).

Ronneberger, O., Fischer, P.&Brox, T. U-Net: convolutional networks for biomedical image segmentation. Preprint athttp://arxiv.org/abs/1505.04597http://arxiv.org/abs/1505.04597(2015).

Zhou, Z. W. et al. UNet++: redesigning skip connections to exploit multiscale features in image segmentation.IEEE Trans. Med. Imaging39, 1856-1867 (2020)..

Liu, P. J. et al. Multi-level wavelet-CNN for image restoration. In:Proc. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 886-88609. (IEEE, Salt Lake City, UT, USA, 2018).

Cho, K. et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation. Preprint athttp://arxiv.org/abs/1406.1078http://arxiv.org/abs/1406.1078(2014).

Hochreiter, S.&Schmidhuber, J. Long short-term memory.Neural Comput.9, 1735-1780 (1997)..

Owen, A. B. A robust hybrid of lasso and ridge regression. inPrediction and Discovery(eds Verducci, J. S., Shen, X. T.&Lafferty, J. ) 59-71 (American Mathematical Society, Providence, Rhode Island, 2007).

Laina, I. et al. Deeper depth prediction with fully convolutional residual networks. Preprint athttp://arxiv.org/abs/1606.00373http://arxiv.org/abs/1606.00373(2016).

Wang, Z., Simoncelli, E. P.&Bovik, A. C. Multiscale structural similarity for image quality assessment. In:Proc. 37th Asilomar Conference on Signals, Systems & Computers. (IEEE, Pacific Grove, CA, USA, 2003, 1398-1402).

Goodfellow, I. J. et al. Generative adversarial nets. In:Proc. 27th International Conference on Neural Information Processing Systems. (ACM, Montreal, Quebec, Canada, 2014).

Zhao, H. et al. Loss functions for image restoration with neural networks.IEEE Trans. Computational Imaging3, 47-57 (2017)..

Kingma, D. P.&Ba, J. Adam: a method for stochastic optimization. Preprint athttp://arxiv.org/abs/1412.6980http://arxiv.org/abs/1412.6980(2017).

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰