Deep learning autofluorescence-harmonic microscopy

Binglin Shen; Shaowen Liu; Yanping Li; Ying Pan; Yuan Lu; Rui Hu; Junle Qu; Liwei Liu

doi:10.1038/s41377-022-00768-x

Go to Nature Website

Your Location：

Home >

Browse articles >

Deep learning autofluorescence-harmonic microscopy

Articles

- Deep learning autofluorescence-harmonic microscopy
- Light: Science & Applications Vol. 11, Issue 4, Pages: 697-710(2022)
- Affiliations：
  
  1.Key Laboratory of Optoelectronic Devices and Systems of Guangdong Province and Ministry of Education, College of Physics and Optoelectronic Engineering, Shenzhen University, 518060, Shenzhen, China
  2.Shenzhen Meitu Innovation Technology LTD, 518060, Shenzhen, China
  3.China–Japan Union Hospital of Jilin University, 130033, Changchun, China
  4.The Sixth People's Hospital of Shenzhen, 518052, Shenzhen, China
- Author bio：
  
  Liwei Liu (liulw@szu.edu.cn)
- Funds：
- DOI：10.1038/s41377-022-00768-x
  CLC：
- Published：30 April 2022，
  
  Published Online：29 March 2022，
  
  Received：09 August 2021，
  
  Revised：05 March 2022，
  
  Accepted：10 March 2022
- Accepted：
Scan QR Code
Shen, B. L. et al. Deep learning autofluorescence-harmonic microscopy. Light: Science & Applications, 11, 697-710 (2022).
DOI：

Shen, B. L. et al. Deep learning autofluorescence-harmonic microscopy. Light: Science & Applications, 11, 697-710 (2022). DOI： 10.1038/s41377-022-00768-x.

摘要

Abstract

Laser scanning microscopy has inherent tradeoffs between imaging speed

field of view (FOV)

and spatial resolution due to the limitations of sophisticated mechanical and optical setups

and deep learning networks have emerged to overcome these limitations without changing the system. Here

we demonstrate deep learning autofluorescence-harmonic microscopy (DLAM) based on self-alignment attention-guided residual-in-residual dense generative adversarial networks to close the gap between speed

FOV

and quality. Using the framework

we demonstrate label-free large-field multimodal imaging of clinicopathological tissues with enhanced spatial resolution and running time advantages. Statistical quality assessments show that the attention-guided residual dense connections minimize the persistent noise

distortions

and scanning fringes that degrade the autofluorescence-harmonic images and avoid reconstruction artifacts in the output images. With the advantages of high contrast

high fidelity

and high speed in image reconstruction

DLAM can act as a powerful tool for the noninvasive evaluation of diseases

neural activity

and embryogenesis.

关键词

Keywords

references

Ji, M. B. et al. Rapid, label-free detection of brain tumors with stimulated Raman scattering microscopy.Sci. Transl. Med.5, 201ra119 (2013)..

Tao, Y. K. et al. Assessment of breast pathologies using nonlinear microscopy.Proc. Natl Acad. Sci. USA111, 15304–15309 (2014)..

Tu, H. H. et al. Stain-free histopathology by programmable supercontinuum pulses.Nat. Photonics10, 534–540 (2016)..

Shen, B. L. et al. Label-free whole-colony imaging and metabolic analysis of metastatic pancreatic cancer by an autoregulating flexible optical system.Theranostics10, 1849–1860 (2020)..

Kasischke, K. A. et al. Neural activity triggers neuronal oxidative metabolism followed by astrocytic glycolysis.Science305, 99–103 (2004)..

Yang, W. J.&Yuste, R. In vivo imaging of neural activity.Nat. Methods14, 349–359 (2017)..

Lecoq, J. et al. Visualizing mammalian brain area interactions by dual-axis two-photon calcium imaging.Nat. Neurosci.17, 1825–1829 (2014)..

Ji, N., Freeman, J.&Smith, S. L. Technologies for imaging neural activity in large volumes.Nat. Neurosci.19, 1154–1164 (2016)..

Stirman, J. N. et al. Wide field-of-view, multi-region, two-photon imaging of neuronal activity in the mammalian brain.Nat. Biotechnol.34, 857–862 (2016)..

Sofroniew, N. J. et al. A large field of view two-photon mesoscope with subcellularresolution for in vivo imaging.eLife5, e14472 (2016)..

Gustafsson, M. G. L. Nonlinear structured-illumination microscopy: wide-field fluorescence imaging with theoretically unlimited resolution.Proc. Natl Acad. Sci. USA102, 13081–13086 (2005)..

Zhang, H. et al. High-throughput, high-resolution deep learning microscopy based on registration-free generative adversarial network.Biomed. Opt. Express10, 1044–1063 (2019)..

Rivenson, Y. et al. Deep learning microscopy.Optica4, 1437–1443 (2017)..

Rivenson, Y. et al. PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning.Light: Sci. Appl.8, 23 (2019)..

Wang, H. D. et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy.Nat. Methods16, 103–110 (2019)..

Ouyang, W. et al. Deep learning massively accelerates super-resolution localization microscopy.Nat. Biotechnol.36, 460–468 (2018)..

Ounkomol, C. et al. Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy.Nat. Methods15, 917–920 (2018)..

Wang, Z. Q. et al. Real-time volumetric reconstruction of biological dynamics with light-field microscopy and deep learning.Nat. Methods18, 551–556 (2021)..

Weigert, M. et al. Content-aware image restoration: pushing the limits of fluorescence microscopy.Nat. Methods15, 1090–1097 (2018)..

Lin, H. N. et al. Microsecond fingerprint stimulated Raman spectroscopic imaging by ultrafast tuning and spatial-spectral learning.Nat. Commun.12, 3052 (2021)..

Hollon, T. C. et al. Near real-time intraoperative brain tumor diagnosis using stimulated Raman histology and deep neural networks.Nat. Med.26, 52–58 (2020)..

Li, X. Y. et al. Unsupervised content-preserving transformation for optical microscopy.Light: Sci. Appl.10, 44 (2021)..

Chen, J. J. et al. Three-dimensional residual channel attention networks denoise and sharpen fluorescence microscopy image volumes.Nat. Methods18, 678–687 (2021)..

Li, X. F. et al. Deep learning methods in real-time image super-resolution: a survey.J. Real. -Time Image Process.17, 1885–1909 (2020)..

Belthangady, C.&Royer, L. A. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction.Nat. Methods16, 1215–1225 (2019)..

Johnson, J., Alahi, A.&Li, F. F. Perceptual losses for real-time style transfer and super-resolution. InComputer Vision-ECCV 2016(eds Leibe, B., Matas, J., Sebe, N.&Welling, M. ) 694–711 (Springer, Cham, 2016).

Ledig, C. et al. Photo-realistic single image super-resolution using a generative adversarial network. In30th IEEE Conference on Computer Vision and Pattern Recognition(eds Chellappa, R., Zhang, Z., Hoogs, A. ) 105–114 (IEEE, Honolulu, 2017).

Horiuchi, Y. et al. Spectral normalization and relativistic adversarial training for conditional pose generation with self-attention. InProceedings of the 2019 16th International Conference on Machine Vision Applications(eds Maki, A., Favaro, P. ) 1–5 (IEEE, Tokyo, 2019).

Rublee, E. et al. ORB: an efficient alternative to SIFT or SURF. In2011 IEEE International Conference on Computer Vision(eds Sanfeliu, A., Gool, L.V., Metaxas, D. ) 2564–2571 (IEEE, Barcelona, 2011).

Wang, X. T. et al. EDVR: video restoration with enhanced deformable convolutional networks. In2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops(eds Davis, L., Torr, P., Zhu, S. ) 1954–1963 (IEEE, Long Beach, 2019).

Wang, X. T. et al. ESRGAN: enhanced super-resolution generative adversarial networks. InComputer Vision–ECCV 2018 Workshops(eds Leal-Taixé, L.&Roth, S. ) 63–79 (Springer, Cham, 2019).

Dey, N. et al. Customized VGG19 architecture for pneumonia detection in chest X-rays.Pattern Recognit. Lett.143, 67–74 (2021)..

Provenzano, P. P. et al. Collagen reorganization at the tumor–stromal interface facilitates local invasion.BMC Med.4, 38 (2006)..

Adur, J. et al. Second harmonic generation microscopy as a powerful diagnostic imaging modality for human ovarian cancer.J. Biophotonics7, 37–48 (2014)..

Lecoq, J. et al. Removing independent noise in systems neuroscience data using DeepInterpolation.Nat. Methods18, 1401–1408 (2021)..

Qiao, C. et al. Evaluation and development of deep neural networks for image super-resolution in optical microscopy.Nat. Methods18, 194–202 (2021)..

Koho, S. et al. Fourier ring correlation simplifies image restoration in fluorescence microscopy.Nat. Commun.10, 3103 (2019)..

McCann, M. T., Jin, K. H.&Unser, M. Convolutional neural networks for inverse problems in imaging: a review.IEEE Signal Process. Mag.34, 85–95 (2017)..

Lim, B. et al. Enhanced deep residual networks for single image super-resolution. In2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops1132–1140 (IEEE, Honolulu, 2017).

Huang, X. S. et al. Fast, long-term, super-resolution imaging with Hessian structured illumination microscopy.Nat. Biotechnol.36, 451–459 (2018)..

Tenne, R. et al. Super-resolution enhancement by quantum image scanning microscopy.Nat. Photonics13, 116–122 (2019)..

Nehme, E. et al. Deep-STORM: super-resolution single-molecule microscopy by deep learning.Optica5, 458–464 (2018)..

Chen, H. et al. Low-dose CT via convolutional neural network.Biomed. Opt. Express8, 679–694 (2017)..

Jin,K. H. et al. Deep convolutional neural network for inverse problems in imaging.IEEE Trans. Image Process.26, 4509–4522 (2017)..

Mardani, M. et al. Deep generative adversarial neural networks for compressive sensing MRI.IEEE Trans. Med. Imaging38, 167–179 (2019)..

You, S. X. et al. Intravital imaging by simultaneous label-free autofluorescence-multiharmonic microscopy.Nat. Commun.9, 2125 (2018)..

You, S. X. et al. Label-free deep profiling of the tumor microenvironment.Cancer Res.81, 2534–2544 (2021)..

Débarre, D. et al. Mitigating phototoxicity during multiphoton microscopy of live drosophila embryos in the 1.0–1.2 μm wavelength range.PLoS ONE9, e104250 (2014)..

Lefort, C. A review of biomedical multiphoton microscopy and its laser sources.J. Phys. D: Appl. Phys.50, 423001 (2017)..

Cai, J. R. et al. Toward real-world single image super-resolution: a new benchmark and a new model. In2019 IEEE/CVF International Conference on Computer Vision(eds Lee, K., Forsyth, D., Pollefeys, M., Tang, X. ) 3086–3095 (IEEE, Seoul).

Hu, J. et al. Squeeze-and-excitation networks.IEEE Trans. Pattern Anal. Mach. Intell.42, 2011–2023 (2020)..

Woo, S. et al. CBAM: convolutional block attention module. InComputer Vision-ECCV 2018(eds Ferrari, V., Hebert, M., Sminchisescu, C.&Weiss, Y. ) 3–19 (Springer, Cham, 2018).

Nazeri, K. et al. EdgeConnect: structureguided image inpainting using edge prediction. In2019 IEEE/CVF International Conference on Computer Vision Workshop(eds Lee, K., Forsyth, D., Pollefeys, M., Tang, X. ) 3265–3274 (IEEE, Seoul, 2019).

Cortiñas-Lorenzo, B.&Pérez-González, F. Adam and the ants: on the influence of the optimization algorithm on the detectability of DNN watermarks.Entropy22, 1379 (2020)..

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰