Deep self-learning enables fast, high-fidelity isotropic resolution restoration for volumetric fluorescence microscopy

Kefu Ning; Bolin Lu; Xiaojun Wang; Xiaoyu Zhang; Shuo Nie; Tao Jiang; Anan Li; Guoqing Fan; Xiaofeng Wang; Qingming Luo; Hui Gong; Jing Yuan

doi:10.1038/s41377-023-01230-2

Go to Nature Website

Your Location：

Home >

Browse articles >

Deep self-learning enables fast, high-fidelity isotropic resolution restoration for volumetric fluorescence microscopy

Original Articles

- Deep self-learning enables fast, high-fidelity isotropic resolution restoration for volumetric fluorescence microscopy
- Light: Science & Applications Vol. 12, Issue 9, Pages: 1943-1960(2023)
- Affiliations：
  
  1.Britton Chance Center for Biomedical Photonics, Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, China
  2.MoE Key Laboratory for Biomedical Photonics, School of Engineering Sciences, Huazhong University of Science and Technology, Wuhan, China
  3.HUST-Suzhou Institute for Brainsmatics, Suzhou, China
  4.School of Biomedical Engineering, Hainan University, Haikou, China
- Author bio：
  
  Hui Gong (huigong@hust.edu.cn)
  Jing Yuan (yuanj@hust.edu.cn)
- Funds：
- DOI：10.1038/s41377-023-01230-2
  CLC：
- Published：30 September 2023，
  
  Published Online：28 August 2023，
  
  Received：02 February 2023，
  
  Revised：04 July 2023，
  
  Accepted：12 July 2023
- Accepted：
Scan QR Code
Ning, K. F. et al. Deep self-learning enables fast, high-fidelity isotropic resolution restoration for volumetric fluorescence microscopy. Light: Science & Applications, 12, 1943-1960 (2023).
DOI：

Ning, K. F. et al. Deep self-learning enables fast, high-fidelity isotropic resolution restoration for volumetric fluorescence microscopy. Light: Science & Applications, 12, 1943-1960 (2023). DOI： 10.1038/s41377-023-01230-2.

摘要

Abstract

One intrinsic yet critical issue that troubles the field of fluorescence microscopy ever since its introduction is the unmatched resolution in the lateral and axial directions (i.e.

resolution anisotropy)

which severely deteriorates the quality

reconstruction

and analysis of 3D volume images. By leveraging the natural anisotropy

we present a deep self-learning method termed Self-Net that significantly improves the resolution of axial images by using the lateral images from the same raw dataset as rational targets. By incorporating unsupervised learning for realistic anisotropic degradation and supervised learning for high-fidelity isotropic recovery

our method can effectively suppress the hallucination with substantially enhanced image quality compared to previously reported methods. In the experiments

we show that Self-Net can reconstruct high-fidelity isotropic 3D images from organelle to tissue levels via

raw images from various microscopy platforms

e.g.

wide-field

laser-scanning

or super-resolution microscopy. For the first time

Self-Net enables isotropic whole-brain imaging at a voxel resolution of 0.2 × 0.2 × 0.2 μm

which addresses the last-mile problem of data quality in single-neuron morphology visualization and reconstruction with minimal effort and cost. Overall

Self-Net is a promising approach to overcoming the inherent resolution anisotropy for all classes of 3D fluorescence microscopy.

关键词

Keywords

references

Bodén, A. et al. Volumetric live cell imaging with three-dimensional parallelized RESOLFT microscopy.Nat. Biotechnol.39, 609–618 (2021)..

Chakraborty, T. et al. Light-sheet microscopy of cleared tissues with isotropic, subcellular resolution.Nat. Methods16, 1109–1113 (2019)..

Fischer, R. S. et al. Microscopy in 3D: a biologist's toolbox.Trends Cell Biol.21, 682–691 (2011)..

Economo, M. N. et al. A platform for brain-wide imaging and reconstruction of individual neurons.eLife5, e10566 (2016)..

Zhong, Q. Y. et al. High-definition imaging using line-illumination modulation microscopy.Nat. Methods18, 309–315 (2021)..

Wang, X. J. et al. Chemical sectioning fluorescence tomography: high-throughput, high-contrast, multicolor, whole-brain imaging at subcellular resolution.Cell Rep.34, 108709 (2021)..

Wang, Y. M. et al. TeraVR empowers precise reconstruction of complete 3-D neuronal morphology in the whole brain.Nat. Commun.10, 3474 (2019)..

Hell, S.&Stelzer, E. H. K. Properties of a 4Pi confocal fluorescence microscope.J. Opt. Soc. Am. A9, 2159–2166 (1992)..

Swoger, J., Huisken, J.&Stelzer, E. H. K. Multiple imaging axis microscopy improves resolution for thick-sample applications.Opt. Lett.28, 1654–1656 (2003)..

Krzic, U. et al. Multiview light-sheet microscope for rapid in toto imaging.Nat. Methods9, 730–733 (2012)..

Chhetri, R. K. et al. Whole-animal functional and developmental imaging with isotropic spatial resolution.Nat. Methods12, 1171–1178 (2015)..

Wu, Y. C. et al. Spatially isotropic four-dimensional imaging with dual-view plane illumination microscopy.Nat. Biotechnol.31, 1032–1038 (2013)..

Wu, Y. C. et al. Multiview confocal super-resolution microscopy.Nature600, 279–284 (2021)..

Dean, K. M. et al. Isotropic imaging across spatial scales with axially swept light-sheet microscopy.Nat. Protoc.17, 2025–2053 (2022)..

Zhang, X. Y. et al. Deep learning optical-sectioning method.Opt. Express26, 30762–30772 (2018)..

Wu, Y. C. et al. Three-dimensional virtual refocusing of fluorescence microscopy images using deep learning.Nat. Methods16, 1323–1331 (2019)..

Fang, L. J. et al. Deep learning-based point-scanning super-resolution imaging.Nat. Methods18, 406–416 (2021)..

Li, X. Y. et al. Real-time denoising enables high-sensitivity fluorescence time-lapse imaging beyond the shot-noise limit.Nat. Biotechnol.41, 282–292 (2023)..

Wagner, N. et al. Deep learning-enhanced light-field imaging with continuous validation.Nat. Methods18, 557–563 (2021)..

Qiao, C. et al. Rationalized deep learning super-resolution microscopy for sustained live imaging of rapid subcellular processes.Nat. Biotechnol.41, 367–377 (2023)..

Chen, J. J. et al. Three-dimensional residual channel attention networks denoise and sharpen fluorescence microscopy image volumes.Nat. Methods18, 678–687 (2021)..

Wang, H. D. et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy.Nat. Methods16, 103–110 (2019)..

Ouyang, W. et al. Deep learning massively accelerates super-resolution localization microscopy.Nat. Biotechnol.36, 460–468 (2018)..

Weigert, M. et al. Content-aware image restoration: pushing the limits of fluorescence microscopy.Nat. Methods15, 1090–1097 (2018)..

Zhu, J. Y. et al. Unpaired image-to-image translation using cycle-consistent adversarial networks. InProc.IEEE International Conference on Computer Vision2242–2251 (IEEE, 2017)..

Park, H. et al. Deep learning enables reference-free isotropic super-resolution for volumetric fluorescence microscopy.Nat. Commun.13, 3297 (2022)..

Cohen, J. P., Luck, M.&Honari, S. Distribution matching losses can hallucinate features in medical image translation. InProc. 21st International Conference on Medical Image Computing and Computer Assisted Intervention529–536 (Springer, 2018).

Todorov, M. I. et al. Machine learning analysis of whole mouse brain vasculature.Nat. Methods17, 442–449 (2020)..

Hein, B., Willig, K. I.&Hell, S. W. Stimulated emission depletion (STED) nanoscopy of a fluorescent protein-labeled organelle inside a living cell.Proc. Natl Acad. Sci. USA105, 14271–14276 (2008)..

Richardson, W. H. Bayesian-based iterative method of image restoration.J. Opt. Soc. Am.62, 55–59 (1972)..

Lucy, L. B. An iterative technique for the rectification of observed distributions.Astron. J.79, 745 (1974)..

Shao, L. et al. Super-resolution 3D microscopy of live whole cells using structured illumination.Nat. Methods8, 1044–1046 (2011)..

Shao, L. et al. I5S: wide-field light microscopy with 100-nm-scale resolution in three dimensions.Biophys. J.94, 4971–4983 (2008)..

Li, X. S.et al. Three-dimensional structured illumination microscopy with enhanced axial resolution.Nat. Biotechnol. https://doi.org/10.1038/s41587-022-01651-1 (2023).

York, A. G. et al. Instant super-resolution imaging in live cells and embryos via analog image processing.Nat. Methods10, 1122–1126 (2013)..

Wang, X. J. et al. Genetic single neuron anatomy reveals fine granularity of cortical axo-axonic cells.Cell Rep.26, 3145–3159. e5 (2019)..

Quan, T. W. et al. NeuroGPS-Tree: automatic reconstruction of large-scale neuronal populations with dense neurites.Nat. Methods13, 51–54 (2016)..

Li, R. J. et al. Deep learning segmentation of optical microscopy images improves 3-D neuron reconstruction.IEEE Trans. Med. Imaging36, 1533–1541 (2017)..

McNally, J. G. et al. Three-dimensional imaging by deconvolution microscopy.Methods19, 373–385 (1999)..

Zhou, H. et al. GTree: an open-source tool for dense reconstruction of brain-wide neuronal population.Neuroinformatics19, 305–317 (2021)..

Gao, L. et al. Single-neuron projectome of mouse prefrontal cortex.Nat. Neurosci.25, 515–529 (2022)..

Winnubst, J. et al. Reconstruction of 1, 000 projection neurons reveals new cell types and organization of long-range connectivity in the mouse brain.Cell179, 268–281. e13 (2019)..

Sun, Q. T. et al. A whole-brain map of long-range inputs to GABAergic interneurons in the mouse medial prefrontal cortex.Nat. Neurosci.22, 1357–1370 (2019)..

Muñoz-Castañeda, R. et al. Cellular anatomy of the mouse primary motor cortex.Nature598, 159–166 (2021)..

Peng, H. C. et al. Morphological diversity of single neurons in molecularly defined cell types.Nature598, 174–181 (2021)..

Foster, N. N. et al. The mouse cortico–basal ganglia–thalamic network.Nature598, 188–194 (2021)..

Zeng, H. K. What is a cell type and how to define it?Cell185, 2739–2755 (2022)..

BRAIN Initiative Cell Census Network (BICCN). A multimodal cell census and atlas of the mammalian primary motor cortex.Nature598, 86–102 (2021)..

Kirshner, H. et al. 3-D PSF fitting for fluorescence microscopy: implementation and localization application.J. Microsc.249, 13–25 (2013)..

Susaki, E. A. et al. Whole-brain imaging with single-cell resolution using chemical cocktails and computational analysis.Cell157, 726–739 (2014)..

Ertürk, A. et al. Three-dimensional imaging of solvent-cleared organs using 3DISCO.Nat. Protoc.7, 1983–1995 (2012)..

Gang, Y. D. et al. Embedding and chemical reactivation of green fluorescent protein in the whole mouse brain for optical micro-imaging.Front. Neurosci.11, 121 (2017)..

Yang, M. C.&Wang, Y. C. F. A self-learning approach to single image super-resolution.IEEE Trans. Multimed.15, 498–508 (2013)..

Han, J. F., Luo, P.&Wang, X. G. Deep self-learning from noisy labels. InProc. IEEE/CVF International Conference on Computer Vision5137–5146 (IEEE, 2019).

Mao, X. D. et al. Least squares generative adversarial networks. InProc. IEEE International Conference on Computer Vision. 2813–2821 (IEEE, 2017).

Lai, W. S. et al. Fast and accurate image super-resolution with deep laplacian pyramid networks.IEEE Trans. Pattern Anal. Mach. Intell.41, 2599–2613 (2019)..

He, K. M. et al. Deep residual learning for image recognition. InProc. IEEE Conference on Computer Vision and Pattern Recognition. 770–778 (IEEE, 2016).

Maas, A. L., Hannun, A. Y.&Ng, A. Y. Rectifier nonlinearities improve neural network acoustic models. InProc. 30th International Conference on Machine Learning(JMLR, 2013).

Isola, P. et al. Image-to-image translation with conditional adversarial networks. InProc. IEEE Conference on Computer Vision and Pattern Recognition(5967–5976) (IEEE, 2017).

Ulyanov, D., Vedaldi, A.&Lempitsky, V. Instance normalization: the missing ingredient for fast stylization. Preprint athttps://doi.org/10.48550/arXiv.1607.08022https://doi.org/10.48550/arXiv.1607.08022(2016).

Kingma, D. P.&Ba, J. Adam: a method for stochasticoptimization. InProc. 3rd International Conference on Learning Representations(ICLR, 2014).

Guo, M. et al. Rapid image deconvolution and multiview fusion for optical microscopy.Nat. Biotechnol.38, 1337–1346 (2020)..

Wang, Z. et al. Image quality assessment: from error visibility to structural similarity.IEEE Trans. Image Process.13, 600–612 (2004)..

Ming, X. et al. Rapid reconstruction of 3D neuronal morphology from light microscopy images with augmented rayburst sampling.PLoS ONE8, e84557 (2013)..

Li, S. W. et al. Brain-wide shape reconstruction of a traced neuron using the convex image segmentation method.Neuroinformatics18, 199–218 (2020)..

Li, Y. X. et al. TDat: an efficient platform for processing petabyte-scale whole-brain volumetric images.Front. Neural Circuits11, 51 (2017)..

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰