Deep learning acceleration of multiscale superresolution localization photoacoustic imaging

Jongbeom Kim; Gyuwon Kim; Lei Li; Pengfei Zhang; Jin Young Kim; Yeonggeun Kim; Hyung Ham Kim; Lihong V. Wang; Seungchul Lee; Chulhong Kim

doi:10.1038/s41377-022-00820-w

Go to Nature Website

Your Location：

Home >

Browse articles >

Deep learning acceleration of multiscale superresolution localization photoacoustic imaging

Articles

- Deep learning acceleration of multiscale superresolution localization photoacoustic imaging
- Light: Science & Applications Vol. 11, Issue 6, Pages: 1166-1177(2022)
- Affiliations：
  
  1.Departments of Electrical Engineering, Mechanical Engineering, Convergence IT Engineering, and Interdisciplinary Bioscience and Bioengineering, Graduate School of Artificial Intelligence, Medical Device Innovation Center, Pohang University of Science and Technology (POSTECH), 77 Cheongam-ro, Nam-gu, Pohang, Gyeongbuk 37673, Republic of Korea
  2.Caltech Optical Imaging Laboratory, Andrew and Peggy Cherng Department of Medical Engineering, Department of Electrical Engineering, California Institute of Technology, 1200 E. California Blvd., MC 138-78, Pasadena, CA 91125, USA
  3.School of Precision Instruments and Optoelectronics Engineering, Tianjin University, 92 Weijin Road, Nankai District, Tianjin 300072, China
  4.Opticho, 532, CHANGeUP GROUND, 87 Cheongam-ro, Nam-gu, Pohang, Gyeongsangbuk 37673, Republic of Korea
- Author bio：
  
  Lihong V. Wang (LVW@caltech.edu)
  Seungchul Lee (seunglee@postech.ac.kr)
  Chulhong Kim (chulhong@postech.edu)
- Funds：
- DOI：10.1038/s41377-022-00820-w
  CLC：
- Published：30 June 2022，
  
  Published Online：12 May 2022，
  
  Received：21 November 2021，
  
  Revised：24 April 2022，
  
  Accepted：26 April 2022
- Accepted：
Scan QR Code
Kim, J. et al. Deep learning acceleration of multiscale superresolution localization photoacoustic imaging. Light: Science & Applications, 11, 1166-1177 (2022).
DOI：

Kim, J. et al. Deep learning acceleration of multiscale superresolution localization photoacoustic imaging. Light: Science & Applications, 11, 1166-1177 (2022). DOI： 10.1038/s41377-022-00820-w.

摘要

Abstract

A superresolution imaging approach that localizes very small targets

such as red blood cells or droplets of injected photoacoustic dye

has significantly improved spatial resolution in various biological and medical imaging modalities. However

this superior spatial resolution is achieved by sacrificing temporal resolution because many raw image frames

each containing the localization target

must be superimposed to form a sufficiently sampled high-density superresolution image. Here

we demonstrate a computational strategy based on deep neural networks (DNNs) to reconstruct high-density superresolution images from far fewer raw image frames. The localization strategy can be applied for both 3D label-free localization optical-resolution photoacoustic microscopy (OR-PAM) and 2D labeled localization photoacoustic computed tomography (PACT). For the former

the required number of raw volumetric frames is reduced from tens to fewer than ten. For the latter

the required number of raw 2D frames is reduced by 12 fold. Therefore

our proposed method has simultaneously improved temporal (via the DNN) and spatial (via the localization method) resolutions in both label-free microscopy and labeled tomography. Deep-learning powered localization PA imaging can potentially provide a practical tool in preclinical and clinical studies requiring fast temporal and fine spatial resolutions.

关键词

Keywords

references

Wang, L. V.&Hu, S. Photoacoustic tomography: in vivo imaging from organelles to organs.Science335, 1458–1462 (2012)..

Jeon, S. et al. Review on practical photoacoustic microscopy.Photoacoustics15, 100141 (2019)..

Jeon, S. et al. In vivo photoacoustic imaging of anterior ocular vasculature: a random sample consensus approach.Sci. Rep.7, 4318 (2017)..

Kim, H. et al. PAExM: label-free hyper-resolution photoacoustic expansion microscopy.Opt. Lett.45, 6755–6758 (2020)..

Baik, J. W. et al. Super wide-field photoacoustic microscopy of animals and humans in vivo.IEEE Trans. Med. Imaging39, 975–984 (2020)..

Kim, J. Y. et al. Fast optical-resolution photoacoustic microscopy using a 2-axis water-proofing MEMS scanner.Sci. Rep.5, 7932 (2015)..

Wong, T. T. W. et al. Label-free automated three-dimensional imaging of whole organs by microtomy-assisted photoacoustic microscopy.Nat. Commun.8, 1386 (2017)..

Shi, J. H. et al. High-resolution, high-contrast mid-infrared imaging of fresh biological samples with ultraviolet-localized photoacoustic microscopy.Nat. Photonics13, 609–615 (2019)..

Yao, J. J. et al. High-speed label-free functional photoacoustic microscopy of mouse brain in action.Nat. Methods12, 407–410 (2015)..

Kim, J. et al. Superresolution localization photoacoustic microscopy using intrinsic red blood cells as contrast absorbers.Light. Sci. Appl.8, 103 (2019)..

Baik, J. W. et al. Intraoperative label-free photoacoustic histopathology of clinical specimens.Laser Photonics Rev.15, 2100124 (2021)..

Ahn, J. et al. High-resolution functional photoacoustic monitoring of vascular dynamics in human fingers.Photoacoustics23, 100282 (2021)..

Cho, S. W. et al. High-speed photoacoustic microscopy: a review dedicated on light sources.Photoacoustics24, 100291 (2021)..

Park, J. et al. Quadruple ultrasound, photoacoustic, optical coherence, and fluorescence fusion imaging with a transparent ultrasound transducer.Proc. Natl Acad. Sci. USA118, e1920879118 (2021)..

Lin, L. et al. Single-breath-hold photoacoustic computed tomography of the breast.Nat. Commun.9, 2352 (2018)..

Park, B. et al. 3D wide-field multispectral photoacoustic imaging of human melanomas in vivo: a pilot study.J. Eur. Acad. Dermatol. Venereol.35, 669–676 (2021)..

Na, S. et al. Massively parallel functional photoacoustic computed tomography of the human brain.Nat. Biomed. Eng.1–9 (2021).

Kim, J. et al. Multiparametric photoacoustic analysis of human thyroid cancers in vivo.Cancer Res.81, 4849–4860 (2021)..

Choi, W. et al. Clinical photoacoustic imaging platforms.Biomed. Eng. Lett.8, 139–155 (2018)..

Yao, J. J.&Wang, L. V. Photoacoustic microscopy.Laser Photonics Rev.7, 758–778 (2013)..

Yao, J. J. et al. Photoimprint photoacoustic microscopyfor three-dimensional label-free subdiffraction imaging.Phys. Rev. Lett.112, 014302 (2014)..

Betzig, E. et al. Imaging intracellular fluorescent proteins at nanometer resolution.Science313, 1642–1645 (2006)..

Rust, M. J., Bates, M.&Zhuang, X. W. Sub-diffraction-limit imaging by stochastic optical reconstruction microscopy (STORM).Nat. Methods3, 793–796 (2006)..

Danielli, A. et al. Label-free photoacoustic nanoscopy.J. Biomed. Opt.19, 086006 (2014)..

Zhang, P. F. et al. In vivo superresolution photoacoustic computed tomography by localization of single dyed droplets.Light. Sci. Appl.8, 36 (2019)..

Dean-Ben, X. L.&Razansky, D. Localization optoacoustic tomography.Light. Sci. Appl.7, 18004 (2018)..

Vilov, S., Arnal, B.&Bossy, E. Overcoming the acoustic diffraction limit in photoacoustic imaging by the localization of flowing absorbers.Opt. Lett.42, 4379–4382 (2017)..

Choi, W.&Kim, C. Toward in vivo translation of super-resolution localization photoacoustic computed tomography using liquid-state dyed droplets.Light. Sci. Appl.8, 57 (2019)..

Zhao, H. X. et al. Deep learning enables superior photoacoustic imaging at ultralow laser dosages.Adv. Sci.8, 2003097 (2021)..

Ouyang, W. et al. Deep learning massively accelerates super-resolution localization microscopy.Nat. Biotechnol.36, 460–468 (2018)..

DiSpirito, A. et al. Reconstructing undersampled photoacoustic microscopy images using deep learning.IEEE Trans. Med. Imaging40, 562–570 (2021)..

Wang, H. D. et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy.Nat. Methods16, 103–110 (2019)..

Nehme, E. et al. DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning.Nat. Methods17, 734–740 (2020)..

Qiao, C. et al. Evaluation and development of deep neural networks for image super-resolution in optical microscopy.Nat. Methods18, 194–202 (2021)..

Milecki, L. et al. A deep learning framework for spatiotemporal ultrasound localization microscopy.IEEE Trans. Med. Imaging40, 1428–1437 (2021)..

Masutani, E. M., Bahrami, N.&Hsiao, A. Deep learning single-frame and multiframe super-resolution for cardiac MRI.Radiology295, 552–561 (2020)..

Brady, S. L. et al. Improving image quality and reducing radiation dose for pediatric CT by using deep learning reconstruction.Radiology298, 180–188 (2021)..

Ronneberger, O., Fischer, P.&Brox, T. U-Net: convolutional networks for biomedical image segmentation. In: Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention. Munich: Springer, 234–241 (2015).

Goodfellow, I. J. et al. Generative adversarial nets. In: Proceedings of the 27th International Conference on Neural Information Processing Systems. Montreal: MIT Press, 2672–2680 (2014).

Isola, P. et al. Image-to-image translation with conditional adversarial networks. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 5967–5976 (2017).

Vu, T. et al. Deep image prior for undersampling high-speed photoacoustic microscopy.Photoacoustics22, 100266 (2021)..

Shi, W. Z. et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 1874–1883 (2016).

Drozdzal, M. et al. The importance of skip connections in biomedical image segmentation. In: Proceedings of the 1st International Workshop on Deep Learning in Medical Image Analysis. Athens. Greece: Springer, 179–187 (2016).

Raghu, M. et al. Transfusion: understanding transfer learning for medical imaging.Adv. Neural Inf. Process. Syst.32, 3347–3357 (2019)..

Cho, S. et al. 3D PHOVIS: 3D photoacoustic visualization studio.Photoacoustics18, 100168 (2020)..

Wang, Z., Simoncelli, E. P.&Bovik, A. C. Multiscale structural similarity for image quality assessment. In: Proceedings of the Thrity-Seventh Asilomar Conference on Signals, Systems&Computers. Pacific Grove: IEEE, 1398–1402 (2003).

Selvaraju, R. R. et al. Grad-CAM: visual explanations from deep networks via gradient-based localization. In: Proceedings of 2017 IEEE International Conference on Computer Vision. Venice: IEEE, 618–626 (2017).

Bach, S. et al. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.PLoS One10, e0130140 (2015)..

Li, L. et al. Single-impulse panoramic photoacoustic computed tomography of small-animal whole-body dynamics at high spatiotemporal resolution.Nat. Biomed. Eng.1, 1–11 (2017)..

Srivastava, N. et al. Dropout: a simple way to prevent neural networks from overfitting.J. Mach. Learn. Res.15, 1929–1958 (2014)..

Ioffe, S.&Szegedy, C. Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd International Conference on International Conference on Machine Learning. Lille, France: PMLR, 448–456 (2015).

He, K. M. et al. Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile: IEEE, 1026–1034 (2015).

Zhao, H. et al. Loss functions for image restoration with neural networks.IEEE Trans. Comput. Imaging3, 47–57 (2017)..

Kingma, D. P.&Ba, L. J. Adam: a method for stochastic optimization. In: Proceedings of the 3rd International Conference on Learning Representations. San Diego, 2015.

Goodfellow, I., Bengio, Y.&Courville, A.Deep Learning.(MIT Press, Cambridge, 2016).

Feng, J. et al. Catheterization of the carotid artery and jugular vein to perform hemodynamic measures, infusions and blood sampling in a conscious rat model.J. Vis. Exp.30, 51881 (2015)..

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰