Deep learning-based virtual staining, segmentation, and classification in label-free photoacoustic histology of human specimens

Chiho Yoon; Eunwoo Park; Sampa Misra; Jin Young Kim; Jin Woo Baik; Kwang Gi Kim; Chan Kwon Jung; Chulhong Kim

doi:10.1038/s41377-024-01554-7

Your Location：

Home >

Browse articles >

Deep learning-based virtual staining, segmentation, and classification in label-free photoacoustic histology of human specimens

Original Articles

- Deep learning-based virtual staining, segmentation, and classification in label-free photoacoustic histology of human specimens
- Light: Science & Applications Vol. 13, Issue 10, Pages: 2353-2366(2024)
- Affiliations：
  
  1.Departments of Electrical Engineering, Convergence IT Engineering, Mechanical Engineering, Medical Science and Engineering, Graduate School of Artificial Intelligence, and Medical Device Innovation Center, Pohang University of Science and Technology (POSTECH), Pohang, Republic of Korea
  2.Opticho Inc., Pohang, Republic of Korea
  3.Department of Health Sciences and Technology, Gachon Advanced Institute for Health Sciences and Technology (GAIHST), Gachon University, Incheon, Republic of Korea
  4.Cancer Research Institute, College of Medicine, The Catholic University of Korea, Seoul, Republic of Korea
  5.Department of Hospital Pathology, Seoul St. Mary's Hospital, College of Medicine, The Catholic University of Korea, Seoul, Republic of Korea
- Author bio：
  
  Chan Kwon Jung (ckjung@catholic.ac.kr)
  Chulhong Kim (chulhong@postech.edu)
- Funds：
- DOI：10.1038/s41377-024-01554-7
  CLC：
- Received：31 January 2024，
  
  Revised：08 July 2024，
  
  Accepted：2024-07-24，
  
  Published Online：02 September 2024，
  
  Published：31 October 2024
- Accepted：
Scan QR Code
Yoon, C. et al. Deep learning-based virtual staining, segmentation, and classification in label-free photoacoustic histology of human specimens. Light: Science & Applications, 13, 2353-2366 (2024).
DOI：

Yoon, C. et al. Deep learning-based virtual staining, segmentation, and classification in label-free photoacoustic histology of human specimens. Light: Science & Applications, 13, 2353-2366 (2024). DOI： 10.1038/s41377-024-01554-7.

摘要

Abstract

In pathological diagnostics

histological images highlight the oncological features of excised specimens

but they require laborious and costly staining procedures. Despite recent innovations in label-free microscopy that simplify complex staining procedures

technical limitations and inadequate histological visualization are still problems in clinical settings. Here

we demonstrate an interconnected deep learning (DL)-based framework for performing automated virtual staining

segmentation

and classification in label-free photoacoustic histology (PAH) of human specimens. The framework comprises three components: (1) an explainable contrastive unpaired translation (E-CUT) method for virtual H&E (VHE) staining

(2) an U-net architecture for feature segmentation

and (3) a DL-based stepwise feature fusion method (StepFF) for classification. The framework demonstrates promising performance at each step of its application to human liver cancers. In virtual staining

the E-CUT preserves the morphological aspects of the cell nucleus and cytoplasm

making VHE images highly similar to real H&E ones. In segmentation

various features (e.g.

the cell area

number of cells

and the distance between cell nuclei) have been successfully segmented in VHE images. Finally

by using deep feature vectors from PAH

VHE

and segmented images

StepFF has achieved a 98.00% classification accuracy

compared to the 94.80% accuracy of conventional PAH classification. In particular

StepFF's classification reached a sensitivity of 100% based on the evaluation of three pathologists

demonstrating its applicability in real clinical settings. This series of DL methods for label-free PAH has great potential as a practical clinical strategy for digital pathology.

关键词

Keywords

references

Bancroft, J. D.&Gamble, M. Theory and practice of histological techniques. 6th edn. (Philadelphia: Elsevier, 2008).

Musumeci, G. Past, present and future: overview on histology and histopathology. J. Histol. Histopathol. 1 , 5 (2014)..

Gurcan, M. N. et al. Histopathological image analysis: a review. IEEE Rev. Biomed. Eng. 2 , 147–171(2009)..

Zhu, E. J. et al. Intraoperatively measured tumor size and frozen section results should be considered jointly to predict the final pathology for lung adenocarcinoma. Mod. Pathol. 31 , 1391–1399 (2018)..

Ounkomol, C. et al. Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy. Nat. Methods 15 , 917–920 (2018)..

Zhang, Y. J. et al. Digital synthesis of histological stains using micro-structured and multiplexed virtual staining of label-free tissue. Light Sci. Appl. 9 , 78 (2020)..

Cheng, S. Y. et al. Single-cell cytometry via multiplexed fluorescence prediction by label-free reflectance microscopy. Sci. Adv. 7 , eabe0431 (2021)..

Visscher, M. et al. Label-free analytic histology of carotid atherosclerosis by mid-infrared optoacoustic microscopy. Photoacoustics 26 , 100354 (2022)..

Pleitez, M. A. et al. Label-free metabolic imaging by mid-infrared optoacoustic microscopy in living cells. Nat. Biotechnol. 38 , 293–296 (2020)..

Glaser, A. K. et al. Light-sheet microscopy for slide-free non-destructive pathology of large clinical specimens. Nat. Biomed. Eng. 1 , 0084 (2017)..

Olarte, O. E. et al. Light-sheet microscopy: a tutorial. Adv. Opt. Photonics 10 , 111–179 (2018)..

Selinummi, J. et al. Bright field microscopy as an alternative to whole cell fluorescence in automated analysis of macrophage images. PLoS One 4 , e7497 (2009)..

Bouma, B. E. et al. Optical coherence tomography. Nat. Rev. Methods Prim. 2 , 79 (2022)..

You, S. X. et al. Intravital imaging by simultaneous label-free autofluorescence-multiharmonic microscopy. Nat. Commun. 9 , 2125 (2018)..

Jones, R. R. et al. Raman techniques: fundamentals and frontiers. Nanoscale Res. Lett. 14 , 231 (2019)..

Petersen, D. et al. Virtual staining of colon cancer tissue by label-free Raman micro-spectroscopy. Analyst 142 , 1207–1215 (2017)..

Xu, C. Y. et al. Spectroscopic spectral-domain optical coherence microscopy. Opt. Lett. 31 , 1079–1081 (2006)..

Ojaghi, A. et al. Label-free deep-UV microscopy detection and grading of neutropenia using a passive microfluidic device. Opt. Lett. 47 , 6005–6008 (2022)..

Kim, J. et al. Super-resolution localization photoacoustic microscopy using intrinsic red blood cells as contrast absorbers. Light Sci. Appl. 8 , 103 (2019)..

Lee, H. et al. Transportable multispectral optical-resolution photoacoustic microscopy using stimulated Raman scattering spectrum. IEEE Trans. Instrum. Meas. 7 , 4502309 (2024)..

Park, E. et a l. Effective photoacoustic absorption spectrum for collagen-based tissue imaging. J. Biomed. Opt. 25 , 056002 (2020)..

Kim, D. et al. In vivo quantitative photoacoustic monitoring of corticosteroid-induced vasoconstriction. J. Biomed. Opt. 28 , 082805 (2023)..

Martell, M. T. et al. Deep learning-enabled realistic virtual histology with ultraviolet photoacoustic remote sensing microscopy. Nat. Commun. 14 , 5967 (2023)..

Park, B. et al. Functional photoacoustic imaging: from nano-and micro-to macro-scale. Nano Converg. 10 , 29 (2023)..

Ahn, J. et al. In vivo photoacoustic monitoring of vasoconstriction induced by acute hyperglycemia. Photoacoustics 30 , 100485 (2023)..

Choi, W. et al. Recent advances in contrast-enhanced photoacoustic imaging: overcoming the physical and practical challenges. Chem. Rev. 123 , 7379–7419 (2023)..

Ahn, J. et al. Fully integrated photoacoustic microscopy and photoplethysmography of human in vivo. Photoacoustics 27 , 100374 (2022)..

Cho, S. W. et al. High-speed photoacoustic microscopy: a review dedicated on light sources. Photoacoustics 24 , 100291 (2021)..

Park, E. et al. Azimuth mapping of fibrous tissue in linear dichroism-sensitive photoacoustic microscopy. Photoacoustics 31 , 100510 (2023)..

Park, J. et al. Quadruple ultrasound, photoacoustic, optical coherence, and fluorescence fusion imaging with a transparent ultrasound transducer. Proc. Natl Acad. Sci. USA 118 , e1920879118 (2021)..

Cho, S. et al. An ultrasensitive and broadband transparent ultrasound transducer for ultrasound and photoacoustic imaging in-vivo. Nat. Commun. 15 , 1444 (2024)..

Wong, T. T. et al. Label-free automated three-dimensional imaging of whole organs by microtomy-assisted photoacoustic microscopy. Nat. Commun. 8 , 1386 (2017)..

Wong, T. T. W. et al. Fast label-free multilayered histology-like imaging of human breast cancer by photoacoustic microscopy. Sci. Adv. 3 , e1602168 (2017)..

Kim, D. et al. An ultraviolet-transparent ultrasound transducer enables high-resolution label-free photoacoustic histopathology. Laser Photonics Rev. 18 , 2300652 (2024)..

Yang, J. G. et al. Recent advances in deep-learning-enhanced photoacoustic imaging. Adv. Photonics Nexus 2 , 054001 (2023)..

Choi, S. et al. Deep learning enhances multiparametric dynamic volumetric photoacoustic computed tomography in vivo (DL‐PACT). Adv. Sci. 10 , 2202089 (2023)..

Kim, J. et al. Deep learning acceleration of multiscale superresolution localization photoacoustic imaging. Light Sci. Appl. 11 , 131 (2022)..

Jeon, S. et al. A deep learning-based model that reduces speed of sound aberrations for improved in vivo photoacoustic imaging. IEEE Trans. Image Process. 30 , 8773–8784 (2021)..

Li, Y. Z. et al. Virtual histological staining of unlabeled autopsy tissue. Nat. Commun. 15 , 1684 (2024)..

Bai, B. J. et al. Deep learning-enabled virtual histological staining of biological samples. Light Sci. Appl. 12 , 57 (2023)..

Kendall, W. Y. et al. Deep learning classification of ex vivo human colon tissues using spectroscopic OCT. bioRxiv . https://doi.org/10.1101/2023.09.04.555974 (2023).

Zhang, Y. J. et al. Virtual staining of defocused autofluorescence images of unlabeled tissue using deep neural networks. Intell. Comput. 2022 , 9818965 (2022)..

Zhang, G. H. et al. Image-to-images translation for multiple virtual histological staining of unlabeled human carotid atherosclerotic tissue. Mol. Imaging Biol. 24 , 31–41 (2022)..

Winetraub, Y. et al. OCT2Hist: non-invasive virtual biopsy using optical coherence tomography. medRxiv . https://doi.org/10.1101/2021.03.31.21254733 (2021).

Picon, A. et al. Autofluorescence image reconstruction and virtual staining for in-vivo optical biopsying. IEEE Access 9 , 32081–32093 (2021)..

Kaza, N., Ojaghi, A. & Robles, F. E. Virtual staining, segmentation, and classification of blood smears for lab el-free hematology analysis. BME Front. 2022 , 9853606 (2022)..

Boktor, M. et al. Virtual histological staining of label-free total absorption photoacoustic remote sensing (TA-PARS). Sci. Rep. 12 , 10296 (2022)..

Rivenson, Y. et al. Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning. Nat. Biomed. Eng. 3 , 466–477 (2019)..

Zhu, J. Y. et al. Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the 2017 IEEE International Conference on Computer Vision, 2223–2232. Venice, Italy: IEEE, 2017.

Tsai, S. T. et al. H & E-like staining of OCT images of human skin via generative adversarial network. Appl. Phys. Lett. 121 , 134102 (2022)..

Meng, X. Y., Li, X. & Wang, X. A computationally virtual histological staining method to ovarian cancer tissue by deep generative adversarial networks. Comput. Math. Methods Med. 2021 , 4244157 (2021)..

Li, X. Y. et al. Unsupervised content-preserving transformation for optical microsco py. Light Sci. Appl. 10 , 44 (2021)..

Cao, R. et al. Label-free intraoperative histology of bone tissue via deep-learning-assisted ultraviolet photoacoustic microscopy. Nat. Biomed. Eng. 7 , 124–134 (2023)..

Kang, L. et al. Deep learning enables ultraviolet photoacoustic microscopy based histological imaging with near real-time virtual staining. Photoacoustics 25 , 100308 (2022)..

Liu, Z. J. et al. Virtual formalin-fixed and paraffin-embedded staining of fresh brain tissue via stimulated Raman CycleGAN model. Sci. Adv. 10 , eadn3426 (2024)..

Pérez, J. C. G., Baguer, D. O. & Maass, P. StainCUT: stain normalization with contrastive learning. J. Imaging 8 , 202 (2022)..

Park, T. et al. Contrastive learning for unpaired image-to-image translation. Proceedings of the 16th European Conference on Computer Vision (pp. 319–345. Springer, Glasgow, UK, 2020).

Bayramoglu, N., Kannala, J. & Heikkilä, J. Deep learning for magnification independent breast cancer histopathology image classification. In : Proceedings of the 23rd International Conference on Pattern Recognition (ICPR), 2440–2445. Cancun, Mexico: IEEE, 2016.

Coudray, N. et al. Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning. Nat. Med. 24 , 1559–1567 (2018)..

Wang, W., Ozolek, J. A. & Rohde, G. K. Detection and classification of thyroid follicular lesions based on nuclear structure from histopathology images. Cytom. Part A 77A , 485–494 (2010)..

Mahmood, F. et al. Deep adversarial training for multi-organ nuclei segmentation in histopathology images. IEEE Trans. Med. Imaging 39 , 3257–3267 (2020)..

Wu, J. C. et al. Learned end-to-end high-resolution lensless fiber imaging towards real-time cancer diagnosis. Sci. Rep. 12 , 18846 (2022)..

Baik, J. W. et al. Intraoperative label‐free photoacoustic histopathology of clinical specimens. Laser Photonics Rev. 15 , 2100124 (2021)..

Stoscheck, C. M. [6 ] Quantitation of protein. Methods Enzymol. 182 , 50–68 (1990) ..

Yao, D. K. et al. In vivo label-free photoacoustic microscopy of cell nuclei by excitation of DNA and RNA. Opt. Lett. 35 , 4139–4141 (2010)..

Tjoa, E. & Guan, C. T. Quantifying explainability of saliency methods in deep neural networks with a synthetic dataset. IEEE Trans. Artif. Intell. 4 , 858–870 (2023)..

Borji, A. Saliency prediction in the deep learning era: successes and limitations. IEEE Trans. Pattern Anal. Mach. Intell. 43 , 679–700 (2021)..

Alqaraawi, A. et al. Evaluating saliency map explanations for convolutional neural networks: a user study. In : Proceedings of the 25th International Conference on Intelligent User Interfaces, 275–285. Cagliari, Italy: ACM, 2020.

Saadallah, A., Jakobs, M. & Morik, K. Explainable online deep neural network selection using adaptive saliency maps for time series forecasting. In : Proceedings of the European Conference on Machine Learning and Knowledge Discovery in Databases, 404–420. Bilbao, Spain: Springer, 2021.

Sundararajan, M., Taly, A. & Yan, Q. Q. Axiomatic attribution for deep net works. In : Proceedings of the 34th International Conference on Machine Learning, 3319–3328. Sydney, NSW, Australia: JMLR. org, 2017.

Heusel, M. et al. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In : Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach, California, USA, 6629–6640, Curran Associates Inc., 2017.

Bińkowski, M. et al. Demystifying MMD GANs. In : Proceedings of the6th International Conference on Learning Representations. Vancouver, BC, Canada: OpenReview. net, 2018.

Ronneberger, O., Fischer, P. & Brox, T. U-Net: convolutional networks for biomedical image segmentation. In : Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention. Munich, Germany, 234–241 (Springer, 2015).

Vu, Q. D. et al. Methods for segmentation and classification of digital microscopy tissue images. Front. Bioeng. Biotechnol. 7 , 53 (2019)..

Kumar, N. et al. A dataset and a technique for generalized nuclear segmentation for computational pathology. IEEE Trans. Med. Imaging 36 , 1550–1560 (2017)..

Naylor, P. et al. Segmentation of nuclei in histopathology images by deep regression of the distance map. IEEE Trans. Med. Imaging 38 , 448–459 (2019)..

Bradski, G. The openCV library. Dr. Dobb's. J. Softw. Tools 25 , 120–125 (2000)..

Vinutha, H. P. et al. Detection of outliers using interquartile range technique from intrusion dataset. In : Information and Decision Sciences. 511–518 (Singapore: Springer, 2018).

He, K. M. et al. Deep residual learning for image recognition. In : Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA: IEEE, 2016, 770–778.

Tan, M. X. & Le, Q. V. Efficientnet: rethinking model scaling for convolutional neural networks. In: Proceedings of the 36th International Conference on Machine Learning . Long Beach, California, USA: PMLR, 2019, 6105-6114.

Szegedy, C. et al. Rethinking the inception architecture for computer vision. In: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition . Las Vegas, NV, USA: IEEE, 2016, 2818–2826.

Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. In: Proceedings of the 3rd International Conference on Learning Representations . San Diego, CA, USA, 2015.

Liu, Z. et al. Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision . Montreal, QC, Canada: IEEE, 2021, 10012–10022.

Bosman, F. T. et al. WHO Classification of Tumours of the Digestive System . 4th edn. (World Health Organization, Geneva, 2010).

McHugh, M. L. Interrater reliability: the kappa statistic. Biochem. Med. 22 , 276–282 (2012)..

Selvaraju, R. R. et al. Grad-CAM: Visual explanations from deep networks via gradient-based localization. Proceedingsof the 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017, 618–626.

Petsiuk, V., Das, A.&Saenko, K. RISE: randomized input sampling for explanation of black-box models. Proceedings of the British Machine Vision Conference 2018. Newcastle, UK: BMVA Press, 2018.

Fong, R., Patrick, M.&Vedaldi, A. Understanding deep networks via extremal perturbations and smooth masks. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Seoul, Korea (South): IEEE, 2950-2958.

Johnson, J., Alahi, A.&Fei-Fei, L. Perceptual losses for real-time style transfer and super-resolution. Proceedings of the 14th European Conference on Computer Vision. Amsterdam, The Netherlands: Springer, 2016, 694–711.

Isola, P. et al. A. Image-to-image translation with conditional adversarial networks. In: Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu, HI, USA: IEEE, 2017, 1125–1134.

Kokhlikyan, N. et al. Captum: a unified and generic model interpretability library for PyTorch. Print at https://arxiv.org/abs/2009.07896 https://arxiv.org/abs/2009.07896 (2020).

Mao, X. D. et al. Least squares generative adversarial networks. Proceedings of the 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017, 2794–2802.

Kingma, D. P.&Ba, J. Adam: a method for stochastic optimization. Proceedings of the 3rd International Conference on Learning Representations. San Diego, CA, USA,2015.

Jadon, S. A survey of loss functions for semantic segmentation. Proceedings of the 2020 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB). Via del Mar, Chile: IEEE, 2020, 1–7.

Plotly Technologies Inc. Collaborative Data Science. (Montréal, QC: Plotly Technologies Inc., 2015).

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰