Biopsy-free in vivo virtual histology of skin using deep learning

Jingxi Li; Jason Garfinkel; Xiaoran Zhang; Di Wu; Yijie Zhang; Kevin de Haan; Hongda Wang; Tairan Liu; Bijie Bai; Yair Rivenson; Gennady Rubinstein; Philip O. Scumpia; Aydogan Ozcan

doi:10.1038/s41377-021-00674-8

Go to Nature Website

Your Location：

Home >

Browse articles >

Biopsy-free in vivo virtual histology of skin using deep learning

Articles

- Biopsy-free in vivo virtual histology of skin using deep learning
- Light: Science & Applications Vol. 10, Issue 12, Pages: 2421-2442(2021)
- Affiliations：
  
  1.Electrical and Computer Engineering Department, University of California, Los Angeles, CA 90095, USA
  2.Bioengineering Department, University of California, Los Angeles, CA 90095, USA
  3.California NanoSystems Institute (CNSI), University of California, Los Angeles, CA 90095, USA
  4.Dermatology and Laser Centre, Studio City, CA 91604, USA
  5.Computer Science Department, University of California, Los Angeles, CA 90095, USA
  6.Division of Dermatology, University of California, Los Angeles, CA 90095, USA
  7.Department of Dermatology, Veterans Affairs Greater Los Angeles Healthcare System, Los Angeles, CA 90073, USA
  8.Department of Surgery, University of California, Los Angeles, CA 90095, USA
- Author bio：
  
  Gennady Rubinstein (grmd@sbcglobal.net)
  Philip O. Scumpia (pscumpia@mednet.ucla.edu)
  Aydogan Ozcan (ozcan@ucla.edu)
- Funds：
- DOI：10.1038/s41377-021-00674-8
  CLC：
- Published：31 December 2021，
  
  Published Online：18 November 2021，
  
  Received：30 July 2021，
  
  Revised：22 October 2021，
  
  Accepted：28 October 2021
- Accepted：
Scan QR Code
Li, J. X. et al. Biopsy-free in vivo virtual histology of skin using deep learning. Light: Science & Applications, 10, 2421-2442 (2021).
DOI：

Li, J. X. et al. Biopsy-free in vivo virtual histology of skin using deep learning. Light: Science & Applications, 10, 2421-2442 (2021). DOI： 10.1038/s41377-021-00674-8.

摘要

Abstract

An invasive biopsy followed by histological staining is the benchmark for pathological diagnosis of skin tumors. The process is cumbersome and time-consuming

often leading to unnecessary biopsies and scars. Emerging noninvasive optical technologies such as reflectance confocal microscopy (RCM) can provide label-free

cellular-level resolution

in vivo images of skin without performing a biopsy. Although RCM is a useful diagnostic tool

it requires specialized training because the acquired images are grayscale

lack nuclear features

and are difficult to correlate with tissue pathology. Here

we present a deep learning-based framework that uses a convolutional neural network to rapidly transform in vivo RCM images of unstained skin into virtually-stained hematoxylin and eosin-like images with microscopic resolution

enabling visualization of the epidermis

dermal-epidermal junction

and superficial dermis layers. The network was trained under an adversarial learning scheme

which takes ex vivo RCM images of excised unstained/label-free tissue as inputs and uses the microscopic images of the same tissue labeled with acetic acid nuclear contrast staining as the ground truth. We show that this trained neural network can be used to rapidly perform virtual histology of in vivo

label-free RCM images of normal skin structure

basal cell carcinoma

and melanocytic nevi with pigmented melanocytes

demonstrating similar histological features to traditional histology from the same excised tissue. This application of deep learning-based virtual staining to noninvasive imaging technologies may permit more rapid diagnoses of malignant skin neoplasms and reduce invasive skin biopsies.

关键词

Keywords

references

Muzic, J. G. et al. Incidence and trends of basal cell carcinoma and cutaneous squamous cell carcinoma: a population-based study in Olmsted county, Minnesota, 2000 to 2010.Mayo Clin. Proc.92, 890–898 (2017)..

Lim, H. W. et al. The burden of skin disease in the United States.J. Am. Acad. Dermatol.76, 958–972. e2 (2017)..

Grajdeanu, I. A. et al. Use of imaging techniques for melanocytic naevi and basal cell carcinoma in integrative analysis (Review).Exp. Therapeutic Med.20, 78–86 (2020)..

Murzaku, E. C., Hayan, S.&Rao, B. K. Methods and rates of dermoscopy usage: a cross-sectional survey of US dermatologists stratified by years in practice.J. Am. Acad. Dermatol.71, 393–395 (2014)..

Kittler, H. et al. Diagnostic accuracy of dermoscopy.Lancet Oncol.3, 159–165 (2002)..

Koenig, K.&Riemann, I. High-resolution multiphoton tomography of human skin with subcellular spatial resolution and picosecond time resolution.J. Biomed. Opt.8, 432–439 (2003)..

Heibel, H. D., Hooey, L.&Cockerell, C. J. A review of noninvasive techniques for skin cancer detection in dermatology.Am. J. Clin. Dermatol.21, 513–524 (2020)..

Rajadhyaksha, M. et al. In vivo confocal scanning laser microscopy of human skin: melanin provides strong contrast.J. Invest. Dermatol.104, 946–952 (1995)..

Serban, E. D. et al. Role of in vivo reflectance confocal microscopy in the analysis of melanocytic lesions.Acta Dermatovenerol. Croat.26, 64–67 (2018)..

Rajadhyaksha, M. et al. Reflectance confocal microscopy of skin in vivo: from bench to bedside.Lasers Surg. Med.49, 7–19 (2017)..

Rao, B. K. et al. Diagnostic accuracy of reflectance confocal microscopy for diagnosis of skin lesions: an update.Arch. Pathol. Lab. Med.143, 326–329 (2019)..

Jain, M. et al. Evaluation of bedside diagnostic accuracy, learning curve, and challenges for a novice reflectance confocal microscopy reader for skin cancer detection in vivo.JAMA Dermatol.154, 962–965 (2018)..

Gareau, D. S. Feasibility of digitally stained multimodal confocal mosaics to simulate histopathology.J. Biomed. Opt.14, 034050 (2009)..

Puliatti, S. et al. Ex vivo fluorescence confocal microscopy: the first application for real-time pathological examination of prostatic tissue.BJU Int.124, 469–476 (2019)..

Haenssle, H. A. et al. Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists.Ann. Oncol.29, 1836–1842 (2018)..

Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks.Nature542, 115–118 (2017)..

Ehteshami Bejnordi, B. et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer.JAMA318, 2199–2210 (2017)..

Xia, D. et al. Computationally-guided development of a stromal inflammation histologic biomarker in lung squamous cell carcinoma.Sci. Rep.8, 3941 (2018)..

Abels, E. et al. Computational pathology definitions, best practices, and recommendations for regulatory guidance: a white paper from the Digital Pathology Association.J. Pathol.249, 286–294 (2019)..

Amgad, M. et al. Report on computational assessment of tumor infiltrating lymphocytes from the International Immuno-Oncology Biomarker Working Group.npj Breast Cancer6, 16 (2020)..

Diao, J. A. et al. Human-interpretable image features derived from densely mapped cancer pathology slides predict diverse molecular phenotypes.Nat. Commun.12, 1613 (2021)..

Taylor-Weiner, A. et al. A machine learning approach enables quantitative measurement of liver histology and disease monitoring in NASH.Hepatology74, 133–147 (2021)..

Kose, K. et al. Segmentation of cellular patterns in confocal images of melanocytic lesions in vivo via a Multiscale Encoder-Decoder Network (MED-Net).Med. Image Anal.67, 101841 (2021)..

D'Alonzo, M. et al. Semantic segmentation of reflectance confocal microscopy mosaics of pigmented lesions using weak labels.Sci. Rep.11, 3679 (2021)..

Wang, H. D. et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy.Nat. Methods16, 103–110 (2019)..

Wu, Y. C. et al. Three-dimensional virtual refocusing of fluorescence microscopy images using deep learning.Nat. Methods16, 1323–1331 (2019)..

Mela, C. A.&Liu, Y. Application of convolutional neural networks towards nuclei segmentation in localization-based super-resolution fluorescence microscopyimages.BMC Bioinforma.22, 325 (2021)..

Jiao, Y. H. et al. Computational interference microscopy enabled by deep learning.APL Photonics6, 046103 (2021)..

You, S. X. et al. Real-time intraoperative diagnosis by deep neural network driven multiphoton virtual histology.npj Precis. Oncol.3, 33 (2019)..

Rivenson, Y. et al. Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning.Nat. Biomed. Eng.3, 466–477 (2019)..

Rivenson, Y. et al. Emerging advances to transform histopathology using virtual staining.BME Front.2020, 9647163 (2020)..

Zhang, Y. J. et al. Digital synthesis of histological stains using micro-structured and multiplexed virtual staining of label-free tissue.Light. Sci. Appl.9, 78 (2020)..

Rivenson, Y. et al. PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning.Light. Sci. Appl.8, 23 (2019)..

Borhani, N. et al. Digital staining through the application of deep neural networks to multi-modal multi-photon microscopy.Biomed. Opt. Express10, 1339–1350 (2019)..

Goodfellow, I. J. et al. Generative adversarial nets. InProc. 27th International Conference on Neural Information Processing Systems2672–2680 (MIT Press, 2014).

de Haan, K. et al. Deep-learning-based image reconstruction and enhancement in optical microscopy.Proc. IEEE108, 30–50 (2020)..

Drezek, R. A. et al. Laser scanning confocal microscopy of cervical tissue before and after application of acetic acid.Am. J. Obstet. Gynecol.182, 1135–1139 (2000)..

Patel, Y. G. et al. Confocal reflectance mosaicing of basal cell carcinomas in Mohs surgical skin excisions.J. Biomed. Opt.12, 034027 (2007)..

Gareau, D. S. et al. Confocal mosaicing microscopy in skin excisions: a demonstration of rapid surgical pathology.J. Microsc.233, 149–159 (2009)..

Wang, Z. et al. Image quality assessment: from error visibility to structural similarity.IEEE Trans. Image Process.13, 600–612 (2004)..

Ulrich, M. et al. Peritumoral clefting in basal cell carcinoma: correlation of in vivo reflectance confocal microscopy and routine histology.J. Cutan. Pathol.38, 190–195 (2011)..

Pellacani, G. et al. Reflectance confocal microscopy made easy: the 4 must-know key features for the diagnosis of melanoma and nonmelanoma skin cancers.J. Am. Acad. Dermatol.81, 520–526 (2019)..

Navarrete-Dechent, C. et al. Reflectance confocal microscopy terminology glossary for nonmelanocytic skin lesions: a systematic review.J. Am. Acad. Dermatol.80, 1414–1427. e3 (2019)..

Zhang, M. Y. et al. Weakly paired multi-domain image translation. InProc. 31st British Machine Vision Conference. Virtual Event(BMVA Press, 2020).

Mackiewicz-Wysocka, M. et al. Basal cell carcinoma – diagnosis.Contemp. Oncol.17, 337–342 (2013)..

Lee, N. R. et al. Periadnexal mucin as an additional histopathologic feature of chronic eczematous dermatitis.Ann. Dermatol.27, 133–141 (2015)..

Ko, C. J. et al. Visual perception, cognition, and error in dermatologic diagnosis: key cognitive principles.J. Am. Acad. Dermatol.81, 1227–1234 (2019)..

Smyth, M. J. et al. Combination cancer immunotherapies tailored to the tumour microenvironment.Nat. Rev. Clin. Oncol.13, 143–158 (2016)..

Paolino, G. et al. Histology of non-melanoma skin cancers: an update.Biomedicines5, 71 (2017)..

Henke, E., Nandigama, R.&Ergün, S. Extracellular matrix in the tumor microenvironment and its impact on cancer therapy.Front. Mol. Biosci.6, 160 (2020)..

Mugdha, A. C.&Wilson, J. W. Machine learning approach to synthesizing multiphoton microscopic images from reflectance confocal (Conference Presentation). InProc. SPIE 10851, Photonics in Dermatology and Plastic Surgery 2019P 1085106 (SPIE, 2019).

Wilson, J. W.&Mugdha, A. C. Progress in synthetic multiphoton contrast for in vivo microscopy of mucosal melanoma. InProc. Biophotonics Congress: Biomedical Optics 2020(Optical Society of America, 2020).

de Haan, K. et al. Deep learning-based transformation of H&E stained tissues into special stains.Nat. Commun.12, 4884 (2021)..

Balakrishnan, G. et al. VoxelMorph: a learning framework for deformable medical image registration.IEEE Trans. Med. Imaging38, 1788–1800 (2019)..

Isola, P. et al. Image-to-image translation with conditional adversarial networks. InProc. 2017 IEEE Conference on Computer Vision and Pattern Recognition5967–5976 (IEEE, 2017).

Mao, X. D. et al. Least squares generative adversarial networks. InProc. 2017 IEEE International Conference on Computer Vision2813–2821 (IEEE, 2017).

Miyato, T. et al. Spectral normalization for generative adversarial networks. InProc. 6th International Conference on Learning Representations(ICLR, 2018).

Ronneberger, O., Fischer, P.&Brox, T. U-Net: convolutional networks for biomedical image segmentation. InProc. 18th International Conference on Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015234–241 (Springer, 2015).

Schlemper, J. et al. Attention gated networks: learning to leverage salient regions in medical images.Med. Image Anal.53, 197–207 (2019)..

Wong, S. C. et al. Understanding data augmentation for classification: when to warp? InProc. 2016 International Conference on Digital Image Computing: Techniques and Applications1–6 (IEEE, 2016).

Kingma, D. P.&Ba, J. Adam: a method for stochastic optimization. InProc. 3rd International Conference on Learning Representations(ICLR, 2014).

Giacomelli, M. G. et al. Virtual hematoxylin and eosin transillumination microscopy using Epi-fluorescence imaging.PLoS ONE11, e0159337 (2016)..

Carpenter, A. E. et al. CellProfiler: image analysis software for identifying and quantifying cell phenotypes.Genome Biol.7, R100 (2006)..

Waskom, M. L. Seaborn: statistical data visualization.J. Open Source Softw.6, 3021 (2021)..

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰