Unsupervised content-preserving transformation for optical microscopy

Xinyang Li; Guoxun Zhang; Hui Qiao; Feng Bao; Yue Deng; Jiamin Wu; Yangfan He; Jingping Yun; Xing Lin; Hao Xie; Haoqian Wang; Qionghai Dai

doi:10.1038/s41377-021-00484-y

Go to Nature Website

Your Location：

Home >

Browse articles >

Unsupervised content-preserving transformation for optical microscopy

Articles

- Unsupervised content-preserving transformation for optical microscopy
- Light: Science & Applications Vol. 10, Issue 3, Pages: 390-400(2021)
- Affiliations：
  
  1.Department of Automation, Tsinghua University, Beijing, 100084, China
  2.Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, 518055, China
  3.Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing, 100084, China
  4.School of Astronautics, Beihang University, Beijing, 100191, China
  5.Beijing Advanced Innovation Center for Big Data and Brain Computing, Beihang University, Beijing, 100191, China
  6.Department of Pathology, Sun Yat-sen University Cancer Center, Guangzhou, 510060, China
  7.State Key Laboratory of Oncology in South China, Guangzhou, 510060, China
  8.Collaborative Innovation Center for Cancer Medicine, Guangzhou, 510060, China
  9.Beijing Innovation Center for Future Chips, Tsinghua University, Beijing, 100084, China
- Author bio：
  
  Haoqian Wang (wanghaoqian@tsinghua.edu.cn)
  Qionghai Dai (qhdai@mail.tsinghua.edu.cn)
- Funds：
- DOI：10.1038/s41377-021-00484-y
  CLC：
- Published：31 March 2021，
  
  Published Online：01 March 2021，
  
  Received：13 November 2020，
  
  Revised：29 January 2021，
  
  Accepted：30 January 2021
- Accepted：
Scan QR Code
Li, X. y. et al. Unsupervised content-preserving transformation for optical microscopy. Light: Science & Applications 10, 390-400 (2021).
DOI：

Li, X. y. et al. Unsupervised content-preserving transformation for optical microscopy. Light: Science & Applications 10, 390-400 (2021). DOI： 10.1038/s41377-021-00484-y.

摘要

Abstract

The development of deep learning and open access to a substantial collection of imaging data together provide a potential solution for computational image transformation

which is gradually changing the landscape of optical imaging and biomedical research. However

current implementations of deep learning usually operate in a supervised manner

and their reliance on laborious and error-prone data annotation procedures remains a barrier to more general applicability. Here

we propose an unsupervised image transformation to facilitate the utilization of deep learning for optical microscopy

even in some cases in which supervised models cannot be applied. Through the introduction of a saliency constraint

the unsupervised model

named Unsupervised content-preserving Transformation for Optical Microscopy (UTOM)

can learn the mapping between two image domains without requiring paired training data while avoiding distortions of the image content. UTOM shows promising performance in a wide range of biomedical image transformation tasks

including in silico histological staining

fluorescence image restoration

and virtual fluorescence labeling. Quantitative evaluations reveal that UTOM achieves stable and high-fidelity image transformations across different imaging conditions and modalities. We anticipate that our framework will encourage a paradigm shift in training neural networks and enable more applications of artificial intelligence in biomedical imaging.

关键词

Keywords

references

LeCun, Y., Bengio, Y.&Hinton, G. Deep learning.Nature521, 436-444 (2015)..

Barbastathis, G., Ozcan, A.&Situ, G. On the use of deep learning for computational imaging.Optica6, 921-943 (2019)..

Moen, E. et al. Deep learning for cellular image analysis.Nat. Methods16, 1233-1246 (2019)..

Hornik, K., Stinchcombe, M.&White, H. Multilayer feedforward networks are universal approximators.Neural Netw.2, 359-366 (1989)..

Belthangady, C.&Royer, L. A. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction.Nat. Methods16, 1215-1225 (2019)..

Ronneberger, O., Fischer, P.&Brox, T. U-net: convolutional networks for biomedical image segmentation. Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention. Munich, Germany: Springer, 2015, 234-241.

Falk, T. et al. U-Net: deep learning for cell counting, detection, and morphometry.Nat. Methods16, 67-70 (2019)..

Weigert, M. et al. Content-aware image restoration: pushing the limits of fluorescence microscopy.Nat. Methods15, 1090-1097 (2018)..

Ounkomol,C. et al. Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy.Nat. Methods15, 917-920 (2018)..

Rivenson, Y. et al. Deep learning microscopy.Optica4, 1437-1443 (2017)..

Christiansen, E. M. et al. In silico labeling: predicting fluorescent labels in unlabeled images.Cell173, 792-803. e19 (2018)..

Goodfellow, I. J. et al. Generative adversarial nets. Proceedings of the 27th International Conference on Neural Information Processing Systems. Long Beach, USA: NIPS, 2014, 2672-2680.

Isola, P. et al. Image-to-image translation with conditional adversarial networks. Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE, 2017, 5967-5976.

Wang, H. D. et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy.Nat. Methods16, 103-110 (2019)..

Ouyang, W. et al. Deep learning massively accelerates super-resolution localization microscopy.Nat. Biotechnol.36, 460-468 (2018)..

Wu, Y. C. et al. Bright-field holography: cross-modality deep learning enables snapshot 3D imaging with bright-field contrast using a single hologram.Light. : Sci. Appl.8, 25 (2019)..

Rivenson, Y. et al. Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning.Nat. Biomed. Eng.3, 466-477 (2019)..

Zhu, J. Y. et al. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017, 2242-2251.

Zhang, Y. B. et al. Multiple cycle-in-cycle generative adversarial networks for unsupervised image super-resolution.IEEE Trans. Image Process.29, 1101-1112 (2019)..

Choi, Y. et al. StarGAN: unified generative adversarial networks for multi-domain image-to-image translation. Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE, 2018, 8789-8797.

Yi, Z. L. et al. DualGAN: unsupervised dual learning for image-to-image translation. Proceedings of 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017, 2868-2876.

Kang, E. et al. Cycle-consistent adversarial denoising network for multiphase coronary CT angiography.Med. Phys.46, 550-562 (2019)..

You, C. Y. et al. CT super-resolution GAN Constrained by the Identical, Residual, and Cycle Learning Ensemble (GAN-CIRCLE).IEEE Transactions on Medical.Imaging39, 188-203 (2020)..

Sim, B. et al. Optimal transport driven CycleGAN for unsupervised learning in inverse problems (2019). athttps://arxiv.org/abs/1909.12116https://arxiv.org/abs/1909.12116.

Choi, G. et al. Cycle-consistent deep learning approach to coherent noise reduction in optical diffraction tomography.Opt. Express27, 4927-4943 (2019)..

Ihle, S. J. et al. Unsupervised data to content transformation with histogram-matching cycle-consistent generative adversarial networks.Nat. Mach. Intell.1, 461-470 (2019)..

Gharesifard, B.&Cortés, J. Distributed convergence to Nash equilibria in two-network zero-sum games.Automatica49, 1683-1692 (2013)..

Coudray, N. etal. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning.Nat. Med.24, 1559-1567 (2018)..

Lu, F. K. et al. Label-free neurosurgical pathology with stimulated Raman imaging.Cancer Res.76, 3451-3462 (2016)..

Orringer, D. A. et al. Rapid intraoperative histology of unprocessed surgical specimens via fibre-laser-based stimulated Raman scattering microscopy.Nat. Biomed. Eng.1, 0027 (2017)..

Hollon, T. C. et al. Near real-time intraoperative brain tumor diagnosis using stimulated Raman histology and deep neural networks.Nat. Med.26, 52-58 (2020)..

Zhang, Y. H. et al. PhaseGAN: a deep-learning phase-retrieval approach for unpaired datasets (2020). athttps://arxiv.org/abs/2011.08660https://arxiv.org/abs/2011.08660.

Johnson, J., Alahi, A.&Li, F. F. Perceptual losses for real-time style transfer and super-resolution. Proceedings of the 14th European Conference on Computer Vision. Amsterdam, The Netherlands: Springer, 2016, 694-711.

Li, C.&Wand, M. Precomputed real-time texture synthesis with markovian generative adversarial networks. Proceedings of the 14th European Conference on Computer Vision. Amsterdam, The Netherlands: Springer, 2016, 702-716.

Kingma, D. P.&Ba, J. Adam: a method for stochastic optimization (2014). athttps://arxiv.org/abs/1412.6980https://arxiv.org/abs/1412.6980.

Sirinukunwattana, K. et al. Gland segmentation in colon histology images: the glas challenge contest.Med. Image Anal.35, 489-502 (2017)..

Sirinukunwattana, K., Snead, D. R. J.&Rajpoot, N. M. A stochastic polygons model for glandular structures in colon histology images.IEEE Transactions on Medical.Imaging34, 2366-2378 (2015)..

Caicedo, J. C. et al. Nucleus segmentation across imaging experiments: the 2018 Data Science Bowl.Nat. Methods16, 1247-1253 (2019)..

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰