GANscan: continuous scanning microscopy using deep learning deblurring

Michael John Fanous; Gabriel Popescu

doi:10.1038/s41377-022-00952-z

Go to Nature Website

Your Location：

Home >

Browse articles >

GANscan: continuous scanning microscopy using deep learning deblurring

Original Articles

- GANscan: continuous scanning microscopy using deep learning deblurring
- Light: Science & Applications Vol. 11, Issue 10, Pages: 2378-2387(2022)
- Affiliations：
  
  1.Quantitative Light Imaging Laboratory, Beckman Institute for Advanced Science and Technology, University of Illinois at Urbana-Champaign, Urbana, IL, 61801, USA
  2.Department of Bioengineering, University of Illinois at Urbana-Champaign, 306 N. Wright Street, Urbana, IL, 61801, USA
  3.Department of Electrical and Computer Engineering, University of Illinois at Urbana-Champaign, 306 N. Wright Street, Urbana, IL, 61801, USA
- Author bio：
  
  Michael John Fanous (mfanous2@illinois.edu)
  Gabriel Popescu (gpopescu@illinois.edu)
- Funds：
- DOI：10.1038/s41377-022-00952-z
  CLC：
- Published：31 October 2022，
  
  Published Online：07 September 2022，
  
  Received：15 March 2022，
  
  Revised：31 July 2022，
  
  Accepted：07 August 2022
- Accepted：
Scan QR Code
Fanous, M. J. & Popescu, G. et al. GANscan: continuous scanning microscopy using deep learning deblurring. Light: Science & Applications, 11, 2378-2387 (2022).
DOI：

Fanous, M. J. & Popescu, G. et al. GANscan: continuous scanning microscopy using deep learning deblurring. Light: Science & Applications, 11, 2378-2387 (2022). DOI： 10.1038/s41377-022-00952-z.

摘要

Abstract

Most whole slide imaging (WSI) systems today rely on the "stop-and-stare" approach

where

at each field of view

the scanning stage is brought to a complete stop before the camera snaps a picture. This procedure ensures that each image is free of motion blur

which comes at the expense of long acquisition times. In order to speed up the acquisition process

especially for large scanning areas

such as pathology slides

we developed an acquisition method in which the data is acquired continuously while the stage is moving at high speeds. Using generative adversarial networks (GANs)

we demonstrate this ultra-fast imaging approach

referred to as GANscan

which restores sharp images from motion blurred videos. GANscan allows us to complete image acquisitions at 30x the throughput of stop-and-stare systems. This method is implemented on a Zeiss Axio Observer Z1 microscope

requires no specialized hardware

and accomplishes successful reconstructions at stage speeds of up to 5000 μm/s. We validate the proposed method by imaging H & E stained tissue sections. Our method not only retrieves crisp images from fast

continuous scans

but also adjusts for defocusing that occurs during scanning within +/− 5 μm. Using a consumer GPU

the inference runs at < 20 ms/ image.

关键词

Keywords

references

R. Horstmeyer, 等 . Digital pathology with Fourier ptychography . Computerized Med. Imaging Graph. , 2015 . 42 38 -43 . DOI:10.1016/j.compmedimag.2014.11.005http://doi.org/10.1016/j.compmedimag.2014.11.005 .

B. Potsaid, Y. Bellouard, J. T. Wen . Adaptive Scanning Optical Microscope (ASOM): A multidisciplinary optical microscope design for large field of view and high resolution imaging . Opt. Express , 2005 . 13 6504 -6518 . DOI:10.1364/OPEX.13.006504http://doi.org/10.1364/OPEX.13.006504 .

R. H. Webb, F. Rogomentich . Confocal microscope with large field and working distance . Appl. Opt. , 1999 . 38 4870 -4875 . DOI:10.1364/AO.38.004870http://doi.org/10.1364/AO.38.004870 .

M. Alegro, 等 . Automating cell detection and classification in human brain fluorescent microscopy images using dictionary learning and sparse coding . J. Neurosci. Methods , 2017 . 282 20 -33 . DOI:10.1016/j.jneumeth.2017.03.002http://doi.org/10.1016/j.jneumeth.2017.03.002 .

P. Brodin, T. Christophe . High-content screening in infectious diseases . Curr. Opin. Chem. Biol. , 2011 . 15 534 -539 . DOI:10.1016/j.cbpa.2011.05.023http://doi.org/10.1016/j.cbpa.2011.05.023 .

C. B. Messner, 等 . Ultra-fast proteomics with Scanning SWATH . Nat. Biotechnol. , 2021 . 39 846 -854 . DOI:10.1038/s41587-021-00860-4http://doi.org/10.1038/s41587-021-00860-4 .

M. Remmelinck, 等 . How could static telepathology improve diagnosis in neuropathology? . Anal. Cell. Pathol. , 2000 . 21 177 -182 . DOI:10.1155/2000/838615http://doi.org/10.1155/2000/838615 .

D. S. Gareau, 等 . Confocal mosaicing microscopy in Mohs skin excisions: feasibility of rapid surgical pathology . J. Biomed. Opt. , 2008 . 13 054001 DOI:10.1117/1.2981828http://doi.org/10.1117/1.2981828 .

Z. F. Phillips, 等 . High-throughput fluorescence microscopy using multi-frame motion deblurring . Biomed. Opt. express , 2020 . 11 281 -300 . DOI:10.1364/BOE.11.000281http://doi.org/10.1364/BOE.11.000281 .

J. Ho, 等 . Use of whole slide imaging in surgical pathology quality assurance: design and pilot validation studies . Hum. Pathol. , 2006 . 37 322 -331 . DOI:10.1016/j.humpath.2005.11.005http://doi.org/10.1016/j.humpath.2005.11.005 .

Hamamatsu. High throughput imaging in low light applications. https://www.hamamatsu.comhttps://www.hamamatsu.com (2011).

De Moor, P. et al. Enhanced time delay integration iamging using embedded CCD in CMOS technology. Proceedings of 2014 IEEE International Electron Devices Meeting. San Francisco, CA, USA, IEEE, 4.6. 1–4.6. 4.

N. V. Iftimia, 等 . Adaptive ranging for optical coherence tomography . Opt. Express , 2004 . 12 4025 -4034 . DOI:10.1364/OPEX.12.004025http://doi.org/10.1364/OPEX.12.004025 .

P. Prabhat, 等 . Simultaneous imaging of different focal planes in fluorescence microscopy for the study of cellular dynamics in three dimensions . IEEE Trans. Nanobiosci. , 2004 . 3 237 -242 . DOI:10.1109/TNB.2004.837899http://doi.org/10.1109/TNB.2004.837899 .

S. Abrahamsson, 等 . Fast multicolor 3D imaging using aberration-corrected multifocus microscopy . Nat. Methods , 2013 . 10 60 -63 . DOI:10.1038/nmeth.2277http://doi.org/10.1038/nmeth.2277 .

M. B. Bouchard, 等 . Swept confocally-aligned planar excitation (SCAPE) microscopy for high-speed volumetric imaging of behaving organisms . Nat. Photonics , 2015 . 9 113 -119 . DOI:10.1038/nphoton.2014.323http://doi.org/10.1038/nphoton.2014.323 .

A. Nakano . Spinning-disk confocal microscopy—a cutting-edge tool for imaging of membrane traffic . Cell Struct. Funct. , 2002 . 27 349 -355 . DOI:10.1247/csf.27.349http://doi.org/10.1247/csf.27.349 .

H. Y. Li, 等 . Fast, volumetric live-cell imaging using high-resolution light-field microscopy . Biomed. Opt. Express , 2019 . 10 29 -49 . DOI:10.1364/BOE.10.000029http://doi.org/10.1364/BOE.10.000029 .

M. Martínez-Corral, B. Javidi . Fundamentals of 3D imaging and displays: a tutorial on integral imaging, light-field, and plenoptic systems . Adv. Opt. Photonics , 2018 . 10 512 -566 . DOI:10.1364/AOP.10.000512http://doi.org/10.1364/AOP.10.000512 .

C. F. Hu, 等 . Synthetic aperture interference light (SAIL) microscopy for high-throughput label-free imaging . Appl. Phys. Lett. , 2021 . 119 233701 DOI:10.1063/5.0065628http://doi.org/10.1063/5.0065628 .

N. Farahani, A. V. Parwani, L. Pantanowitz . Whole slide imaging in pathology: advantages, limitations, and emerging perspectives . Pathol. Lab. Med. Int. , 2015 . 7 23 -33. .

A. W. Lohmann, 等 . Space–bandwidth product of optical signals and systems . J. Optical Soc. Am. A , 1996 . 13 470 -473 . DOI:10.1364/JOSAA.13.000470http://doi.org/10.1364/JOSAA.13.000470 .

M. G. L. Gustafsson . Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy . J. Microsc. , 2000 . 198 82 -87 . DOI:10.1046/j.1365-2818.2000.00710.xhttp://doi.org/10.1046/j.1365-2818.2000.00710.x .

J. M. Rodenburg, H. M. L. Faulkner . A phase retrieval algorithm for shifting illumination . Appl. Phys. Lett. , 2004 . 85 4795 -4797 . DOI:10.1063/1.1823034http://doi.org/10.1063/1.1823034 .

L. Tian, 等 . Multiplexed coded illumination for Fourier Ptychography with an LED array microscope . Biomed. Opt. Express , 2014 . 5 2376 -2389 . DOI:10.1364/BOE.5.002376http://doi.org/10.1364/BOE.5.002376 .

T.G. Nguyen, 等 . Deep learning approach for Fourier ptychography microscopy . Opt. Express , 2018 . 26 26470 -26484 . DOI:10.1364/OE.26.026470http://doi.org/10.1364/OE.26.026470 .

Y. Rivenson, 等 . Deep learning microscopy . Optica , 2017 . 4 1437 -1443 . DOI:10.1364/OPTICA.4.001437http://doi.org/10.1364/OPTICA.4.001437 .

Y. J. Xue, 等 . Reliable deep-learning-based phase imaging with uncertainty quantification . Optica , 2019 . 6 618 -629 . DOI:10.1364/OPTICA.6.000618http://doi.org/10.1364/OPTICA.6.000618 .

E. Nehme, 等 . Deep-STORM: super-resolution single-molecule microscopy by deep learning . Optica , 2018 . 5 458 -464 . DOI:10.1364/OPTICA.5.000458http://doi.org/10.1364/OPTICA.5.000458 .

Bayramoglu, N. et al. Towards virtual H & E staining of hyperspectral lung histology images using conditional generative adversarial networks. Proceedings of 2017 IEEE International Conference on Computer Vision Workshops. Venice, Italy, IEEE, 2017, 64–71.

Y. Rivenson, 等 . PhaseStain: the digital staining of label-free quantitative phase microscopy images using deep learning . Light Sci. Appl. , 2019 . 8 23 DOI:10.1038/s41377-019-0129-yhttp://doi.org/10.1038/s41377-019-0129-y.

E. M. Christiansen, 等 . In silico labeling: predicting fluorescent labels in unlabeled images . Cell , 2018 . 173 792 -803.e19 . DOI:10.1016/j.cell.2018.03.040http://doi.org/10.1016/j.cell.2018.03.040 .

C. Ounkomol, 等 . Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy . Nat. Methods , 2018 . 15 917 -920 . DOI:10.1038/s41592-018-0111-2http://doi.org/10.1038/s41592-018-0111-2 .

M. Fanous, 等 . Label-free screening of brain tissue myelin content using phase imaging with computational specificity (PICS) . Apl. Photonics , 2021 . 6 076103 DOI:10.1063/5.0050889http://doi.org/10.1063/5.0050889 .

M. E. Kandel, 等 . Phase Imaging with Computational Specificity (PICS) for measuring dry mass changes in sub-cellular compartments . Nat. Commun. , 2020 . 11 6256 DOI:10.1038/s41467-020-20062-xhttp://doi.org/10.1038/s41467-020-20062-x .

Goswami, N. et al. Label-free SARS-CoV-2 detection and classification using phase imaging with computational specificity. Light Sci. Appl. 10, 176 (2021).

C. F. Hu, 等 . Live-dead assay on unlabeled cells using phase imaging with computational specificity . Nat. Commun. , 2022 . 13 713 DOI:10.1038/s41467-022-28214-xhttp://doi.org/10.1038/s41467-022-28214-x .

H. Pinkard, 等 . Deep learning for single-shot autofocus microscopy . Optica , 2019 . 6 794 -797 . DOI:10.1364/OPTICA.6.000794http://doi.org/10.1364/OPTICA.6.000794 .

G. Barbastathis, A. Ozcan, G. H. Situ . On the use of deep learning for computational imaging . Optica , 2019 . 6 921 -943 . DOI:10.1364/OPTICA.6.000921http://doi.org/10.1364/OPTICA.6.000921 .

K. de Haan, 等 . Deep learning-based transformation of H & E stained tissues into special stains . Nat. Commun. , 2021 . 12 488 DOI:10.1038/s41467-020-20718-8http://doi.org/10.1038/s41467-020-20718-8 .

Popescu, G. Principles of Biophotonics, Volume 1: Linear systems and the Fourier transform in optics. Philidelphia, IOP PublishingLtd, 2018.

Popescu, G. Quantitative phase imaging of cells and tissues. New York, McGraw Hill Professional, 2011.

Y. Luo, 等 . Single-shot autofocusing of microscopy images using deep learning . ACS Photonics , 2021 . 8 625 -638 . DOI:10.1021/acsphotonics.0c01774http://doi.org/10.1021/acsphotonics.0c01774 .

Isola, P. et al. Image-to-image translation with conditional adversarial networks. Proceedings of 2017 IEEE conference on computer vision and pattern recognition. Honolulu, HI, USA, IEEE, 2017, 5967–5976.

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰