Neural network-based processing and reconstruction of compromised biophotonic image data

Michael John Fanous; Paloma Casteleiro Costa; Çağatay Işıl; Luzhe Huang; Aydogan Ozcan

doi:10.1038/s41377-024-01544-9

Go to Nature Website

Your Location：

Home >

Browse articles >

Neural network-based processing and reconstruction of compromised biophotonic image data

Reviews

- Neural network-based processing and reconstruction of compromised biophotonic image data
- Light: Science & Applications Vol. 13, Issue 10, Pages: 2101-2113(2024)
- Affiliations：
  
  1.Electrical and Computer Engineering Department, University of California, Los Angeles, CA, USA
  2.Bioengineering Department, University of California, Los Angeles, CA, USA
  3.California NanoSystems Institute (CNSI), University of California, Los Angeles, CA, USA
  4.Department of Surgery, David Geffen School of Medicine, University of California, Los Angeles, CA, USA
- Author bio：
  
  Aydogan Ozcan (ozcan@ucla.edu)
- Funds：
- DOI：10.1038/s41377-024-01544-9
  CLC：
- Published：31 October 2024，
  
  Published Online：04 September 2024，
  
  Received：22 March 2024，
  
  Revised：16 July 2024，
  
  Accepted：18 July 2024
- Accepted：
Scan QR Code
Fanous, M. J. et al. Neural network-based processing and reconstruction of compromised biophotonic image data. Light: Science & Applications, 13, 2101-2113 (2024).
DOI：

Fanous, M. J. et al. Neural network-based processing and reconstruction of compromised biophotonic image data. Light: Science & Applications, 13, 2101-2113 (2024). DOI： 10.1038/s41377-024-01544-9.

摘要

Abstract

In recent years

the integration of deep learning techniques with biophotonic setups has opened new horizons in bioimaging. A compelling trend in this field involves deliberately compromising certain measurement metrics to engineer better bioimaging tools in terms of e.g.

cost

speed

and form-factor

followed by compensating for the resulting defects through the utilization of deep learning models trained on a large amount of ideal

superior or alternative data. This strategic approach has found increasing popularity due to its potential to enhance various aspects of biophotonic imaging. One of the primary motivations for employing this strategy is the pursuit of higher temporal resolution or increased imaging speed

critical for capturing fine dynamic biological processes. Additionally

this approach offers the prospect of simplifying hardware requirements and complexities

thereby making advanced imaging standards more accessible in terms of cost and/or size. This article provides an in-depth review of the diverse measurement aspects that researchers intentionally impair in their biophotonic setups

including the point spread function (PSF)

signal-to-noise ratio (SNR)

sampling density

and pixel resolution. By deliberately compromising these metrics

researchers aim to not only recuperate them through the application of deep learning networks

but also bolster in return other crucial parameters

such as the field of view (FOV)

depth of field (DOF)

and space-bandwidth product (SBP). Throughout this article

we discuss various biophotonic methods that have successfully employed this strategic approach. These techniques span a wide range of applications and showcase the versatility and effectiveness of deep learning in the context of compromised biophotonic data. Finally

by offering our perspectives on the exciting future possibilities of this rapidly evolving concept

we hope to motivate our readers from various disciplines to explore novel ways of balancing hardware compromises with compensation via artificial intelligence (AI).

关键词

Keywords

references

Prasad, P. N.Introduction to Biophotonics(Hoboken: John Wiley&Sons, 2003).

Marcu, L. et al. Biophotonics: the big picture.J. Biomed. Opt.23, 021103 (2017)..

Tian, L. et al. Deep learning in biomedical optics.Lasers Surg. Med.53, 748–775 (2021)..

Pradhan, P. et al. Deep learning a boon for biophotonics?J. Biophotonics13, e201960186 (2020)..

Icha, J. et al. Phototoxicity in live fluorescence microscopy, and how to avoid it.BioEssays39, 1700003 (2017)..

Diaspro, A. et al. inHandbook of Biological Confocal Microscopy3rd edn (ed Pawley, J. B.) (New York: Springer, 2006).

Demchenko, A. P. Photobleaching of organic fluorophores: quantitative characterization, mechanisms, protection.Methods Appl. Fluoresc.8, 022001 (2020)..

Luo, Y. L. et al. Single-shot autofocusing of microscopy images using deep learning.ACS Photonics8, 625–638 (2021)..

Yang, X. L. et al. Deep-learning-based virtual refocusing of images using an engineered point-spread function.ACS Photonics8, 2174–2182 (2021)..

Fanous, M. J.&Popescu, G. GANscan: continuous scanning microscopy using deep learning deblurring.Light Sci. Appl.11, 265, https://doi.org/10.1038/s41377-022-00952-z (2022)..

Chen, H. L. et al. eFIN: enhanced Fourier Imager Network for generalizable autofocusing and pixel super-resolution in holographic imaging.IEEE J. Sel. Top. Quantum Electron.29, 6800810 (2023)..

Huang, L. Z. et al. Recurrent neural network-based volumetric fluorescence microscopy.Light Sci. Appl.10, 62 (2021)..

Huang, L. Z. et al. Few-shot transfer learning for holographic image reconstruction using a recurrent neural network.APL Photonics7, 070801 (2022)..

Zhang, Y. J. et al. Neural network-based image reconstruction in swept-source optical coherence tomography using undersampled spectral data.Light Sci. Appl.10, 155 (2021)..

Cheng, Y. F. et al. Illumination pattern design with deep learning for single-shot Fourier ptychographic microscopy.Opt. express27, 644–656 (2019)..

Rivenson, Y. et al. Deep learning enhanced mobile-phone microscopy.ACS Photonics5, 2354–2364 (2018)..

Yao, X. et al. Increasing a microscope's effective field of view via overlapped imaging and machine learning.Opt. Express30, 1745–1761, https://doi.org/10.1364/OE.445001 (2022)..

Jin, L. H. et al. Deep learning enables structured illumination microscopy with low light levels and enhanced speed.Nat. Commun.11, 1934, https://doi.org/10.1038/s41467-020-15784-x (2020)..

Manifold, B. et al. Denoising of stimulated Raman scattering microscopy images via deep learning.Biomed. Opt. Express10, 3860–3874 (2019)..

Pinkard, H. et al. Deep learning for single-shot autofocus microscopy.Optica6, 794–797 (2019)..

Ebrahimi, V. et al. Deep learning enables fast, gentle STED microscopy.Commun. Biol.6, 674 (2023)..

Botcherby, E. J. et al. An optical technique for remote focusing in microscopy.Opt. Commun.281, 880–887 (2008)..

Botcherby, E. J. et al. Aberration-free optical refocusing in high numerical aperture microscopy.Opt. Lett.32, 2007–2009 (2007)..

Mohanan, S.&Corbett, A. D. Understanding the limits of remote focusing.Opt. Express31, 16281–16294 (2023)..

Rossmann, K. Point spread-function, line spread-function, and modulation transfer function: tools for the study of imaging systems.Radiology93, 257–272 (1969)..

Jouchet, P., Roy, A. R.&Moerner, W. E. Combining deep learning approaches and point spread function engineering for simultaneous 3D position and 3D orientation measurements of fluorescent single molecules.Opt. Commun.542, 129589 (2023)..

Astratov, V. N. et al. Roadmap on label‐free super‐resolution imaging.Laser Photonics Rev.17, 2200029 (2023)..

Nehme, E. et al. DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning.Nat. Methods17, 734–740 (2020)..

Vaquero, D. et al. Generalized autofocus. InProc. 2011 IEEE Workshop on applications of computer vision (WACV). 511–518 (IEEE: Kona, HI, USA, 2011).

Pavani, S. R. P. et al. Three-dimensional, single-molecule fluorescence imaging beyond the diffraction limit by using a double-helix point spread function.Proc. Natl Acad. Sci. U. Stateds Am.106, 2995–2999 (2009)..

Wang, H. D. et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy.Nat. Methods16, 103–110 (2019)..

Wu, Y. C. et al. Extended depth-of-field in holographic imaging using deep-learning-based autofocusing and phase recovery.Optica5, 704–710 (2018)..

Huang, L. Z. et al. Holographic image reconstruction with phase recovery and autofocusing using recurrent neural networks.ACS Photonics8, 1763–1774 (2021)..

Huang, L. Z. et al. Self-supervised learning of hologram reconstruction using physics consistency.Nat. Mach. Intell.5, 895–907 (2023)..

Pirone, D. et al. Speeding up reconstruction of 3D tomograms in holographic flow cytometry via deep learning.Lab Chip22, 793–804 (2022)..

Park, J. et al. Revealing 3D cancer tissue structures using holotomography and virtual hematoxylin and eosin staining via deep learning. Preprint athttps://www.biorxiv.org/content/10.1101/2023.12.04.569853v2https://www.biorxiv.org/content/10.1101/2023.12.04.569853v2(2023).

Barbastathis, G., Ozcan, A.&Situ, G. H. On the use of deep learning for computational imaging.Optica6, 921–943 (2019)..

Situ, G. H. Deep holography.Light Adv. Manuf.3, 13 (2022)..

Kakkava, E. et al. Imaging through multimode fibers using deep learning: the effects of intensity versus holographic recording of the speckle pattern.Opt. Fiber Technol.52, 101985 (2019)..

Park, J. et al. Artificial intelligence-enabled quantitative phase imaging methods for life sciences.Nat. Methods20, 1645–1660 (2023)..

Chen, H. L. et al. Fourier Imager Network (FIN): a deep neural network for hologram reconstruction with superior external generalization.Light Sci. Appl.11, 254 (2022)..

Goodfellow, I. et al. Generative adversarial networks.Commun. ACM63, 139–144 (2020)..

Lepage, G., Bogaerts, J.&Meynants, G. Time-delay-integration architectures in CMOS image sensors.IEEE Trans. Electron Devices56, 2524–2533 (2009)..

Ren, Z. B., Xu, Z. M.&Lam, E. Y. Learning-based nonparametric autofocusing for digital holography.Optica5, 337–344 (2018)..

Konda, P. C. et al. Fourier ptychography: current applications and future promises.Opt. express28, 9603–9630 (2020)..

Zheng, G. A., Horstmeyer, R.&Yang, C. Wide-field, high-resolution Fourier ptychographic microscopy.Nat. Photonics7, 739–745 (2013)..

Tian, L. et al. Multiplexed coded illumination for Fourier Ptychography with an LED array microscope.Biomed. Opt. Express5, 2376–2389 (2014)..

Nguyen, T. et al. Deep learning approach for Fourier ptychography microscopy.Opt. Express26, 26470–26484 (2018)..

Grossberg, S. Recurrent neural networks.Scholarpedia8, 1888 (2013)..

Podoleanu, A. G. Optical coherence tomography.Br. J. Radiol.78, 976–988 (2005)..

Kim, M. K. Principles and techniques of digital holographic microscopy.SPIE Rev.1, 018005 (2010)..

Stelzer Contrast, resolution, pixelation, dynamic range and signal‐to‐noise ratio: fundamental limits to resolution in fluorescence light microscopy.J. Microsc.189, 15–24 (1998)..

Rittweger, E. et al. STED microscopy reveals crystal colour centres with nanometric resolution.Nat. Photonics3, 144–147 (2009)..

Tipping, W. J. et al. Stimulated Raman scattering microscopy: an emerging tool for drug discovery.Chem. Soc. Rev.45, 2075–2089 (2016)..

Saxena, M., Eluru, G.&Gorthi, S. S. Structured illumination microscopy.Adv. Opt. Photonics7, 241–275 (2015)..

Wu, Y. C. et al. Three-dimensional virtual refocusing of fluorescence microscopy images using deep learning.Nat. Methods16, 1323–1331 (2019)..

Repetti, A., Pereyra, M.&Wiaux, Y. Scalable Bayesian uncertainty quantification in imaging inverse problems via convex optimization.SIAM J. Imaging Sci.12, 87–118 (2019)..

Zhou, Q. P. et al. Bayesian inference and uncertainty quantification for medical image reconstruction with Poisson data.SIAM J. Imaging Sci.13, 29–52 (2020)..

Xue, Y. J. et al. Reliable deep-learning-based phase imaging with uncertainty quantification.Optica6, 618–629 (2019)..

Hoffmann, L., Fortmeier, I.&Elster, C. Uncertainty quantification by ensemble learning for computational optical form measurements.Mach. Learn. Sci. Technol.2, 035030 (2021)..

Huang, L. Z. et al. Cycle-consistency-based uncertainty quantification of neural networks in inverse imaging problems.Intell. Comput.2, 0071 (2023)..

Chen, J. T. et al. A transfer learning based super-resolution microscopy for biopsy slice images: the joint methods perspective.IEEE/ACM Trans. Comput. Biol. Bioinform.18, 103–113 (2021)..

Christensen, C. N. et al. ML-SIM: universal reconstruction of structured illumination microscopy images using transfer learning.Biomed. Opt. Express12, 2720–2733 (2021)..

Shi, X. J. et al. Convolutional LSTM network: a machine learning approach for precipitation nowcasting. InProc. 28thInternal Conference on Neural Information Processing Systems(Montreal, Canada: MIT Press, 2015)..

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰