Universal adaptive optics for microscopy through embedded neural network control

Qi Hu; Martin Hailstone; Jingyu Wang; Matthew Wincott; Danail Stoychev; Huriye Atilgan; Dalia Gala; Tai Chaiamarit; Richard M. Parton; Jacopo Antonello; Adam M. Packer; Ilan Davis; Martin J. Booth

doi:10.1038/s41377-023-01297-x

Go to Nature Website

Your Location：

Home >

Browse articles >

Universal adaptive optics for microscopy through embedded neural network control

Original Articles

- Universal adaptive optics for microscopy through embedded neural network control
- Light: Science & Applications Vol. 12, Issue 12, Pages: 2597-2609(2023)
- Affiliations：
  
  1.Department of Engineering Science, University of Oxford, Oxford, UK
  2.Department of Biochemistry, University of Oxford, Oxford, UK
  3.Department of Physiology, Anatomy, and Genetics, University of Oxford, Oxford, UK
- Author bio：
  
  Martin J. Booth (martin.booth@eng.ox.ac.uk)
- Funds：
- DOI：10.1038/s41377-023-01297-x
  CLC：
- Published：31 December 2023，
  
  Published Online：13 November 2023，
  
  Received：20 April 2023，
  
  Revised：24 September 2023，
  
  Accepted：01 October 2023
- Accepted：
Scan QR Code
Hu, Q. et al. Universal adaptive optics for microscopy through embedded neural network control. Light: Science & Applications, 12, 2597-2609 (2023).
DOI：

Hu, Q. et al. Universal adaptive optics for microscopy through embedded neural network control. Light: Science & Applications, 12, 2597-2609 (2023). DOI： 10.1038/s41377-023-01297-x.

摘要

Abstract

The resolution and contrast of microscope imaging is often affected by aberrations introduced by imperfect optical systems and inhomogeneous refractive structures in specimens. Adaptive optics (AO) compensates these aberrations and restores diffraction limited performance. A wide range of AO solutions have been introduced

often tailored to a specific microscope type or application. Until now

a universal AO solution – one that can be readily transferred between microscope modalities – has not been deployed. We propose versatile and fast aberration correction using a physics-based machine learning assisted wavefront-sensorless AO control (MLAO) method. Unlike previous ML methods

we used a specially constructed neural network (NN) architecture

designed using physical understanding of the general microscope image formation

that was embedded in the control loop of different microscope systems. The approach means that not only is the resulting NN orders of magnitude simpler than previous NN methods

but the concept is translatable across microscope modalities. We demonstrated the method on a two-photon

a three-photon and a widefield three-dimensional (3D) structured illumination microscope. Results showed that the method outperformed commonly-used modal-based sensorless AO methods. We also showed that our ML-based method was robust in a range of challenging imaging conditions

such as 3D sample structures

specimen motion

low signal to noise ratio and activity-induced fluorescence fluctuations. Moreover

as the bespoke architecture encapsulated physical understanding of the imaging process

the internal NN configuration was no-longer a "black box"

but provided physical insights on internal workings

which could influence future designs.

关键词

Keywords

references

Booth, M. J. Adaptive optics in microscopy.Philos. Trans. R. Soc. A Math., Phys. Eng. Sci.365, 2829–2843 (2007)..

Booth, M. J. Adaptive optical microscopy: the ongoing quest for a perfect image.Light Sci. Appl.3, e165 (2014)..

Booth, M. J.&Patton, B. R. Adaptive optics for fluorescence microscopy. in Fluorescence Microscopy: Super-Resolution and other Novel Techniques (eds Cornea, A.&Conn, P. M. ) (London: Academic, 2014), 15-33.

Booth, M. et al. Aberrations and adaptive optics in super-resolution microscopy.Microscopy64, 251–261 (2015)..

Ji, N. Adaptive optical fluorescence microscopy.Nat. Methods14, 374–380 (2017)..

Hampson, K. M. et al. Adaptive optics for high-resolution imaging.Nat. Rev. Methods Prim.1, 68 (2021)..

Hartmann, J. Objektivuntersuchungen.Z. f. ür. Instrumentenkunde24, 1–21 (1904). 33-47, 97-117..

Shack, R. V.&Platt, B. C. Production and use of a lenticular Hartmann screen.J. Opt. Soc. Am.61, 656–661 (1971)..

Schwertner, M., Booth, M. J.&Wilson, T. Characterizing specimen induced aberrations for high NA adaptive optical microscopy.Opt. Express12, 6540–6552 (2004)..

Booth, M. et al. Methods for the characterization of deformable membrane mirrors.Appl. Opt.44, 5131–5139 (2005)..

Hu, Q. et al. A universal framework for microscope sensorless adaptive optics: Generalized aberration representations.APL Photonics5, 100801 (2020)..

Hu, Q. Adaptive optics for corrections of phase and polarisation state aberrations in microscopes. PhD thesis, University of Oxford, Oxford, 2021.

Booth, M. J., Neil, M. A. A.&Wilson, T. New modal wave-front sensor: application to adaptive confocal fluorescence microscopy and two-photon excitation fluorescence microscopy.J. Opt. Soc. Am. A19, 2112–2120 (2002)..

Sherman, L. et al. Adaptive correction of depth-induced aberrations in multiphoton scanning microscopy using a deformable mirror.J. Microsc.206, 65–71 (2002)..

Marsh, P. N., Burns, D.&Girkin, J. M. Practical implementation of adaptive optics in multiphoton microscopy.Opt. Expr.11, 1123–1130 (2003)..

Wright, A. J. et al. Exploration of the optimisation algorithms used in the implementation of adaptive optics in confocal and multiphoton microscopy.Microsc. Res. Tech.67, 36–44 (2005)..

Débarre, D. et al. Image-based adaptive optics for two-photon microscopy.Opt. Lett.34, 2495–2497 (2009)..

Tang, J. Y., Germain, R. N.&Cui, M. Superpenetration optical microscopy by iterative multiphoton adaptive compensation technique.Proc. Natl Acad. Sci. USA109, 8434–8439 (2012)..

Facomprez, A., Beaurepaire, E.&Débarre, D. Accuracy of correction in modal sensorless adaptive optics.Opt. Expr.20, 2598–2612 (2012)..

Katz, O. et al. Noninvasive nonlinear focusing and imaging through strongly scattering turbid layers.Optica1, 170–174 (2014)..

Sinefeld, D. et al. Adaptive optics in multiphoton microscopy: comparison of two, three and four photon fluorescence.Opt. Expr.23, 31472–31483 (2015)..

Galwaduge, P. T. et al. Simple wavefront correction framework for two-photon microscopy of in-vivo brain.Biomed. Opt. Expr.6, 2997–3013 (2015)..

Streich, L. etal. High-resolution structural and functional deep brain imaging using adaptive optics three-photon microscopy.Nat. Methods18, 1253–1258 (2021)..

Débarre, D., Booth, M. J.&Wilson, T. Image based adaptive optics through optimisation of low spatial frequencies.Opt. Expr.15, 8176–8190 (2007)..

Gould, T. J. et al. Adaptive optics enables 3D STED microscopy in aberrating specimens.Opt. Expr.20, 20998–21009 (2012)..

Bourgenot, C. et al. 3D adaptive optics in a light sheet microscope.Opt. Expr.20, 13252–13261 (2012)..

Burke, D. et al. Adaptive optics correction of specimen-induced aberrations in single-molecule switching microscopy.Optica2, 177–185 (2015)..

Patton, B. R. et al. Three-dimensional STED microscopy of aberrating tissue using dual adaptive optics.Opt. Expr.24, 8862–8876 (2016)..

Wang, B. R.&Booth, M. J. Optimum deformable mirror modes for sensorless adaptive optics.Opt. Commun.282, 4467–4474 (2009)..

Milkie, D. E., Betzig, E.&Ji, N. Pupil-segmentation-based adaptive optical microscopy with full-pupil illumination.Opt. Lett.36, 4206–4208 (2011)..

Booth, M. J. et al. Adaptive aberration correction in a confocal microscope.Proc. Natl. Acad. Sci. USA99, 5788–5792 (2002)..

Wang, F. L. Wavefront sensing through measurements of binary aberration modes.Appl. Opt.48, 2865–2870 (2009)..

Antonello, J. et al. Semidefinite programming for model-based sensorless adaptive optics.J. Opt. Soc. Am. A29, 2428–2438 (2012)..

Antonello, J. et al. Multi-scale sensorless adaptive optics: application to stimulated emission depletion microscopy.Opt. Expr.28, 16749–16763 (2020)..

Jin, Y. C. et al. Machine learning guided rapid focusing with sensor-less aberration corrections.Opt. Expr.26, 30162–30171 (2018)..

Möckl, L., Petrov, P. N.&Moerner, W. E. Accurate phase retrieval of complex 3D point spread functions with deep residual neural networks.Appl. Phys. Lett.115, 251106 (2019)..

Vishniakou, I.&Seelig, J. D. Wavefront correction for adaptive optics with reflected light and deep neural networks.Opt. Expr.28, 15459–15471 (2020)..

Cumming, B. P.&Gu, M. Direct determination of aberration functions in microscopy by an artificial neural network.Opt. Expr.28, 14511–14521 (2020)..

Khorin, P. A. et al. Neural networks application to determine the types and magnitude of aberrations from the pattern of the point spread function out of the focal plane.J. Phys. Conf. Ser.2086, 012148 (2021)..

Zhang, H. et al. Application of adamspgd algorithm to sensor-less adaptive optics in coherent free-space optical communication system.Opt. Expr.30, 7477–7490 (2022)..

Saha, D. et al. Practical sensorless aberration estimation for 3D microscopy with deep learning.Opt. Expr.28, 29044–29053 (2020)..

Durech, E. et al. Wavefront sensor-less adaptive optics using deep reinforcement learning.Biomed. Opt. Expr.12, 5423–5438 (2021)..

Wang, F. et al. Phase imaging with an untrained neural network.Light Sci. Appl.9, 77 (2020)..

Bostan, E. et al. Deep phase decoder: self-calibrating phase microscopy with an untrained deep neural network.Optica7, 559–562 (2020)..

Noll, R. J. Zernike polynomials and atmospheric turbulence.J. Opt. Soc. Am.66, 207–211 (1976)..

Hall, N. Accessible adaptive optics and super-resolution microscopy to enable improved imaging. PhD thesis, University of Oxford, Oxford, 2020.

Žurauskas, M. et al. Isosense: frequency enhanced sensorless adaptive optics through structured illumination.Optica6, 370–379 (2019)..

Xin, Q. et al. Object-independent image-based wavefront sensing approach using phase diversity images and deep learning.Opt. Expr.27, 26102–26119 (2019)..

Thévenaz, P., Ruttimann, U. E.&Unser, M. A pyramid approach to subpixel registration based on intensity.IEEE Trans. Image Process.7, 27–41 (1998)..

Dougherty, R. Extensions of DAMAS and benefits and limitations of deconvolution in beamforming. Proceedings of the 11th AIAA/CEAS Aeroacoustics Conference. Monterey: AIAA, 2005.

Gustafsson, M. G. L. et al. Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination.Biophys. J.94, 4957–4970 (2008)..

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰