E2E-BPF microscope: extended depth-of-field microscopy using learning-based implementation of binary phase filter and image deconvolution

Baekcheon Seong; Woovin Kim; Younghun Kim; Kyung-A Hyun; Hyo-Il Jung; Jong-Seok Lee; Jeonghoon Yoo; Chulmin Joo

doi:10.1038/s41377-023-01300-5

Go to Nature Website

Your Location：

Home >

Browse articles >

E2E-BPF microscope: extended depth-of-field microscopy using learning-based implementation of binary phase filter and image deconvolution

Original Articles

- E2E-BPF microscope: extended depth-of-field microscopy using learning-based implementation of binary phase filter and image deconvolution
- Light: Science & Applications Vol. 12, Issue 12, Pages: 2610-2623(2023)
- Affiliations：
  
  1.Department of Mechanical Engineering, Yonsei University, Seoul 03722, Republic of Korea
  2.The DABOM Inc, Seoul 03722, Republic of Korea
  3.School of Integrated Technology, Yonsei University, Incheon 21983, Republic of Korea
- Author bio：
  
  Chulmin Joo (cjoo@yonsei.ac.kr)
- Funds：
- DOI：10.1038/s41377-023-01300-5
  CLC：
- Published：31 December 2023，
  
  Published Online：13 November 2023，
  
  Received：16 March 2023，
  
  Revised：09 September 2023，
  
  Accepted：07 October 2023
- Accepted：
Scan QR Code
Seong, B. et al. E2E-BPF microscope: extended depth-of-field microscopy using learning-based implementation of binary phase filter and image deconvolution. Light: Science & Applications, 12, 2610-2623 (2023).
DOI：

Seong, B. et al. E2E-BPF microscope: extended depth-of-field microscopy using learning-based implementation of binary phase filter and image deconvolution. Light: Science & Applications, 12, 2610-2623 (2023). DOI： 10.1038/s41377-023-01300-5.

摘要

Abstract

Several image-based biomedical diagnoses require high-resolution imaging capabilities at large spatial scales. However

conventional microscopes exhibit an inherent trade-off between depth-of-field (DoF) and spatial resolution

and thus require objects to be refocused at each lateral location

which is time consuming. Here

we present a computational imaging platform

termed E2E-BPF microscope

which enables large-area

high-resolution imaging of large-scale objects without serial refocusing. This method involves a physics-incorporated

deep-learned design of binary phase filter (BPF) and jointly optimized deconvolution neural network

which altogether produces high-resolution

high-contrast images over extended depth ranges. We demonstrate the method through numerical simulations and experiments with fluorescently labeled beads

cells and tissue section

and present high-resolution imaging capability over a 15.5-fold larger DoF than the conventional microscope. Our method provides highly effective and scalable strategy for DoF-extended optical imaging system

and is expected to find numerous applications in rapid image-based diagnosis

optical vision

and metrology.

关键词

Keywords

references

Ortyn, W. E. et al. Extended depth of field imaging for high speed cell analysis.Cytometry A71, 215–231 (2007)..

Meng, Q., Li, Y., Yu, Y., Chu, K.&Smith, Z. J. A drop-in, focus-extending phase mask simplifies microscopic and microfluidic imaging systems for cost-effective point-of-care diagnostics.Anal. Chem.94, 11000–11007 (2022)..

Jin, L. B. et al. Deep learning extended depth-of-field microscope for fast and slide-free histology.Proc. Natl Acad. Sci. USA.117, 33051–33060 (2020)..

Lorenser, D., Yang, X. J.&Sampson, D. D. Ultrathin fiber probes with extended depth of focus for optical coherence tomography.Opt. Lett.37, 1616–1618 (2012)..

Greene, J. et al. Pupil engineering for extended depth-of-field imaging in a fluorescence miniscope.Neurophotonics10, 044302–044302 (2023)..

Kim, J. et al. Endoscopic micro-optical coherence tomography with extended depth of focus using a binary phase spatial filter.Opt. Lett.42, 379–382 (2017)..

Dowski, E. R.&Cathey, W. T. Extended depth of field through wave-front coding.Appl. Opt.34, 1859–1866 (1995)..

Zhao, H.&Li, Y. C. Optimized sinusoidal phase mask to extend the depth of field of an incoherent imaging system.Opt. Lett.35, 267–269 (2010)..

Sherif, S. S., Cathey, W. T.&Dowski, E. R. Phase plate to extend the depth of field of incoherent hybrid imaging systems.Appl. Opt.43, 2709–2721 (2004)..

Le, V. N., Chen, S. Q.&Fan, Z. G. Optimized asymmetrical tangent phase mask to obtain defocus invariant modulation transfer function in incoherent imaging systems.Opt. Lett.39, 2171–2174 (2014)..

Flores, A., Wang, M. R.&Yang, J. J. Achromatic hybrid refractive-diffractive lens with extended depth of focus.Appl. Opt.43, 5618–5630 (2004)..

Gao, X. M., Gan, F. X.&Xu, W. D. Superresolution by three-zone pure phase plate with 0, π, 0 phase variation.Opt. Laser Technol.39, 1074–1080 (2007)..

Sheppard, C. J. R., Campos, J., Escalera, J. C.&Ledesma, S. Three-zone pupil filters.Opt. Commun.281, 3623–3630 (2008)..

Sheppard, C. J. R., Campos, J., Escalera, J. C.&Ledesma, S. Two-zone pupil filters.Opt. Commun.281, 913–922 (2008)..

Ben-Eliezer, E., Konforti, N., Milgrom, B.&Marom, E. An optimal binary amplitude-phase mask for hybrid imaging systems that exhibit high resolution and extended depth of field.Opt. Express16, 20540–20561 (2008)..

Liu, L. B. et al. Superresolution along extended depth of focus with binary-phase filters for the Gaussian beam.J. Opt. Soc. Am. A25, 2095–2101 (2008)..

Diaz, F., Goudail, F., Loiseaux, B.&Huignard, J. P. Design of a complex filter for depth of focus extension.Opt. Lett.34, 1171–1173 (2009)..

Milgrom, B., Konforti, N., Golub, M. A.&Marom, E. Pupil coding masks for imaging polychromatic scenes with high resolution and extended depth of field.Opt. Express18, 15569–15584 (2010)..

Xing, J. C., Kim, J.&Yoo, H. Design and fabrication of an optical probe with a phase filter for extended depth of focus.Opt. Express24, 1037–1044 (2016)..

Canales, V. F., Oti, J. E.&Cagigal, M. P. Three-dimensional control of the focal light intensity distribution by analytically designed phase masks.Opt. Commun.247, 11–18 (2005)..

Zalevsky, Z., Shemer, A., Zlotnik, A., Eliezer, E. B.&Marom, E. All-optical axial super resolving imaging using a low-frequency binary-phase mask.Opt. Express14, 2631–2643 (2006)..

Sheppard, C. J. R. Binary phase filters with a maximally flat response.Opt. Lett.36, 1386–1388 (2011)..

Cagigal, M. P., Oti, J. E., Canales, V. F.&Valle, P. J. Analytical design of superresolving phase filters.Opt. Commun.241, 249–253 (2004)..

Ryu, S.&Joo, C. Design of binary phase filters for depth-of-focus extension via binarization of axisymmetric aberrations.Opt. Express25, 30312–30326 (2017)..

Katkovnik, V., Hogasten, N.&Egiazarian, K. A novel binary and multilevel phase masks for enhanced depth-of-focus infrared imaging. In Proc.52nd Asilomar Conference on Signals, Systems, and Computers386–390 (Pacific Grove: IEEE, 2018).

Yang, G. G. An optical pickup using a diffractive optical element for a high-density optical disc.Opt. Commun.159, 19–22 (1999)..

Roper, S. W. K., Ryu, S., Seong, B., Joo, C.&Kim, I. Y. A topology optimization implementation for depth-of-focus extension of binary phase filters.Struct. Multidiscip. Optim.62, 2731–2748 (2020)..

Falcón, R., Goudail, F., Kulcsár, C.&Sauer, H. Performance limits of binary annular phase masks codesigned for depth-of-field extension.Opt. Eng.56, 065104 (2017)..

Kennedy, J.&Eberhart, R. Particle swarm optimization. InProc. ICNN'95-International Conference on Neural Networks1942–1948 (Perth: IEEE, 1995).

Fontbonne, A., Sauer, H., Kulcsár, C., Coutrot, A.&Goudail, F. Experimental validation of hybrid optical–digital imaging system for extended depth-of-field based on co-optimized binary phase masks.Opt. Eng.58, 113107 (2019)..

Fontbonne, A., Sauer, H.&Goudail, F. Theoretical and experimental analysis of co-designed binary phase masks for enhancing the depth of field of panchromatic cameras.Opt. Eng.60, 033101 (2021)..

Fontbonne, A., Sauer, H.&Goudail, F. End-to-end optimization of optical systems with extended depth of field under wide spectrum illumination.Appl. Opt.61, 5358–5367 (2022)..

Lin, J., Zhao, H., Ma, Y., Tan, J.&Jin, P. New hybrid genetic particle swarm optimization algorithm to design multi-zone binary filter.Opt. Express24, 10748–10758 (2016)..

Li, M., Du, W. Q.&Nian, F. Z. An adaptive particle swarm optimization algorithm based on directed weighted complex network.Math. Probl. Eng.2014, 1–7 (2014)..

Oldenbourg, R.&Shribak, M. Microscopes.Handbook of Optics. I: Geometrical And Physical Optics, Polarized Light, Components And Instruments3rd ed (eds Bass, M.&Mahajan, V. N. ) 28.1–28.62.0 (New York: McGraw-Hill, 2010)..

Ronneberger, O., Fischer, P.&Brox, T. U-net: convolutional networks for biomedical image segmentation. InProc. 18th International Conference on Medical Image Computing and Computer-Assisted Intervention234–241 (Munich: Springer, 2015).

Borkowski, A. A. et al. Lung and colon cancer histopathological image dataset (lc25000). Preprint athttps://arxiv.org/abs/1912.12142https://arxiv.org/abs/1912.12142(2019).

Nikon Microscopy, U. Triple band excitation (Dapi: FITC-TRITC).https://www.microscopyu.com/techniques/fluorescence/nikon-fluorescence-filter-sets/triple-band-excitation-filter-sets/triple-band-excitation-dapi-fitc-tritchttps://www.microscopyu.com/techniques/fluorescence/nikon-fluorescence-filter-sets/triple-band-excitation-filter-sets/triple-band-excitation-dapi-fitc-tritc(n. d. ).

Lévêque, O., Kulcsár, C.&Goudail, F. Can better performance be obtained when an imaging system is co-optimized with a nonlinear deconvolution algorithm?Proc. SPIE 11866, 118660A, (2021).

Elmalem, S., Giryes, R.&Marom, E. Learned phase coded aperture for the benefit of depth of field extension.Opt. Express26, 15316–15331 (2018)..

Preibisch, S., Saalfeld, S.&Tomancak, P. Globally optimal stitching of tiled 3D microscopic image acquisitions.Bioinformatics25, 1463–1465 (2009)..

Ji, N. Adaptive optical fluorescence microscopy.Nat. Methods14, 374–380 (2017)..

Gibson, S. F.&Lanni, F. Experimental test of an analytical model of aberration in an oil-immersion objective lens used in three-dimensional light microscopy.J. Opt. Soc. Am. A9, 154–166 (1992)..

Sheppard, C. J. R.&Hegedus, Z. S. Axial behavior of pupil-plane filters.J. Opt. Soc. Am. A5, 643–647 (1988)..

Nehme, E. et al. Learning optimal wavefront shaping for multi-channel imaging.IEEE Trans. Pattern Anal. Mach. Intell.43, 2179–2192 (2021)..

Nehme, E. et al. DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning.Nat. Methods17, 734–740 (2020)..

Levin, A., Fergus, R., Durand, F.&Freeman, W. T. Image and depth from a conventional camera with a coded aperture.ACM Trans. Graph26, 70-es (2007)..

Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A.&Tumblin, J. Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing.ACM Trans. Graph26, 69-es (2007)..

Pavani, S. R. P. et al. Three-dimensional, single-molecule fluorescence imaging beyond the diffraction limit by using a double-helix point spread function.Proc. Natl Acad. Sci. USA106, 2995–2999 (2009)..

Shechtman, Y., Sahl, S. J., Backer, A. S.&Moerner, W. E. Optimal point spread function design for 3D imaging.Phys. Rev. Lett.113, 133902 (2014)..

Ryu, S. et al. Light sheet fluorescence microscopy using axi-symmetric binary phase filters.Biomed. Opt. Express11, 3936–3951 (2020)..

Zhao, J. J. et al. Flexible method for generating needle-shaped beams and its application in optical coherence tomography.Optica9, 859–867 (2022)..

Liu, L. B., Liu, C., Howe, W. C., Sheppard, C. J.&Chen, N. Binary-phase spatial filter forreal-time swept-source optical coherence microscopy.Opt. Lett.32, 2375–2377 (2007)..

Cao, R. et al. Optical-resolution photoacoustic microscopy with a needle-shaped beam.Nat. Photon.17, 89–95 (2023)..

Takezawa, A., Nishiwaki, S.&Kitamura, M. Shape and topology optimization based on the phase field method and sensitivity analysis.J. Comp. Phys.229, 2697–2718 (2010)..

Vettenburg, T. et al. Light-sheet microscopy using an Airy beam.Nat. Methods11, 541–544 (2014)..

Wang, Z., Bovik, A. C., Sheikh, H. R.&Simoncelli, E. P. P. Image quality assessment: from error visibility to structural similarity.IEEE Trans. Image Process13, 600–612 (2004)..

Ren, J. H.&Han, K. Y. 2.5-D microscopy: fast, high-throughput imaging via volumetric projection for quantitative subcellular analysis.ACS Photonics8, 933–942 (2021)..

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰