Fourier Imager Network (FIN): A deep neural network for hologram reconstruction with superior external generalization

Hanlong Chen; Luzhe Huang; Tairan Liu; Aydogan Ozcan

doi:10.1038/s41377-022-00949-8

Go to Nature Website

Your Location：

Home >

Browse articles >

Fourier Imager Network (FIN): A deep neural network for hologram reconstruction with superior external generalization

Original Articles

- Fourier Imager Network (FIN): A deep neural network for hologram reconstruction with superior external generalization
- Light: Science & Applications Vol. 11, Issue 9, Pages: 2225-2234(2022)
- Affiliations：
  
  1.Electrical and Computer Engineering Department, University of California, Los Angeles, CA 90095, USA
  2.Bioengineering Department, University of California, Los Angeles, CA 90095, USA
  3.California Nano Systems Institute (CNSI), University of California, Los Angeles, CA 90095, USA
  4.David Geffen School of Medicine, University of California Los Angeles, Los Angeles, CA 90095, USA
- Author bio：
  
  Aydogan Ozcan (ozcan@ucla.edu)
- Funds：
- DOI：10.1038/s41377-022-00949-8
  CLC：
Scan for full text
Chen, H. L. et al. Fourier Imager Network (FIN): A deep neural network for hologram reconstruction with superior external generalization. Light: Science & Applications, 11, 2225-2234 (2022).
DOI：

Chen, H. L. et al. Fourier Imager Network (FIN): A deep neural network for hologram reconstruction with superior external generalization. Light: Science & Applications, 11, 2225-2234 (2022). DOI： 10.1038/s41377-022-00949-8.

摘要

Abstract

Deep learning-based image reconstruction methods have achieved remarkable success in phase recovery and holographic imaging. However, the generalization of their image reconstruction performance to new types of samples never seen by the network remains a challenge. Here we introduce a deep learning framework, termed Fourier Imager Network (FIN), that can perform end-to-end phase recovery and image reconstruction from raw holograms of new types of samples, exhibiting unprecedented success in external generalization. FIN architecture is based on spatial Fourier transform modules that process the spatial frequencies of its inputs using learnable filters and a global receptive field. Compared with existing convolutional deep neural networks used for hologram reconstruction, FIN exhibits superior generalization to new types of samples, while also being much faster in its image inference speed, completing the hologram reconstruction task in ~0.04 s per 1 mm,2, of the sample area. We experimentally validated the performance of FIN by training it using human lung tissue samples and blindly testing it on human prostate, salivary gland tissue and Pap smear samples, proving its superior external generalization and image reconstruction speed. Beyond holographic microscopy and quantitative phase imaging, FIN and the underlying neural network architecture might open up various new opportunities to design broadly generalizable deep learning models in computational imaging and machine vision fields.

关键词

Keywords

references

S. S. Kou, 等. Transport-of-intensity approach to differential interference contrast (TI-DIC) microscopy for quantitative phase imaging. Opt. Lett., 2010. 35447-449. DOI:10.1364/OL.35.000447http://doi.org/10.1364/OL.35.000447.

Popescu, G. Quantitative phase imaging of cells and tissues. (New York: McGraw-Hill, 2011).

V. Chhaniwal, 等. Quantitative phase-contrast imaging with compact digital holographic microscope employing Lloyd's mirror. Opt. Lett., 2012. 375127-5129. DOI:10.1364/OL.37.005127http://doi.org/10.1364/OL.37.005127.

M. H. Jericho, 等. Quantitative phase and refractive index measurements with point-source digital in-line holographic microscopy. Appl. Opt., 2012. 511503-1515. DOI:10.1364/AO.51.001503http://doi.org/10.1364/AO.51.001503.

T. W. Su, L. Xue, A. Ozcan. High-throughput lensfree 3D tracking of human sperms reveals rare statistics of helical trajectories. Proc. Natl Acad. Sci. USA, 2012. 10916018-16022. DOI:10.1073/pnas.1212506109http://doi.org/10.1073/pnas.1212506109.

A. Greenbaum, 等. Wide-field computational color imaging using pixel super-resolved on-chip microscopy. Opt. Express, 2013. 2112469-12483. DOI:10.1364/OE.21.012469http://doi.org/10.1364/OE.21.012469.

A. Greenbaum, 等. Wide-field computational imaging of pathology slides using lens-free on-chip microscopy. Sci. Transl. Med., 2014. 6267ra175DOI:10.1126/scitranslmed.3009850http://doi.org/10.1126/scitranslmed.3009850.

L. Tian, L. Waller. Quantitative differential phase contrast imaging in an LED array microscope. Opt. Express, 2015. 2311394-11403. DOI:10.1364/OE.23.011394http://doi.org/10.1364/OE.23.011394.

F. Merola, 等. Tomographic flow cytometry by digital holography. Light Sci. Appl., 2017. 6e16241DOI:10.1038/lsa.2016.241http://doi.org/10.1038/lsa.2016.241.

Y. Park, C. Depeursinge, G. Popescu. Quantitative phase imaging in biomedicine. Nat. Photonics, 2018. 12578-589. DOI:10.1038/s41566-018-0253-xhttp://doi.org/10.1038/s41566-018-0253-x.

G. Barbastathis, A. Ozcan, G. H. Situ. On the use of deep learning for computational imaging. Optica, 2019. 6921-943. DOI:10.1364/OPTICA.6.000921http://doi.org/10.1364/OPTICA.6.000921.

B. Javidi, 等. Roadmap on digital holography [Invited]. Opt. Express, 2021. 2935078-35118. DOI:10.1364/OE.435915http://doi.org/10.1364/OE.435915.

J. R. Fienup. Phase retrieval algorithms: a comparison. Appl. Opt., 1982. 212758-2769. DOI:10.1364/AO.21.002758http://doi.org/10.1364/AO.21.002758.

M. R. Teague. Deterministic phase retrieval: a Green's function solution. J. Optical Soc. Am., 1983. 731434-1441. DOI:10.1364/JOSA.73.001434http://doi.org/10.1364/JOSA.73.001434.

G. Z. Yang, 等. Gerchberg–Saxton and Yang–Gu algorithms for phase retrieval in a nonunitary transform system: a comparison. Appl. Opt., 1994. 33209-218. DOI:10.1364/AO.33.000209http://doi.org/10.1364/AO.33.000209.

L. J. Allen, M. P. Oxley. Phase retrieval from series of images obtained by defocus variation. Opt. Commun., 2001. 19965-75. DOI:10.1016/S0030-4018(01)01556-5http://doi.org/10.1016/S0030-4018(01)01556-5.

S. Marchesini. Invited article: a unified evaluation of iterative projection algorithms for phase retrieval. Rev. Sci. Instrum., 2007. 78011301DOI:10.1063/1.2403783http://doi.org/10.1063/1.2403783.

A. Greenbaum, A. Ozcan. Maskless imaging of dense samples using pixel super-resolution based multi-height lensfree on-chip microscopy. Opt. Express, 2012. 203129-3143. DOI:10.1364/OE.20.003129http://doi.org/10.1364/OE.20.003129.

A. Greenbaum, U. Sikora, A. Ozcan. Field-portable wide-field microscopy of dense samples using multi-height pixel super-resolution based lensfree imaging. Lab a Chip, 2012. 121242-1245. DOI:10.1039/c2lc21072jhttp://doi.org/10.1039/c2lc21072j.

W. Luo, 等. Synthetic aperture-based on-chip microscopy. Light Sci. Appl., 2015. 4e261DOI:10.1038/lsa.2015.34http://doi.org/10.1038/lsa.2015.34.

Y. Rivenson, 等. Sparsity-based multi-height phase recovery in holographic microscopy. Sci. Rep., 2016. 637862DOI:10.1038/srep37862http://doi.org/10.1038/srep37862.

W. Luo, 等. Propagation phasor approach for holographic image reconstruction. Sci. Rep., 2016. 622738DOI:10.1038/srep22738http://doi.org/10.1038/srep22738.

Y. Rivenson, 等. Phase recovery and holographic image reconstruction using deep learning in neural networks. Light Sci. Appl., 2018. 717141DOI:10.1038/lsa.2017.141http://doi.org/10.1038/lsa.2017.141.

H. Wang, M. Lyu, G. H. Situ. eHoloNet: a learning-based end-to-end approach for in-line digital holographic reconstruction. Opt. Express, 2018. 2622603-22614. DOI:10.1364/OE.26.022603http://doi.org/10.1364/OE.26.022603.

Y. C. Wu, 等. Extended depth-of-field in holographic imaging using deep-learning-based autofocusing and phase recovery. Optica, 2018. 5704-710. DOI:10.1364/OPTICA.5.000704http://doi.org/10.1364/OPTICA.5.000704.

A. Goy, 等. Low photon count phase retrieval using deep learning. Phys. Rev. Lett., 2018. 121243902DOI:10.1103/PhysRevLett.121.243902http://doi.org/10.1103/PhysRevLett.121.243902.

G. Zhang, 等. Fast phase retrieval in off-axis digital holographic microscopy through deep learning. Opt. Express, 2018. 2619388-19405. DOI:10.1364/OE.26.019388http://doi.org/10.1364/OE.26.019388.

Z. B. Ren, Z. M. Xu, E. Y. Lam. Learning-based nonparametric autofocusing for digital holography. Optica, 2018. 5337-344. DOI:10.1364/OPTICA.5.000337http://doi.org/10.1364/OPTICA.5.000337.

Y. Rivenson, Y. C. Wu, A. Ozcan. Deep learning in holography and coherent imaging. Light Sci. Appl., 2019. 885DOI:10.1038/s41377-019-0196-0http://doi.org/10.1038/s41377-019-0196-0.

T. R. Liu, 等. Deep learning-based color holographic microscopy. J. Biophotonics, 2019. 12e201900107.

Y. C. Wu, 等. Bright-field holography: cross-modality deep learning enables snapshot 3D imaging with bright-field contrast using a single hologram. Light Sci. Appl., 2019. 825DOI:10.1038/s41377-019-0139-9http://doi.org/10.1038/s41377-019-0139-9.

Y. Jo, 等. Quantitative phase imaging and artificial intelligence: a review. IEEE J. Sel. Top. Quantum Electron., 2019. 256800914DOI:10.1109/JSTQE.2018.2859234http://doi.org/10.1109/JSTQE.2018.2859234.

K. Q. Wang, 等. Y-Net: a one-to-two deep learning framework for digital holographic reconstruction. Opt. Lett., 2019. 444765-4768. DOI:10.1364/OL.44.004765http://doi.org/10.1364/OL.44.004765.

H. Byeon, T. Go, S. J. Lee. Deep learning-based digital in-line holographic microscopy for high resolution with extended field of view. Opt. Laser Technol., 2019. 11377-86. DOI:10.1016/j.optlastec.2018.12.014http://doi.org/10.1016/j.optlastec.2018.12.014.

Z. B. Ren, Z. M. Xu, E. Y. M. Lam. End-to-end deep learning framework for digital holographic reconstruction. Adv. Photonics, 2019. 1016004DOI:10.1117/1.AP.1.1.016004http://doi.org/10.1117/1.AP.1.1.016004.

H. Y. Li, 等. Deep DIH: single-shot digital in-line holography reconstruction by deep learning. IEEE Access, 2020. 8202648DOI:10.1109/ACCESS.2020.3036380http://doi.org/10.1109/ACCESS.2020.3036380.

I. Moon, 等. Noise-free quantitative phase imaging in Gabor holography with conditional generative adversarial network. Opt. Express, 2020. 2826284-26301. DOI:10.1364/OE.398528http://doi.org/10.1364/OE.398528.

T. Zeng, H. K. H. So, E. Y. Lam. RedCap: residual encoder-decoder capsule network for holographic image reconstruction. Opt. Express, 2020. 284876-4887. DOI:10.1364/OE.383350http://doi.org/10.1364/OE.383350.

T. R. Liu, 等. Deep learning-based holographic polarization microscopy. ACS Photonics, 2020. 73023-3034. DOI:10.1021/acsphotonics.0c01051http://doi.org/10.1021/acsphotonics.0c01051.

M. Deng, 等. Learning to synthesize: robust phase retrieval at low photon counts. Light Sci. Appl., 2020. 936DOI:10.1038/s41377-020-0267-2http://doi.org/10.1038/s41377-020-0267-2.

D. Yin, 等. Digital holographic reconstruction based on deep learning framework with unpaired data. IEEE Photonics J., 2020. 123900312DOI:10.1109/JPHOT.2019.2961137http://doi.org/10.1109/JPHOT.2019.2961137.

L. Z. Huang, 等. Holographic image reconstruction with phase recovery and autofocusing using recurrent neural networks. ACS Photonics, 2021. 81763-1774. DOI:10.1021/acsphotonics.1c00337http://doi.org/10.1021/acsphotonics.1c00337.

X. Yang, 等. High imaging quality of Fourier single pixel imaging based on generative adversarial networks at low sampling rate. Opt. Lasers Eng., 2021. 140106533DOI:10.1016/j.optlaseng.2021.106533http://doi.org/10.1016/j.optlaseng.2021.106533.

T. Shimobaba, 等. Deep-learning computational holography: a review (invited). Front. Photonics, 2022. 3854391DOI:10.3389/fphot.2022.854391http://doi.org/10.3389/fphot.2022.854391.

D. Pirone, 等. Speeding up reconstruction of 3D tomograms in holographic flow cytometry via deep learning. Lab a Chip, 2022. 22793-804. DOI:10.1039/D1LC01087Ehttp://doi.org/10.1039/D1LC01087E.

L. Z. Huang, 等. Recurrent neural network-based volumetric fluorescence microscopy. Light Sci. Appl., 2021. 1062DOI:10.1038/s41377-021-00506-9http://doi.org/10.1038/s41377-021-00506-9.

I. Goodfellow, 等. Generative adversarial networks. Commun. ACM, 2020. 63139-144. DOI:10.1145/3422622http://doi.org/10.1145/3422622.

S. J. Kim, 等. Deep transfer learning-based hologram classification for molecular diagnostics. Sci. Rep., 2018. 817003DOI:10.1038/s41598-018-35274-xhttp://doi.org/10.1038/s41598-018-35274-x.

Y. M. Zhu, C. H. Yeung, E. Y. Lam. Digital holographic imaging and classification of microplastics using deep transfer learning. Appl. Opt., 2021. 60A38-A47. DOI:10.1364/AO.403366http://doi.org/10.1364/AO.403366.

L. Z. Huang, 等. Few-shot transfer learning for holographic image reconstruction using a recurrent neural network. APL Photonics, 2022. 7070801DOI:10.1063/5.0090582http://doi.org/10.1063/5.0090582.

Wang, Z., Simoncelli, E. P. & Bovik, A. C. Multiscale structural similarity for image quality assessment. The Thirty-Seventh Asilomar Conference on Signals, Systems & Computers, 2003. Pacific Grove: IEEE, 2003, 1398-1402.

Li, Z. Y. et al. Fourier neural operator for parametric partial differential equations. The 9th International Conference on Learning Representations (ICLR-21). https://iclr.cc/virtual/2021/poster/3281https://iclr.cc/virtual/2021/poster/3281 (2021).

W. Bishara, 等. Lensfree on-chip microscopy over a wide field-of-view using pixel super-resolution. Opt. Express, 2010. 1811181-11191. DOI:10.1364/OE.18.011181http://doi.org/10.1364/OE.18.011181.

A. Greenbaum, 等. Imaging without lenses: achievements and remaining challenges of wide-field on-chip microscopy. Nat. Methods, 2012. 9889-895. DOI:10.1038/nmeth.2114http://doi.org/10.1038/nmeth.2114.

Y. B. Zhang, 等. Edge sparsity criterion for robust holographic autofocusing. Opt. Lett., 2017. 423824-3827. DOI:10.1364/OL.42.003824http://doi.org/10.1364/OL.42.003824.

Goodman, J. W. Introduction to Fourier Optics. 3rd edn. (Greenwoood Village: Roberts & Company Publishers, 2005).

Zhang, Y. L. et al. Image super-resolution using very deep residual channel attention networks. The 15th European Conference on Computer Vision. Munich, Germany: Springer, 2018, 294-310.

Paszke, A. et al. PyTorch: an imperative style, high-performance deep learning library. Proceedings of the 33rd International Conference on Neural Information Processing Systems. Vancouver: ACM, 2019, 8026-8037.

Johnson, J., Alahi, A. & Fei-Fei, L. Perceptual losses for real-time style transfer and super-resolution. The 14th European Conference on Computer Vision. Amsterdam: Springer, 2016, 694-711.

Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. Print at https://doi.org/10.48550/arXiv.1412.6980https://doi.org/10.48550/arXiv.1412.6980 (2014).

Loshchilov, I. & Hutter, F. SGDR: stochastic gradient descent with warm restarts. Print at https://doi.org/10.48550/arXiv.1608.03983https://doi.org/10.48550/arXiv.1608.03983 (2016).

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰