Adaptive 3D descattering with a dynamic synthesis network

Waleed Tahir; Hao Wang; Lei Tian

doi:10.1038/s41377-022-00730-x

Go to Nature Website

Your Location：

Home >

Browse articles >

Adaptive 3D descattering with a dynamic synthesis network

Articles

- Adaptive 3D descattering with a dynamic synthesis network
- Light: Science & Applications Vol. 11, Issue 5, Pages: 766-786(2022)
- Affiliations：
  
  1.Department of Electrical and Computer Engineering, Boston University, Boston, MA, 02215, USA
  2.Department of Biomedical Engineering, Boston University, Boston, MA, 02215, USA
- Author bio：
  
  Lei Tian (leitian@bu.edu)
- Funds：
- DOI：10.1038/s41377-022-00730-x
  CLC：
- Published：31 May 2022，
  
  Published Online：24 February 2022，
  
  Received：30 June 2021，
  
  Revised：22 January 2022，
  
  Accepted：02 February 2022
- Accepted：
Scan QR Code
Tahir, W., Wang, H. & Tian, L. Adaptive 3D descattering with a dynamic synthesis network. Light: Science & Applications, 11, 766-786 (2022).
DOI：

Tahir, W., Wang, H. & Tian, L. Adaptive 3D descattering with a dynamic synthesis network. Light: Science & Applications, 11, 766-786 (2022). DOI： 10.1038/s41377-022-00730-x.

摘要

Abstract

Deep learning has been broadly applied to imaging in scattering applications. A common framework is to train a descattering network for image recovery by removing scattering artifacts. To achieve the best results on a broad spectrum of scattering conditions

individual "expert" networks need to be trained for each condition. However

the expert's performance sharply degrades when the testing condition differs from the training. An alternative brute-force approach is to train a "generalist" network using data from diverse scattering conditions. It generally requires a larger network to encapsulate the diversity in the data and a sufficiently large training set to avoid overfitting. Here

we propose an

adaptive learning

framework

termed dynamic synthesis network (DSN)

which

dynamically

adjusts the model weights and

adapts

to different scattering conditions. The adaptability is achieved by a novel "mixture of experts" architecture that enables dynamically synthesizing a network by blending multiple experts using a gating network. We demonstrate the DSN in holographic 3D particle imaging for a variety of scattering conditions. We show in simulation that our DSN provides generalization across a

continuum

of scattering conditions. In addition

we show that by training the DSN entirely on simulated data

the network can generalize to experiments and achieve robust 3D descattering. We expect the same concept can find many other applications

such as denoising and imaging in scattering media. Broadly

our dynamic synthesis framework opens up a new paradigm for designing highly

adaptive

deep learning and computational imaging techniques.

关键词

Keywords

references

Barbastathis, G., Ozcan, A.&Situ, G. H. On the use of deep learning for computational imaging.Optica6, 921–943 (2019)..

Sinha, A. et al. Lensless computational imaging through deep learning.Optica4, 1117–1125 (2017)..

Xue, Y. J. et al. Reliable deep-learning-based phase imaging with uncertainty quantification.Optica6, 618–629 (2019)..

Wang, F. et al. Phase imaging with an untrained neural network.Light. Sci. Appl.9, 77 (2020)..

Matlock, A.&Tian, L. Physical model simulator-trained neural network for computational 3D phase imaging of multiple-scattering samples. Preprint athttps://arxiv.org/abs/2103.15795https://arxiv.org/abs/2103.15795(2021).

Wang, G., Ye,J. C.&De Man, B. Deep learning for tomographic image reconstruction.Nat. Mach. Intell.2, 737–748 (2020)..

Liu, J. M. et al. RARE: image reconstruction using deep priors learned without groundtruth.IEEE J. Sel. Top. Signal Process.14, 1088–1099 (2020)..

Wu, Z. H. et al. SIMBA: scalable inversion in optical tomography using deep denoising priors.IEEE J. Sel. Top. Signal Process.14, 1163–1175 (2020)..

Gupta, H. et al. CNN-based projected gradient descent for consistent CT image reconstruction.IEEE Trans. Med. Imaging37, 1440–1453 (2018)..

Wang, F. et al. Learning from simulation: an end-to-end deep-learning approach for computational ghost imaging.Opt. Express27, 25560–25572 (2019)..

Rizvi, S. et al. DeepGhost: real-time computational ghost imaging via deep learning.Sci. Rep.10, 11400 (2020)..

Lyu, M. et al. Deep-learning-based ghost imaging.Sci. Rep.7, 17865 (2017)..

Li, F. Q. et al. Compressive ghost imaging through scattering media with deep learning.Opt. Express28, 17395–17408 (2020)..

Wagner, N. et al. Deep learning-enhanced light-field imaging with continuous validation.Nat. Methods18, 557–563 (2021)..

Wang, Z. Q. et al. Real-time volumetric reconstruction of biological dynamics with light-field microscopy and deep learning.Nat. Methods18, 551–556 (2021)..

Wang, H. D. et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy.Nat. Methods16, 103–110 (2019)..

Liu, T. R. et al. Deep learning-based super-resolution in coherent imaging systems.Sci. Rep.9, 3926 (2019)..

Rivenson, Y. et al. Deep learning microscopy.Optica4, 1437–1443 (2017)..

Rivenson, Y. et al. Phase recovery and holographic image reconstruction using deep learning in neural networks.Light. Sci. Appl.7, 17141 (2018)..

Ren, Z. B., Xu, Z. M.&Lam, E. Y. End-to-end deep learning framework for digital holographic reconstruction.Adv. Photon.1, 016004 (2019)..

Rivenson, Y., Wu, Y. C.&Ozcan, A. Deep learning in holography and coherent imaging.Light Sci. Appl.8, 85 (2019)..

Wu, Y. C. et al. Bright-field holography: cross-modality deep learning enables snapshot 3D imaging with bright-field contrast using a single hologram.Light Sci. Appl.8, 25 (2019)..

Li, Y. Z., Xue, Y. J.&Tian, L. Deep speckle correlation: a deep learning approach toward scalable imaging through scattering media.Optica5, 1181–1190 (2018)..

Sun, Y. W. et al. Image reconstruction through dynamic scattering media based on deep learning.Opt. Express27, 16032–16046 (2019)..

Li, Y. Z. et al. Displacement-agnostic coherent imaging through scatter with an interpretable deep neural network.Opt. Express29, 2244–2257 (2021)..

Li, S. et al. Imaging through glass diffusers using densely connected convolutional networks.Optica5, 803–813 (2018)..

Sun, Y., Xia, Z. H.&Kamilov, U. S. Efficient and accurate inversion of multiple scattering with deep learning.Opt. Express26, 14678–14688 (2018)..

Yuksel, S. E., Wilson, J. N.&Gader, P. D. Twenty years of mixture of experts.IEEE Trans. Neural Netw. Learn. Syst.23, 1177–1193 (2012)..

Agostinelli, F., Anderson, M. R.&Lee, H. Adaptive multi-column deep neuralnetworks with application to robust image denoising. InProc 26th International Conference on Neural Information Processing Systems, 1493–1501 (ACM, 2013).

Choi, J. H., Elgendy, O. A.&Chan, S. H. Optimal combination of image denoisers.IEEE Trans. Image Process.28, 4016–4031 (2019)..

Yang, C. Y. et al. Multi-expert learning of adaptive legged locomotion.Sci. Robot.5, eabb2174 (2020)..

Deng, M. et al. Learning to synthesize: robust phase retrieval at low photon counts.Light Sci. Appl.9, 36 (2020)..

Katz, J.&Sheng, J. Applications of holography in fluid mechanics and particle dynamics.Annu. Rev. Fluid Mech.42, 531–555 (2010)..

Wang, H. et al. Large-scale holographic particle 3D imaging with the beam propagation model.Opt. Express29, 17159–17172 (2021)..

Weigert, M. et al. Content-aware image restoration: pushing the limits of fluorescence microscopy.Nat. Methods15, 1090–1097 (2018)..

Wang, D. et al. Non-invasive super-resolution imaging through dynamic scattering media.Nat. Commun.12, 3150 (2021)..

Zheng, S. S. et al. Incoherent imaging through highly nonstatic and optically thick turbid media based on neural network.Photon. Res.9, B220–B228 (2021)..

Pégard, N. C. et al. Compressive light-field microscopy for 3D neural activity recording.Optica3, 517–524 (2016)..

Xue, Y. J. et al. Single-shot 3D wide-field fluorescence imaging with a computational miniature mesoscope.Sci. Adv.6, eabb7508 (2020)..

Turpin, A., Vishniakou, I.&Seelig, J. D. Light scattering control in transmission and reflection with neural networks.Opt. Express26, 30911–30929 (2018)..

Rahmani, B. et al. Actor neural networks for the robust control of partially measured nonlinear systems showcased for image propagation through diffuse media.Nat. Mach. Intell.2, 403–410 (2020)..

Turpin, A. et al. Spatial images from temporal data.Optica7, 900–905 (2020)..

Skarsoulis, K., Kakkava, E.&Psaltis, D. Predicting optical transmission through complex scattering media from reflection patterns with deep neural networks.Opt. Commun.492, 126968 (2021)..

Tahir, W., Kamilov, U. S.&Tian, L. Holographic particle localization under multiple scattering.Adv. Photon.1, 036003 (2019)..

Milletari, F., Navab, N.&Ahmadi, S. A. V-Net: fully convolutional neural networksfor volumetric medical image segmentation. InProc 4th International Conference on 3D Vision (3DV)565–571 (IEEE, 2016).

Glorot, X.&Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. InProc 13th International Conference on Artificial Intelligence and Statistics, 249–256 (JMLR, 2010).

McInnes, L., Healy, J.&Melville, J. UMAP: uniform manifold approximation and projection for dimension reduction. Preprint athttps://arxiv.org/abs/1802.03426https://arxiv.org/abs/1802.03426(2018).

Tan, C. Q. et al. A survey on deep transfer learning. InProc 27th International Conference on Artificial Neural Networks, 270–279 (Springer, 2018).

Ganin, Y. et al. Domain-adversarial training of neural networks.J. Mach. Learn. Res.17, 2096–2030 (2016)..

Tian, L. et al. Quantitative measurement of size and three-dimensional position of fast-moving bubbles in air-water mixture flows using digital holography.Appl. Opt.49, 1549–1554 (2010)..

Hinsch, K. D. Holographic particle image velocimetry.Meas. Sci. Technol.13, R61–R72 (2002)..

Chen, N., Wang, C. L.&Heidrich, W. Snapshot space–time holographic 3D particle tracking velocimetry.Laser Photon. Rev.15, 2100008 (2021)..

Cheong, F. C. et al. Flow visualization and flow cytometry with holographic video microscopy.Opt. Express17, 13071–13079 (2009)..

Seo, S. et al. Lensfree holographic imaging for on-chip cytometry and diagnostics.Lab Chip9, 777–787 (2009)..

Merola, F. et al. Tomographic flow cytometry by digital holography.Light Sci. Appl.6, e16241 (2017)..

Moon, I. et al. Automated three-dimensional identification and tracking of micro/nanobiological organisms by computational holographic microscopy.Proc. IEEE97, 990–1010 (2009)..

Su, T. W., Xue, L.&Ozcan, A. High-throughput lensfree 3D tracking of human sperms reveals rare statistics of helical trajectories.Proc. Natl Acad. Sci. USA109, 16018–16022 (2012)..

Faccio, D., Velten, A.&Wetzstein, G. Non-line-of-sight imaging.Nat. Rev. Phys.2, 318–327 (2020)..

Badon, A. et al. Smart optical coherence tomography for ultra-deep imaging through highly scattering media.Sci. Adv.2, e1600370 (2016)..

Kang, S. et al. Imaging deep within a scattering medium using collective accumulation of single-scattered waves.Nat. Photon.9, 253–258 (2015)..

Huang, L. Z. et al. Recurrent neural network-based volumetric fluorescence microscopy.Light Sci. Appl.10, 62 (2021)..

Kang, I., Goy, A.&Barbastathis, G. Dynamical machine learning volumetric reconstruction of objects' interiors from limited angular views.Light Sci. Appl.10, 74 (2021)..

Goy, A. et al. High-resolution limited-angle phase tomography of dense layered objects using deep neural networks.Proc. Natl Acad. Sci. USA116, 19848–19856 (2019)..

Roy, A. G., Navab, N.&Wachinger, C. Recalibrating fully convolutional networks with spatial and channel "squeeze and excitation" blocks.IEEE Trans. Med. Imaging.38, 540–549 (2019)..

Simonyan, K.&Zisserman, A. Very deep convolutional networks for large-scale image recognition. Preprint athttps://arxiv.org/abs/1409.1556https://arxiv.org/abs/1409.1556(2014).

Bridson, R. Fast Poisson disk sampling in arbitrary dimensions. InProc ACM SIGGRAPH 2007 Sketches, 22-es (ACM, 2007).

Chen, W. S. et al. Empirical concentration bounds for compressive holographic bubble imaging based on a Mie scattering model.Opt. Express23, 4715–4725 (2015)..

Otsu, N. A threshold selection method from gray-level histograms.IEEE Trans. Syst. Man Cybern.9, 62–66 (1979)..

Burkard, R. E.&Çela, E.Handbook of Combinatorial Optimization: Supplement Volume A(eds. Du, D. Z.&Pardalos, P. M. ) 75–149 (Springer, 1999).

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰