Dynamical machine learning volumetric reconstruction of objects' interiors from limited angular views

Iksung Kang; Alexandre Goy; George Barbastathis

doi:10.1038/s41377-021-00512-x

Go to Nature Website

Your Location：

Home >

Browse articles >

Dynamical machine learning volumetric reconstruction of objects' interiors from limited angular views

Articles

- Dynamical machine learning volumetric reconstruction of objects' interiors from limited angular views
- Light: Science & Applications Vol. 10, Issue 5, Pages: 824-844(2021)
- Affiliations：
  
  1.Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, 77 Massachusetts Ave, Cambridge, MA, USA
  2.Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
  3.Singapore-MIT Alliance for Research and Technology (SMART) Centre, 1 Create Way, Singapore 117543, Singapore
  4.Present address: Omnisens SA, Morges 1110, Switzerland
- Author bio：
  
  Iksung Kang (iskang@mit.edu)
- Funds：
- DOI：10.1038/s41377-021-00512-x
  CLC：
- Published：31 May 2021，
  
  Published Online：07 April 2021，
  
  Received：17 September 2020，
  
  Revised：04 March 2021，
  
  Accepted：13 March 2021
- Accepted：
Scan QR Code
Kang, I., Goy, A. & Barbastathis G. Dynamical machine learning volumetric reconstruction of objects' interiors from limited angular views. Light: Science & Applications, 10, 824-844 (2021).
DOI：

Kang, I., Goy, A. & Barbastathis G. Dynamical machine learning volumetric reconstruction of objects' interiors from limited angular views. Light: Science & Applications, 10, 824-844 (2021). DOI： 10.1038/s41377-021-00512-x.

摘要

Abstract

Limited-angle tomography of an interior volume is a challenging

highly ill-posed problem with practical implications in medical and biological imaging

manufacturing

automation

and environmental and food security. Regularizing priors are necessary to reduce artifacts by improving the condition of such problems. Recently

it was shown that one effective way to learn the priors for strongly scattering yet highly structured 3D objects

e.g. layered and Manhattan

is by a static neural network [Goy et al.

Proc. Natl. Acad. Sci

. 116

19848-19856 (2019)

]

. Here

we present a radically different approach where the collection of raw images from multiple angles is viewed analogously to a dynamical system driven by the object-dependent forward scattering operator. The sequence index in the angle of illumination plays the role of discrete time in the dynamical system analogy. Thus

the imaging problem turns into a problem of nonlinear system identification

which also suggests dynamical learning as a better fit to regularize the reconstructions. We devised a Recurrent Neural Network (RNN) architecture with a novel Separable-Convolution Gated Recurrent Unit (SC-GRU) as the fundamental building block. Through a comprehensive comparison of several quantitative metrics

we show that the dynamic method is suitable for a generic interior-volumetric reconstruction under a limited-angle scheme. We show that this

approach accurately reconstructs volume interiors under two conditions: weak scattering

when the Radon transform approximation is applicable and the forward operator well defined; and strong scattering

which is nonlinear with respect to the 3D refractive index distribution and includes uncertainty in the forward operator.

关键词

Keywords

references

Radon, J. On the determination of functions from their integral values along certain manifolds.IEEE Trans. Med. Imaging5, 170-176 (1986)..

Radon, J. On the determination of functions from their integrals along certain manifolds.Ber. Saechsische Akademie Wissenschaften29, 262-277 (1917)..

Bracewell, R. N.&Riddle, A. C. Inversion of fan-beam scans in radio astronomy.Astrophysical J.150, 427 (1967)..

Feldkamp, L. A., Davis, L. C.&Kress, J. W. Practical cone-beam algorithm.J. Optical Soc. Am. A1, 612-619 (1984)..

Dreike, P.&Boyd, D. P. Convolution reconstruction of fan beam projections.Computer Graph. Image Process.5, 459-469 (1976)..

Wang, G. et al. A general cone-beam reconstruction algorithm.IEEE Trans. Med. Imaging12, 486-496 (1993)..

Kudo, H.&Saito, T. Helical-scan computed tomography using cone-beam projections. InProc. Conference Record of the 1991 IEEE Nuclear Science Symposium and Medical Imaging Conference1958-1962 (IEEE, 1991).

Grangeat, P. inMathematical Methods in Tomography(eds Herman, G. T., Louis, A. K.&Natterer, F. ) 66-97 (Springer, 1991).

Katsevich, A. Analysis of an exact inversion algorithm for spiral cone-beam CT.Phys. Med. Biol.47, 2583-2597 (2002)..

Choi, W. et al. Tomographic phase microscopy.Nat. Methods4, 717-719 (2007)..

Delaney, A. H.&Bresler, Y. Globally convergent edge-preserving regularized reconstruction: an application to limited-angle tomography.IEEE Trans. Image Process.7, 204-221 (1998)..

Bartolac, S. et al. A local shift‐variant Fourier model and experimental validation of circular cone‐beam computed tomography artifacts.Med. Phys.36, 500-512 (2009)..

Lim, J. W. et al. Comparative study of iterative reconstruction algorithms for missing cone problems in optical diffraction tomography.Opt. Express23, 16933-16948 (2015)..

Ishimaru, A.Electromagnetic Wave Propagation, Radiation, and Scattering: From Fundamentals to Applications. 1st edn (Prentice-Hall, 1991).

Tatarski, V. I.Wave Propagation in a Turbulent Medium(Dover Publications, 2016).

Wolf, E. Three-dimensional structure determination of semi-transparent objects from holographic data.Opt. Commun.1, 153-156 (1969)..

Devaney, A. J. Inverse-scattering theory within the Rytov approximation.Opt. Lett.6, 374-376 (1981)..

Pham, T. A. et al. Three-dimensional optical diffraction tomography with Lippmann-Schwinger model.IEEE Trans. Comput. Imaging6, 727-738 (2020)..

Marks, D. L. A family of approximations spanning the Born and Rytov scattering series.Opt. Express14, 8837-8848 (2006)..

Milgram, J. H.&Li, W. C. Computational reconstruction of images from holograms.Appl. Opt.41, 853-864 (2002)..

Tian, L. et al. Quantitative measurement of size and three-dimensional position of fast-moving bubbles in air-water mixture flows using digital holography.Appl. Opt.49, 1549-1554 (2010)..

Hahn, J. et al. Wide viewing angle dynamic holographic stereogram with a curved array of spatial light modulators.Opt. Express16, 12372-12386 (2008)..

Park, J. H., Hong, K.&Lee, B. Recent progress in three-dimensional information processing based on integral imaging.Appl. Opt.48, H77-H94 (2009)..

Nehmetallah, G.&Banerjee, P. P. Applications of digital and analog holography in three-dimensional imaging.Adv. Opt. Photonics4, 472-553 (2012)..

Williams, L., Nehmetallah, G.&Banerjee, P. P. Digital tomographic compressive holographic reconstruction of three-dimensional objects in transmissive and reflective geometries.Appl. Opt.52, 1702-1710 (2013)..

Brady, D. J. et al. Compressive holography.Opt. Express17, 13040-13049 (2009)..

Choi, K. et al. Compressive holography of diffuse objects.Appl. Opt.49, H1-H10 (2010)..

Rivenson, Y. et al. Phase recovery and holographic image reconstruction using deep learning in neural networks.Light. : Sci. Appl.7, 17141 (2018)..

Wu, Y. C. et al. Bright-field holography: cross-modality deep learning enables snapshot 3d imaging with bright-field contrast using a single hologram.Light. : Sci. Appl.8, 25 (2019)..

Rivenson, Y., Wu, Y. C.&Ozcan, A. Deep learning in holography and coherent imaging.Light. : Sci. Appl.8, 85 (2019)..

Zhang, W. H. et al. Twin-image-free holography: a compressive sensing approach.Phys. Rev. Lett.121, 093902 (2018)..

Kamilov, U. S. et al. A recursive born approach to nonlinear inverse scattering.IEEE Signal Process. Lett.23, 1052-1056 (2016)..

Kamilov, U. S. et al. Optical tomographic image reconstruction based on beam propagation and sparse regularization.IEEE Trans. Comput. Imaging2, 59-70 (2016)..

Giorgi, G. et al. Application of the inhomogeneous Lippmann-Schwinger equation to inverse scattering problems.SIAM J. Appl. Math.73, 212-231 (2013)..

Chew, W. C.&Wang, Y. M. Reconstruction of two-dimensional permittivity distribution using the distorted Born iterative method.IEEE Trans. Med. Imaging9, 218-225 (1990)..

Sun, Y., Xia, Z. H.&Kamilov, U. S. Efficient and accurate inversion of multiple scattering with deep learning.Opt. Express26, 14678-14688 (2018)..

Lu, Z. Q. Multidimensional structure diffraction tomography for varying object orientation through generalised scattered waves.Inverse Probl.1, 339-356 (1985)..

Lu, Z. Q. JKM perturbation theory, relaxation perturbation theory, and their applications to inverse scattering: theory and reconstruction algorithms.IEEE Trans. Ultrason. Ferroelectr. Frequency Control33, 722-730 (1986)..

Tsihrintzis, G. A.&Devaney, A. J. Higher order (nonlinear) diffraction tomography: Inversion of the Rytov series.IEEE Trans. Inf. Theory46, 1748-1761 (2000)..

Feit, M. D.&Fleck, J. A. Computation of mode properties in optical fiber waveguides by a propagating beam method.Appl. Opt.19, 1154-1164 (1980)..

Kamilov, U. S. et al. Learning approach to optical tomography.Optica2, 517-522 (2015)..

Shoreh, M. H. et al. Optical tomography based on a nonlinear model that handles multiple scattering. InProc. 2017 IEEE International Conference on Acoustics, Speech and Signal Processing6220-6224 (IEEE, 2017).

Lim, J. et al. Learning tomography assessed using Mie theory.Phys. Rev. Appl.9, 034027 (2018)..

Chowdhury, S. et al. High-resolution 3D refractive index microscopy of multiple-scattering samples from intensity images.Optica6, 1211-1219 (2019)..

Goy, A. et al. High-resolution limited-angle phase tomography of dense layered objects using deep neural networks.Proc. Natl Acad. Sci. USA116, 19848-19856 (2019)..

Bertero, M.&Boccacci, P.Introduction to Inverse Problems in Imaging(IOP Publishing Ltd., 1998).

Candès, E. J., Romberg, J.&Tao, T. Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information.IEEE Trans. Inf. Theory52, 489-509 (2006)..

Gregor, K.&LeCun, Y. Learning fast approximations of sparse coding. InProc. 27th International Conference on Machine Learning399-406 (ACM, 2010).

Barbastathis, G., Ozcan, A.&Situ, G. On the use of deep learning for computational imaging.Optica6, 921-943 (2019)..

Jin, K. H. et al. Deep convolutional neural network for inverse problems in imaging.IEEE Trans. Image Process.26, 4509-4522 (2017)..

Jacobs, O. L. R.Introduction to Control Theory(Oxford University Press, 1993).

Mardani, M. et al. Deep generative adversarial networks for compressed sensing automates MRI. Preprint athttps://arxiv.org/abs/1706.00051https://arxiv.org/abs/1706.00051(2017).

Mardani, M. et al. Recurrent generative adversarial networks for proximal learning and automated compressive image recovery. Preprint athttps://arxiv.org/abs/1711.10046https://arxiv.org/abs/1711.10046(2017).

Daubechies, I., Defrise, M.&De Mol, C. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint.Commun. Pure Appl. Math.57, 1413-1457 (2004)..

Williams, R. J.&Zipser, D. A learning algorithm for continually running fully recurrent neural networks.Neural Comput.1, 270-280 (1989)..

Hochreiter, S.&Schmidhuber, J. Long short-term memory.Neural Comput.9, 1735-1780 (1997)..

Shi, X. J. et al. Convolutional LSTM network: a machine learning approach for precipitation nowcasting. InProc. 28th International Conference on Neural Information Processing Systems802-810 (MIT Press, 2015).

Wang, Y. B. et al. Eidetic 3D LSTM: A model for video prediction and beyond. InProc. International Conference on Learning Representations(OpenReview. net, 2019).

Wang, Y. B. et al. PredRNN: recurrent neural networks for predictive learning using spatiotemporal LSTMs. InProc. 31st Conference on Neural Information Processing Systems879-888 (ACM, 2017).

Wang, Y. B. et al. PredRNN+ +: towards a resolution of the deep-in-time dilemma in spatiotemporal predictive learning. InProc. 35th International Conference on Machine Learning. (PMLR, 2018).

Kumar, A. C. S., Bhandarkar, S. M.&Prasad, M. DepthNet: a recurrent neural network architecture for monocular depth prediction. InProc. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops283-291 (IEEE, 2018).

Wang, W. Y. et al. Shape inpainting using 3D generative adversarial network and recurrent convolutional networks.Proc. 2017 IEEE International Conference on Computer Vision (IEEE, 2017, 2298-2306.

Liu, J.&Ji, S. P. A novel recurrent encoder-decoder structure for large-scale multi-view stereo reconstruction from an open aerial dataset. InProc 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition6050-6059(IEEE, 2020).

Choy, C. B. et al. 3D-R2N2: a unified approach for single and multi-view 3D object reconstruction. InProc. 14th European Conference on Computer Vision628-644 (Springer, 2016).

Le, T., Bui, G.&Duan, Y. A multi-view recurrent neural network for 3D mesh segmentation.Comput. Graph.66, 103-112 (2017)..

Stollenga, M. F. et al. Parallel multi-dimensional LSTM, with application to fast biomedical volumetric image segmentation. InProc. 28th International Conference on Neural Information Processing Systems(MIT Press, 2015).

Hou, Y. X., Kannala, J.&Solin, A. Multi-view stereo by temporal nonparametric fusion. InProc. 2019 IEEE/CVF International Conference on Computer Vision2651-2660 (IEEE, 2019).

Cierniak, R. A new approach to image reconstruction from projections using a recurrent neural network.Int. J. Appl. Math. Comput. Sci.18, 147-157 (2008)..

Cho, K. et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In:Proc. 2014 Conference on Empirical Methods in Natural Language Processing1724-1734 (Association for Computational Linguistics, 2014).

Kang, I., Goy, A.&Barbastathis, G. Limited-angle tomographic reconstruction of dense layered objects by dynamical machine learning. Preprint athttps://arxiv.org/abs/2007.10734https://arxiv.org/abs/2007.10734(2020).

Bioucas-Dias, J. M.&Figueiredo, M. A. T. A new TwIST: two-step iterative shrinkage/thresholding algorithms for image restoration.IEEE Trans. Image Process.16, 2992-3004 (2007)..

Nair, V.&Hinton, G. E. Rectified linear units improve restricted Boltzmann machines. InProc. 27th International Conference on Machine Learning(ACM, 2010).

Deng, M. et al. On the interplay between physical and content priors in deep learning for computational imaging.Opt. Express28, 24152-24170 (2020)..

Bahdanau, D., Cho, K.&Bengio, Y. Neural machine translation by jointly learning to align and translate. Preprint athttps://arxiv.org/abs/1409.0473https://arxiv.org/abs/1409.0473(2014).

Lukoševicius, M.&Jaeger, H. Reservoir computing approaches to recurrent neural network training.Comput. Sci. Rev.3, 127-149 (2009)..

Lukoševicius, M., Jaeger, H.&Schrauwen, B. Reservoir computing trends.KI-K. ünstliche Intell.26, 365-371 (2012)..

Schrauwen, B., Verstraeten, D.&Van Campenhout, J. An overview of reservoir computing: theory, applications and implementations. InProc. 15th European Symposium on Artificial Neural Networks471-482 (Catholic University of Louvain, 2007).

Goy, A. et al.Low photon count phase retrieval using deep learning.Phys. Rev. Lett.121, 243902 (2018)..

Beck, A.&Teboulle, M. Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems.IEEE Trans. Image Process.18, 2419-2434 (2009)..

Chambolle, A. An algorithm for total variation minimization and applications.J. Math. Imaging Vis.20, 89-97 (2004)..

Beck, A.&Teboulle, M. A fast iterative shrinkage-thresholding algorithm for linear inverse problems.SIAM J. Imaging Sci.2, 183-202 (2009)..

Chollet, F. Xception: deep learning with depthwise separable convolutions. InProc. 2017 IEEE Conference on Computer Vision and Pattern Recognition(IEEE, 2017).

Gonda, F. et al. Parallel separable 3D convolution for video and volumetric data understanding. Preprint athttps://arxiv.org/abs/1809.04096https://arxiv.org/abs/1809.04096(2018).

Dey, R.&Salem, F. M. Gate-variants of gated recurrent unit (GRU) neural networks. InProc. 2017 IEEE 60th International Midwest Symposium on Circuits and Systems1597-1600 (IEEE, 2017).

Glorot, X., Bordes, A.&Bengio, Y. Deep sparse rectifier neuralnetworks. InProc. 14th International Conference on Artificial Intelligence and Statistics315-323 (Society for Artificial Intelligence and Statistics, 2011).

Sinha, A. et al. Lensless computational imaging through deep learning.Optica4, 1117-1125 (2017)..

Gehring, J. et al. A convolutional encoder model for neural machine translation. InProc. 55th Annual Meeting of the Association for Computational Linguistics(Association for Computational Linguistics, 2016).

Hori, T. et al. Advances in joint CTC-attention based end-to-end speech recognition with a deep CNN encoder and RNN-LM. InProc. Interspeech 2017949-953 (International Speech Communication Association, 2017).

Zhao, R. et al. Learning to monitor machine health with convolutional bi-directional LSTM networks.Sensors17, 273 (2017)..

He, K. M. et al. Deep residual learning for image recognition. InProc. 2016 IEEE Conference on Computer Vision and Pattern Recognition770-778 (IEEE, 2016).

Ronneberger, O., Fischer, P.&Brox, T. U-Net: convolutional networks for biomedical image segmentation. InProc. 18th International Conference on Medical Image Computing and Computer-Assisted Intervention234-241(Springer, 2015).

Vaswani, A. et al. Attention is all you need. InProc. 31st International Conference on Neural Information Processing Systems5998-6008(NIPS, 2017).

Li, S. et al. Imaging through glass diffusers using densely connected convolutional networks.Optica5, 803-813 (2018)..

Kingma, D. P.&Ba, J. Adam: a method for stochastic optimization. Preprint athttps://arxiv.org/abs/1412.6980https://arxiv.org/abs/1412.6980(2014).

Reuther, A. et al. Interactive supercomputing on 40, 000 cores for machine learning and data analysis. InProc. 2018 IEEE High Performance Extreme Computing Conference1-6 (IEEE, 2018).

Villani, C.Topics in Optimal Transportation(American Mathematical Society, 2003).

Kolouri, S. et al. Optimal mass transport: signal processing and machine-learning applications.IEEE Signal Process. Mag.34, 43-59 (2017)..

Wang, Z. et al. Image quality assessment: from error visibility to structural similarity.IEEE Trans. Image Process.13, 600-612 (2004)..

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰