Non-line-of-sight reconstruction with signal–object collaborative regularization

Xintong Liu; Jianyu Wang; Zhupeng Li; Zuoqiang Shi; Xing Fu; Lingyun Qiu

doi:10.1038/s41377-021-00633-3

Go to Nature Website

Your Location：

Home >

Browse articles >

Non-line-of-sight reconstruction with signal–object collaborative regularization

Articles

- Non-line-of-sight reconstruction with signal–object collaborative regularization
- Non-line-of-sight reconstruction with signal–object collaborative regularization
- LSA 2021年10卷第10期页码：1967-1986
- Affiliations：
  
  1.Yau Mathematical Sciences Center, Tsinghua University, 100084, Beijing, China
  2.State Key Laboratory of Precision Measurement Technology and Instruments, Department of Precision Instrument, Tsinghua University, 100084, Beijing, China
  3.Key Laboratory of Photonic Control Technology (Tsinghua University), Ministry of Education, 100084, Beijing, China
  4.Department of Mathematical Sciences, Tsinghua University, 100084, Beijing, China
  5.Yanqi Lake Beijing Institute of Mathematical Sciences and Applications, 101408, Beijing, China
- Author bio：
  
  Xing Fu (fuxing@tsinghua.edu.cn)
  Lingyun Qiu (lyqiu@tsinghua.edu.cn)
- Funds：
- DOI：10.1038/s41377-021-00633-3
  中图分类号：
- 纸质出版日期：2021-10-31，
  
  网络出版日期：2021-09-24，
  
  收稿日期：2021-05-25，
  
  修回日期：2021-08-18，
  
  录用日期：2021-09-03
- Accepted：
Scan QR Code
Non-line-of-sight reconstruction with signal–object collaborative regularization[J]. LSA, 2021,10(10):1967-1986.

Liu, X. T. et al. Non-line-of-sight reconstruction with signal–object collaborative regularization. Light: Science & Applications, 10, 1967-1986 (2021).
Non-line-of-sight reconstruction with signal–object collaborative regularization[J]. LSA, 2021,10(10):1967-1986. DOI： 10.1038/s41377-021-00633-3.

Liu, X. T. et al. Non-line-of-sight reconstruction with signal–object collaborative regularization. Light: Science & Applications, 10, 1967-1986 (2021). DOI： 10.1038/s41377-021-00633-3.

摘要

Abstract

Non-line-of-sight imaging aims at recovering obscured objects from multiple scattered lights. It has recently received widespread attention due to its potential applications

such as autonomous driving

rescue operations

and remote sensing. However

in cases with high measurement noise

obtaining high-quality reconstructions remains a challenging task. In this work

we establish a unified regularization framework

which can be tailored for different scenarios

including indoor and outdoor scenes with substantial background noise under both confocal and non-confocal settings. The proposed regularization framework incorporates sparseness and non-local self-similarity of the hidden objects as well as the smoothness of the signals. We show that the estimated signals

albedo

and surface normal of the hidden objects can be reconstructed robustly even with high measurement noise under the proposed framework. Reconstruction results on synthetic and experimental data show that our approach recovers the hidden objects faithfully and outperforms state-of-the-art reconstruction algorithms in terms of both quantitative criteria and visual quality.

关键词

Keywords

references

Liu, X. C., Bauer, S.&Velten, A. Analysis of feature visibility in non-line-of-sight measurements. InProc. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), 10132–10140 (IEEE, 2019).

Velten, A. et al. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging.Nat. Commun.3, 745 (2012)..

O'Toole, M., Lindell, D. B.&Wetzstein, G. Confocal non-line-of-sight imaging based on the light-cone transform.Nature555, 338–341 (2018)..

Tsai, C. Y. et al. The geometry of first-returning photons for non-line-of-sight imaging. InProc. 2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR), 2336–2344 (IEEE, 2017).

Xin, S. M. et al. A theory of fermat paths for non-line-of-sight shape reconstruction. InProc. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR) (IEEE, 2019).

Lindell, D. B., Wetzstein, G.&O'Toole, M. Wave-based non-line-of-sight imaging using fastf-Kmigration.ACM Trans. Graph.38, 116 (2019)..

Liu, X. C. et al. Non-line-of-sight imaging using phasor-field virtual wave optics.Nature572, 620–623 (2019)..

Liu, X. C., Bauer, S.&Velten, A. Phasor field diffraction based reconstruction for fast non-line-of-sight imaging systems.Nat. Commun.11, 1645 (2020)..

Liu, X. C.&Velten, A. The role of wigner distribution function in non-line-of-sight imaging. InProc. 2020 IEEE International Conference on Computational Photography(ICCP) (IEEE, 2020).

Tancik, M., Satat, G.&Raskar, R. Flash photography for data-driven hidden scene recovery. Preprint at arXiv: 1810.11710 (2018).https://www.media.mit.edu/publications/flash-photography-for-data-driven-hidden-scene-recovery/https://www.media.mit.edu/publications/flash-photography-for-data-driven-hidden-scene-recovery/.

Chen, W. Z. et al. Steady-state non-line-of-sight imaging. InProc. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), 6783–6792 (IEEE, 2019).

Chen, W. Z. et al. Learned feature embeddings for non-line-of-sight imaging and recognition.ACM Trans. Graph.39, 230 (2020)..

Chopite, J. G. etal. Deep non-line-of-sight reconstruction. InProc. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR) (IEEE, 2020).

Metzler, C. A. et al. Deep-inverse correlography: towards real-time high-resolution non-line-of-sight imaging.Optica7, 63–71 (2020)..

Buttafava, M. et al. Non-line-of-sight imaging using a time-gated single photon avalanche diode.Opt. Express23, 20997–21011 (2015)..

Saunders, C., Murray-Bruce, J.&Goyal, V. K. Computational periscopy with an ordinary digital camera.Nature565, 472–475 (2019)..

Pediredla, A., Dave, A.&Veeraraghavan, A. SNLOS: non-line-of-sight scanning through temporal focusing. InProc. 2019 IEEE International Conference on Computational Photography(ICCP) (IEEE, 2019).

La Manna, M. et al. Non-line-of-sight-imaging using dynamic relay surfaces.Opt. Express28, 5331–5339 (2020)..

Tanaka, K., Mukaigawa, Y.&Kadambi, A. Polarized non-line-of-sight imaging. InProc. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), 2133–2142 (IEEE, 2020).

Metzler, C. A., Lindell, D. B.&Wetzstein, G. Keyhole imaging: non-line-of-sight imaging and tracking of moving objects along a single optical path.IEEE Trans. Comput. Imaging7, 1–12 (2020)..

Wu, C. et al. Non–line-of-sight imaging over 1.43 km.Proc. Natl Acad. Sci. USA118, e2024468118 (2021)..

Ye, J. T. et al. Compressed sensing for active non-line-of-sight imaging.Opt. Express29, 1749–1763 (2021)..

Arellano, V., Gutierrez, D.&Jarabo, A. Fast back-projection for non-line of sight reconstruction. InProc. SIGGRAPH '17: Special Interest Group on Computer Graphics and Interactive Techniques Conference(SIGGRAPH, 2017).

Young, S. I. et al. Non-line-of-sight surface reconstruction using the directional light-cone transform. InProc. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), 1404–1413 (IEEE, 2020).

Heide, F. et al. Non-line-of-sight imaging with partialoccluders and surface normals.ACM Trans. Graph.38, 22 (2019)..

Yilmaz, Ö.Seismic Data Analysis: Processing, Inversion, and Interpretation of Seismic Data(Society of Exploration Geophysicists, 2001).

Laurenzis, M.&Velten, A. Feature selection and back-projection algorithms for nonline-of-sight laser-gated viewing.J. Electron. Imaging23, 063003 (2014)..

Feng, X. H.&Gao, L. Improving non-line-of-sight image reconstruction with weighting factors.Opt. Lett.45, 3921–3924 (2020)..

Thrampoulidis, C. et al. Exploiting occlusion in non-line-of-sight active imaging.IEEE Trans. Comput. Imaging4, 419–431 (2018)..

Ahn, B. et al. Convolutional approximations to the general non-line-of-sight imaging operator. InProc. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), 7888–7898 (IEEE, 2019).

Tsai, C. Y., Sankaranarayanan, A. C.&Gkioulekas, I. Beyond volumetric albedo—a surface optimization framework for non-line-of-sight imaging. InProc. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), 1545–1555 (IEEE, 2019).

Iseringhausen, J.&Hullin, M. B. Non-line-of-sight reconstruction using efficient transient rendering.ACM Trans. Graph.39, 8 (2020)..

Lebrun, M. An analysis and implementation of the BM3D image denoising method.Image Process. Line2, 175–213 (2012)..

Cai, J. F. et al. Data-driven tight frame construction and image denoising.Appl. Comput. Harmon. Anal.37, 89–105 (2014)..

Dabov, K. et al. Image denoising with block-matching and 3D filtering. InProc. SPIE 6064, Image Processing: Algorithms and Systems, Neural Networks, and Machine Learning(SPIE, 2006).

Galindo, M. et al. A dataset for benchmarking time-resolved non-line-of-sight imaging. InProc. SIGGRAPH '19: Special Interest Group on Computer Graphics and Interactive Techniques Conference(SIGGRAPH, 2019).

Veach, E.&Guibas, L. J. Optimally combining sampling techniques for Monte Carlo rendering. InProc. 22nd Annual Conference on Computer Graphics and Interactive Techniques419–428 (ACM Press, 1995).

Klein, J. et al. A quantitative platform for non-line-of-sight imaging problems. InProc. British Machine Vision Conference(BMV C, 2018).

Lebrun, M., Buades, A.&Morel, J. M. Implementation of the "Non-Local Bayes" (NL-Bayes) image denoising algorithm.Image Process. Line3, 1–42 (2013)..

Herlihy, M. et al.The Art of Multiprocessor Programming2nd edn (Newnes, 2020).

Goldstein, T.&Osher, S. The split bregman method for L1-regularized problems.SIAM J. Imaging Sci.2, 323–343 (2009)..

浏览量

Downloads

CSCD

文章被引用时，请邮件提醒。

Submit

工具集

关联资源

暂无数据