Aberration-robust monocular passive depth sensing using a meta-imaging camera

Zhexuan Cao; Ning Li; Laiyu Zhu; Jiamin Wu; Qionghai Dai; Hui Qiao

doi:10.1038/s41377-024-01609-9

Your Location：

Home >

Browse articles >

Aberration-robust monocular passive depth sensing using a meta-imaging camera

Original Articles

- Aberration-robust monocular passive depth sensing using a meta-imaging camera
- Light: Science & Applications Vol. 13, Issue 11, Pages: 2556-2565(2024)
- Affiliations：
  
  1.Department of Automation, Tsinghua University, Beijing 100084, China
  2.Institute for Brain and Cognitive Sciences, Tsinghua University, Beijing 100084, China
  3.Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China
- Author bio：
  
  Qionghai Dai (qhdai@tsinghua.edu.cn)
  Hui Qiao (qiaohui@mail.tsinghua.edu.cn)
- Funds：
- DOI：10.1038/s41377-024-01609-9
  CLC：
- Received：09 April 2024，
  
  Revised：23 August 2024，
  
  Accepted：2024-08-26，
  
  Published Online：05 September 2024，
  
  Published：30 November 2024
- Accepted：
Scan QR Code
Cao, Z. X. et al. Aberration-robust monocular passive depth sensing using a meta-imaging camera. Light: Science & Applications, 13, 2556-2565 (2024).
DOI：

Cao, Z. X. et al. Aberration-robust monocular passive depth sensing using a meta-imaging camera. Light: Science & Applications, 13, 2556-2565 (2024). DOI： 10.1038/s41377-024-01609-9.

摘要

Abstract

Depth sensing plays a crucial role in various applications

including robotics

augmented reality

and autonomous driving. Monocular passive depth sensing techniques have come into their own for the cost-effectiveness and compact design

offering an alternative to the expensive and bulky active depth sensors and stereo vision systems. While the light-field camera can address the defocus ambiguity inherent in 2D cameras and achieve unambiguous depth perception

it compromises the spatial resolution and usually struggles with the effect of optical aberration. In contrast

our previously proposed meta-imaging sensor

has overcome such hurdles by reconciling the spatial-angular resolution trade-off and achieving the multi-site aberration correction for high-resolution imaging. Here

we present a compact meta-imaging camera and an analytical framework for the quantification of monocular depth sensing precision by calculating the Cramér–Rao lower bound of depth estimation. Quantitative evaluations reveal that the meta-imaging camera exhibits not only higher precision over a broader depth range than the light-field camera but also superior robustness against changes in signal-background ratio. Moreover

both the simulation and experimental results demonstrate that the meta-imaging camera maintains the capability of providing precise depth information even in the presence of aberrations. Showing the promising compatibility with other point-spread-function engineering methods

we anticipate that the meta-imaging camera may facilitate the advancement of monocular passive depth sensing in various applications.

关键词

Keywords

references

Wu, J. M. et al. An integrated imaging sensor for aberration-corrected 3D photography. Nature 612 , 62–71 (2022)..

Biswas, J. & Veloso, M. Depth camera based indoor mobile robot localization and navigation. In Proc. IEEE International Conference on Robotics and Automation Saint Paul, MN, USA 1697–1702 (IEEE, 2012).

Schmidt, T. et al. Depth-based tracking with physical constraints for robot manipulation. In Proc. IEEE International Conference on Robotics and Automation. Seattle, WA, USA 119–126 (IEEE, 201 5).

Peng, W. L. et al. IDA-3D: Instance-depth-aware 3D object detection from stereo vision for autonomous driving. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, WA, USA 13012–13021 (IEEE, 2020).

Wang, Y. et al. Pseudo-LiDAR from visual depth estimation: bridging the gap in 3D object detection for autonomous driving. In Proc. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA 8437–8445 (IEEE, 2019).

Xiong, J. H. & Wu, S. T. Planar liquid crystal polarization optics for augmented reality and virtual reality: from fundamentals to applications. ELight 1 , 1–20 (2021)..

Fan, Z. B. et al. Integral imaging near-eye 3D display using a nanoimprint metalens array. ELight 4 , 1–10 (2024)..

Park, J.-H. & Lee, B. Holographic techniques for augmented reality and virtual reality near-eye displays. Light Adv. Manuf. 3 , 137–150 (2022)..

Jiang, H. Z. et al. Self-supervised relative depth learning for urban scene understanding. In Proc. 15th European Conference on Computer Vision. Munich Springer 20–37 (ECCV, 2018)

Pavlakos, G., Zhou, X. W. & Daniilidis, K. Ordinal depth supervision for 3D human pose estimation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA 7307–7316 (IEEE, 2018).

Zhu, L., Zhang, Y. L. & Sun, H. B. Miniaturising artificial compound eyes based on advanced micronanofabrication techniques. Light Adv. Manuf. 2 , 84–100 (2021)..

Wu, Z. J. et al. Dynamic 3D shape reconstruction under complex reflection and transmission conditions using multi-scale parallel single-pixel imaging. Light Adv. Manuf. 5 , 1–12 (2024)..

McCarthy, A. et al. Long-range time-of-flight scanning sensor based on high-speed time-correlated single-photon counting. Appl. opt. 48 , 6241–6251 (2009)..

Lange, R. 3D time-of-flight distance measurement with custom solid-state image sensors in CMOS/CCD-technology. PhD thesis, University of Siegen, North Rhine-Westphalia (2000).

Scharstein, D. & Szeliski, R. High-accuracy stereo depth maps using structured light. In Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings. Madison, WI, USA (IEEE, 2003).

Furukawa, Y. & Hernández, C. Multi-View Stereo: A Tutorial Vol. 2, 1–148 (Foundations and Trends ® in Computer Graphics and Vision, 2015)..

Özyeşil, O. et al. A survey of structure from motion. Acta Numerica 26 , 305–364 (2017)..

Aguilar, J.-J., Torres, F. & Lope, M. A. Stereo vision for 3D measurement: accuracy analysis, calibration and industrial applications. Measurement 18 , 193–200 (1996)..

Eigen, D., Puhrsch, C. & Fergus, R.Depth map prediction from a single image using a multi-scale deep network. Proc. 27th International Conference on Neural Information Processing Systems (pp. 2366–2374. MIT Press, 2014).

Fu, H. et al. Deep ordinal regression network for monocular depth estimation. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA 2002–2011 (IEEE, 2018).

Chang, J. L. & Wetzstein, G. Deep optics for monocular depth estimation and 3D object detection. In Proc. IEEE/CVF International Conference on Computer Vision. Seoul, Korea (South) 10192–10201 (IEEE, 2019).

Ikoma, H. et al. Depth from defocus with learned optics for imaging and occlusion-aware depth estimation. In Proc. IEEE International Conference on Computational Photography. Haifa, Israel 1–12 (IEEE, 2021).

Carvalho, M et al. Deep depth from defocus: how can defocus blur improve 3D estimation using dense neural networks? Proc. European Conference on Computer Vision (ECCV) Workshops 307–323 (Springer, 2018).

Levin, A. et al. Image and depth from a conventional camera with a coded aperture. ACM Trans. Graph. 26 , 70–es (2007)..

Ng, R. Digital light field photography. PhD thesis, Stanford University (2006).

Kay, S. M. Fundamentals of Statistical Signal Processing: Estimation Theory (Prentice-Hall, 1993).

Hahne, C. et al. Baseline and triangulation geometry in a standard plenoptic camera. Int. J. Comput. Vis. 126 , 21–35 (2018)..

Chao, J., Ward, E. S. & Ober, R. J. Fisher information theory for parameter estimation in single molecule microscopy: tutorial. J. Opt. Soc. Am. A 33 , B36–B57 (2016)..

Li, Y. M. et al. Real-time 3D single-molecule localization using experimental point spread functions. Nat. Methods 15 , 367–369 (2018)..

Aristov, A. et al. ZOLA-3D allows flexible 3D localization microscopy over an adjustable axial range. Nat. Commun. 9 , 2409 (2018)..

Yang, F. et al. Monocular depth sensing using metalens. Nanophotonics 12 , 2987–2996 (2023)..

Shen, Z. C. et al. Monocular metasurface camera for passive single-shot 4D imaging. Nat. Commun. 14 , 1035 (2023)..

Opatovski, N. et al. Monocular kilometer-scale passive ranging by point-spread function engineering. Opt. Express 30 , 37925–37937 (2022)..

Prasad, S. Rotating point spread functionvia pupil-phase engineering. Opt. Lett. 38 , 585–587 (2013)..

Berlich, R. & Stallinga, S. High-order-helix point spread functions for monocular three-dimensional imaging with superior aberration robustness. Opt. Express 26 , 4873–4891 (2018)..

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰