Single-shot super-resolved fringe projection profilometry (SSSR-FPP): 100,000 frames-per-second 3D imaging with deep learning

Bowen Wang; Wenwu Chen; Jiaming Qian; Shijie Feng; Qian Chen; Chao Zuo

doi:10.1038/s41377-024-01721-w

Go to Nature Website

Your Location：

Home >

Browse articles >

Single-shot super-resolved fringe projection profilometry (SSSR-FPP): 100,000 frames-per-second 3D imaging with deep learning

Original Articles

- Single-shot super-resolved fringe projection profilometry (SSSR-FPP): 100,000 frames-per-second 3D imaging with deep learning
- Light: Science & Applications Vol. 14, Issue 2, Pages: 551-563(2025)
- Affiliations：
  
  1.Smart Computational Imaging Laboratory (SCILab), Nanjing University of Science and Technology, Nanjing, Jiangsu Province, China
  2.Jiangsu Key Laboratory of Spectral Imaging & Intelligent Sense, Nanjing University of Science and Technology, Nanjing, Jiangsu Province, China
- Author bio：
  
  Shijie Feng (shijiefeng@njust.edu.cn)
  Qian Chen (chenqian@njust.edu.cn)
  Chao Zuo (zuochao@njust.edu.cn)
- Funds：
- DOI：10.1038/s41377-024-01721-w
  CLC：
- Received：05 September 2024，
  
  Revised：09 December 2024，
  
  Accepted：2024-12-13，
  
  Published Online：07 February 2025，
  
  Published：28 February 2025
- Accepted：
Scan QR Code
Wang, B W. et al. Single-shot super-resolved fringe projection profilometry (SSSR-FPP): 100,000 frames-per-second 3D imaging with deep learning. Light: Science & Applications, 14, 551-563 (2025).
DOI：

Wang, B W. et al. Single-shot super-resolved fringe projection profilometry (SSSR-FPP): 100,000 frames-per-second 3D imaging with deep learning. Light: Science & Applications, 14, 551-563 (2025). DOI： 10.1038/s41377-024-01721-w.

摘要

Abstract

To reveal the fundamental aspects hidden behind a variety of transient events in mechanics

physics

and biology

the highly desired ability to acquire three-dimensional (3D) images with ultrafast temporal resolution has been long sought. As one of the most commonly employed 3D sensing techniques

fringe projection profilometry (FPP) reconstructs the depth of a scene from stereo images taken with sequentially structured illuminations. However

the imaging speed of current FPP methods is generally capped at several kHz

which is limited by the projector-camera hardware and the number of fringe patterns required for phase retrieval and unwrapping. Here we report a novel learning-based ultrafast 3D imaging technique

termed single-shot super-resolved FPP (SSSR-FPP)

which enables ultrafast 3D imaging at 100

000 Hz. SSSR-FPP uses only one pair of low signal-to-noise ratio (SNR)

low-resolution

and pixelated fringe patterns as input

while the high-resolution unwrapped phase and fringe orders can be deciphered with a specific trained deep neural network. Our approach exploits the significant speed gain achieved by reducing the imaging window of conventional high-speed cameras

while “regenerating” the lost spatial resolution through deep learning. To demonstrate the high spatio-temporal resolution of SSSR-FPP

we present 3D videography of several transient scenes

including rotating turbofan blades

exploding building blocks

and the reciprocating motion of a steam engine

etc.

which were previously challenging or even impossible to capture with conventional methods. Experimental results establish SSSR-FPP as a significant step forward in the field of 3D optical sensing

offering new insights into a broad spectrum of dynamic processes across various scientific disciplines.

关键词

Keywords

references

Beurg, M., Fettiplace, R., Nam, J.-H. & Ricci, A. J. Localization of inner hair cell mechanotransducer channels using high-speed calcium imaging. Nat. Neurosci. 12 , 553–558 (2009)..

Hockett, P., Bisgaard, C. Z., Clarkin, O. J. & Stolow, A. Time-resolved imaging of purely valence-electron dynamics during a chemical reaction. Nat. Phys. 7 , 612–615 (2011)..

Wang, H. D. et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy. Nat. Methods 16 , 103–110 (2019)..

Chen, Z. G. & Segev, M. Highlighting photonics: looking into the next decade. eLight 1 , 2 (2021)..

Kleinfelder, S., Lim, S.-I., Liu, X. & El Gamal, A. A 10 kframe/s 0.18/spl mu/m CMOS digital pixel sensor with pixel-level memory. 2001 IEEE International Solid-State Circuits Conference. Digest of Technical Papers. San Francisco, CA, USA: IEEE, 2001 88–89.

El-Desouki, M. et al. CMOS image sensors for high speed applications. Sensors 9 , 430–444 (2009)..

Salvi, J., Fernandez, S., Pribanic, T. & Llado, X. A state of the art in structured light patterns for surface profilometry. Pattern Recognit. 43 , 2666–2680 (2010)..

Geng, J. Structured-light 3D surface imaging: a tutorial. Adv. Opt. Photonics 3 , 128–160 (2011)..

Gorthi, S. S. & Rastogi, P. Fringe projection techniques: whither we are? Opt. Lasers Eng. 48 , 133–140 (2010)..

Boyer, K. L. & Kak, A. C. Color-encoded structured light for rapid active ranging. IEEE Trans. Pattern Anal. Mach. Intell. , PAMI-9 , 14–28 (1987)..

Zhang, L., Curless, B. & Seitz, S. M. Rapid shape acquisition using color structured light and multi-pass dynamic programming. Rapid shape acquisition using color structured light and multi-pass dynamic programming. First International Symposium on 3D Data Processing Visualization and Transmission. Padova, Italy: IEEE, 2002 24–36.

Scharstein, D. & Szeliski, R. High-accuracy stereo depth maps using structured light. IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Madison, WI, USA: IEEE, 2003 .

Zuo, C. et al. Micro Fourier Transform Profilometry ( μ FTP): 3D shape measurement at 10,000 frames per second. Opt. Lasers Eng. 102 , 70–91 (2018)..

Lei, S. Y. & Zhang, S. Flexible 3-d shape measurement using projector defocusing. Opt. Lett. 34 , 3080–3082 (2009)..

Zuo, C. et al. Optimized pulse width modulation pattern strategy for three-dimensional profilometry with projector defocusing. Appl. Opt. 51 , 4477–4490 (2012)..

Heist, S. et al. High-speed three-dimensional shape measurement using gobo projection. Opt. Lasers Eng. 87 , 90–96 (2016)..

Zhang, S., Van Der Weide, D. & Oliver, J. Superfast phase-shifting method for 3-d shape measurement. Opt. Express 18 , 9684–9689 (2010)..

Gong, Y. Z. & Zhang, S. Ultrafast 3-D shape measurement with an off-the-shelf DLP projector. Opt. Express 18 , 19743–19754 (2010)..

Liu, K., Wang, Y., Lau, D. L., Hao, Q. & Hassebrook, L. G. Dual-frequency pattern scheme for high-speed 3-d shape measurement. Opt. Express 18 , 5229–5244 (2010)..

Zuo, C. et al. High-speed three-dimensional shape measurement for dynamic scenes using bi-frequency tripolar pulse-width-modulation fringe projection. Opt. Lasers Eng. 51 , 953–960 (2013)..

Zuo, C., Chen, Q., Gu, G., Feng, S. & Feng, F. High-speed three-dimensional profilometry for multiple objects with complex shapes. Opt. Express 20 , 19493–19510 (2012)..

An, Y. T., Hyun, J. S. & Zhang, S. Pixel-wise absolute phase unwrapping using geometric constraints of structured light system. Opt. Express 24 , 18445–18459 (2016)..

Feng, S. J., Chen, Q. & Zuo, C. Graphics processing unit–assisted real-time three-dimensional measurement using speckle-embedded fringe. Appl. Opt. 54 , 6865–6873 (2015)..

LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521 , 436–444 (2015)..

Schmidhuber, J. Deep learning in neural networks: An overview. Neural Netw. 61 , 85–117 (2015)..

Saguy, A. et al. Dblink: Dynamic localization microscopy in super spatiotemporal resolution via deep l earning. Nat. Methods 20 , 1939–1948 (2023)..

Wang, D. et al. Liquid lens based holographic camera for real 3d scene hologram acquisition using end-to-end physical model-driven network. Light Sci. Appl. 13 , 62 (2024)..

Liu, H. S. et al. Learning-based real-time imaging through dynamic scattering media. Light Sci. Appl. 13 , 194 (2024)..

Seong, B. et al. E2e-bpf microscope: extended depth-of-field microscopy using learning-based implementation of binary phase filter and image deconvolution. Light Sci. Appl. 12 , 269 (2023)..

Wang, F. et al. Phase imaging with an untrained neural network. Light Sci. Appl. 9 , 77 (2020)..

Zuo, C. et al. Deep learning in optical metrology: a review. LightSci. Appl. 11 , 39 (2022)..

Feng, S. J. et al. Fringe pattern analysis using deep learning. Adv. Photonics 1 , 025001 (2019)..

Wang, K. Q., Dou, J., Kemao, Q., Di, J. & Zhao, J. Y-net: a one-to-two deep learning framework for digital holographic reconstruction. Opt. Lett. 44 , 4765–4768 (2019)..

Yang, T., Zhang, Z., Li, H., Li, X. & Zhou, X. Single-shot phase extraction for fringe projection profilometry using deep convolutional generative adversarial network. Meas. Sci. Technol. 32 , 015007 (2020)..

Yan, K. T. et al. Fringe pattern denoising based on deep learning. Opt. Commun. 437 , 148–152 (2019)..

Lin, B. W., Fu, S., Zhang, C., Wang, F. & Li, Y. Optical fringe patterns filtering based on multi-stage convolution neural network. Opt. Lasers Eng. 126 , 105853 (2020)..

Reyes-Figueroa, A., Flores, V. H. & Rivera, M. Deep neural network for fringe pattern filtering and normalization. Appl. Opt. 60 , 2022–2036 (2021)..

Pritt, M. D. & Shipman, J. S. Least-squares two-dimensional phase unwrapping using FFT’s. IEEE Trans. Geosci. Remote Sens. 32 , 706–708 (1994)..

Wang, K. Q., Li, Y., Kemao, Q., Di, J. & Zhao, J. One-step robust deep learning phase unwrapping. Opt. Express 27 , 15100–15115 (2019)..

Qian, J. M. et al. Deep-learning-enabled geometric constraints and phase unwrapping for single-shot absolute 3d shape measurement. APL Photonics 5 , 046105 (2020)..

Spoorthi, G. E., Gorthi, R. K. S. S. & Gorthi, S. PhaseNet 2.0: phase unwrapping of noisy data based on deep learning approach. IEEE Trans. Image Process. 29 , 4862–4872 (2020)..

Feng, S. J., Zuo, C., Yin, W., Gu, G. & Chen, Q. Micro deep learning profilometry for high-speed 3d surface imaging. Opt. Lasers Eng. 121 , 416–427 (2019)..

Rivenson, Y. et al. Deep learning microscopy. Optica 4 , 1437–1443 (2017)..

Feng, S. J., Zuo, C., Hu, Y., Li, Y. & Chen, Q. Deep-learning-based fringe-pattern analysis with uncertainty estimation. Optica 8 , 1507–1510 (2021)..

Chen, W. W. et al. Deep-learning-enabled temporally super-resolved multiplexed fringe projection profilometry: high-speed khz 3d imaging with low-speed camera. PhotoniX 5 , 25 (2024)..

Yao, Y., Luo, Z., Li, S., Fang, T. & Quan, L. MVSNet: depth inference for unstructured multi-view stereo. Proceedings of the 15th European Conference on Computer Vision (ECCV). Munich, Germany: Springer, 2018 785–801.

Mayya, V., Pai, R. M. & Pai, M. M. Automatic facial expression recognition using dcnn. Procedia Comput. Sci. 93 , 453–461 (2016)..

Schönberger, J.L. & Frahm, J.M. Structure-from-motion revisited. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA: IEEE, 2016 4104–4113.

Durrant-Whyte, H. & Bailey, T. Simultaneous localization and mapping: part I. IEEE Robot. Autom. Mag. 13 , 99–110 (2006)..

Seitz, S. M., Curless, B., Diebel, J., Scharstein, D. & Szeliski, R. A comparison and evaluation of multi-view stereo reconstruction algorithms. 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06). New York, NY, USA: IEEE, 2006 519–528.

Furukawa, Y. & Hernández, C. Multi-view stereo: a tutorial. Found. Trends® Comput. Graph. Vis. 9 , 1–148 (2015)..

Zheng, Y., Wang, S., Li, Q. & Li, B. Fringe projection profilometry by conducting deep learning from its digital twin. Opt. Express 28 , 36568–36583 (2020)..

Wang, F. Z., Wang, C. X. & Guan, Q. Z. Single-shot fringe projection profilometry based on deep learning and computer graphics. Opt. Express 29 , 8024–8040 (2021)..

Feng, S. J. et al. General solution for high dynamic range three-dimensional shape measurement using the fringe projection technique. Opt. Lasers Eng. 59 , 56–71 (2014)..

Chen, H. G. et al. Real-world single image super-resolution: A brief review. Inf. Fusion 79 , 124–145 (2022)..

Ji, X. Z. et al. Real-world super-resolution via kernel estimation and noise injection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Seattle, WA, USA: IEEE, 2020 1914–1923.

Lv, S. Z. & Kemao, Q. Modeling the measurement precision of fringe projection profilometry. Light Sci. Appl. 12 , 257 (2023)..

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰