End-to-end learning of 3D phase-only holograms for holographic display

Liang Shi; Beichen Li; Wojciech Matusik

doi:10.1038/s41377-022-00894-6

Go to Nature Website

Your Location：

Home >

Browse articles >

End-to-end learning of 3D phase-only holograms for holographic display

Original Articles

- End-to-end learning of 3D phase-only holograms for holographic display
- Light: Science & Applications Vol. 11, Issue 9, Pages: 2176-2193(2022)
- Affiliations：
  
  1.Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, 32 Vassar St, Cambridge, MA 02139, USA
- Author bio：
  
  Liang Shi (liangs@mit.edu)
  Wojciech Matusik (wojciech@mit.edu)
- Funds：
- DOI：10.1038/s41377-022-00894-6
  CLC：
Scan for full text
Shi, L. et al. End-to-end learning of 3D phase-only holograms for holographic display. Light: Science & Applications, 11, 2176-2193 (2022).
DOI：

Shi, L. et al. End-to-end learning of 3D phase-only holograms for holographic display. Light: Science & Applications, 11, 2176-2193 (2022). DOI： 10.1038/s41377-022-00894-6.

摘要

Abstract

Computer-generated holography (CGH) provides volumetric control of coherent wavefront and is fundamental to applications such as volumetric 3D displays, lithography, neural photostimulation, and optical/acoustic trapping. Recently, deep learning-based methods emerged as promising computational paradigms for CGH synthesis that overcome the quality-runtime tradeoff in conventional simulation/optimization-based methods. Yet, the quality of the predicted hologram is intrinsically bounded by the dataset's quality. Here we introduce a new hologram dataset, MIT-CGH-4K-V2, that uses a layered depth image as a data-efficient volumetric 3D input and a two-stage supervised+unsupervised training protocol for direct synthesis of high-quality 3D phase-only holograms. The proposed system also corrects vision aberration, allowing customization for end-users. We experimentally show photorealistic 3D holographic projections and discuss relevant spatial light modulator calibration procedures. Our method runs in real-time on a consumer GPU and 5 FPS on an iPhone 13 Pro, promising drastically enhanced performance for the applications above.

关键词

Keywords

references

Benton, S. A. & Bove, V. M. Jr. Holographic Imaging (Wiley-Interscience, 2008).

N. C. Pégard , 等 . Three-dimensional scanless holographic optogenetics with temporal focusing (3D-SHOT) . Nat. Commun. , 2017 . 8 1228 DOI:10.1038/s41467-017-01031-3http://doi.org/10.1038/s41467-017-01031-3 .

M. H. Eybposh , 等 . DeepCGH: 3D computer-generated holography using deep learning . Opt. Express , 2020 . 28 26636 -26650 . DOI:10.1364/OE.399624http://doi.org/10.1364/OE.399624 .

M. Reicherter , 等 . Optical particle trapping with computer-generated holograms written on a liquid-crystal display . Opt. Lett. , 1999 . 24 608 -610 . DOI:10.1364/OL.24.000608http://doi.org/10.1364/OL.24.000608 .

K. Melde , 等 . Holograms for acoustics . Nature , 2016 . 537 518 -522 . DOI:10.1038/nature19755http://doi.org/10.1038/nature19755 .

L. Shi , 等 . Towards real-time photorealistic 3D holography with deep neural networks . Nature , 2021 . 591 234 -239 . DOI:10.1038/s41586-020-03152-0http://doi.org/10.1038/s41586-020-03152-0 .

J. An , 等 . Slim-panel holographic video display . Nat. Commun. , 2020 . 11 5568 DOI:10.1038/s41467-020-19298-4http://doi.org/10.1038/s41467-020-19298-4 .

Z. C. Zhang , Z. You , D. P. Chu . Fundamentals of phase-only liquid crystal on silicon (LCOS) devices . Light Sci. Appl. , 2014 . 3 e213 DOI:10.1038/lsa.2014.94http://doi.org/10.1038/lsa.2014.94 .

J. P. Waters . Holographic image synthesis utilizing theoretical methods . Appl. Phys. Lett. , 1966 . 9 405 -407 . DOI:10.1063/1.1754630http://doi.org/10.1063/1.1754630 .

S. Hamann , 等 . Time-multiplexed light field synthesis via factored Wigner distribution function . Opt. Lett. , 2018 . 43 599 -602 . DOI:10.1364/OL.43.000599http://doi.org/10.1364/OL.43.000599 .

L. Shi , 等 . Near-eye light field holographic rendering with spherical waves for wide field of view interactive 3D computer graphics . ACM Trans. Graph. , 2017 . 36 236 DOI:10.1145/3130800.3130832http://doi.org/10.1145/3130800.3130832 .

K. Matsushima , S. Nakahara . Extremely high-definition full-parallax computer-generated hologram created by the polygon-based method . Appl. Opt. , 2009 . 48 H54 -H63 . DOI:10.1364/AO.48.000H54http://doi.org/10.1364/AO.48.000H54 .

K. Matsushima . Computer-generated holograms for three-dimensional surface objects with shade and texture . Appl. Opt. , 2005 . 44 4607 -4614 . DOI:10.1364/AO.44.004607http://doi.org/10.1364/AO.44.004607 .

A. Maimone , A. Georgiou , J. S. Kollin . Holographic near-eye displays for virtual and augmented reality . ACM Trans. Graph. , 2017 . 36 85 DOI:10.1145/3072959.3073624http://doi.org/10.1145/3072959.3073624 .

H. Zhang , L. C. Cao , G. F. Jin . Computer-generated hologram with occlusion effect using layer-based processing . Appl. Opt. , 2017 . 56 F138 -F143 . DOI:10.1364/AO.56.00F138http://doi.org/10.1364/AO.56.00F138 .

K. Matsushima , T. Shimobaba . Band-limited angular spectrum method for numerical simulation of free-space propagation in far and near fields . Opt. Express , 2009 . 17 19662 -19673 . DOI:10.1364/OE.17.019662http://doi.org/10.1364/OE.17.019662 .

C. K. Hsueh , A. A. Sawchuk . Computer-generated double-phase holograms . Appl. Opt. , 1978 . 17 3874 -3883 . DOI:10.1364/AO.17.003874http://doi.org/10.1364/AO.17.003874 .

X. Sui , 等 . Band-limited double-phase method for enhancing image sharpness in complex modulated computer-generated holograms . Opt. Express , 2021 . 29 2597 -2612 . DOI:10.1364/OE.414299http://doi.org/10.1364/OE.414299 .

Y. Shechtman , 等 . Phase retrieval with application to optical imaging: A contemporary overview . IEEE Signal Process. Mag. , 2015 . 32 87 -109 . DOI:10.1109/MSP.2014.2352673http://doi.org/10.1109/MSP.2014.2352673 .

J. Zhang , 等 . 3D computer-generated holography by non-convex optimization . Optica , 2017 . 4 1306 -1313 . DOI:10.1364/OPTICA.4.001306http://doi.org/10.1364/OPTICA.4.001306 .

P. Chakravarthula , 等 . Wirtinger holography for near-eye displays . ACM Trans. Graph. , 2019 . 38 213 DOI:10.1145/3355089.3356539http://doi.org/10.1145/3355089.3356539 .

G. Kuo , 等 . High resolution étendue expansion for holographic displays . ACM Trans. Graph. , 2020 . 39 66 DOI:10.1145/3386569.3392414http://doi.org/10.1145/3386569.3392414 .

P. Chakravarthula , 等 . Learned hardware-in-the-loop phase retrieval for holographic near-eye displays . ACM Trans. Graph. , 2020 . 39 186 DOI:10.1145/3414685.3417846http://doi.org/10.1145/3414685.3417846 .

Y. F. Peng , 等 . Neural holography with camera-in-the-loop training . ACM Trans. Graph. , 2020 . 39 185 DOI:10.1145/3414685.3417802http://doi.org/10.1145/3414685.3417802 .

S. Choi , 等 . Optimizing image quality for holographic near-eye displays with Michelson Holography . Optica , 2021 . 8 143 -146 . DOI:10.1364/OPTICA.410622http://doi.org/10.1364/OPTICA.410622 .

K. Kavaklı , H. Urey , K. Akşit . Learned holographic light transport: Invited . Appl. Opt. , 2022 . 61 DOI:10.1364/AO.439401http://doi.org/10.1364/AO.439401 .

S. Choi , 等 . Neural 3D holography: learning accurate wave propagation models for 3D holographic virtual and augmented reality displays . ACM Trans. Graph. , 2021 . 40 240 DOI:10.1145/3478513.3480542http://doi.org/10.1145/3478513.3480542 .

R. Horisaki , R. Takagi , J. Tanida . Deep-learning-generated holography . Appl. Opt. , 2018 . 57 3859 -3863 . DOI:10.1364/AO.57.003859http://doi.org/10.1364/AO.57.003859 .

R. Horisaki , 等 . Three-dimensional deeply generated holography [Invited] . Appl. Opt. , 2021 . 60 A323 -A328 . DOI:10.1364/AO.404151http://doi.org/10.1364/AO.404151 .

Shade, J. et al. Layered depth images. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques 231–242 (ACM, New York, 1998).

D. Hiyama , 等 . Acceleration of color computer-generated hologram from RGB–D images using color space conversion . Opt. Commun. , 2015 . 340 121 -125 . DOI:10.1016/j.optcom.2014.11.099http://doi.org/10.1016/j.optcom.2014.11.099 .

N. Padmanaban , Y. F. Peng , G. Wetzstein . Holographic near-eye displays based on overlap-add stereograms . ACM Trans. Graph. , 2019 . 38 214 DOI:10.1145/3355089.3356517http://doi.org/10.1145/3355089.3356517 .

T. Ichikawa , K. Yamaguchi , Y. Sakamoto . Realistic expression for full-parallax computer-generated holograms with the ray-tracing method . Appl. Opt. , 2013 . 52 A201 -A209 . DOI:10.1364/AO.52.00A201http://doi.org/10.1364/AO.52.00A201 .

K. Matsushima , M. Nakamura , S. Nakahara . Silhouette method for hidden surface removal in computer holography and its acceleration using the switch-back technique . Opt. Express , 2014 . 22 24450 -24465 . DOI:10.1364/OE.22.024450http://doi.org/10.1364/OE.22.024450 .

H. Zhang , 等 . Fully computed holographic stereogram based algorithm for computer-generated holograms with accurate depth cues . Opt. Express , 2015 . 23 3901 -3913 . DOI:10.1364/OE.23.003901http://doi.org/10.1364/OE.23.003901 .

Z. Wang , 等 . Image quality assessment: From error visibility to structural similarity . IEEE Trans. Image Process. , 2004 . 13 600 -612 . DOI:10.1109/TIP.2003.819861http://doi.org/10.1109/TIP.2003.819861 .

C. L. Chang , W. Cui , L. Gao . Foveated holographic near-eye 3D display . Opt. Express , 2020 . 28 1345 -1356 . DOI:10.1364/OE.384421http://doi.org/10.1364/OE.384421 .

A. Cem , 等 . Foveated near-eye display using computational holography . Sci. Rep. , 2020 . 10 14905 DOI:10.1038/s41598-020-71986-9http://doi.org/10.1038/s41598-020-71986-9 .

P. Chakravarthula , 等 . Gaze-contingent retinal speckle suppression for perceptually-matched foveated holographic displays . IEEE Trans. Vis. Computer Graph. , 2021 . 27 4194 -4203 . DOI:10.1109/TVCG.2021.3106433http://doi.org/10.1109/TVCG.2021.3106433 .

Walton, D. R. et al. Metameric varifocal holograms. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) 746–755 (IEEE, Christchurch, 2022).

B. Guenter , 等 . Foveated 3D graphics . ACM Trans. Graph. , 2012 . 31 164 DOI:10.1145/2366145.2366183http://doi.org/10.1145/2366145.2366183 .

A. Patney , 等 . Towards foveated rendering for gaze-tracked virtual reality . ACM Trans. Graph. , 2016 . 35 179 DOI:10.1145/2980179.2980246http://doi.org/10.1145/2980179.2980246 .

Q. Sun , 等 . Perceptually-guided foveation for light field displays . ACM Trans. Graph. , 2017 . 36 192 DOI:10.1145/3130800.3130807http://doi.org/10.1145/3130800.3130807 .

Mildenhall, B. et al. NeRF: representing scenes as neural radiance fields for view synthesis. In Proceedings of the 16th European Conference on Computer Vision 405–421 (Springer, Glasgow, 2020).

H. Zhang , L. C. Cao , G. F. Jin . Three-dimensional computer-generated hologram with Fourier domain segmentation . Opt. Express , 2019 . 27 11689 -11697 . DOI:10.1364/OE.27.011689http://doi.org/10.1364/OE.27.011689 .

M. Gopakumar , 等 . Unfiltered holography: Optimizing high diffraction orders without optical filtering for compact holographic displays . Opt. Lett. , 2021 . 46 5822 -5825 . DOI:10.1364/OL.442851http://doi.org/10.1364/OL.442851 .

B. Lee , 等 . High-contrast, speckle-free, true 3D holography via binary CGH optimization . Sci. Rep. , 2022 . 12 2811 DOI:10.1038/s41598-022-06405-2http://doi.org/10.1038/s41598-022-06405-2 .

A. Maimone , J. R. Wang . Holographic optics for thin and lightweight virtual reality . ACM Trans. Graph. , 2020 . 39 67 DOI:10.1145/3386569.3392416http://doi.org/10.1145/3386569.3392416 .

F. S. Roux . Geometric phase lens . J. Optical Soc. Am. A , 2006 . 23 476 -482 . DOI:10.1364/JOSAA.23.000476http://doi.org/10.1364/JOSAA.23.000476 .

R. J. Li , L. C. Cao . Progress in phase calibration for liquid crystal spatial light modulators . Appl. Sci. , 2019 . 9 2012 DOI:10.3390/app9102012http://doi.org/10.3390/app9102012 .

C. Kim , 等 . Scene reconstruction from high spatio-angular resolution light fields . ACM Trans. Graph. , 2013 . 32 73 DOI:http://doi.org/ .

L. Xiao , 等 . DeepFocus: Learned image synthesis for computational displays . ACM Trans. Graph. , 2018 . 37 200 DOI:http://doi.org/ .

B. Mildenhall , 等 . Local light field fusion: Practical view synthesis with prescriptive sampling guidelines . ACM Trans. Graph. , 2019 . 38 29 DOI:10.1145/3306346.3322980http://doi.org/10.1145/3306346.3322980 .

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰