All-optical image denoising using a diffractive visual processor

Çağatay Işıl; Tianyi Gan; Fazil Onuralp Ardic; Koray Mentesoglu; Jagrit Digani; Huseyin Karaca; Hanlong Chen; Jingxi Li; Deniz Mengu; Mona Jarrahi; Kaan Akşit; Aydogan Ozcan

doi:10.1038/s41377-024-01385-6

Your Location：

Home >

Browse articles >

All-optical image denoising using a diffractive visual processor

Original Articles

- All-optical image denoising using a diffractive visual processor
- Light: Science & Applications Vol. 13, Issue 3, Pages: 429-445(2024)
- Affiliations：
  
  1.Electrical and Computer Engineering Department, University of California, Los Angeles, CA 90095, USA
  2.Bioengineering Department, University of California, Los Angeles, CA 90095, USA
  3.California NanoSystems Institute (CNSI), University of California, Los Angeles, CA 90095, USA
  4.University College London, Department of Computer Science, London, United Kingdom
- Author bio：
  
  Aydogan Ozcan (ozcan@ucla.edu)
- Funds：
- DOI：10.1038/s41377-024-01385-6
  CLC：
- Received：17 September 2023，
  
  Revised：14 January 2024，
  
  Accepted：2024-01-15，
  
  Published Online：04 February 2024，
  
  Published：31 March 2024
- Accepted：
Scan QR Code
Işıl, Ç. et al. All-optical image denoising using a diffractive visual processor. Light: Science & Applications, 13, 429-445 (2024).
DOI：

Işıl, Ç. et al. All-optical image denoising using a diffractive visual processor. Light: Science & Applications, 13, 429-445 (2024). DOI： 10.1038/s41377-024-01385-6.

摘要

Abstract

Image denoising

one of the essential inverse problems

targets to remove noise/artifacts from input images. In general

digital image denoising algorithms

executed on computers

present latency due to several iterations implemented in

e.g.

graphics processing units (GPUs). While deep learning-enabled methods can operate non-iteratively

they also introduce latency and impose a significant computational burden

leading to increased power consumption. Here

we introduce an analog diffractive image denoiser to all-optically and non-iteratively clean various forms of noise and artifacts from input images – implemented at the speed of light propagation within a thin diffractive visual processor that axially spans <250 × λ

where λ is the wavelength of light. This all-optical image denoiser comprises passive transmissive layers optimized using deep learning to physically scatter the optical modes that represent various noise features

causing them to miss the output image Field-of-View (FoV) while retaining the object features of interest. Our results show that these diffractive denoisers can efficiently remove salt and pepper noise and image rendering-related spatial artifacts from input phase or intensity images while achieving an output power efficiency of ~30–40%. We experimentally demonstrated the effectiveness of this analog denoiser architecture using a 3D-printed diffractive visual processor operating at the terahertz spectrum. Owing to their speed

power-efficiency

and minimal computational overhead

all-optical diffractive denoisers can be transformative for various image display and projection systems

including

e.g.

holographic displays.

关键词

Keywords

references

Bertero, M., Boccacci, P. & De Mol, C. Introduction to Inverse Problems in Imaging (CRC Press, 2021).

Gonzalez, R. C. & Woods, R. E. Digital Image Processing (Pearson Education, 2009).

Haines, E. & Akenine-Möller, T. Ray Tracing Gems: High-Quality and Real-Time Rendering with DXR and Other APIs (Apress, 2019), https://doi.org/10.1007/978-1-4842-4427-2

Blinder, D. et al. The state-of-the-art in computer generated holography for 3D display. Light.: Adv. Manuf. 3 , 35, https://doi.org/10.37188/lam.2022.035 (2022)..

Bai, B. et al. To image, or not to image: class-specific diffractive cameras with all-optical erasure of undesired objects. eLight 2 , 14, https://doi.org/10.1186/s43593-022-00021-3 (2022)..

Lundberg, K. H. Noise Sources in Bulk CMOS. https://www.mit.edu/people/klund/papers/UNP_noise.pdf https://www.mit.edu/people/klund/papers/UNP_noise.pdf (2002).

Noise, dynamic range and bit depth in digital SLRs. https://homes.psd.uchicago.edu/~ejmartin/pix/20d/tests/noise/ https://homes.psd.uchicago.edu/~ejmartin/pix/20d/tests/noise/ (2023).

Chaitanya, C. R. A. et al. Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder. ACM Trans. Graph. 36 , 98, https://doi.org/10.1145/3072959.3073601 (2017)..

Işık, M. et al. Interactive Monte Carlo denoising using affinity of neural features. ACM Trans. Graph. 40 , 37, https://doi.org/10.1145/3450626.3459793 (2021)..

Introducing the NVIDIA RTX Ray Tracing Platform. NVIDIA developer. at https://developer.nvidia.com/rtx/ray-tracing https://developer.nvidia.com/rtx/ray-tracing (2018).

Elad, M., Kawar, B. & Vaksman, G. Image denoising: The deep learning revolution and beyond - a survey paper. SIAM J. Imaging Sci. 16 , 1594–1654, https://doi.org/10.48550/ar2301.03362 (2023)..

Buades, A., Coll, B. & Morel, J. M. A review of image denoising algorithms, with a new one. Multiscale Model. Simul. 4 , 490–530, https://doi.org/10.1137/040616024 (2005)..

Milanfar, P. A tour of modern image filtering: new insights and methods, both practical and theoretical. IEEE Signal Process. Mag. 30 , 106–128, https://doi.org/10.1109/MS2011.2179329 (2013)..

Lindenbaum, M., Fischer, M. & Bruckstein, A. On Gabor's contribution to image enhancement. Pattern Recognit. 27 , 1–8, https://doi.org/10.1016/0031-3203(94)90013-2 (1994)..

Perona, P. & Malik, J. Scale-space and edge detection using anisotropic diffusion. IEEE Trans. Pattern Anal. Mach. Intell. 12 , 629–639, https://doi.org/10.1109/34.56205 (1990)..

Rudin, L. I., Osher, S. & Fatemi, E. Nonlinear total variation based noise removal algorithms. Phys. D: Nonlinear Phenom. 60 , 259–268, https://doi.org/10.1016/0167-2789(92)90242-F (1992)..

Donoho, D. L. De-noising by soft-thresholding. IEEE Trans. Inf. Theory 41 , 613–627, https://doi.org/10.1109/18.382009 (1995)..

Tomasi, C. & Manduchi, R. Bilateral filtering for gray and color images. In Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH36271) . 839–846 (IEEE, 1998), https://doi.org/10.1109/I1998.710815 https://doi.org/10.1109/I1998.710815 ..

Buades, A., Coll, B. & Morel, J. M. A non-local algorithm for image denoising. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) . 60–65 (IEEE, 2005), https://doi.org/10.1109/C2005.38 https://doi.org/10.1109/C2005.38 ..

Dabov, K. et al. Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process. 16 , 2080–2095, https://doi.org/10.1109/TI2007.901238, (2007)..

Lucas, A. et al. Using deep neural networks for inverse problems in imaging: beyond analytical methods. IEEE Signal Process. Mag. 35 , 20–36, https://doi.org/10.1109/MS2017.2760358 (2018)..

McCann, M. T., J in, K. H. & Unser, M. Convolutional neural networks for inverse problems in imaging: a review. IEEE Signal Process. Mag. 34 , 85–95, https://doi.org/10.1109/MSP.2017.2739299 (2017)..

Zhang, K. et al. Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process. 26 , 3142–3155, https://doi.org/10.1109/TI2017.2662206 (2017)..

Burger, H. C., Schuler, C. J. & Harmeling, S. Image denoising: Can plain neural networks compete with BM3D? In 2012 IEEE Conference on Computer Vision and Pattern Recognition . 2392–2399 (IEEE, 2012), https://doi.org/10.1109/CVPR.2012.6247952 https://doi.org/10.1109/CVPR.2012.6247952 ..

Plötz, T. & Roth, S. Neural nearest neighbors networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems (Curran Associates Inc., 2018).

Valsesia, D., Fracastoro, G. & Magli, E. Deep graph-convolutional image denoising. IEEE Trans. Image Process. 29 , 8226–8237, https://doi.org/10.1109/TIP.2020.3013166 (2020)..

Liang, J. Y. et al. SwinIR: Image restoration using swin transformer. In Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision Workshops . 1833–1844 (IEEE, 2021).

Zhang, K., Zuo, W. M. & Zhang, L. FFDNet: Toward a fast and flexible solution for CNN-based image denoising. IEEE Trans. Image Process. 27 , 4608–4622, https://doi.org/10.1109/TIP.2018.2839891 (2018)..

Tu, Z. et al. MAXIM: Multi-axis MLP for image processing. In Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition . 5759–5770 (IEEE, 2022).

Wang, Z. D. et al. Uformer: a general U-shaped transformer for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 17662–17672 (IEEE, 2022).

Li, Y. H. et al. Quantitative phase imaging (QPI) through random diffusers using a diffractive optical network. Light.: Adv. Manuf. 4 , 19, https://doi.org/10.37188/lam.2023.017 (2023)..

Situ, G. H. Deep holography. Light.: Adv. Manuf. 3 , 8, https://doi.org/10.37188/lam.2022.013 (2022)..

Lin, H. & Cheng, J. X. Computational coherent Raman scattering imaging: breaking physical barriers by fusion of advanced instrumentation and data science. eLight 3 , 6, https://doi.org/10.1186/s43593-022-00038-8 (2023)..

Koskela, M. et al. Blockwise multi-order feature regression for real-time path-tracing reconstruction. ACM Trans. Graph. 38 , 138, https://doi.org/10.1145/3269978 (2019)..

Hasselgren, J. et al. Neural temporal adaptive sampling and denoising. Comput. Graph. Forum 39 , 147–155, https://doi.org/10.1111/cgf.13919 (2020)..

Meng, X. X. et al. Real-time Monte Carlo Denoising with the neural bilateral grid. In 31st Eurographics Symposium on Rendering (Eurographics Association, 2020).

Mengu, D. et al. Analysis of diffractive optical neural networks and their integration with electronic neural networks. IEEE J. Sel. Top. Quantum Electron. 26 , 1–14, https://doi.org/10.1109/JSTQE.2019.2921376 (2020)..

Justus, D. et al. Predicting the computational cost of deep learning models. In 2018 IEEE International Conference on Big Data (Big Data) . 3873–3882, (IEEE, 2018), https://doi.org/10.1109/BigData.2018.8622396 https://doi.org/10.1109/BigData.2018.8622396 ..

Luo, Y. et al. Computational imaging without a computer: seeing through random diffusers at the speed of light. eLight 2 , 4, https://doi.org/10.1186/s43593-022-00012-4 (2022)..

Wang, T. Y. et al. Image sensing with multilayer nonlinear optical neural networks. Nat. Photonics 17 , 408–415, https://doi.org/10.1038/s41566-023-01170-8 (2023)..

Wetzstein, G. et al. Inference in artificial intelligence with deep optics and photonics. Nature 588 , 39–47, https://doi.org/10.1038/s41586-020-2973-6 (2020)..

Goi, E., Schoenhardt, S. & Gu, M. Direct retrieval of Zernike-based pupil functions using integrated diffractive deep neural networks. Nat. Commun. 13 , 7531, https://doi.org/10.1038/s41467-022-35349-4 (2022)..

Chen, Y. T. et al. Photonic unsupervised learning variational autoencoder for high-throughput and low-latency image transmission. Sci. Adv. 9 , eadf8437, https://doi.org/10.1126/sciadv.adf8437 (2023)..

Mengu, D. & Ozcan, A. All-optical phase recovery: diffractive computing for quantitative phase imaging. Adv. Optical Mater. 10 , 2200281, https://doi.org/10.1002/adom.202200281 (2022)..

Ha, D. & Eck, D. A neural representation of sketch drawings. In 6th International Conference on Learning Representations . (OpenReview. net, 2018).

Wang, Z. et al. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13 , 600–612, https://doi.org/10.1109/TIP.2003.819861 (2004)..

Xiao, H., Rasul, K. & Vollgraf, R. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. Preprint at https://arxiv.org/abs/1708.07747 https://arxiv.org/abs/1708.07747 .

Cohen, G. et al. EMNIST: Extending MNIST to handwritten letters. In 2017 International Joint Conference on Neural Networks (IJCNN) . 2921–2926 (IEEE, 2017). https://doi.org/10.1109/IJCNN.2017.7966217 https://doi.org/10.1109/IJCNN.2017.7966217 ..

Mengu, D. et al. Misalignment resilient diffractive optical networks. Nanophotonics 9 , 4207–4219, https://doi.org/10.1515/nanoph-2020-0291 (2020)..

Mengu, D., Rivenson, Y. & Ozcan, A. Scale-, shift-, and rotation-invariant diffractive optica l networks. ACS Photonics 8 , 324–334, https://doi.org/10.1021/acsphotonics.0c01583 (2021)..

Wang, J. et al. Speckle suppression using F-D 2 NN in holographic display. Displays 74 , 102231, https://doi.org/10.1016/j.displa.2022.102231 (2022)..

Deng, Y. B. & Chu, D. P. Coherence properties of different light sources and their effect on the image sharpness and speckle of holographic displays. Sci. Rep. 7 , 5893, https://doi.org/10.1038/s41598-017-06215-x (2017)..

Li, J. X. et al. Unidirectional imaging using deep learning–designed materials. Sci. Adv. 9 , eadg1505, https://doi.org/10.1126/sciadv.adg1505 (2023)..

Rahman, M. S. S. et al. Universal linear intensity transformations using spatially incoherent diffractive processors. Light Sci. Appl. 12 , 195, https://doi.org/10.1038/s41377-023-01234-y (2023)..

Gharbi, M. et al. Deep bilateral learning for real-time image enhancement. ACM Trans. Graph. 36 , 118, https://doi.org/10.1145/3072959.3073592 (2017)..

Tang, Y. H. et al. GhostNetV2: Enhance cheap opera tion with long-range attention. In Proceedings of the 36th International Conference on Neural Information Processing Systems (neurips. cc, 2022).

Desislavov, R., Martínez-Plumed, F. & Hernández-Orallo, J. Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning. Sustain. Comput. Inform. Syst. 38 , 100857, https://doi.org/10.1016/j.suscom.2023.100857 (2023)..

Rahman, M. S. S. et al. Learning diffractive optical communication around arbitrary opaque occlusions. Nat. Commun. 14 , 6830, https://doi.org/10.1038/s41467-023-42556-0 (2023)..

Işıl, Ç. et al. Super-resolution image display using diffractive decoders. Sci. Adv. 8 , eadd3433, https://doi.org/10.1126/sciadv.add3433 (2022)..

Li, Y. H. et al. Optical information transfer through random unknown diffusers using electronic encoding and diffractive decoding. Adv. Photonics 5 , 046009, https://doi.org/10.1117/1.AP.5.4.046009 (2023)..

Kulce, O. et al. All-optical information-processing capacity of diffractive surfaces. Light Sci. Appl. 10 , 25, https://doi.org/10.1038/s41377-020-00439-9 (2021)..

Kulce, O. et al. All-optical synthesis of an arbitrary linear transformation using diffractive surfaces. Light Sci. Appl. 10 , 196, https://doi.org/10.1038/s41377-021-00623-5 (2021)..

Li, J. X. et al. Polarization multiplexed diffractive computing: all-optical implementation of a group of linear transformations through a polarization-encoded diffractive network. Light Sci. Appl. 11 , 153, https://doi.org/10.1038/s41377-022-00849-x (2022)..

Goi, E. et al. Nanoprinted high-neuron-density optical linear perceptrons performing near-infrared inference on a CMOS chip. Light Sci. Appl. 10 , 40, https://doi.org/10.1038/s41377-021-00483-z (2021)..

Chen, H. et al. Diffractive deep neural networks at visible wavelengths. Engineering 7 , 1483–1491, https://doi.org/10.1016/j.eng.2020.07.032 (2021)..

Bai, B. J. et al. Data-class-specific all-optical transformations and encryption. Adv. Mater. 35 , 2212091, https://doi.org/10.1002/adma.202212091 (2023)..

Luo, Y. et al. Design of task-specific optical systems using broadband diffractive neural networks. Light Sci. Appl. 8 , 112, https://doi.org/10.1038/s4 1377-019-0223-1 (2019)..

Li, J. X. et al. Spectrally encoded single-pixel machine vision using diffractive networks. Sci. Adv. 7 , eabd7690, https://doi.org/10.1126/sciadv.abd7690 (2021)..

Mengu, D. et al. Snapshot multispectral imaging using a diffractive optical network. Light Sci. Appl. 12 , 86, https://doi.org/10.1038/s41377-023-01135-0 (2023)..

Shen, C. Y. et al. Multispectral quantitative phase imaging using a diffractive optical network. Adv. Intell. Syst. 5 , 2300300, https://doi.org/10.1002/aisy.202300300 (2023)..

Li, J. X. et al. Massively parallel universal linear transformations using a wavelength-multiplexed diffractive optical network. Adv. Photonics 5 , 016003, https://doi.org/10.1117/1.AP.5.1.016003 (2023)..

Lin, X. et al. All-optical machine learning using diffractive deep neural networks. Science 361 , 1004–1008, https://doi.org/10.1126/science.aat8084 (2018)..

LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521 , 436–444, https://doi.org/10.1038/nature14539 (2015)..

Rumelhart, D. E., Hinton, G. E. & Williams, R. J. Learning representations by back-propagating errors. Nature 323 , 533–536, https://doi.org/10.1038/323533a0 (1986)..

Quan, Y. H. et al. Self2Self with dropout: learning self-supervised denoising from single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 1887–1895 (IEEE, 2020).

Loshchilov, I. & Hutter, F. SGDR: stochastic gradient descent with warm restarts. In Presented at the 5th International Conference on Learning Representations (OpenReview. net, 2017).

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰