Spectral imaging with deep learning

Longqian Huang; Ruichen Luo; Xu Liu; Xiang Hao

doi:10.1038/s41377-022-00743-6

Go to Nature Website

Your Location：

Home >

Browse articles >

Spectral imaging with deep learning

Reviews

- Spectral imaging with deep learning
- Spectral imaging with deep learning
- LSA 2022年11卷第6期页码：1040-1058
- Affiliations：
  
  1.State Key Laboratory of Modern Optical Instrumentation, College of Optical Science and Technology, Zhejiang University, Hangzhou, 310027, China
  2.College of Computer Science and Technology, Zhejiang University, Hangzhou, 310027, China
  3.Jiaxing Key Laboratory of Photonic Sensing & Intelligent Imaging, Jiaxing, 314000, China
  4.Intelligent Optics & Photonics Research Center, Jiaxing Research Institute Zhejiang University, Jiaxing, 314000, China
- Author bio：
  
  Correspondence: Xiang Hao (haox@zju.edu.cn)
- Funds：
- DOI：10.1038/s41377-022-00743-6
  中图分类号：
- 纸质出版日期：2022-06-30，
  
  网络出版日期：2022-03-16，
  
  收稿日期：2021-09-30，
  
  修回日期：2022-01-30，
  
  录用日期：2022-02-15
- Accepted：
Scan QR Code
Spectral imaging with deep learning[J]. LSA, 2022,11(6):1040-1058.

Huang, L. Q. et al. Spectral imaging with deep learning. Light: Science & Applications, 11, 1040-1058 (2022).
Spectral imaging with deep learning[J]. LSA, 2022,11(6):1040-1058. DOI： 10.1038/s41377-022-00743-6.

Huang, L. Q. et al. Spectral imaging with deep learning. Light: Science & Applications, 11, 1040-1058 (2022). DOI： 10.1038/s41377-022-00743-6.

摘要

Abstract

The goal of spectral imaging is to capture the spectral signature of a target. Traditional scanning method for spectral imaging suffers from large system volume and low image acquisition speed for large scenes. In contrast

computational spectral imaging methods have resorted to computation power for reduced system volume

but still endure long computation time for iterative spectral reconstructions. Recently

deep learning techniques are introduced into computational spectral imaging

witnessing fast reconstruction speed

great reconstruction quality

and the potential to drastically reduce the system volume. In this article

we review state-of-the-art deep-learning-empowered computational spectral imaging methods. They are further divided into amplitude-coded

phase-coded

and wavelength-coded methods

based on different light properties used for encoding. To boost future researches

we've also organized publicly available spectral datasets.

关键词

Keywords

references

Shaw, G. A.&Burke, H. H. K. Spectral imaging for remote sensing.Lincoln Lab. J.14, 3–28 (2003)..

Lu, G. L.&Fei, B. W. Medical hyperspectral imaging: a review.J. Biomed. Opt.19, 010901 (2014)..

Li, Q. L. et al. Review of spectral imaging technology in biomedical engineering: achievements and challenges.J. Biomed. Opt.18, 100901 (2013)..

Liang, H. D. Advances in multispectral and hyperspectral imaging for archaeology and art conservation.Appl. Phys. A106, 309–323 (2012)..

Feng, Y. Z.&Sun, D. W. Application of hyperspectral imaging in food safety inspection and control: a review.Crit. Rev. Food Sci. Nutr.52, 1039–1058 (2012)..

Vane, G. et al. The airborne visible/infrared imaging spectrometer (AVIRIS).Remote Sens. Environ.44, 127–143 (1993)..

Green, R. O. et al. Imaging spectroscopy and the airborne visible/infrared imaging spectrometer (AVIRIS).Remote Sens. Environ.65, 227–248 (1998)..

Rickard, L. J. et al. HYDICE: an airborne system for hyperspectral imaging. Proceedings of SPIE 1937, Imaging Spectrometry of the Terrestrial Environment, 173–179 (SPIE, 1993).

Basedow, R. W., Carmer, D. C.&Anderson, M. E. HYDICE system: implementation and performance. Proceedings of SPIE 2480, Imaging Spectrometry, 258–267 (SPIE, 1995).

Gat, N. Imaging spectroscopy using tunable filters: a review. Proceedings of SPIE 4056, Wavelet Applications Ⅶ, 50–64 (SPIE, 2000).

Gupta, N. Hyperspectral imager development at army research laboratory. Proceedings of SPIE 6940, Infrared Technology and Applications XXXIV, 69401P (SPIE, 2008).

Hagen, N. A.&Kudenov, M. W. Review of snapshot spectral imaging technologies.Opt. Eng.52, 090901 (2013)..

Candès, E. J., Romberg, J.&Tao, T. Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information.IEEE Trans. Inform. Theory52, 489–509 (2006)..

Donoho, D. L. Compressed sensing.IEEE Trans. Inform. Theory52, 1289–1306 (2006)..

Jeon, D. S. et al. Compact snapshot hyperspectral imaging with diffracted rotation.ACM Trans. Graph.38, 117 (2019)..

Hauser, J. et al. DD-Net: spectral imaging from a monochromatic dispersed and diffused snapshot.Appl. Opt.59, 11196–11208 (2020)..

Glassner, A. S. How to derive a spectrum from an RGB triplet.IEEE Computer Graph. Appl.9, 95–99 (1989)..

Sun, Y. L. et al. Deriving spectra from colors and rendering light interference.IEEE Computer Graph. Appl.19, 61–67 (1999)..

Smits, B. An RGB-to-spectrum conversion for reflectances.J. Graph. Tools4, 11–22 (1999)..

Nguyen, R. M. H., Prasad, D. K.&Brown, M. S. Training-based spectral reconstruction from a single RGB image. Proceedings of the 13th European Conference on Computer Vision, 186–201 (Springer, 2014).

Arad, B.&Ben-Shahar, O. Sparse recovery of hyperspectral signal from natural RGB images. Proceedings of the 14th European Conference on Computer Vision, 19–34 (Springer, 2016).

Chakrabarti, A.&Zickler, T. Statistics of real-world hyperspectral images. Proceedings of CVPR 2011. Colorado Springs: IEEEE, 2011, 193–200.

Xiong, Z. W. et al. HSCNN: CNN-based hyperspectral image recovery from spectrally undersampled projections. Proceedings of 2017 IEEE International Conference on Computer Vision Workshops, 518–525 (IEEE, 2017).

Wang, L. Z. et al. Hyperspectral image reconstruction using a deep spatial-spectral prior. Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8024–8033 (IEEE, 2019).

Arad, B. et al. NTIRE 2020 challenge on spectral reconstruction from an RGB image. Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 1806–1822 (IEEE, 2020).

Gehm, M. E. et al. Single-shot compressive spectral imaging with a dual-disperser architecture.Opt. Express15, 14013–14027 (2007)..

Wagadarikar, A. et al. Single disperser design for coded aperture snapshot spectral imaging.Appl. Opt.47, B44–B51 (2008)..

Correa, C. V., Arguello, H.&Arce, G. R. Snapshot colored compressive spectral imager.J. Opt. Soc. Am. A32, 1754–1763 (2015)..

Lin, X. et al. Spatial-spectral encoded compressive hyperspectral imaging.ACM Trans. Graph.33, 233 (2014)..

Choi, I. et al. High-quality hyperspectral reconstruction using a spectral prior.ACM Trans. Graph.36, 218 (2017)..

Sun, Y. B. et al. Unsupervised spatial-spectral network learning for hyperspectral compressive snapshot reconstruction.IEEE Trans. Geosci. Remote Sens.60, 1–14 (2022)..

Rueda, H., Arguello, H.&Arce, G. R. Compressive spectral imaging based on colored coded apertures. Proceedings of 2014 IEEE International Conference on Acoustics, Speech and Signal Processing, 7799–7803 (IEEE, 2014).

Yuan, X. Generalized alternating projection based total variation minimization for compressive sensing. Proceedings of 2016 IEEE International Conference on Image Processing, 2539–2543 (IEEE, 2016).

Boyd, S. et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Found.Trends Machine Learn.3, 1–122 (2011)..

Zhang, T. et al. Hyperspectral image reconstruction using deep external and internal learning. Proceedings of 2019 IEEE/CVF International Conference on Computer Vision, 8558–8567 (IEEE, 2019).

Miao, X. et al.λ-Net: reconstruct hyperspectral images from a snapshot measurement. Proceedings of 2019 IEEE/CVF International Conference on Computer Vision, 4058–4068 (IEEE, 2019).

Candès, E. J. The restricted isometry property and its implications for compressed sensing.Comptes Rendus Math.346, 589–592 (2008)..

Arce, G. R. et al. Snapshot compressive multispectral cameras. in Wiley Encyclopedia of Electrical and Electronics Engineering (ed. Webster, J. G.) 1–22 (Wiley, 2017).

Wang, L. Z. et al. HyperReconNet: Joint coded aperture optimization and image reconstruction for compressive hyperspectral imaging.IEEE Trans. Image Proc.28, 2257–2270 (2019)..

Courbariaux, M., Bengio, Y.&David, J. P. BinaryConnect: training deep neural networks with binary weights during propagations. Proceedings of the 28th International Conference on Neural Information Processing Systems, 3123–3131 (NIPS, 2015).

Geman, D.&Yang, C. D. Nonlinear image recovery with half-quadratic regularization.IEEE Trans. Image Proc.4, 932–946 (1995)..

Wang, L. Z. et al. DNU: Deep non-local unrolling for computational spectral imaging. Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1658–1668 (IEEE, 2020).

Sogabe, Y. et al. Admm-inspired reconstruction network for compressive spectral imaging. Proceedings of 2020 IEEE International Conference on Image Processing, 2865–2869(IEEE, 2020).

Lempitsky, V., Vedaldi, A.&Ulyanov, D. Deep image prior. Proceedingsof 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9446–9454 (IEEE, 2018).

Bacca, J., Fonseca, Y.&Arguello, H. Compressive spectral image reconstruction using deep prior and low-rank tensor representation.Appl. Opt.60, 4197–4207 (2021)..

Kim, Y. D.&Choi, S. Nonnegative tucker decomposition. Proceedings of 2007 IEEE Conference on Computer Vision and Pattern Recognition, 1–8 (IEEE, 2007).

Antipa, N. et al. DiffuserCam: lensless single-exposure 3D imaging.Optica5, 1–9 (2018)..

Monakhova, K. et al. Spectral DiffuserCam: lensless snapshot hyperspectral imaging with a spectral filter array.Optica7, 1298–1307 (2020)..

Golub, M. A. et al. Compressed sensing snapshot spectral imaging by a regular digital camera with an added optical diffuser.Appl. Opt.55, 432–443 (2016)..

Baek, S. H. et al. Single-Shot Hyperspectral-Depth Imaging With Learned Diffractive Optics. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2651–2660 (IEEE, 2021).

Peng, Y. F. et al. Computational imaging using lightweight diffractive-refractive optics.Opt. Express23, 31393–31407 (2015)..

Heide, F. et al. Encoded diffractive optics for full-spectrum computational imaging.Sci. Rep.6, 33543 (2016)..

Peng, Y. F. et al. The diffractive achromat full spectrum computational imaging with diffractive optics.ACM Trans. Graph.35, 31 (2016)..

Kar, O. F.&Oktem, F. S. Compressive spectral imaging with diffractive lenses.Opt. Lett.44, 4582–4585 (2019)..

Monakhova, K. et al. Learned reconstructions for practical mask-based lensless imaging.Opt. Express27, 28075–28090 (2019)..

Jia, Y. et al. From RGB to spectrum for natural scenes via manifold-based mapping. Proceedings of 2017 IEEE International Conference on Computer Vision, 4715–4723 (IEEE, 2017).

Tenenbaum, J. B., De Silva, V.&Langford, J. C. A global geometric framework for nonlinear dimensionality reduction.Science290, 2319–2323 (2000)..

Fubara, B. J., Sedky, M.&Dyke, D. RGB to spectral reconstruction via learnedbasis functions and weights. Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 1984–1993 (IEEE, 2020).

Robles-Kelly, A. Single image spectral reconstruction for multimedia applications. Proceedings of the 23rd ACM International Conference on Multimedia, 251–260 (ACM, 2015).

Galliani, S. et al. Learned spectral super-resolution.https://arxiv.org/abs/1703.09470https://arxiv.org/abs/1703.09470(2017).

Arad, B. et al. NTIRE 2018 challenge on spectral reconstruction from RGB images. Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 1042–104209 (IEEE, 2018).

Alvarez-Gila, A., Van De Weijer, J.&Garrote, E. Adversarial networks for spatial context-aware spectral image reconstruction from RGB. Proceedings of 2017 IEEE International Conference on Computer Vision Workshops, 480–490 (Venice: IEEE, 2017).

Isola, P. et al. Image-to-image translation with conditional adversarial networks. Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 5967–5976 (IEEE, 2017).

He, K. M. et al. Deep residual learning for image recognition. Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 770–778 (Las Vegas: IEEE, 2016).

Huang, G. et al. Densely connected convolutional networks. Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2261–2269 (IEEE, 2017).

Vaswani, A. et al. Attention is all you need. Proceedings of the 31st International Conference on Neural Information Processing Systems, 6000–6010 (NIPS, 2017).

Zhang, H. et al. Self-attention generative adversarial networks. Proceedings of the 36th International Conference on Machine Learning, 7354–7363 (PMLR, 2019).

Zhang, L. et al. Pixel-aware deep function-mixture network for spectral super-resolution. Proceedings of the 34th AAAI Conference on Artificial Intelligence, 12821–12828 (AAAI, 2020).

Zhang, Y. L. et al. Image super-resolution using very deep residual channel attention networks. Proceedings of the 15th European Conference on Computer Vision, 294–310 (Springer, 2018).

Zhao, L. M. et al. On the connection of deep fusion to ensembling.http://arxiv.org/abs/1611.07718http://arxiv.org/abs/1611.07718(2016).

Shi, Z. et al. HSCNN+: advanced CNN-based hyperspectral recovery from RGB images. Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 1052–10528 (IEEE, 2018).

Peng, H., Chen, X. M.&Zhao, J. Residual pixel attention network for spectral reconstruction from RGB images. Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2012–2020 (IEEE, 2020).

Zhao, Y. Z. et al. Hierarchical regression network for spectral reconstruction from RGB images. Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 1695–1704 (IEEE, 2020).

Ronneberger, O., Fischer, P.&Brox, T. U-Net: convolutional networks for biomedical image segmentation. Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention, 234–241 (Springer, 2015).

Shi, W. Z. et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 1874–1883 (IEEE, 2016).

Gatys, L., Ecker, A.&Bethge, M. A neural algorithm of artistic style.J. Vis.16, 326 (2016)..

Johnson, J., Alahi, A.&Fei-Fei, L. Perceptual losses for real-time style transfer and super-resolution. Proceedings of the 14th European Conference on Computer Vision, 694–711 (Springer, 2016).

Windrim, L. et al. A physics-based deep learning approach to shadow invariant representations of hyperspectral images.IEEE Trans. Image Proc.27, 665–677 (2018)..

Kaya, B., Can, Y. B.&Timofte, R. Towards spectral estimation from a single RGB image in the wild. Proceedings of 2019 IEEE/CVF International Conference on Computer Vision Workshop, 3546–3555 (IEEE, 2019).

Li, J. J. et al. Adaptive weighted attention network with camera spectral sensitivity prior for spectral reconstruction from RGB images. Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 1894–1903 (IEEE, 2020).

Schanda, J. CIE 1931 and 1964 standard colorimetric observers: history, data, and recent assessments. in Encyclopedia of Color Science and Technology (ed. Luo, M. R.), 125–129 (Springer, 2016).

Lin, Y. T.&Finlayson, G. D. Physically plausible spectral reconstruction.Sensors20, 6399 (2020)..

Nie, S. J. et al. Deeply learned filter response functions for hyperspectral reconstruction. Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4767–4776 (IEEE, 2018).

Song, H. Y. et al. Deep-learned broadband encoding stochastic filters for computational spectroscopic instruments.Adv. Theory Simul.4, 2000299 (2021)..

Zhang, W. Y. et al. Deeply learned broadband encoding stochastic hyperspectral imaging.Light: Sci. Appl.10, 108 (2021)..

Han, X. et al. Inverse design of metasurface optical filters using deep neural network with high degrees of freedom.InfoMat3, 432–442 (2021)..

Oiknine, Y. et al. Compressive sensing hyperspectral imaging by spectral multiplexing with liquid crystal.J. Imag.5, 3 (2019)..

Gedalin, D., Oiknine, Y.&Stern, A. DeepCubeNet: reconstruction of spectrally compressive sensed hyperspectral images with deep neural networks.Opt. Express27, 35811–35822 (2019)..

Yasuma, F. et al. Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum.IEEE Trans. Image Proc.19, 2241–2253 (2010)..

Hyperspectral&Color Imaging. Hyperspectral Images Database.https://sites.google.com/site/hyperspectralcolorimaging/datasethttps://sites.google.com/site/hyperspectralcolorimaging/dataset.

Habili, N., Oorloff, J.&Wei, R. Scyllarus hyperspectral dataset.https://scyllarus.data61.csiro.au/data/https://scyllarus.data61.csiro.au/data/.

Yan, L. B. et al. Reconstruction of hyperspectral data from RGB images with prior category information.IEEE Trans. Comput. Imag.6, 1070–1081 (2020)..

Shrestha, R. et al. Quality evaluation in spectral imaging-quality factors and metrics.J. Int. Colour Assoc.10, 22–35 (2014)..

Redmon, J. et al. You only look once: unified, real-time object detection. Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition, 779–788 (IEEE, 2016).

Yan, L. B. et al. Object detection in hyperspectral images.IEEE Signal Proc. Lett.28, 508–512 (2021)..

Zoph, B.&Le, Q. V. Neural architecture search with reinforcement learning.https://arxiv.org/abs/1611.01578https://arxiv.org/abs/1611.01578(2016).

Han, K. et al. A survey on vision transformer.https://arxiv.org/abs/2012.12556https://arxiv.org/abs/2012.12556(2020).

Stiebel, T. et al. Reconstructing spectral images from RGB-images using a convolutional neural network. Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 948–953 (IEEE, 2018).

Can, Y. B.&Timofte, R. An efficient CNN for spectral reconstruction from RGB images.https://arxiv.org/abs/1804.04647https://arxiv.org/abs/1804.04647(2018).

Koundinya, S. et al. 2D-3D CNN based architectures for spectral reconstruction from RGB images. Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 957–9577 (IEEE, 2018).

Ji, S. W. et al. 3D convolutional neural networks for human action recognition.IEEE Trans. Pattern Anal. Mach. Intel.35, 221–231 (2013)..

Banerjee, A.&Palrecha, A. MXR-U-nets for real time hyperspectral reconstruction.https://arxiv.org/abs/2004.07003https://arxiv.org/abs/2004.07003(2020).

He, T. et al. Bag of tricks for image classification with convolutional neural networks. Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 558–567 (IEEE, 2019).

Misra, D. Mish: a self regularized non-monotonic activation function. The 31st British Machine Vision Virtual Conference: online (2020).

Wang, W. J.&Wang, J. W. Double ghost convolution attention mechanism network: a framework for hyperspectral reconstruction of a single RGB image.Sensors21, 666 (2021)..

Han, K. et al. GhostNet: more features from cheap operations. Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1577–1586 (IEEE, 2020).

Wang, X. L. et al. Non-local neural networks. Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7794–7803 (IEEE, 2018).

浏览量

Downloads

CSCD

文章被引用时，请邮件提醒。

Submit

工具集

关联资源

深度学习赋能光谱成像

Light | 深度学习赋能光谱成像

暂无数据