Snapshot multispectral imaging using a diffractive optical network

Deniz Mengu; Anika Tabassum; Mona Jarrahi; Aydogan Ozcan

doi:10.1038/s41377-023-01135-0

Go to Nature Website

Your Location：

Home >

Browse articles >

Snapshot multispectral imaging using a diffractive optical network

Original Articles

- Snapshot multispectral imaging using a diffractive optical network
- Light: Science & Applications Vol. 12, Issue 5, Pages: 789-808(2023)
- Affiliations：
  
  1.Electrical and Computer Engineering Department, University of California, Los Angeles, CA 90095, USA
  2.Bioengineering Department, University of California, Los Angeles, CA 90095, USA
  3.California NanoSystems Institute, University of California, Los Angeles, CA 90095, USA
- Author bio：
  
  Aydogan Ozcan (ozcan@ucla.edu)
- Funds：
- DOI：10.1038/s41377-023-01135-0
  CLC：
- Published：31 May 2023，
  
  Published Online：6 April 2023，
  
  Received：30 November 2022，
  
  Revised：14 March 2023，
  
  Accepted：17 March 2023
Scan for full text
Mengu, D. et al. Snapshot multispectral imaging using a diffractive optical network. Light: Science & Applications, 12, 789-808 (2023).
DOI：

Mengu, D. et al. Snapshot multispectral imaging using a diffractive optical network. Light: Science & Applications, 12, 789-808 (2023). DOI： 10.1038/s41377-023-01135-0.

摘要

Abstract

Multispectral imaging has been used for numerous applications in e.g.

environmental monitoring

aerospace

defense

and biomedicine. Here

we present a diffractive optical network-based multispectral imaging system trained using deep learning to create a virtual spectral filter array at the output image field-of-view. This diffractive multispectral imager performs spatially-coherent imaging over a large spectrum

and at the same time

routes a pre-determined set of spectral channels onto an array of pixels at the output plane

converting a monochrome focal-plane array or image sensor into a multispectral imaging device without any spectral filters or image recovery algorithms. Furthermore

the spectral responsivity of this diffractive multispectral imager is not sensitive to input polarization states. Through numerical simulations

we present different diffractive network designs that achieve snapshot multispectral imaging with 4

9 and 16 unique spectral bands within the visible spectrum

based on passive spatially-structured diffractive surfaces

with a compact design that axially spans ~72

where

is the mean wavelength of the spectral band of interest. Moreover

we experimentally demonstrate a diffractive multispectral imager based on a 3D-printed diffractive network that creates at its output image plane a spatially repeating virtual spectral filter array with 2 × 2 = 4 unique bands at terahertz spectrum. Due to their compact form factor and computation-free

power-efficient and polarization-insensitive forward operation

diffractive multispectral imagers can be transformative for various imaging and sensing applications and be used at different parts of the electromagnetic spectrum where high-density and wide-area multispectral pixel arrays are not widely available.

关键词

Keywords

references

Kargel,J.S.etal.Multispectralimagingcontributionstogloballandicemeasurementsfromspace.Remote Sens. Environ.99,187–219(2005)..

Bell,J.F.etal.PancammultispectralimagingresultsfromtheopportunityroveratMeridianiPlanum.Science306,1703–1709(2004)..

Dinguirard,M.&Slater,P.N.Calibrationofspace-multispectralimagingsensors:areview.Remote Sens. Environ.68,194–205(1999)..

Berry,S.etal.AnalysisofmultispectralimagingwiththeAstroPathplatforminformsefficacyofPD-1blockade.Science372,eaba2609(2021)..

Boelt,B.etal.Multispectralimaging–anewtoolinseedqualityassessment?Seed Sci. Res.28,222–228(2018)..

DeOca,A.M.etal.Low-costmultispectralimagingsystemforcropmonitoring.InProc 2018 International Conference on Unmanned Aircraft Systems (ICUAS), 443–451.(IEEE,Dallas,2018).

Rouse,A.R.&Gmitro,A.F.Multispectralimagingwithaconfocalmicroendoscope.Opt. Lett.25,1708–1710(2000)..

Levenson,R.M.&Mansfield,J.R.Multispectralimaginginbiologyandmedicine:Slicesoflife.Cytometry69A,748–758(2006)..

McGrath,K.E.,Bushnell,T.P.&Palis,J.Multispectralimagingofhematopoieticcells:whereflowmeetsmorphology.J. Immunol. Methods336,91–97(2008)..

Halicek,M.etal.In-vivoandex-vivotissueanalysisthroughhyperspectralimagingtechniques:revealingtheinvisiblefeaturesofcancer.Cancers11,756(2019)..

Levenson,R.M.,Fornari,A.&Loda,M.Multispectralimagingandpathology:seeinganddoingmore.Expert Opin. Med. Diagnostics2,1067–1081(2008)..

Feng,C.H.etal.Hyperspectralimagingandmultispectralimagingasthenoveltechniquesfordetectingdefectsinrawandprocessedmeatproducts:currentstate-of-the-artresearchadvances.Food Control84,165–176(2018)..

Qin,J.etal.Hyperspectralandmultispectralimagingforevaluatingfoodsafetyandquality.J. Food Eng.118,157–171(2013)..

Elias,M.&Cotte,P.MultispectralcameraandradiativetransferequationusedtodepictLeonardo'ssfumatoinMonaLisa.Appl. Opt.47,2146–2154(2008)..

Pelagotti,A.etal.Multispectralimagingofpaintings.IEEE Signal Process. Mag.25,27–36(2008)..

Cosentino,A.Identificationofpigmentsbymultispectralimaging;aflowchartmethod.Herit. Sci.2,8(2014)..

Easton,R.L.,Knox,K.T.&Christens-Barry,W.A.MultispectralimagingoftheArchimedespalimpsest.InProc 32nd Applied Imagery Pattern Recognition Workshop, 2003,111–116(IEEE,Washington,2003).

Ortega,S.etal.Useofhyperspectral/multispectralimagingingastroenterology.sheddingsome–different–lightintothedark.J. Clin. Med.8,36(2019)..

Eisenbeiß,W.,Marotz,J.&Schrade,J.P.Reflection-opticalmultispectralimagingmethodforobjectivedeterminationofburndepth.Burns25,697–704(1999)..

Shimoni,M.,Haelterman,R.&Perneel,C.Hypersectralimagingformilitaryandsecurityapplications:combiningmyriadprocessingandsensingtechniques.IEEE Geosci. Remote Sens. Mag.7,101–117(2019)..

Zhang,C.etal.Anovel3Dmultispectralvisionsystembasedonfilterwheelcameras.InProc 2016 IEEE International Conference on Imaging Systems and Techniques (IST),267–272(IEEE,Chania,2016).

Thompson,L.L.Remote Sensing Using Solid-state Array Technology(NTRS,1979).

Chen,Z.Y.,Wang,X.&Liang,R.G.RGB-NIRmultispectralcamera.Opt. Express22,4985–4994(2014)..

Fletcher-Holmes,D.W.&Harvey,A.R.Real-timeimagingwithahyperspectralfovea.J. Opt. A Pure Appl. Opt.7,S298–S302(2005)..

Weitzel,L.etal.3D:thenextgenerationnear-infraredimagingspectrometer.Astron. Astrophys. Suppl. Ser.119,531–546(1996)..

Wagadarikar,A.etal.Singledisperserdesignforcodedaperturesnapshotspectralimaging.Appl. Opt.47,B44–B51(2008)..

Arguello,H.&Arce,G.R.Coloredcodedaperturedesignbyconcentrationofmeasureincompressivespectralimaging.IEEE Trans. Image Process.23,1896–1908(2014)..

Correa,C.V.,Arguello,H.&Arce,G.R.Snapshotcoloredcompressivespectralimager.J. Opt. Soc. Am. A32,1754–1763(2015)..

Lin,X.etal.Dual-codedcompressivehyperspectralimaging.Opt. Lett.39,2044–2047(2014)..

Lin,X.etal.Spatial-spectralencodedcompressivehyperspectralimaging.ACM Trans. Graph.33,233(2014)..

August,Y.etal.Compressivehyperspectralimagingbyrandomseparableprojectionsinboththespatialandthespectraldomains.Appl. Opt.52,D46–D54(2013)..

Wu,Y.H.etal.Developmentofadigital-micromirror-device-basedmultishotsnapshotspectralimagingsystem.Opt. Lett.36,2692–2694(2011)..

Wang,P.&Menon,R.Ultra-high-sensitivitycolorimagingviaatransparentdiffractive-filterarrayandcomputationaloptics.Optica2,933–939(2015)..

Heide,F.etal.Encodeddiffractiveopticsforfull-spectrumcomputationalimaging.Sci. Rep.6,33543(2016)..

Jeon,D.S.etal.Compactsnapshothyperspectralimagingwithdiffractedrotation.ACM Trans. Graph.38,117(2019)..

Arguello,H.etal.Shift-variantcolor-codeddiffractivespectralimagingsystem.Optica8,1424–1434(2021)..

Dun,X.etal.Learnedrotationallysymmetricdiffractiveachromatforfull-spectrumcomputationalimaging.Optica7,913–922(2020)..

Ballard,Z.etal.Machinelearningandcomputation-enabledintelligentsensordesign.Nat. Mach. Intell.3,556–565(2021)..

Wetzstein,G.etal.Inferenceinartificialintelligencewithdeepopticsandphotonics.Nature588,39–47(2020)..

Chen,M.J.etal.Full-colornanorouterforhigh-resolutionimaging.Nanoscale13,13024–13029(2021)..

Shegai,T.etal.Abimetallicnanoantennafordirectionalcolourrouting.Nat. Commun.2,481(2011)..

Chen,B.H.etal.GaNMetalensforpixel-levelfull-colorroutingatvisiblelight.Nano Lett.17,6345–6352(2017)..

Zou,X.J.etal.Pixel-levelBayer-typecolourrouterbasedonmetasurfaces.Nat. Commun.13,3288(2022)..

Li,J.H.etal.Single-layerBayermetasurfaceviainversedesign.ACS Photonics9,2607–2613(2022)..

Miyata,M.,Nakajima,M.&Hashimoto,T.High-sensitivitycolorimagingusingpixel-scalecolorsplittersbasedondielectricmetasurfaces.ACS Photonics6,1442–1450(2019)..

Nishiwaki,S.etal.Efficientcoloursplittersforhigh-pixel-densityimagesensors.Nat. Photonics7,240–246(2013)..

Sell,D.etal.Periodicdielectricmetasurfaceswithhigh-efficiency,multiwavelengthfunctionalities.Adv. Opt. Mater.5,1700645(2017)..

Camayd-Muñoz,P.etal.Multifunctionalvolumetricmeta-opticsforcolorandpolarizationimagesensors.Optica7,280–283(2020)..

Zhao,N.,Catrysse,P.B.&Fan,S.H.PerfectRGB-IRcolorroutersforsub-wavelengthsizeCMOSimagesensorpixels.Adv. Photonics Res.2,2000048(2021)..

Brauers,J.&Aach,T.Acolorfilterarraybasedmultispectralcamera.In12. Workshop Farbbildverarbeitung, 55–64(Ilmenau,2006).

Lin,X.etal.All-opticalmachinelearningusingdiffractivedeepneuralnetworks.Science361,1004–1008(2018)..

Mengu,D.etal.Analysisofdiffractiveopticalneuralnetworksandtheirintegrationwithelectronicneuralnetworks.IEEE J. Sel. Top. Quantum Electron.26,3700114(2020)..

Mengu,D.etal.Misalignmentresilientdiffractiveopticalnetworks.Nanophotonics9,4207–4219(2020)..

Bai,B.J.etal.Toimage,ornottoimage:class-specificdiffractivecameraswithall-opticalerasureofundesiredobjects.eLight2,14(2022)..

Kulce,O.etal.All-opticalinformation-processingcapacityofdiffractivesurfaces.Light Sci. Appl.10,25(2021)..

Kulce,O.etal.All-opticalsynthesisofanarbitrarylineartransformationusingdiffractivesurfaces.Light Sci. Appl.10,196(2021)..

Luo,Y.etal.Computationalimagingwithoutacomputer:seeingthroughrandomdiffusersatthespeedoflight.eLight2,4(2022)..

Mengu,D.&Ozcan,A.All-opticalphaserecovery:diffractivecomputingforquantitativephaseimaging.Adv. Opt. Mater.10,2200281(2022)..

Goi,E.etal.Nanoprintedhigh-neuron-densityopticallinearperceptronsperformingnear-infraredinferenceonaCMOSchip.Light Sci. Appl.10,40(2021)..

Li,J.X.etal.Massivelyparalleluniversallineartransformationsusingawavelength-multiplexeddiffractiveopticalnetwork.Adv. Photonics5,016003(2023)..

Hasegawa,T.etal.Anew0.8μmCMOSimagesensorwithlowRTSnoiseandhighfullwellcapacity.IISW Dig. Tech. Pap.1,24–27(2019)..

Li,J.X.etal.Polarizationmultiplexeddiffractivecomputing:all-opticalimplementationofagroupoflineartransformationsthroughapolarization-encodeddiffractivenetwork.Light Sci. Appl.11,153(2022)..

Li,J.X.etal.Spectrallyencodedsingle-pixelmachinevisionusingdiffractivenetworks.Sci. Adv.7,eabd7690(2021)..

Veli,M.etal.Terahertzpulseshapingusingdiffractivesurfaces.Nat. Commun.12,37(2021)..

Mengu,D.etal.Diffractiveinterconnects:all-opticalpermutationoperationusingdiffractivenetworks.Nanophotonics.https://doi.org/10.1515/nanoph-2022-0358https://doi.org/10.1515/nanoph-2022-0358(2022)..

Matsushima,K.,Schimmel,H.&Wyrowski,F.Fastcalculationmethodforopticaldiffractionontiltedplanesbyuseoftheangularspectrumofplanewaves.J. Opt. Soc. Am. A20,1755–1762(2003)..

Delen,N.&Hooker,B.Free-spacebeampropagationbetweenarbitrarilyorientedplanesbasedonfulldiffractiontheory:afastFouriertransformapproach.J. Opt. Soc. Am. A15,857–867(1998)..

N-BK7|SCHOTTadvancedoptics.http://www.schott.com/shop/advanced-optics/en/Optical-Glass/N-BK7/c/glass-N-BK7http://www.schott.com/shop/advanced-optics/en/Optical-Glass/N-BK7/c/glass-N-BK7(2022).

Kingma,D.P.&Ba,J.Adam:amethodforstochasticoptimization.InProc 3rd International Conference on Learning Representations(ICLR,SanDiego,CA,USA,2015).

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰