Attentional Ptycho-Tomography (APT) for three-dimensional nanoscale X-ray imaging with minimal data acquisition and computation time

Iksung Kang; Ziling Wu; Yi Jiang; Yudong Yao; Junjing Deng; Jeffrey Klug; Stefan Vogt; George Barbastathis

doi:10.1038/s41377-023-01181-8

Go to Nature Website

Your Location：

Home >

Browse articles >

Attentional Ptycho-Tomography (APT) for three-dimensional nanoscale X-ray imaging with minimal data acquisition and computation time

Original Articles

- Attentional Ptycho-Tomography (APT) for three-dimensional nanoscale X-ray imaging with minimal data acquisition and computation time
- Light: Science & Applications Vol. 12, Issue 6, Pages: 1127-1140(2023)
- Affiliations：
  
  1.Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
  2.Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
  3.Argonne National Laboratory, Lemont, IL 60439, USA
  4.Singapore-MIT Alliance for Research and Technology (SMART) Centre, 1 CREATE Way, Singapore 138602, Singapore
  5.Present address: Department of Molecular and Cell Biology, University of California, Berkeley, CA 94720, USA
  6.Present address: Singapore-MIT Alliance for Research and Technology (SMART) Centre, 1 CREATE Way, Singapore 138602, Singapore
  7.Present address: Center for Transformative Science, ShanghaiTech University, 201210 Shanghai, China
- Author bio：
  
  George Barbastathis (gbarb@mit.edu)
- Funds：
- DOI：10.1038/s41377-023-01181-8
  CLC：
- Published：30 June 2023，
  
  Published Online：30 May 2023，
  
  Received：16 January 2023，
  
  Revised：9 May 2023，
  
  Accepted：10 May 2023
Scan for full text
Kang, I. et al. Attentional Ptycho-Tomography (APT) for three-dimensional nanoscale X-ray imaging with minimal data acquisition and computation time. Light: Science & Applications, 12, 1127-1140 (2023).
DOI：

Kang, I. et al. Attentional Ptycho-Tomography (APT) for three-dimensional nanoscale X-ray imaging with minimal data acquisition and computation time. Light: Science & Applications, 12, 1127-1140 (2023). DOI： 10.1038/s41377-023-01181-8.

摘要

Abstract

Noninvasive X-ray imaging of nanoscale three-dimensional objects

such as integrated circuits (ICs)

generally requires two types of scanning: ptychographic

which is translational and returns estimates of the complex electromagnetic field through the IC; combined with a tomographic scan

which collects these complex field projections from multiple angles. Here

we present Attentional Ptycho-Tomography (APT)

an approach to drastically reduce the amount of angular scanning

and thus the total acquisition time. APT is machine learning-based

utilizing axial self-Attention for Ptycho-Tomographic reconstruction. APT is trained to obtain accurate reconstructions of the ICs

despite the incompleteness of the measurements. The training process includes regularizing priors in the form of typical patterns found in IC interiors

and the physics of X-ray propagation through the IC. We show that APT with ×12 reduced angles achieves fidelity comparable to the gold standard Simultaneous Algebraic Reconstruction Technique (SART) with the original set of angles. When using the same set of reduced angles

then APT also outperforms Filtered Back Projection (FBP)

Simultaneous Iterative Reconstruction Technique (SIRT) and SART. The time needed to compute the reconstruction is also reduced

because the trained neural network is a forward operation

unlike the iterative nature of these alternatives. Our experiments show that

without loss in quality

for a 4.48 × 93.2 × 3.92 µm

IC (

$$\simeq $$

6 × 10

voxels)

APT reduces the total data acquisition and computation time from 67.96 h to 38 min. We expect our physics-assisted and attention-utilizing machine learning framework to be applicable to other branches of nanoscale imaging

including materials science and biological imaging.

关键词

Keywords

references

Hoppe,W.Beugungiminhomogenenprimärstrahlwellenfeld.I.Prinzipeinerphasenmessungvonelektronenbeungungsinterferenzen.Acta Crystallogr. Sect. A25,495–501(1969)..

Hegerl,R.&Hoppe,W.Dynamischetheoriederkristallstrukturanalysedurchelektronenbeugungiminhomogenenprimärstrahlwellenfeld.Ber. Bunsenges. Phys. Chem.74,1148–1154(1970)..

Nellist,P.D.,McCallum,B.C.&Rodenburg,J.M.Resolutionbeyondthe'informationlimit'intransmissionelectronmicroscopy.Nature374,630–632(1995)..

Gerchberg,R.W.Apracticalalgorithmforthedeterminationofphasefromimageanddiffractionplanepictures.Optik35,237–246(1972)..

Faulkner,H.M.L.&Rodenburg,J.M.Movableaperturelenslesstransmissionmicroscopy:anovelphaseretrievalalgorithm.Phys. Rev. Lett.93,023903(2004)..

Rodenburg,J.M.&Faulkner,H.M.L.Aphaseretrievalalgorithmforshiftingillumination.Appl. Phys. Lett.85,4795–4797(2004)..

Thibault,P.etal.High-resolutionscanningX-raydiffractionmicroscopy.Science321,379–382(2008)..

Thibault,P.&Guizar-Sicairos,M.Maximum-likelihoodrefinementforcoherentdiffractiveimaging.New J. Phys.14,063004(2012)..

Radon,J.Onthedeterminationoffunctionsfromtheirintegralsalongcertainmanifolds.MathematischePhysikalische69,262–277(1917)..

Radon,J.Onthedeterminationoffunctionsfromtheirintegralvaluesalongcertainmanifolds.IEEE Trans. Med. Imag.5,170–176(1986)..

Bracewell,R.N.Stripintegrationinradioastronomy.Aust. J. Phys.9,198–217(1956)..

Bracewell,R.N.&Riddle,A.C.Inversionoffan-beamscansinradioastronomy.Astrophys. J.150,427(1967)..

Gordon,R.,Bender,R.&Herman,G.T.Algebraicreconstructiontechniques(ART)forthree-dimensionalelectronmicroscopyandX-rayphotography.J. Theor. Biol.29,471–481(1970)..

Gilbert,P.Iterativemethodsforthethree-dimensionalreconstructionofanobjectfromprojections.J. Theor. Biol.36,105–117(1972)..

Andersen,A.H.&Kak,A.C.Simultaneousalgebraicreconstructiontechnique(SART):asuperiorimplementationoftheartalgorithm.Ultrason. Imaging6,81–94(1984)..

Bouman,C.&Sauer,K.AgeneralizedGaussianimagemodelforedge-preservingmapestimation.IEEE Trans. Image Process.2,296–310(1993)..

Dierolf,M.etal.PtychographicX-raycomputedtomographyatthenanoscale.Nature467,436–439(2010)..

Holler,M.etal.High-resolutionnon-destructivethree-dimensionalimagingofintegratedcircuits.Nature543,402–406(2017)..

Deng,J.J.etal.Thevelociprobe:anultrafasthardX-raynanoprobeforhigh-resolutionptychographicimaging.Rev. Sci. Instrum.90,083701(2019)..

Huang,X.J.etal.Fly-scanptychography.Sci. Rep.5,9074(2015)..

Lempitsky,V.,Vedaldi,A.&Ulyanov,D.Deepimageprior.2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.p.9446-9454(IEEE,2018).

Mataev,G.,Elad,M.&Milanfar,P.DeepRED:deepimagepriorpoweredbyRED.arXivhttps://doi.org/10.48550/arXiv.1903.10176https://doi.org/10.48550/arXiv.1903.10176(2019)..

Kappeler,A.etal.Ptychnet:CNNbasedFourierptychography.2017 IEEE International Conference on Image Processing (ICIP).p.1712-1716(IEEE,2017).

Nguyen,T.etal.DeeplearningapproachforFourierptychographymicroscopy.Opt. Express26,26470–26484(2018)..

Cherukara,M.J.etal.Ai-enabledhigh-resolutionscanningcoherentdiffractionimaging.Appl. Phys. Lett.117,044103(2020)..

Goy,A.etal.Lowphotoncountphaseretrievalusingdeeplearning.Phys. Rev. Lett.121,243902(2018)..

Ronneberger,O.,Fischer,P.&Brox,T.U-net:convolutionalnetworksforbiomedicalimagesegmentation.18th International Conference on Medical Image Computing and Computer-Assisted Intervention.p.234-241(Springer,2015).

Çiçek,Ö.etal.3DU-net:learningdensevolumetricsegmentationfromsparseannotation.18th International Conference on Medical Image Computing and Computer-Assisted Intervention.p.424-432(Springer,2016).

Wang,H.Y.etal.Axial-DeepLab:stand-aloneaxial-attentionforpanopticsegmentation.16th European Conference on Computer Vision.p.108-126(Springer,2020).

Vaswani,A.etal.Attentionisallyouneed.Proceedings of the 31st International Conference on Neural Information Processing Systems.p.6000-6010(CurranAssociatesInc.,2017).

Goy,A.etal.High-resolutionlimited-anglephasetomographyofdenselayeredobjectsusingdeepneuralnetworks.Proc. Natl Acad. Sci. USA116,19848–19856(2019)..

Kang,I.,Goy,A.&Barbastathis,G.Dynamicalmachinelearningvolumetricreconstructionofobjects'interiorsfromlimitedangularviews.Light Sci. Appl.10,74(2021)..

Benesty,J.etal.Pearsoncorrelationcoefficient.inNoiseReductioninSpeech Processing(edsBenesty,J.etal)p.1-4(Springer,2009).

Wang,Z.,Simoncelli,E.P.&Bovik,A.C.Multiscalestructuralsimilarityforimagequalityassessment.The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003.p.1398-1402(IEEE,2003).

Dice,L.R.Measuresoftheamountofecologicassociationbetweenspecies.Ecology26,297–302(1945)..

Pelt,D.M.etal.IntegrationoftomopyandtheASTRAtoolboxforadvancedprocessingandreconstructionoftomographicsynchrotrondata.J. Synchrotron Radiat.23,842–849(2016)..

Chambolle,A.&Pock,T.Afirst-orderprimal-dualalgorithmforconvexproblemswithapplicationstoimaging.J. Math. Imaging Vis.40,120–145(2011)..

Wolf,E.Newtheoryofpartialcoherenceinthespace–frequencydomain.PartI:spectraandcrossspectraofsteady-statesources.J. Opt. Soc. Am. A72,343–351(1982)..

Thibault,P.&Menzel,A.Reconstructingstatemixturesfromdiffractionmeasurements.Nature494,68–71(2013)..

Tsai,E.H.R.etal.X-rayptychographywithextendeddepthoffield.Opt. Express24,29089–29108(2016)..

Suzuki,A.etal.High-resolutionmultisliceX-rayptychographyofextendedthickobjects.Phys. Rev. Lett.112,053903(2014)..

Chen,Z.etal.Electronptychographyachievesatomic-resolutionlimitssetbylatticevibrations.Science372,826–831(2021)..

Baguer,D.O.,Leuschner,J.&Schmidt,M.Computedtomographyreconstructionusingdeepimagepriorandlearnedreconstructionmethods.Inverse Probl.36,094004(2020)..

Kang,I.etal.Accelerateddeepself-supervisedptycho-laminographyforthree-dimensionalnanoscaleimagingofintegratedcircuits.arXivhttps://doi.org/10.48550/arXiv.2304.04597https://doi.org/10.48550/arXiv.2304.04597(2023)..

Helfen,L.etal.High-resolutionthree-dimensionalimagingofflatobjectsbysynchrotron-radiationcomputedlaminography.Appl. Phys. Lett.86,071915(2005)..

Holler,M.etal.Three-dimensionalimagingofintegratedcircuitswithmacro-tonanoscalezoom.Nat. Electron.2,464–470(2019)..

Odstrčil,M.,Menzel,A.&Guizar-Sicairos,M.Iterativeleast-squaressolverforgeneralizedmaximum-likelihoodptychography.Opt. Express26,3108–3123(2018)..

Wakonig,K.etal.Ptychoshelves,aversatilehigh-levelframeworkforhigh-performanceanalysisofptychographicdata.J. Appl. Crystallogr.53,574–586(2020)..

Dong,C.etal.Learningadeepconvolutionalnetworkforimagesuper-resolution.13th European Conference on Computer Vision.p.184-199(Springer,2014).

Dong,C.etal.Imagesuper-resolutionusingdeepconvolutionalnetworks.IEEE Trans. Pattern Anal. Mach. Intell.38,295–307(2016)..

Pelt,D.M.&Sethian,J.A.Amixed-scaledenseconvolutionalneuralnetworkforimageanalysis.Proc. Natl Acad. Sci. USA115,254–259(2018)..

Chen,L.C.etal.Rethinkingatrousconvolutionforsemanticimagesegmentation.arXivhttps://doi.org/10.48550/arXiv.1706.05587https://doi.org/10.48550/arXiv.1706.05587(2017)..

Wu,Z.L.etal.Computational Optical Sensing and Imaging 2022(OpticaPublishingGroup,2022).

He,K.M.etal.Deepresiduallearningforimagerecognition.2016 IEEE Conference on Computer Vision and Pattern Recognition.p.770-778(IEEE,2016).

Li,S.etal.Imagingthroughglassdiffusersusingdenselyconnectedconvolutionalnetworks.Optica5,803–813(2018)..

Kingma,D.P.&Ba,J.Adam:amethodforstochasticoptimization.3rd International Conference on Learning Representations.(ICLR,2015).

Cheng,B.W.etal.Panoptic-DeepLab:asimple,strong,andfastbaselineforbottom-uppanopticsegmentation.2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition.p.12472-12482(IEEE,2020).

Liu,W.,Rabinovich,A.&Berg,A.C.ParseNet:lookingwidertoseebetter.arXivhttps://doi.org/10.48550/arXiv.1506.04579https://doi.org/10.48550/arXiv.1506.04579(2015)..

Li,M.etal.Efficientmini-batchtrainingforstochasticoptimization.Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.p.661-670(ACM,2014).

Milletari,F.,Navab,N.&Ahmadi,S.A.V-net:fullyconvolutionalneuralnetworksforvolumetricmedicalimagesegmentation.2016 Fourth International Conference on 3D Vision (3DV).p.565-571(IEEE,2016).

Zou,K.H.etal.Statisticalvalidationofimagesegmentationqualitybasedonaspatialoverlapindex1:scientificreports.Acad. Radiol.11,178–189(2004)..

Koho,S.etal.Fourierringcorrelationsimplifiesimagerestorationinfluorescencemicroscopy.Nat. Commun.10,3103(2019)..

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

⁰