Abstract
We propose in this paper a texture-invariant 2D keypoints descriptor specifically designed for matching preoperative Magnetic Resonance (MR) images with intraoperative Ultrasound (US) images. We introduce a matching-by-synthesis strategy, where intraoperative US images are synthesized from MR images accounting for multiple MR modalities and intraoperative US variability. We build our training set by enforcing keypoints localization over all images then train a patient-specific descriptor network that learns texture-invariant discriminant features in a supervised contrastive manner, leading to robust keypoints descriptors. Our experiments on real cases with ground truth show the effectiveness of the proposed approach, outperforming the state-of-the-art methods and achieving \(80.35\%\) matching precision on average.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Baruch, E.B., Keller, Y.: Joint detection and matching of feature points in multimodal images. IEEE Transactions on Pattern Analysis and Machine Intelligence 44(10), 6585–6593 (2021)
Christy, D., Moses, C.J.: Retinal image registration feature descriptors-a survey. In: 2014 International Conference on Electronics and Communication Systems (ICECS). pp. 1–5. IEEE (2014)
DeTone, D., Malisiewicz, T., Rabinovich, A.: Superpoint: Self-supervised interest point detection and description. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops. pp. 224–236 (2018)
Dixon, L., Lim, A., Grech-Sollars, M., Nandi, D., Camp, S.: Intraoperative ultrasound in brain tumor surgery: a review and implementation guide. Neurosurgical Review 45(4), 2503–2515 (2022)
Dorent, R., Haouchine, N., Kogl, F., Joutard, S., Juvekar, P., Torio, E., Golby, A.J., Ourselin, S., Frisken, S., Vercauteren, T., et al.: Unified brain mr-ultrasound synthesis using multi-modal hierarchical representations. In: International conference on medical image computing and computer-assisted intervention. pp. 448–458. Springer (2023)
Evan, M.Y., Wang, A.Q., Dalca, A.V., Sabuncu, M.R.: Keymorph: Robust multi-modal affine registration via unsupervised keypoint detection. In: Medical Imaging with Deep Learning (2021)
Ferrante, E., Paragios, N.: Slice-to-volume medical image registration: A survey. Medical image analysis 39, 101–123 (2017)
Gonzalez-Darder, J.M.: State of the Art of the Craniotomy in the Early Twenty-First Century and Future Development, pp. 421–427. Springer International Publishing, Cham (2019)
Haouchine, N., Juvekar, P., Nercessian, M., Wells III, W.M., Golby, A., Frisken, S.: Pose estimation and non-rigid registration for augmented reality during neurosurgery. IEEE Transactions on Biomedical Engineering 69(4), 1310–1317 (2022)
Heinrich, M.P., Jenkinson, M., Bhushan, M., Matin, T., Gleeson, F.V., Brady, M., Schnabel, J.A.: Mind: Modality independent neighbourhood descriptor for multi-modal deformable registration. Medical image analysis 16(7), 1423–1435 (2012)
Jiang, X., Ma, J., Xiao, G., Shao, Z., Guo, X.: A review of multimodal image matching: Methods and applications. Information Fusion 73, 22–71 (2021)
Joutard, S., Dorent, R., Ourselin, S., Vercauteren, T., Modat, M.: Driving points prediction for abdominal probabilistic registration. In: International Workshop on Machine Learning in Medical Imaging. pp. 288–297. Springer (2022)
Juvekar, P., Dorent, R., Kögl, F., Torio, E., Barr, C., Rigolo, L., Galvin, C., Jowkar, N., Kazi, A., Haouchine, N., et al.: Remind: The brain resection multimodal imaging database. medRxiv (2023)
Kumar, A., Kim, J., Cai, W., Fulham, M., Feng, D.: Content-based medical image retrieval: a survey of applications to multidimensional and multimodality data. Journal of digital imaging 26, 1025–1039 (2013)
Lindenberger, P., Sarlin, P.E. and Pollefeys, M.: Lightglue: Local feature matching at light speed. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 17627–17638 (2023)
Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International journal of computer vision 60, 91–110 (2004)
Luo, J., Toews, M., Machado, I., Frisken, S., Zhang, M., Preiswerk, F., Sedghi, A., Ding, H., Pieper, S., Golland, P., Golby, A., Sugiyama, M., Wells III, W.M.: A feature-driven active framework for ultrasound-based brain shift compensation. In: MICCAI 2018. pp. 30–38 (2018)
Machado, I., Toews, M., Luo, J., Unadkat, P., Essayed, W., George, E., Teodoro, P., Carvalho, H., Martins, J., Golland, P., Pieper, S., Frisken, S., Golby, A., III, W.: Non-rigid registration of 3d ultrasound for neurosurgery using automatic feature detection and matching. International Journal of Computer Assisted Radiology and Surgery 13 (06 2018)
Paulus, C.J., Haouchine, N., Kong, S.H., Soares, R.V., Cazier, D., Cotin, S.: Handling topological changes during elastic registration: Application to augmented reality in laparoscopic surgery. International journal of computer assisted radiology and surgery 12, 461–470 (2017)
Sanai, N., Polley, M.Y., McDermott, M.W., Parsa, A.T., Berger, M.S.: An extent of resection threshold for newly diagnosed glioblastomas: Clinical article. Journal of Neurosurgery JNS 115(1), 3–8 (2011)
Talbot, H., Haouchine, N., Peterlik, I., Dequidt, J., Duriez, C., Delingette, H., Cotin, S.: Surgery Training, Planning and Guidance Using the SOFA Framework. In: Hege, H.C., Ropinski, T. (eds.) Eurographics 2015 - Dirk Bartz Prize. The Eurographics Association (2015). 10.2312/egm.20151028
Verdie, Y., Yi, K., Fua, P., Lepetit, V.: Tilde: A temporally invariant learned detector. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5279–5288 (2015)
Wu, M., Goodman, N.: Multimodal Generative Models for Scalable Weakly-Supervised Learning. NeurIPS 31 (2018)
Acknowledgement
This work was supported by the National Institutes of Health grants R01EB032387, R01EB034223, R03EB033910, and K25EB035166.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Rasheed, H. et al. (2025). Learning to Match 2D Keypoints Across Preoperative MR and Intraoperative Ultrasound. In: Gomez, A., Khanal, B., King, A., Namburete, A. (eds) Simplifying Medical Ultrasound. ASMUS 2024. Lecture Notes in Computer Science, vol 15186. Springer, Cham. https://doi.org/10.1007/978-3-031-73647-6_8
Download citation
DOI: https://doi.org/10.1007/978-3-031-73647-6_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-73646-9
Online ISBN: 978-3-031-73647-6
eBook Packages: Computer ScienceComputer Science (R0)
