Skip to main content
Springer Nature Link
Log in

A bronchoscopic navigation method based on neural radiation fields

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

We introduce a novel approach for bronchoscopic navigation that leverages neural radiance fields (NeRF) to passively locate the endoscope solely from bronchoscopic images. This approach aims to overcome the limitations and challenges of current bronchoscopic navigation tools that rely on external infrastructures or require active adjustment of the bronchoscope.

Methods

To address the challenges, we leverage NeRF for bronchoscopic navigation, enabling passive endoscope localization from bronchoscopic images. We develop a two-stage pipeline: offline training using preoperative data and online passive pose estimation during surgery. To enhance performance, we employ Anderson acceleration and incorporate semantic appearance transfer to deal with the sim-to-real gap between training and inference stages.

Results

We assessed the viability of our approach by conducting tests on virtual bronchscopic images and a physical phantom against the SLAM-based methods. The average rotation error in our virtual dataset is about 3.18\(^\circ \) and the translation error is around 4.95 mm. On the physical phantom test, the average rotation and translation error are approximately 5.14\(^\circ \) and 13.12 mm.

Conclusion

Our NeRF-based bronchoscopic navigation method eliminates reliance on external infrastructures and active adjustments, offering promising advancements in bronchoscopic navigation. Experimental validation on simulation and real-world phantom models demonstrates its efficacy in addressing challenges like low texture and challenging lighting conditions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Mildenhall B, Srinivasan PP, Tancik M, Barron JT, Ramamoorthi R, Ng R (2021) Nerf: representing scenes as neural radiance fields for view synthesis. Commun ACM 65(1):99–106

    Article  Google Scholar 

  2. Schwarz Y, Greif J, Becker HD, Ernst A, Mehta A (2006) Real-time electromagnetic navigation bronchoscopy to peripheral lung lesions using overlaid CT images: the first human study. Chest 129(4):988–994

    Article  PubMed  Google Scholar 

  3. Gildea TR, Mazzone PJ, Karnak D, Meziane M, Mehta AC (2006) Electromagnetic navigation diagnostic bronchoscopy: a prospective study. Am J Respir Crit Care Med 174(9):982–989

    Article  PubMed  PubMed Central  Google Scholar 

  4. Luo X, Mori K (2013) Beyond current guided bronchoscopy: A robust and real-time bronchoscopic ultrasound navigation system. In: Medical Image computing and computer-assisted intervention–MICCAI 2013: 16th international conference, Nagoya, Japan, September 22–26, 2013, Proceedings, Part I 16, pp. 388–395. Springer

  5. Sganga J, Eng D, Graetzel C, Camarillo DB (2019) Autonomous driving in the lung using deep learning for localization. arXiv preprint arXiv:1907.08136

  6. Chien J-C, Lee J-D, Su E, Li S-H (2020) A bronchoscope localization method using an augmented reality co-display of real bronchoscopy images with a virtual 3d bronchial tree model. Sensors 20(23):6997

    Article  PubMed  PubMed Central  Google Scholar 

  7. Gu Y, Gu C, Yang J, Sun J, Yang G-Z (2022) Vision-kinematics interaction for robotic-assisted bronchoscopy navigation. IEEE Trans Med Imaging 41(12):3600–3610. https://doi.org/10.1109/TMI.2022.3191317

    Article  PubMed  Google Scholar 

  8. Zhao C, Shen M, Sun L, Yang G-Z (2019) Generative localization with uncertainty estimation through video-ct data for bronchoscopic biopsy. IEEE Robot Autom Lett 5(1):258–265

    Article  Google Scholar 

  9. Banach A, King F, Masaki F, Tsukada H, Hata N (2021) Visually navigated bronchoscopy using three cycle-consistent generative adversarial network for depth estimation. Med Image Anal 73:102164

    Article  PubMed  PubMed Central  Google Scholar 

  10. Wang C, Oda M, Hayashi Y, Kitasaka T, Itoh H, Honma H, Takebatake H, Mori M, Natori H, Mori K (2023) Anatomy aware-based 2.5 d bronchoscope tracking for image-guided bronchoscopic navigation. Comput Methods Biomech Biomed Eng Imaging Vis 11(4):1122–1129

  11. Fried I, Hoelscher J, Akulian JA, Pizer S, Alterovitz R (2023) Landmark based bronchoscope localization for needle insertion under respiratory deformation. In: 2023 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 6593–6600. IEEE

  12. Deguchi D, Mori K, Feuerstein M, Kitasaka T, Maurer CR Jr, Suenaga Y, Takabatake H, Mori M, Natori H (2009) Selective image similarity measure for bronchoscope tracking based on image registration. Med Image Anal 13(4):621–633

    Article  PubMed  Google Scholar 

  13. Merritt SA, Khare R, Bascom R, Higgins WE (2013) Interactive CT-video registration for the continuous guidance of bronchoscopy. IEEE Trans Med Imaging 32(8):1376–1396

    Article  PubMed  PubMed Central  Google Scholar 

  14. Fielding DI, Bashirzadeh F, Son JH, Todman M, Chin A, Tan L, Steinke K, Windsor MN, Sung AW (2019) First human use of a new robotic-assisted fiber optic sensing navigation system for small peripheral pulmonary nodules. Respiration 98(2):142–150

    Article  PubMed  Google Scholar 

  15. Rojas-Solano JR, Ugalde-Gamboa L, Machuzak M (2018) Robotic bronchoscopy for diagnosis of suspected lung cancer: a feasibility study. J Bronchol Intervent Pulmonol 25(3):168–175

    Article  Google Scholar 

  16. Ghosh S, Patel S (2023) Robotic bronchoscopy for diagnosis of lung nodules using the ion system: a narrative review of the technical aspects and advantages over standard flexible bronchoscopy with electromagnetic navigation. J Vis Surg 9

  17. Ma R, Wang R, Pizer S, Rosenman J, McGill SK, Frahm J-M (2019) Real-time 3d reconstruction of colonoscopic surfaces for determining missing regions. In: Medical image computing and computer assisted intervention–MICCAI 2019: 22nd international conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part V 22, pp 573–582. Springer

  18. Wang C, Oda M, Hayashi Y, Villard B, Kitasaka T, Takabatake H, Mori M, Honma H, Natori H, Mori K (2020) A visual slam-based bronchoscope tracking scheme for bronchoscopic navigation. Int J Comput Assist Radiol Surg 15:1619–1630

    Article  PubMed  Google Scholar 

  19. Recasens D, Lamarca J, Fácil JM, Montiel JMM, Civera J (2021) Endo-depth-and-motion: Reconstruction and tracking in endoscopic videos using depth networks and photometric constraints. IEEE Robot Autom Lett 6(4):7225–7232. https://doi.org/10.1109/LRA.2021.3095528

    Article  Google Scholar 

  20. Gómez-Rodríguez JJ, Lamarca J, Morlana J, Tardós JD, Montiel JMM (2021) Sd-defslam: semi-direct monocular slam for deformable and intracorporeal scenes. In: 2021 IEEE international conference on robotics and automation (ICRA), pp 5170–5177. https://doi.org/10.1109/ICRA48506.2021.9561512

  21. Rodríguez JJG, Montiel JM, Tardós JD (2022) Tracking monocular camera pose and deformation for slam inside the human body. In: 2022 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 5278–5285. IEEE

  22. Chabra R, Lenssen JE, Ilg E, Schmidt T, Straub J, Lovegrove S, Newcombe R (2020) Deep local shapes: learning local sdf priors for detailed 3d reconstruction. In: Computer vision–ECCV 2020: 16th European conference, Glasgow, UK, August 23–28, 2020, proceedings, Part XXIX 16, pp 608–625. Springer

  23. Sitzmann V, Chan E, Tucker R, Snavely N, Wetzstein G (2020) Metasdf: meta-learning signed distance functions. Adv Neural Inf Process Syst 33:10136–10147

    Google Scholar 

  24. Rabby A, Zhang C (2023) Beyondpixels: a comprehensive review of the evolution of neural radiance fields. arXiv preprint arXiv:2306.03000

  25. Chen Z, Yang L, Lai J-H, Xie X (2023) Cunerf: cube-based neural radiance field for zero-shot medical image arbitrary-scale super resolution. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 21185–21195

  26. Corona-Figueroa A, Frawley J, Bond-Taylor S, Bethapudi S, Shum HP, Willcocks CG (2022) Mednerf: medical neural radiance fields for reconstructing 3d-aware ct-projections from a single x-ray. In: 2022 44th Annual international conference of the IEEE engineering in medicine & biology society (EMBC), pp 3843–3848. IEEE

  27. Wang Y, Long Y, Fan SH, Dou Q (2022) Neural rendering for stereo 3d reconstruction of deformable tissues in robotic surgery. In: International Conference on medical image computing and computer-assisted intervention, pp 431–441. Springer

  28. Kajiya JT, Von Herzen BP (1984) Ray tracing volume densities. ACM SIGGRAPH Comput Graph 18(3):165–174

    Article  Google Scholar 

  29. Müller T, Evans A, Schied C, Keller A (2022) Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans Graph (ToG) 41(4):1–15

    Article  Google Scholar 

  30. Lo P, Van Ginneken B, Reinhardt JM, Yavarna T, De Jong PA, Irving B, Fetita C, Ortner M, Pinho R, Sijbers J et al (2012) Extraction of airways from CT (exact’09). IEEE Trans Med Imaging 31(11):2093–2107

    Article  PubMed  Google Scholar 

  31. Fedorov A, Beichel R, Kalpathy-Cramer J, Finet J, Fillion-Robin J-C, Pujol S, Bauer C, Jennings D, Fennessy F, Sonka M et al (2012) 3d slicer as an image computing platform for the quantitative imaging network. Magn Reson Imaging 30(9):1323–1341

    Article  PubMed  PubMed Central  Google Scholar 

  32. Au OK-C, Tai C-L, Chu H-K, Cohen-Or D, Lee T-Y (2008) Skeleton extraction by mesh contraction. ACM Trans Graph (TOG) 27(3):1–10

    Article  Google Scholar 

  33. Sahu M, Strömsdörfer R, Mukhopadhyay A, Zachow S (2020) Endo-sim2real: consistency learning-based domain adaptation for instrument segmentation. In: International conference on medical image computing and computer-assisted intervention, pp 784–794. Springer

  34. Tumanyan N, Bar-Tal O, Bagon S, Dekel T (2022) Splicing vit features for semantic appearance transfer. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 10748–10757

  35. Anderson DG (1965) Iterative procedures for nonlinear integral equations. J ACM (JACM) 12(4):547–560

    Article  Google Scholar 

  36. Walker HF, Ni P (2011) Anderson acceleration for fixed-point iterations. SIAM J Numer Anal 49(4):1715–1735

    Article  Google Scholar 

  37. Schonberger JL, Frahm J-M (2016) Structure-from-motion revisited. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4104–4113

  38. Schönberger JL, Frahm J-M (2016) Structure-from-motion revisited. In: Conference on computer vision and pattern recognition (CVPR)

  39. Engel J, Koltun V, Cremers D (2017) Direct sparse odometry. IEEE Trans Pattern Anal Mach Intell 40(3):611–625

    Article  PubMed  Google Scholar 

  40. Shen M, Gu Y, Liu N, Yang G-Z (2019) Context-aware depth and pose estimation for bronchoscopic navigation. IEEE Robot Autom Lett 4(2):732–739

    Article  Google Scholar 

  41. Qian K, Krimsky WS, Sarkar SA, Deng Y (2020) Efficiency of electromagnetic navigation bronchoscopy and virtual bronchoscopic navigation. Ann Thorac Surg 109(6):1731–1740

    Article  PubMed  Google Scholar 

  42. Nakao M, Kobayashi K, Tokuno J, Chen-Yoshikawa T, Date H, Matsuda T (2021) Deformation analysis of surface and bronchial structures in intraoperative pneumothorax using deformable mesh registration. Med Image Anal 73:102181

    Article  PubMed  Google Scholar 

  43. Zhang C, Yu Z, Zhao S (2021) Path-space differentiable rendering of participating media. ACM Trans Graph (TOG) 40(4):1–15

    Google Scholar 

Download references

Funding

The funding was provided by the NSFC under grants No.62133009 and 92148205, the Natural Science Foundation of Jiangsu Province Major Project under grant BK20232008, Jiangsu Key Research and Development Plan under grant BE2023023-4, the Joint Fund Project 8091B042206 and the Fundamental Research Funds for the Central Universities.

Author information

Authors and Affiliations

  1. State Key Laboratory of Digital Medical Engineering, Jiangsu Key Lab of Robot Sensing and Control, School of Instrument Science and Engineering, Southeast University, Nanjing, China

    Lifeng Zhu, Jianwei Zheng & Aiguo Song

  2. Hanglok-Tech Co., Ltd., Hengqin, China

    Cheng Wang

  3. The First Affiliated Hospital of Soochow University, Suzhou, China

    Junhong Jiang

Authors
  1. Lifeng Zhu
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Jianwei Zheng
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Cheng Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Junhong Jiang
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Aiguo Song
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Lifeng Zhu.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethics approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed consent

This article does not contain patient data.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (pdf 1761 KB)

Supplementary file 2 (mp4 52786 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhu, L., Zheng, J., Wang, C. et al. A bronchoscopic navigation method based on neural radiation fields. Int J CARS 19, 2011–2021 (2024). https://doi.org/10.1007/s11548-024-03243-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-024-03243-7

Keywords