Medical Volumetric Reconstruction with Neural Fields

Abstract

Reconstructing volumetric models from multi-view images is a fundamental task in computer vision and graphics, supporting a wide range of applications in medical imaging, robotics, and extended reality. Historically, volumetric reconstruction has progressed from tomographic reconstruction, which aims to recover physically meaningful volumes for non-intrusive inspection, to inverse rendering, which seeks to reproduce volumetric scenes for photorealistic novel-view synthesis. Traditional voxel-based approaches often yield poor reconstruction quality in challenging settings such as sparse views and dynamic scenes. Recent neural field-based methods have revolutionized inverse rendering through expressive field models and differentiable rendering techniques. However, their potential for tomographic reconstruction remains largely unexplored, and substantial challenges persist when deploying inverse rendering in real-world settings. This thesis explores neural field-based volumetric reconstruction, with a particular emphasis on tomographic reconstruction and demanding medical applications.

The primary contributions of this thesis are organised into three main directions. First, we establish neural field-based tomographic reconstruction as a novel paradigm. In particular, we introduce two foundational frameworks: an implicit field approach, NAF, and an explicit primitive-based approach, R$^2$-Gaussian. While the two methods pursue different design objectives in terms of reconstruction fidelity and computational efficiency, both achieve superior reconstruction quality and demonstrate greater extensibility compared with traditional techniques.

Second, we extend neural field methods beyond controlled experimental settings to real-world medical applications. On the one hand, we adapt R$^2$-Gaussian to digital subtraction angiography, addressing dynamic tomography for blood vessel reconstruction. On the other hand, we explore surgical robotics through inverse rendering by proposing a novel method for reconstructing deformable tissues from highly occluded and motion-constrained endoscopic videos.

Finally, we enhance neural field reconstruction by leveraging large-scale external datasets. This includes a large feedforward model that produces high-fidelity neural fields under extremely sparse-view conditions, as well as a diffusion model-based generative prior that regularizes neural field optimization.

In summary, this thesis advances medical volumetric reconstruction, particularly tomographic reconstruction, by establishing neural fields as a powerful alternative to traditional voxel-based methods. We not only build the theoretical and mathematical foundations of neural field-based reconstruction, but also develop practical solutions that enhance these methods and extend them to real-world applications. Looking ahead, we hope that the ideas and techniques introduced in this thesis will inspire future research in volumetric reconstruction and ultimately contribute to making medical imaging more reliable, efficient, and accessible for improving human healthcare. Show more