Audio-Visual Scene Reconstruction of Geometry and Acoustic Fields

Abstract

Reconstructing 3D scenes from sensor observations is an important problem in computer vision and embodied AI, with broad applications in augmented and virtual reality (AR/VR), robotics, telepresence, and immersive media. As real-world environments are spatially complex and rich in audio-visual information, faithful scene reconstruction requires modeling not only the geometric structure of the scene but also its visual and acoustic representations. In this thesis, we investigate audio-visual scene reconstruction, aiming to recover 3D scene geometry and acoustic fields from multi-view audio-visual recordings. While recent deep learning approaches have achieved promising results, critical challenges remain. At the scene scale, light propagates primarily along line-of-sight paths, leading to hard, near-binary visual occlusions across multiple views, whereas sound, with its longer wavelengths and more extensive reflective interactions, exhibits softer acoustic occlusions and pronounced multi-path propagation. We therefore focus on three key challenges in this area: (1) visual occlusions, (2) acoustic occlusions, (3) multi-path sound propagation, and present a series of methods that advance this line of research in audio-visual scene reconstruction. First, we propose a visibility-aware online reconstruction framework that reconstructs 3D scene geometry from monocular video. To address visual occlusions, we explicitly infer view-dependent voxel visibility and predict aggregation weights for multi-view feature fusion. We also develop a local sparsification strategy to preserve small structures and refine geometry in a coarse-to-fine manner with residual learning, yielding more complete surface reconstructions with finer details. Second, we introduce a scene occlusion-aware neural acoustic field that models the occlusion effects of scene geometry in sound propagation to enhance acoustic field learning. We derive a global acoustic prior via distance-aware parametric propagation and adapt it with room geometry recovered from visual frames to extract local acoustic features. With a direction-aware attention mechanism, our approach generates spatial audio with improved audio quality and binaural perception. Third, we present a physics-guided novel-view acoustic synthesis framework that further incorporates multi-path sound propagation modeling into the learning-based networks. By tracing geometry-dependent early reflections based on reconstructed scene layouts and estimating statistical late reverberation using predicted scene acoustic characteristics, our framework extracts physically meaningful acoustic features and synthesizes more accurate and realistic novel-view spatial audio. In summary, this thesis presents several novel ideas that advance the state of the art in audio-visual scene reconstruction, achieving superior performance in reconstructing 3D scene geometry and acoustic fields, thereby enabling more immersive AR/VR experiences and reliable perception for embodied systems. Show more