Artificial Intelligence Generated Content on Spatio-Temporal Modeling

Abstract

Spatio-temporal relationships are fundamental to many real-world phenomena. Spatio-temporal modeling is also integral to Artificial Intelligence Generated Content (AIGC). Spatio-temporal modeling is essential for various applications, such as motion prediction and video restoration. In this thesis, we address two distinct yet deeply interconnected tasks: hand motion prediction and turbulence-degraded video restoration. To address these tasks, we aim to investigate the following challenges: 1) how to develop efficient and effective models for spatio-temporal data while extracting and disentangling meaningful information across time and space, 2) how to leverage the decoupled motion patterns to predict diverse and physically plausible motion sequences, and 3) how to maintain visual structure consistency throughout the entire video.

In the first part of this thesis, we establish a robust spatio-temporal modeling framework for observed motion sequences. This framework is designed to predict diverse yet plausible future motions. A key challenge in motion prediction lies in decouplin motion patterns from observed sequences and consistently leveraging these patterns for accurate prediction. To address this, we propose PromptFDDM, which utilizes future motion data to enhance spatio-temporal modeling. The training process consists of two stages. In the first stage, real future motion data guides the Spatio-Temporal Extractor Network (STEN ), a framework designed to extract spatio-temporal dependencies and motion patterns. In the second stage, Diffusion Models (DM), a class of generative models that iteratively refine data distributions through a denoising process, generate reference data that sulbstitutes real future motion during inference, further refining the STEN's predictions. Additionally, prompt learning is introduced in the second stage to help the DM better capture and utiize spatio-temporal patterns from observed motion sequences, thereby improving the quality of the generated inference data.

In the second part of this thesis, we focus on extracting and disentangling the stable visual structure from turbulence-degraded video. The goal is to restore the visual structure and mitigate the degradation effects. A critical challenge in turbulence mitigation tasks lies in maintaining temporal consistency to prevent unstable and blurred visual artifacts. To address this challenge, we propose TMamba, a novel framework that capitalizes on the linear complexity of state-space models to process spatio-temporal data efficiently. The core component of TMamba is the Spatial/Channel-Temporal (SCT) block, a core architectural component designed to process spatio-temporal data. This block comprises two specialized modules: the Spatio-Temporal State-Space (STSS) module, which captures long-range dependencies in spatio-temporal data, and the Channel-Temporal Fusion Attention (CTFA) module, which models interactions across temporal and channel dimensions. These modules ensure comprehensive modeling of spatio-temporal dependencies and channel-temporal interactions. Furthermore, a weight-sharing technique is implemented to reduce model complexity and enhance scalability. This approach provides an effective solution to the challenges associated with processing spatio-temporal data in turbulence mitigation tasks.

In conclusion, this thesis addresses two critical tasks in spatio-temporal modeling: PromptFDDM for hand motion prediction and TMamba for turbulence mitigation. Despite their distinct applications, both methods share a common foundation in spatio-temporal modeling, which is efficient and effective in extracting and disentangling meaningful information from spatio-temporal data. They demonstrate significant advancements in addressing spatio-temporal modeling challenges within their respective domains. Furthermore, they provide valuable insights and contribute to the progress of both motion prediction and video restoration. Show more