EMO2

We propose a novel audio-driven talking head method capable of simultaneously generating highly expressive facial expressions and hand gestures. Unlike existing methods that focus on generating full-body or half-body poses, we investigate the challenges of audio-driven gesture generation and identify the weak correspondence between audio features and full-body gestures as a key limitation.

To address this, we redefine the task as a two-stage process. In the first stage, we generate hand poses directly from audio input, leveraging the stronger correlation between audio signals and hand movements. In the second stage, we employ a diffusion model to synthesize video frames, incorporating the hand poses generated in the first stage to produce realistic facial expressions and body movements.