AnyTalk: Speech Animation for Arbitrary Characters
Leveraging a Video Generation Model

Kwan Yun1*, Serin Yoon1,2*, Sunjin Jung1, Jung Eun Yoo1, Inyup Lee1, Junyong Noh1
1Visual Media Lab, KAIST, 2University of Toronto
(*equal contribution)
IEEE TVCG 2026
Paper Supplementary arXiv Code

Abstract

We present AnyTalk, a novel method for generating 3D speech animations for arbitrary characters without requiring any animation data. While existing audio-driven 3D speech animation methods rely on character-specific training data or laborious rigging/re-meshing, AnyTalk circumvents these limitations by leveraging recent video diffusion models trained on extensive video datasets. We first adapt a pre-trained video diffusion model to a target character through our Character-specific Fine-tuning (CsF) technique. By fine-tuning on rendered images of the 3D character paired with zeroed-out audio embeddings (representing “no motion”), we eliminate the need for animation data while preserving the motion prior of large-scale video diffusion model. We then uplift the resulting talking-head video into a 3D speech animation by estimating blendshape parameters through a proposed optimization process. AnyTalk enables lip-synced animations across diverse face meshes and blendshape configurations, significantly reducing manual effort and data requirements. We further enhance usability by distilling AnyTalk into a streamlined network, AnyTalkRT, thereby enabling real-time performance. By leveraging talking-head video generation, our method broadens access to audio-driven speech animation technology for arbitrary characters. The code will be made publicly available.

Method

AnyTalk pipeline overview

AnyTalk leverages an audio-driven video diffusion model to generate 3D speech animation for arbitrary characters. It follows a two-stage pipeline: generating a talking-head video from a rendered image of the character, then uplifting that video into blendshape parameters. No animation data is required at any point in the process.

Character-specific Fine-tuning (CsF)

Character-specific Fine-tuning process

Applying a pre-trained talking-head generation model (Hallo) directly to a 3D character produces visual mismatches such as unnatural clothing, distorted chins, or a drift toward generic human appearance. CsF addresses this by rendering an image of the character for each activated blendshape, duplicating it into a still-image video, and pairing it with a zeroed-out audio embedding representing "no motion." During fine-tuning, the audio, motion, and ReferenceNet attention layers stay frozen, and only the spatial residual network is trained. This lets the model learn the character's appearance without overwriting its pre-trained motion prior.

Video Inference

Inference of the personalized model

At inference, the fine-tuned model takes real speech audio instead of the zero-motion condition used during training. Scaling weights are applied to the pose, expression, and lip attention modules (wpose=0, wexp=1, wlip=2), keeping the head pose stable while driving dynamic lip motion. This stability significantly improves the reliability of the subsequent blendshape optimization stage.

Blendshape Optimization

Blendshape optimization process

Blendshape parameters are estimated from the generated 2D video. Talk-related landmark vertices are identified once via ray casting on the neutral mesh, then expression-invariant landmarks are used to estimate a homography that compensates for small frame-to-frame head shifts. The final parameters are optimized using a talk landmark loss, an asymmetric mouth-opening loss that enforces correct mouth aperture, and a regularization loss to keep unrelated blendshapes untouched.

Results

AnyTalk produces natural, well lip-synced animations across characters with widely varying mesh structures, vertex counts, and artistic styles, from stylized to photorealistic, as shown in the videos above.

Applications

Real-Time Distillation

Distillation process from AnyTalk to AnyTalkRT

Per-frame optimization takes about 3.12 seconds, which limits its use for interactive applications. To close this gap, we distill AnyTalk into AnyTalkRT, a lightweight network that predicts blendshape parameters directly from audio. The student is trained on around 1,600 AnyTalk-generated animations using a feature-matching loss and a blendshape reconstruction loss, learning to mimic the teacher's output without running the optimization loop. AnyTalkRT reaches 9.09 ms per frame (110 FPS) in full precision, trading a modest amount of lip-sync accuracy for real-time speed.

Extension to Different Video Generation Model

CsF and the optimization pipeline are not tied to a single video generator. We repeated the same procedure on MEMO in place of Hallo, fine-tuning its spatial layers with the same zeroed-out audio embedding and applying the same landmark-based optimization afterward. The resulting AnyTalkMEMO reproduces the same character-faithful, lip-synced behavior as our Hallo-based model, showing that AnyTalk transfers cleanly across different audio-driven video diffusion backbones.