MuseTalk

Tencent Music's Lyra Lab built MuseTalk to redraw the mouth area of a face so it follows a new speech track, useful for dubbing footage or animating a still portrait. Rather than running many diffusion steps, it fills in the lower face in one pass inside a frozen VAE latent space, reusing a UNet design taken from Stable Diffusion 1.4 and feeding it Whisper-tiny audio features through cross-attention. Version 1.5 added GAN, perceptual, and sync losses for crisper output, and in April 2025 the team opened the two-stage training pipeline along with preprocessing scripts. The repo includes batch and live inference scripts, a Gradio app, and weight download helpers for Linux and Windows. The face crop is 256 by 256 pixels, so close-ups benefit from a restorer such as GFPGAN. The authors report 30 fps or more on a V100 and show a 4 GB RTX 3050 Ti laptop finishing an 8 second clip in roughly five minutes at fp16. Code is MIT and the weights are cleared for commercial work, though bundled parts like Whisper, DWPose, and S3FD keep their own terms. A sensible pick for avatar, localization, and virtual presenter projects.