LatentSync

LatentSync approaches lip sync as end-to-end generation in latent space: Whisper turns mel spectrograms into audio embeddings that condition a Stable Diffusion U-Net through cross-attention layers, so the model learns audio-visual correlations directly rather than going through the intermediate motion representations used by Wav2Lip-era tools. ByteDance has iterated on it publicly, with version 1.5 adding temporal layers, better performance on Chinese-language video, and inference in 8 GB of VRAM, and version 1.6 retraining at 512x512 resolution to fix output blurriness at the cost of needing 18 GB. Training scripts are included and require 20 to 55 GB of VRAM depending on stage and resolution. Setup is a single setup_env.sh script that installs dependencies and pulls checkpoints from Hugging Face, and the codebase builds on AnimateDiff with components borrowed from MuseTalk and StyleSync. Released under Apache-2.0 with about 5,900 GitHub stars, it is widely used for video dubbing, translation re-sync, and talking-head production in both photorealistic and anime styles.