Latte

Latte applies a DiT-style transformer to video diffusion: clips are encoded into a latent space, cut into spatio-temporal patch tokens, and denoised by blocks that model space and time jointly or in separate factorized stages, with the paper comparing four such variants. The repository holds PyTorch model code, configs, sampling scripts, distributed training via torchrun or PyTorch Lightning, Slurm files and FVD evaluation scripts. Released checkpoints cover 256x256 class-conditional and unconditional models trained on Taichi-HD, UCF101, SkyTimelapse and FaceForensics, plus text-to-video weights, the newest being Latte-1 from May 2024, which can also emit single images. Latte-1 also runs through the LattePipeline class in diffusers 0.30 or newer, where the authors report about 17 GB of GPU memory, falling to roughly 9 GB with 4-bit or 8-bit quantization. Setup uses a supplied Conda environment file aimed at CUDA GPUs. Code and weights are Apache 2.0, which permits commercial use. Published in TMLR 2025, it suits researchers studying transformer-based video generation more than anyone wanting a polished app.