Riffusion

Riffusion turned Stable Diffusion into a music generator by fine-tuning it on spectrogram images, the visual representation of audio frequencies over time. A text prompt produces a spectrogram, which is then converted into playable audio, and prompt interpolation lets the system morph smoothly between styles for continuous, real-time generation. The repository includes the diffusion pipeline with image conditioning, spectrogram conversion utilities, a command-line interface, a Streamlit playground for interactive experimentation, and a Flask server for inference APIs. It targets Python 3.9 to 3.10 and runs on CUDA GPUs, Apple Silicon, or CPU, with a GPU recommended for real-time use. Released under the MIT license, the open source project is no longer actively maintained, but the code remains a widely referenced demonstration of applying image diffusion to audio, used by musicians, researchers, and hobbyists exploring AI music.