MPT

MPT, short for MosaicML Pretrained Transformer, is a family of open decoder-only language models released in 7B and 30B parameter sizes and trained from scratch with commercial use in mind. The 7B base model was trained on 1 trillion tokens of text and code, swaps learned positional embeddings for ALiBi attention biases, and uses FlashAttention for faster training and inference. ALiBi is what enables the long-context variants: MPT-7B-StoryWriter-65k+ was fine-tuned at a 65,000 token context and can extrapolate beyond it. Other variants include MPT-7B-Instruct for instruction following and MPT-7B-Chat for dialogue; the base and StoryWriter models carry the Apache 2.0 license, while the chat model is non-commercial. MosaicML, later acquired by Databricks, positioned the family as a commercially usable alternative at a time when Llama weights were research-only, and teams commonly paired the models with the MosaicML training stack to fine-tune on their own data.