Hugging Face Datasets

Hugging Face Datasets gives machine learning practitioners one-line access to hundreds of thousands of datasets on the Hugging Face Hub through a single load_dataset call. Built on Apache Arrow, it uses zero-copy memory-mapped storage so datasets larger than RAM remain usable, caches processed results to avoid recomputation, and converts natively to NumPy, pandas, PyTorch, TensorFlow, JAX, and Spark. Preprocessing runs through a map method with multiprocess support, and a streaming mode iterates over remote data without downloading it. The library reads CSV, JSON, Parquet, Arrow, XML, text, image, audio, video, and PDF formats, spanning text, vision, audio, and agent trace data in over 400 languages and dialects. Two core classes cover the main access patterns: Dataset for random access over memory-mapped data and IterableDataset for lazy large-scale streaming. Apache 2.0 licensed, it is a standard component of training and evaluation pipelines across the ML ecosystem.