Pathway

Pathway unifies batch and streaming data processing behind a single Python API, executing dataflows on a Rust engine built on Differential Dataflow so results update incrementally as new events arrive. That design lets the same code serve ETL jobs, real-time analytics, and live RAG pipelines whose indexes stay in sync with sources such as Kafka, PostgreSQL, SharePoint, Google Drive, and S3-compatible stores, with hundreds of additional sources reachable through Airbyte connectors. LangChain and LlamaIndex integrations are provided, and the companion llm-app repository ships templates for production RAG over live document stores. Installation is pip install pathway on Python 3.10 or newer for Linux and macOS, and the maintainers publish benchmarks showing higher throughput than Flink, Spark, and Kafka Streams on streaming workloads. The source is available under the Business Source License 1.1, which allows free use but is not an OSI-approved open source license; each release converts to Apache-2.0 after four years, and enterprise tiers add support and scaling features.