Open Source
Explore the latest AI open-source projects from GitHub and HuggingFace.
Explore the latest AI open-source projects from GitHub and HuggingFace.
Pathway is a Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG. Powered by a high-performance Rust engine based on Differential Dataflow, it enables incremental computation while maintaining the ease of Python development. Pathway provides dedicated LLM tooling including wrappers, parsers, embedders, splitters, and an in-memory real-time Vector Index with integrations for LangChain and LlamaIndex. It connects to over 300 data sources via Kafka, PostgreSQL, Google Drive, SharePoint, and Airbyte connectors. Despite being written in Python, the underlying Rust engine enables multithreading, multiprocessing, and distributed computations, solving traditional Python performance limitations. The framework supports both batch and streaming data processing, making it suitable for building real-time RAG applications that stay current as knowledge bases change. With 62,700+ stars and active community support on Discord, Pathway has become a go-to framework for developers building production-grade LLM data pipelines.