Pathway Live Data Framework: Python Stream Processing

What is Pathway Live Data Framework?

Pathway Live Data Framework is a Python ETL framework whose pipelines are executed by a Rust engine built on differential dataflow, so the same code handles batch jobs, CI tests, and live streaming without a rewrite. It installs as a pip package, connects to sources like Kafka and PostgreSQL, and adds an LLM/RAG toolkit with an in-memory vector index for building pipelines over live documents.

Core Capabilities of Pathway Live Data Framework

• ✓A wide connector set out of the box: Kafka, Google Drive, PostgreSQL, and SharePoint are named directly, and an Airbyte connector reaches more than 300 additional data sources, per the README; you can also write your own Python connector.
• ✓Stateful transformations — joins, windowing, and sorting — run through operators implemented in Rust, and you can drop in any Python function or library alongside them.
• ✓Persistence for computation state, so a pipeline can restart after a crash or update instead of reprocessing everything from the start.
• ✓Time-aware consistency handling: results update automatically when late or out-of-order events arrive, rather than staying stale.
• ✓A Rust engine under the Python API does the multithreading, multiprocessing, and distributed computation, per the README.
• ✓An LLM extension with wrappers, parsers, embedders, and splitters, plus a vector index held in memory for live lookups, with integrations for LlamaIndex and LangChain.
• ✓A monitoring dashboard that tracks per-connector message counts and system latency and folds in log messages once a pipeline is running.

Pathway Live Data Framework Use Cases

• •Real-time ETL off a Kafka topic, using the same Python code you would otherwise run against a static file.
• •Event-driven pipelines with alerting, one of the README's own template categories.
• •Turning unstructured documents into a queryable structure on the fly, per one of the README's templates.
• •Building a private RAG pipeline over live documents with local models such as Ollama and Mistral AI, per the README's templates.
• •Multimodal RAG over live documents using a model like gpt-4o, per another of the README's example templates.

Getting Started with Pathway Live Data Framework

Install with pip: `pip install -U pathway`. Pathway Live Data Framework needs Python 3.10 or newer, and the README lists MacOS and Linux as supported platforms; on other systems it says to run it inside a virtual machine. The README also documents an official pathwaycom/pathway Docker image and a cookiecutter template for scaffolding a new project.

Building a Pipeline with Pathway

A minimal pipeline defines a schema with `pw.Schema`, reads rows through a connector such as `pw.io.csv.read`, applies operations like `.filter()` and `.reduce()` with a reducer such as `pw.reducers.sum`, writes results out with something like `pw.io.jsonlines.write`, and starts the computation with one `pw.run()` call. The README's own example computes a running sum of positive values from a folder of CSV files this way, and the same script structure applies whether that folder gets rows once or continuously. Locally, you run the file like any Python script, `python main.py`, or launch it with `pathway spawn python main.py`; add `--threads 3` to spread the work across threads. That's it.

Comparing Pathway to Other Frameworks

Pathway Live Data Framework FAQs

The problem it solves

Python data teams commonly end up maintaining two versions of the same logic: a pandas or SQL script for batch and historical runs, and a separate Kafka-consumer or JVM-based streaming job for the live version, with the two slowly drifting apart. Pathway Live Data Framework's README frames its reason for existing around collapsing that split, so one Python code path is meant to run unmodified across local development, CI tests, batch jobs, stream replays, and live data streams, with a Rust engine handling execution underneath.

Who should try it — and who should skip

Try Pathway Live Data Framework if you're a Python team building real-time ETL, an alerting pipeline, or a RAG system over live documents, and you don't want to hand-roll a second streaming implementation of logic you already wrote for batch. Skip it if your workload is a straightforward batch job a scheduler like airflow already covers, or if the BSL 1.1 license's commercial terms need more legal certainty than 'free for most commercial purposes' before you build on it.

Related repositories