supabase/etl

ETL

Build near-real-time Postgres replication applications in Rust
Documentation · Examples · Issues

ETL is a Rust framework by Supabase for building high-performance, near-real-time data replication apps on Postgres.

This repository contains the open-source Rust framework, destination modules, replicator binary, and developer documentation. If you want the managed product in the Supabase Dashboard, see the canonical Supabase Pipelines documentation for product setup, availability, pricing, and operational guidance.

It sits on top of Postgres logical replication and gives you Rust-native building blocks for copying existing data, streaming ongoing changes, and writing them to your own destination. Run it as a standalone replicator binary or embed it as a library in your own Rust service.

ETL is intentionally cheap to operate: it is one lightweight Rust process on top of Postgres logical replication. You do not need Kafka, Flink, Debezium, or another coordination service to run a pipeline.

What ETL Does

flowchart LR
    Postgres["Postgres publication"] --> ETL["ETL<br/>copy + stream"]
    ETL --> Destination["Destination"]

ETL runs as one process that coordinates an initial sync, continuous replication stream, and a state/schema store for recovery:

1. Initial sync copies the existing rows covered by a Postgres publication.
2. Ongoing replication batches and forwards subsequent inserts, updates, deletes, truncates, and schema events.
3. State recovery lets a durable store resume table state, schema versions, and destination metadata after restarts.

Why ETL?

Requirements

ETL officially supports and tests against PostgreSQL 14, 15, 16, 17, and 18.

• PostgreSQL 15+ is recommended for advanced publication features:
  • Column-level filtering
  • Row-level filtering with WHERE clauses
  • FOR TABLES IN SCHEMA syntax
• PostgreSQL 16+ is required when the ETL replication connection points at a physical read replica. Earlier versions support logical decoding only on the primary.
• PostgreSQL 14 is supported with table-level publication filtering.

For detailed configuration instructions, see the Configure Postgres documentation.

Get Started

ETL is currently installed from Git while we prepare for a crates.io release. Choose the destination features you need.

For DuckLake, external maintenance coordination is selected at runtime with maintenance_mode: disabled, kubernetes, or postgres. The default is disabled; postgres uses the same Postgres catalog connection as DuckLake and stores coordination state in the etl schema.

For a first production deployment, start with the stable BigQuery module:

[dependencies]
etl = { git = "https://github.com/supabase/etl" }
etl-destinations = { git = "https://github.com/supabase/etl", features = ["bigquery"] }
tokio = { version = "1", features = ["full"] }

Then create a pipeline that reads from a Postgres publication and writes to BigQuery.

use etl::{
    config::{
        BatchConfig, InvalidatedSlotBehavior, MemoryBackpressureConfig, PgConnectionConfig,
        PipelineConfig, TableSyncCopyConfig, TcpKeepaliveConfig, TlsConfig,
    },
    pipeline::Pipeline,
    store::PostgresStore,
};
use etl_destinations::bigquery::BigQueryDestination;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let pg = PgConnectionConfig {
        host: "localhost".into(),
        hostaddr: None,
        port: 5432,
        name: "mydb".into(),
        username: "postgres".into(),
        password: Some("password".to_string().into()),
        tls: TlsConfig { enabled: false, trusted_root_certs: String::new() },
        keepalive: TcpKeepaliveConfig::default(),
    };

    let pipeline_id = 1;
    let store = PostgresStore::new(pipeline_id, pg.clone()).await?;
    let destination = BigQueryDestination::new_with_key_path(
        "my-gcp-project".into(),
        "my_dataset".into(),
        "/path/to/service-account-key.json",
        None,
        1,
        pipeline_id,
        store.clone(),
    )
    .await?;

    let config = PipelineConfig {
        id: pipeline_id,
        publication_name: "my_publication".into(),
        pg_connection: pg,
        store_pg_connection: None,
        batch: BatchConfig {
            max_fill_ms: 5000,
            memory_budget_ratio: 0.2,
            max_bytes: 8 * 1024 * 1024,
        },
        table_error_retry_delay_ms: 10_000,
        table_error_retry_max_attempts: 5,
        max_table_sync_workers: 4,
        max_copy_connections_per_table: PipelineConfig::DEFAULT_MAX_COPY_CONNECTIONS_PER_TABLE,
        memory_refresh_interval_ms: 100,
        replication_lag_refresh_interval_ms: 10_000,
        memory_backpressure: Some(MemoryBackpressureConfig::default()),
        table_sync_copy: TableSyncCopyConfig::default(),
        invalidated_slot_behavior: InvalidatedSlotBehavior::default(),
    };

    // Start the pipeline.
    let mut pipeline = Pipeline::new(config, store, destination);
    pipeline.start().await?;

    // Wait for the pipeline indefinitely.
    pipeline.wait().await?;

    Ok(())
}

For a guided walkthrough, start with Your First Pipeline. For runnable destination examples, see etl-examples.

Destinations

ETL is designed to be extensible: you can implement your own destination, or use one of the modules shipped in etl-destinations.

Enable one or more destination modules with crate features:

[dependencies]
etl = { git = "https://github.com/supabase/etl" }
etl-destinations = { git = "https://github.com/supabase/etl", features = ["bigquery", "ducklake"] }

Development

See DEVELOPMENT.md for setup instructions, migration workflows, and development guidelines.

Contributing

We welcome pull requests and GitHub issues. We currently cannot accept new custom destinations unless there is significant community demand, as each destination carries a long-term maintenance cost. We are prioritizing core stability, observability, and ergonomics. If you need a destination that is not yet supported, please start a discussion or issue so we can gauge demand before proposing an implementation.

License

Apache‑2.0. See LICENSE for details.

Made with ❤️ by the Supabase team