ClickHouse/clickhouse-rs

clickhouse-rs

Official pure Rust typed client for ClickHouse DB.

• Uses serde for encoding/decoding rows.
• Supports serde attributes: skip_serializing, skip_deserializing, rename.
• Uses RowBinaryWithNamesAndTypes or RowBinary formats over HTTP transport.
  • By default, RowBinaryWithNamesAndTypes with database schema validation is used.
  • It is possible to switch to RowBinary, which can potentially lead to increased performance (see below).
  • There are plans to implement Native format over TCP.
• Supports TLS (see native-tls and rustls-tls features below).
• Supports compression and decompression (LZ4, LZ4HC, and ZSTD).
• Provides API for selecting.
• Provides API for inserting.
• Provides API for infinite transactional (see below) inserting.
• Provides mocks for unit testing.

Note: ch2rs is useful to generate a row type from ClickHouse.

Validation

Starting from 0.14.0, the crate uses RowBinaryWithNamesAndTypes format by default, which allows row types validation against the ClickHouse schema. This enables clearer error messages in case of schema mismatch at the cost of performance. Additionally, with enabled validation, the crate supports structs with correct field names and matching types, but incorrect order of the fields, with an additional slight (5-10%) performance penalty.

If you are looking to maximize performance, you could disable validation using Client::with_validation(false). When validation is disabled, the client switches to RowBinary format usage instead.

The downside with plain RowBinary is that instead of clearer error messages, a mismatch between Row and database schema will result in a NotEnoughData error without specific details.

However, depending on the dataset, there might be x1.1 to x3 performance improvement, but that highly depends on the shape and volume of the dataset.

It is always recommended to measure the performance impact of validation in your specific use case. Additionally, writing smoke tests to ensure that the row types match the ClickHouse schema is highly recommended, if you plan to disable validation in your application.

Usage

To use the crate, add this to your Cargo.toml:

[dependencies]
clickhouse = "0.14.2"

[dev-dependencies]
clickhouse = { version = "0.14.2", features = ["test-util"] }

use clickhouse::Client;

let client = Client::default()
    .with_url("http://localhost:8123")
    .with_user("name")
    .with_password("123")
    .with_database("test");

use serde::Deserialize;
use clickhouse::Row;

#[derive(Row, Deserialize)]
struct MyRow<'a> {
    no: u32,
    name: &'a str,
}

async fn example(client: clickhouse::Client) -> clickhouse::error::Result<()> {
    let mut cursor = client
        .query("SELECT ?fields FROM some WHERE no BETWEEN ? AND ?")
        .bind(500)
        .bind(504)
        .fetch::<MyRow<'_>>()?;

    while let Some(row) = cursor.next().await? {
        println!("no: {}, name: {}", row.no, row.name);
    }
    
    Ok(())
}

use serde::Serialize;
use clickhouse::Row;

#[derive(Row, Serialize)]
struct MyRow {
    no: u32,
    name: String,
}

async fn example(client: clickhouse::Client) -> clickhouse::error::Result<()> {
    let mut insert = client.insert::<MyRow>("some").await?;
    insert.write(&MyRow { no: 0, name: "foo".into() }).await?;
    insert.write(&MyRow { no: 1, name: "bar".into() }).await?;
    insert.end().await?;
    Ok(())
}

use serde::Serialize;
use clickhouse::Row;
use clickhouse::inserter::Inserter;
use std::time::Duration;

#[derive(Row, Serialize)]
struct MyRow {
    no: u32,
    name: String,
}

async fn example(client: clickhouse::Client) -> clickhouse::error::Result<()> {
    let mut inserter = client.inserter::<MyRow>("some")
        .with_timeouts(Some(Duration::from_secs(5)), Some(Duration::from_secs(20)))
        .with_max_bytes(50_000_000)
        .with_max_rows(750_000)
        .with_period(Some(Duration::from_secs(15)));
    
    inserter.write(&MyRow { no: 0, name: "foo".into() }).await?;
    inserter.write(&MyRow { no: 1, name: "bar".into() }).await?;
    let stats = inserter.commit().await?;
    if stats.rows > 0 {
        println!(
            "{} bytes, {} rows, {} transactions have been inserted",
            stats.bytes, stats.rows, stats.transactions,
        );
    }
    Ok(())
}

inserter.end().await?;

async fn example(client: clickhouse::Client) -> clickhouse::error::Result<()> {
    client.query("DROP TABLE IF EXISTS some").execute().await?;
    Ok(())
}

Feature Flags

• lz4 (enabled by default) — enables Compression::Lz4. If enabled, Compression::Lz4 is used by default for all queries.
• zstd — enables Compression::Zstd(level). If enabled and lz4 is not, Compression::zstd() is used by default for all queries. Uses enable_http_compression for responses instead of native framing.
• inserter — enables client.inserter().
• test-util — adds mocks. See the example. Use it only in dev-dependencies.
• uuid — adds serde::uuid to work with uuid crate.
• time — adds serde::time to work with time crate.
• chrono — adds serde::chrono to work with chrono crate.
• opentelemetry — propagate OpenTelemetry context to ClickHouse server.

TLS

By default, TLS is disabled and one or more following features must be enabled to use HTTPS urls:

• native-tls — uses native-tls, utilizing dynamic linking (e.g. against OpenSSL).
• rustls-tls — enables rustls-tls-aws-lc and rustls-tls-webpki-roots features.
• rustls-tls-aws-lc — uses rustls with the aws-lc cryptography implementation.
• rustls-tls-ring — uses rustls with the ring cryptography implementation.
• rustls-tls-webpki-roots — uses rustls with certificates provided by the webpki-roots crate.
• rustls-tls-native-roots — uses rustls with certificates provided by the rustls-native-certs crate.

If multiple features are enabled, the following priority is applied:

• native-tls > rustls-tls-aws-lc > rustls-tls-ring
• rustls-tls-native-roots > rustls-tls-webpki-roots

How to choose between all these features? Here are some considerations:

• A good starting point is rustls-tls, e.g. if you use ClickHouse Cloud.
• To be more environment-agnostic, prefer rustls-tls over native-tls.
• Enable rustls-tls-native-roots or native-tls if you want to use self-signed certificates.

Data Types

Usage of all mentioned data types are covered in the following examples:

• Simple ClickHouse data types
• Container-like ClickHouse data types
• Variant data type
• JSON data type

Overview

• (U)Int(8|16|32|64|128) maps to/from corresponding (u|i)(8|16|32|64|128) types or newtypes around them.

• (U)Int256 are supported with convenience wrappers over [u8; 32]: clickhouse::types::Int256 and clickhouse::types::UInt256. See the derive example.

• Float(32|64) maps to/from corresponding f(32|64) or newtypes around them.

• Decimal(32|64|128) maps to/from corresponding i(32|64|128) or newtypes around them. It's more convenient to use fixnum or another implementation of signed fixed-point numbers.

• Boolean maps to/from bool or newtypes around it.

• String maps to/from any string or bytes types, e.g. &str, &[u8], String, Vec<u8> or SmartString. Newtypes are also supported. To store bytes, consider using serde_bytes, because it's more efficient.

  Example

  use serde::{Serialize, Deserialize};
  use clickhouse::Row;
  
  #[derive(Row, Debug, Serialize, Deserialize)]
  struct MyRow<'a> {
      str: &'a str,
      string: String,
      #[serde(with = "serde_bytes")]
      bytes: Vec<u8>,
      #[serde(with = "serde_bytes")]
      byte_slice: &'a [u8],
  }
  

• FixedString(N) is supported as an array of bytes, e.g. [u8; N].

  Example

  use clickhouse::Row;
  use serde::{Serialize, Deserialize};
  #[derive(Row, Debug, Serialize, Deserialize)]
  struct MyRow {
      fixed_str: [u8; 16], // FixedString(16)
  }
  

• Enum(8|16) are supported using serde_repr. You could use #[repr(i8)] for Enum8 and #[repr(i16)] for Enum16.

  Example

  use clickhouse::Row;
  use serde::{Serialize, Deserialize};
  use serde_repr::{Deserialize_repr, Serialize_repr};
  
  #[derive(Row, Serialize, Deserialize)]
  struct MyRow {
      level: Level,
  }
  
  #[derive(Debug, Serialize_repr, Deserialize_repr)]
  #[repr(i8)]
  enum Level {
      Debug = 1,
      Info = 2,
      Warn = 3,
      Error = 4,
  }
  

• UUID maps to/from uuid::Uuid by using serde::uuid. Requires the uuid feature.

  Example

  use serde::{Serialize, Deserialize};
  use clickhouse::Row;
  
  #[derive(Row, Serialize, Deserialize)]
  struct MyRow {
      #[serde(with = "clickhouse::serde::uuid")]
      uuid: uuid::Uuid,
  }
  

• IPv6 maps to/from std::net::Ipv6Addr.

• IPv4 maps to/from std::net::Ipv4Addr by using serde::ipv4.

  Example

  use serde::{Serialize, Deserialize};
  use clickhouse::Row;
  
  #[derive(Row, Serialize, Deserialize)]
  struct MyRow {
      #[serde(with = "clickhouse::serde::ipv4")]
      ipv4: std::net::Ipv4Addr,
  }
  

• Date maps to/from u16 or a newtype around it and represents a number of days elapsed since 1970-01-01. The following external types are supported:

  • time::Date is supported by using serde::time::date, requiring the time feature.
  • chrono::NaiveDate is supported by using serde::chrono::date, requiring the chrono feature.
  Example

  use serde::{Serialize, Deserialize};
  use clickhouse::Row;
  use time::Date;
  use chrono::NaiveDate;
  
  #[derive(Row, Serialize, Deserialize)]
  struct MyRow {
      days: u16,
      #[serde(with = "clickhouse::serde::time::date")]
      date: Date,
      // if you prefer using chrono:
      #[serde(with = "clickhouse::serde::chrono::date")]
      date_chrono: NaiveDate,
  }
  
  

• Date32 maps to/from i32 or a newtype around it and represents a number of days elapsed since 1970-01-01. The following external types are supported:

  • time::Date is supported by using serde::time::date32, requiring the time feature.
  • chrono::NaiveDate is supported by using serde::chrono::date32, requiring the chrono feature.
  Example

  use serde::{Serialize, Deserialize};
  use clickhouse::Row;
  use time::Date;
  use chrono::NaiveDate;
  
  #[derive(Row, Serialize, Deserialize)]
  struct MyRow {
      days: i32,
      #[serde(with = "clickhouse::serde::time::date32")]
      date: Date,
      // if you prefer using chrono:
      #[serde(with = "clickhouse::serde::chrono::date32")]
      date_chrono: NaiveDate,
  
  }
  
  

• DateTime maps to/from u32 or a newtype around it and represents a number of seconds elapsed since UNIX epoch. The following external types are supported:

  • time::OffsetDateTime is supported by using serde::time::datetime, requiring the time feature.
  • chrono::DateTime<Utc> is supported by using serde::chrono::datetime, requiring the chrono feature.
  Example

  use serde::{Serialize, Deserialize};
  use clickhouse::Row;
  use time::OffsetDateTime;
  use chrono::{DateTime, Utc};
  #[derive(Row, Serialize, Deserialize)]
  struct MyRow {
      ts: u32,
      #[serde(with = "clickhouse::serde::time::datetime")]
      dt: OffsetDateTime,
      // if you prefer using chrono:
      #[serde(with = "clickhouse::serde::chrono::datetime")]
      dt_chrono: DateTime<Utc>,        
  }
  

• DateTime64(_) maps to/from i64 or a newtype around it and represents a time elapsed since UNIX epoch. The following external types are supported:

  • time::OffsetDateTime is supported by using serde::time::datetime64::*, requiring the time feature.
  • chrono::DateTime<Utc> is supported by using serde::chrono::datetime64::*, requiring the chrono feature.
  Example

  use serde::{Serialize, Deserialize};
  use clickhouse::Row;
  use time::OffsetDateTime;
  use chrono::{DateTime, Utc};
  
  #[derive(Row, Serialize, Deserialize)]
  struct MyRow {
      ts: i64, // elapsed s/us/ms/ns depending on `DateTime64(X)`
      #[serde(with = "clickhouse::serde::time::datetime64::secs")]
      dt64s: OffsetDateTime,  // `DateTime64(0)`
      #[serde(with = "clickhouse::serde::time::datetime64::millis")]
      dt64ms: OffsetDateTime, // `DateTime64(3)`
      #[serde(with = "clickhouse::serde::time::datetime64::micros")]
      dt64us: OffsetDateTime, // `DateTime64(6)`
      #[serde(with = "clickhouse::serde::time::datetime64::nanos")]
      dt64ns: OffsetDateTime, // `DateTime64(9)`
      // if you prefer using chrono:
      #[serde(with = "clickhouse::serde::chrono::datetime64::secs")]
      dt64s_chrono: DateTime<Utc>,  // `DateTime64(0)`
      #[serde(with = "clickhouse::serde::chrono::datetime64::millis")]
      dt64ms_chrono: DateTime<Utc>, // `DateTime64(3)`
      #[serde(with = "clickhouse::serde::chrono::datetime64::micros")]
      dt64us_chrono: DateTime<Utc>, // `DateTime64(6)`
      #[serde(with = "clickhouse::serde::chrono::datetime64::nanos")]
      dt64ns_chrono: DateTime<Utc>, // `DateTime64(9)`
  }
  
  
  

• Time maps to/from i32 or a newtype around it. The Time data type is used to store a time value independent of any calendar date. It is ideal for representing daily schedules, event times, or any situation where only the time component (hours, minutes, seconds) is important.

  • time:Duration is is supported by using serde::time::*, requiring the time feature.
  • chrono::Duration is supported by using serde::chrono::*, which is an alias to TimeDelta, requiring the chrono feature
  Example

  use serde::{Serialize, Deserialize};
  use clickhouse::Row;
  #[derive(Row, Serialize, Deserialize)]
  struct MyRow {
      #[serde(with = "clickhouse::serde::chrono::time64::secs")]
      t0: chrono::Duration,
      #[serde(with = "clickhouse::serde::chrono::time64::secs::option")]
      t0_opt: Option<chrono::Duration>,
  }
  
  

• Time64(_) maps to/from i64 or a newtype around it. The Time data type is used to store a time value independent of any calendar date. It is ideal for representing daily schedules, event times, or any situation where only the time component (hours, minutes, seconds) is important.

  • time:Duration is is supported by using serde::time::*, requiring the time feature.
  • chrono::Duration is supported by using serde::chrono::*, requiring the chrono feature
  Example

  #[derive(Row, Serialize, Deserialize)]
  struct MyRow {
      #[serde(with = "clickhouse::serde::time::time")]
      t0: Time,
  }
  
  

• Tuple(A, B, ...) maps to/from (A, B, ...) or a newtype around it.

• Array(_) maps to/from any slice, e.g. Vec<_>, &[_], and UUID (using serde::uuid_vec). Newtypes are also supported.

  Example

  use serde::{Serialize, Deserialize};
  use clickhouse::Row;
  
  #[derive(Row, Serialize, Deserialize)]
  struct MyRow {
      #[serde(with = "clickhouse::serde::uuid_vec")]
      uuids: Vec<uuid::Uuid>,
  }
  

• Map(K, V) can be deserialized as HashMap<K, V> or Vec<(K, V)>.

• LowCardinality(_) is supported seamlessly.

• Nullable(_) maps to/from Option<_>. For clickhouse::serde::* helpers add ::option.

  Example

  use clickhouse::Row;
  use serde::{Serialize, Deserialize};
  use std::net::Ipv4Addr;
  #[derive(Row, Serialize, Deserialize)]
  struct MyRow {
      #[serde(with = "clickhouse::serde::ipv4::option")]
      ipv4_opt: Option<Ipv4Addr>,
  }
  

• Nested is supported by providing multiple arrays with renaming.

  Example

  // CREATE TABLE test(items Nested(name String, count UInt32))
  use clickhouse::Row;
  use serde::{Serialize, Deserialize};
  #[derive(Row, Serialize, Deserialize)]
  struct MyRow {
      #[serde(rename = "items.name")]
      items_name: Vec<String>,
      #[serde(rename = "items.count")]
      items_count: Vec<u32>,
  }
  

• Geo types are supported. Point behaves like a tuple (f64, f64), and the rest of the types are just slices of points.

  Example

  use clickhouse::Row;
  use serde::{Serialize, Deserialize};
  
  type Point = (f64, f64);
  type Ring = Vec<Point>;
  type Polygon = Vec<Ring>;
  type MultiPolygon = Vec<Polygon>;
  type LineString = Vec<Point>;
  type MultiLineString = Vec<LineString>;
  
  #[derive(Row, Serialize, Deserialize)]
  struct MyRow {
      point: Point,
      ring: Ring,
      polygon: Polygon,
      multi_polygon: MultiPolygon,
      line_string: LineString,
      multi_line_string: MultiLineString,
  }
  

• Variant data type is supported as a Rust enum. As the inner Variant types are always sorted alphabetically, Rust enum variants should be defined in the exactly same order as it is in the data type; their names are irrelevant, only the order of the types matters. This following example has a column defined as Variant(Array(UInt16), Bool, Date, String, UInt32):

  Example

  use clickhouse::Row;
  use serde::{Serialize, Deserialize};
  use time::Date;
  #[derive(Serialize, Deserialize)]
  enum MyRowVariant {
      Array(Vec<i16>),
      Boolean(bool),
      #[serde(with = "clickhouse::serde::time::date")]
      Date(time::Date),
      String(String),
      UInt32(u32),
  }
  
  #[derive(Row, Serialize, Deserialize)]
  struct MyRow {
      id: u64,
      var: MyRowVariant,
  }
  

• New JSON data type is currently supported as a string when using ClickHouse 24.10+.

• Dynamic data type is not supported for now.

Mocking

The crate provides utils for mocking CH server and testing DDL, SELECT and INSERT queries.

The functionality can be enabled with the test-util feature. Use it only in dev-dependencies.

See the example.

Support Policies

Minimum Supported Rust Version (MSRV)

This project's MSRV is the second-to-last stable release as of the beginning of the current release cycle (0.x.0), where it will remain until the beginning of the next release cycle (0.{x+1}.0).

The MSRV for the 0.14.x release cycle is 1.89.0.

This guarantees that clickhouse-rs will compile with a Rust version that is at least six weeks old, which should be plenty of time for it to make it through any packaging system that is being actively kept up to date.

Beware when installing Rust through operating system package managers, as it can often be a year or more out-of-date. For example, Debian Bookworm (released 10 June 2023) shipped with Rust 1.63.0 (released 11 August 2022).

ClickHouse Versions

The supported versions of the ClickHouse database server coincide with the versions currently receiving security updates.

For the list of currently supported versions, see https://github.com/ClickHouse/ClickHouse/blob/master/SECURITY.md#security-change-log-and-support.