ranaroussi/pystore

PyStore - Fast data store for Pandas timeseries data

PyStore is a simple (yet powerful) datastore for Pandas dataframes, and while it can store any Pandas object, it was designed with storing timeseries data in mind.

It's built on top of Pandas, Numpy, Dask, and Parquet (via pyarrow), to provide an easy to use datastore for Python developers that can easily query millions of rows per second per client.

New in 2025 Release (PR #77):

• MultiIndex Support - Store and retrieve DataFrames with Pandas MultiIndex
• Complex Data Types - Full support for Timedelta, Period, Interval, Categorical dtypes
• Timezone-Aware Operations - Proper handling of timezone data with UTC storage
• Async/Await Support - Non-blocking I/O operations for better performance
• Data Validation Framework - Extensible validation rules for data integrity
• Schema Evolution - Handle schema changes over time with flexible strategies
• Transaction Support - Atomic operations with rollback capabilities
• Performance Optimizations - Streaming operations and memory management

Performance Enhancements (Phase 3 Release):

• Streaming Operations - Memory-efficient append for datasets larger than RAM
• Batch Processing - 5-10x faster parallel read/write operations
• Intelligent Partitioning - Automatic time-based and size-based partitioning
• Memory Management - 70-90% memory reduction with monitoring and optimization
• Metadata Caching - 100x faster metadata access with TTL cache
• Query Optimization - Column selection and predicate pushdown at storage level

Performance improvements include:

• Append 1M rows: 3.75x faster, 90% less memory
• Batch operations: 6x faster for multiple items
• Column selection: 4x faster when reading subset of columns
• Filtered reads: 8x faster with predicate pushdown

Check out this blog post for the reasoning and philosophy behind PyStore, as well as a detailed tutorial with code examples.

Follow this PyStore tutorial in Jupyter notebook format.

Quickstart

Install PyStore

Install using pip:

pip install pystore --upgrade --no-cache-dir

Install using conda:

conda install -c ranaroussi pystore

INSTALLATION NOTE: If you don't have Snappy installed (compression/decompression library), you'll need to install it first.

Using PyStore

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import pystore
import yfinance as yf

# Set storage path (optional)
# Defaults to `~/pystore` or `PYSTORE_PATH` environment variable (if set)
pystore.set_path("~/pystore")

# List stores
pystore.list_stores()

# Connect to datastore (create it if not exist)
store = pystore.store("mydatastore")

# List existing collections
store.list_collections()

# Access a collection (create it if not exist)
collection = store.collection("NASDAQ")

# List items in collection
collection.list_items()

# Load some data from yfinance
aapl = yf.download("AAPL", multi_level_index=False)

# Store the first 100 rows of the data in the collection under "AAPL"
collection.write("AAPL", aapl[:100], metadata={"source": "yfinance"})

# Reading the item's data
item = collection.item("AAPL")
data = item.data  # <-- Dask dataframe (see dask.pydata.org)
metadata = item.metadata
df = item.to_pandas()

# Append the rest of the rows to the "AAPL" item
collection.append("AAPL", aapl[100:])

# Reading the item's data
item = collection.item("AAPL")
data = item.data
metadata = item.metadata
df = item.to_pandas()


# --- Query functionality ---

# Query available symbols based on metadata
collection.list_items(some_key="some_value", other_key="other_value")


# --- Snapshot functionality ---

# Snapshot a collection
# (Point-in-time named reference for all current symbols in a collection)
collection.create_snapshot("snapshot_name")

# List available snapshots
collection.list_snapshots()

# Get a version of a symbol given a snapshot name
collection.item("AAPL", snapshot="snapshot_name")

# Delete a collection snapshot
collection.delete_snapshot("snapshot_name")


# ...


# Delete the item from the current version
collection.delete_item("AAPL")

# Delete the collection
store.delete_collection("NASDAQ")

Advanced Features

Async Operations:

import asyncio
from pystore import async_pystore

async def async_example():
    async with async_pystore.store("mydatastore") as store:
        async with store.collection("NASDAQ") as collection:
            # Async write
            await collection.write("AAPL", df)
            # Async read
            df = await collection.item("AAPL").to_pandas()

asyncio.run(async_example())

Data Validation:

from pystore import create_validator, ColumnExistsRule, RangeRule

# Create a validator
validator = create_validator([
    ColumnExistsRule(["Open", "High", "Low", "Close"]),
    RangeRule("Close", min_value=0),
])

# Apply validator to collection
collection.set_validator(validator)

Schema Evolution:

from pystore import SchemaEvolution, EvolutionStrategy

# Enable schema evolution
evolution = collection.enable_schema_evolution(
    "AAPL",
    strategy=EvolutionStrategy.FLEXIBLE,
)

# Schema changes are handled automatically during append
collection.append("AAPL", new_data_with_extra_columns)

Complex Data Types:

# DataFrames with Period, Interval, Categorical types
df = pd.DataFrame({
    "period": pd.period_range("2024-01", periods=12, freq="M"),
    "interval": pd.IntervalIndex.from_tuples([(0, 1), (1, 2)]),
    "category": pd.Categorical(["A", "B", "A"]),
    "nested": [{"key": "value"}, [1, 2, 3], None],
})
collection.write("complex_data", df)

Performance Features:

# Streaming append for large datasets
def data_generator():
    for chunk in pd.read_csv("huge_file.csv", chunksize=100000):
        yield chunk

collection.append_stream("large_data", data_generator())

# Batch operations
items_to_write = {
    "item1": df1,
    "item2": df2,
    "item3": df3,
}
collection.write_batch(items_to_write, parallel=True)

# Read multiple items efficiently
results = collection.read_batch(["item1", "item2", "item3"])

# Memory-optimized reading
from pystore.memory import optimize_dataframe_memory, read_in_chunks

# Optimize DataFrame memory usage
df = collection.item("large_item").to_pandas()
df_optimized = optimize_dataframe_memory(df)  # Up to 70% memory reduction

# Read in chunks for processing
for chunk in read_in_chunks(collection, "large_item", chunk_size=50000):
    # Process chunk - automatically garbage collected
    process(chunk)

Query Optimization:

# Column selection - read only what you need
item = collection.item("data")
df = item.to_pandas(columns=["price", "volume"])  # 4x faster for subset

# Filter at storage level
df = item.to_pandas(filters=[("price", ">", 100)])  # 8x faster

Using Dask schedulers

PyStore supports using Dask distributed.

To use a local Dask scheduler, add this to your code:

from dask.distributed import LocalCluster
pystore.set_client(LocalCluster())

To use a distributed Dask scheduler, add this to your code:

pystore.set_client("tcp://xxx.xxx.xxx.xxx:xxxx")
pystore.set_path("/path/to/shared/volume/all/workers/can/access")

Concepts

PyStore provides namespaced collections of data. These collections allow bucketing data by source, user or some other metric (for example, frequency: End-Of-Day, Minute Bars, etc.). Each collection (or namespace) maps to directory containing partitioned parquet files for each item (e.g., symbol).

A good practice it to create collections that may look something like this:

• collection.EOD
• collection.ONEMINUTE

Requirements

• Python >= 3.8
• Pandas >= 2.0
• Numpy >= 1.20
• Dask >= 2023.1
• PyArrow >= 10.0 (Parquet engine)
• Snappy (Google's compression/decompression library)
• multitasking
• pytest-asyncio (for async testing)

PyStore was tested to work on *nix-like systems, including macOS.

Dependencies

PyStore utilizes Snappy, a fast and efficient compression/decompression library developed by Google. You'll need to install Snappy on your system before installing PyStore.

See the python-snappy GitHub repo for more information.

*nix Systems:

• APT: sudo apt-get install libsnappy-dev
• RPM: sudo yum install libsnappy-devel

macOS:

First, install Snappy's C library using Homebrew:

brew install snappy

Then, install Python's snappy using conda:

conda install python-snappy -c conda-forge

...or, using pip:

CPPFLAGS="-I/usr/local/include -L/usr/local/lib" pip install python-snappy

Windows:

Windows users should check out Snappy for Windows and this Stack Overflow post for help on installing Snappy and python-snappy.

Current Status

Core Features:

• Local filesystem support with Parquet storage
• Full Pandas DataFrame compatibility, including MultiIndex
• Snapshots for point-in-time data versioning
• Metadata support for data organization

Advanced Features (July 2025 Release):

• Complex data type serialization (Period, Interval, Categorical, nested objects)
• Timezone-aware datetime handling with UTC storage
• Async/await operations for non-blocking I/O
• Data validation framework with extensible rules
• Schema evolution for handling data structure changes
• Transaction support with rollback capabilities

Performance Features:

• Streaming operations for datasets larger than RAM
• Batch read/write with parallel processing
• Intelligent partitioning (time-based and size-based)
• Memory optimization with automatic type downcasting
• Metadata caching for faster access
• Query optimization with column selection and predicate pushdown

Known Limitations:

• MultiIndex append operations have limited support due to Dask limitations - while there's a workaround that converts MultiIndex to regular columns, it may not fully preserve the MultiIndex structure after append (test remains marked as expected failure)
• Some Parquet limitations with preserving exact index metadata

Future Plans:

• Amazon S3 support (via s3fs)
• Google Cloud Storage support (via gcsfs)
• Hadoop Distributed File System support (via hdfs3)

Acknowledgements

PyStore is hugely inspired by Man AHL's Arctic which uses MongoDB for storage and allows for versioning and other features. I highly recommend you check it out.

License

PyStore is licensed under the Apache License, Version 2.0. A copy of which is included in LICENSE.txt.

---

I'm very interested in your experience with PyStore. Please drop me a note with any feedback you have.

Contributions welcome!

• Ran Aroussi