TopGit tracks ranaroussi/pystore on GitHub as part of the Data family. The project has 612 stars. Fast data store for Pandas time-series data
Snapshot summary built from the project's own GitHub metadata — there's no written TopGit review yet. The page will update automatically when a full review is published.
WHY NO REVIEW YET
TopGit writes full reviews for the most-starred, most-requested repositories. This page is a snapshot until then — see the READ ME tab for the original README in full.
PyStore - Fast data store for Pandas timeseries data
PyStore is a simple (yet powerful) datastore for Pandas dataframes, and while it can store any Pandas object, it was designed with storing timeseries data in mind.
It's built on top of Pandas, Numpy, Dask, and Parquet (via pyarrow), to provide an easy to use datastore for Python developers that can easily query millions of rows per second per client.
New in 2025 Release (PR #77):
MultiIndex Support - Store and retrieve DataFrames with Pandas MultiIndex
Complex Data Types - Full support for Timedelta, Period, Interval, Categorical dtypes
Timezone-Aware Operations - Proper handling of timezone data with UTC storage
Async/Await Support - Non-blocking I/O operations for better performance
Data Validation Framework - Extensible validation rules for data integrity
Schema Evolution - Handle schema changes over time with flexible strategies
Transaction Support - Atomic operations with rollback capabilities
Performance Optimizations - Streaming operations and memory management
Performance Enhancements (Phase 3 Release):
Streaming Operations - Memory-efficient append for datasets larger than RAM
Intelligent Partitioning - Automatic time-based and size-based partitioning
Memory Management - 70-90% memory reduction with monitoring and optimization
Metadata Caching - 100x faster metadata access with TTL cache
Query Optimization - Column selection and predicate pushdown at storage level
Performance improvements include:
Append 1M rows: 3.75x faster, 90% less memory
Batch operations: 6x faster for multiple items
Column selection: 4x faster when reading subset of columns
Filtered reads: 8x faster with predicate pushdown
Check out this blog post for the reasoning and philosophy behind PyStore, as well as a detailed tutorial with code examples.
Follow this PyStore tutorial in Jupyter notebook format.
Quickstart
Install PyStore
Install using pip:
pip install pystore --upgrade --no-cache-dir
Install using conda:
conda install -c ranaroussi pystore
INSTALLATION NOTE:
If you don't have Snappy installed (compression/decompression library), you'll need to install it first.
Using PyStore
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import pystore
import yfinance as yf
# Set storage path (optional)
# Defaults to `~/pystore` or `PYSTORE_PATH` environment variable (if set)
pystore.set_path("~/pystore")
# List stores
pystore.list_stores()
# Connect to datastore (create it if not exist)
store = pystore.store("mydatastore")
# List existing collections
store.list_collections()
# Access a collection (create it if not exist)
collection = store.collection("NASDAQ")
# List items in collection
collection.list_items()
# Load some data from yfinance
aapl = yf.download("AAPL", multi_level_index=False)
# Store the first 100 rows of the data in the collection under "AAPL"
collection.write("AAPL", aapl[:100], metadata={"source": "yfinance"})
# Reading the item's data
item = collection.item("AAPL")
data = item.data # <-- Dask dataframe (see dask.pydata.org)
metadata = item.metadata
df = item.to_pandas()
# Append the rest of the rows to the "AAPL" item
collection.append("AAPL", aapl[100:])
# Reading the item's data
item = collection.item("AAPL")
data = item.data
metadata = item.metadata
df = item.to_pandas()
# --- Query functionality ---
# Query available symbols based on metadata
collection.list_items(some_key="some_value", other_key="other_value")
# --- Snapshot functionality ---
# Snapshot a collection
# (Point-in-time named reference for all current symbols in a collection)
collection.create_snapshot("snapshot_name")
# List available snapshots
collection.list_snapshots()
# Get a version of a symbol given a snapshot name
collection.item("AAPL", snapshot="snapshot_name")
# Delete a collection snapshot
collection.delete_snapshot("snapshot_name")
# ...
# Delete the item from the current version
collection.delete_item("AAPL")
# Delete the collection
store.delete_collection("NASDAQ")
Advanced Features
Async Operations:
import asyncio
from pystore import async_pystore
async def async_example():
async with async_pystore.store("mydatastore") as store:
async with store.collection("NASDAQ") as collection:
# Async write
await collection.write("AAPL", df)
# Async read
df = await collection.item("AAPL").to_pandas()
asyncio.run(async_example())
Data Validation:
from pystore import create_validator, ColumnExistsRule, RangeRule
# Create a validator
validator = create_validator([
ColumnExistsRule(["Open", "High", "Low", "Close"]),
RangeRule("Close", min_value=0),
])
# Apply validator to collection
collection.set_validator(validator)
Schema Evolution:
from pystore import SchemaEvolution, EvolutionStrategy
# Enable schema evolution
evolution = collection.enable_schema_evolution(
"AAPL",
strategy=EvolutionStrategy.FLEXIBLE,
)
# Schema changes are handled automatically during append
collection.append("AAPL", new_data_with_extra_columns)
PyStore provides namespaced collections of data. These collections allow bucketing data by source, user or some other metric (for example, frequency: End-Of-Day, Minute Bars, etc.). Each collection (or namespace) maps to directory containing partitioned parquet files for each item (e.g., symbol).
A good practice it to create collections that may look something like this:
PyStore was tested to work on *nix-like systems, including macOS.
Dependencies
PyStore utilizes Snappy, a fast and efficient compression/decompression library developed by Google. You'll need to install Snappy on your system before installing PyStore.
See the python-snappy GitHub repo for more information.
Windows users should check out Snappy for Windows and this Stack Overflow post for help on installing Snappy and python-snappy.
Current Status
Core Features:
Local filesystem support with Parquet storage
Full Pandas DataFrame compatibility, including MultiIndex
Snapshots for point-in-time data versioning
Metadata support for data organization
Advanced Features (July 2025 Release):
Complex data type serialization (Period, Interval, Categorical, nested objects)
Timezone-aware datetime handling with UTC storage
Async/await operations for non-blocking I/O
Data validation framework with extensible rules
Schema evolution for handling data structure changes
Transaction support with rollback capabilities
Performance Features:
Streaming operations for datasets larger than RAM
Batch read/write with parallel processing
Intelligent partitioning (time-based and size-based)
Memory optimization with automatic type downcasting
Metadata caching for faster access
Query optimization with column selection and predicate pushdown
Known Limitations:
MultiIndex append operations have limited support due to Dask limitations - while there's a workaround that converts MultiIndex to regular columns, it may not fully preserve the MultiIndex structure after append (test remains marked as expected failure)
Some Parquet limitations with preserving exact index metadata
Future Plans:
Amazon S3 support (via s3fs)
Google Cloud Storage support (via gcsfs)
Hadoop Distributed File System support (via hdfs3)
Acknowledgements
PyStore is hugely inspired by Man AHL's Arctic which uses MongoDB for storage and allows for versioning and other features. I highly recommend you check it out.
License
PyStore is licensed under the Apache License, Version 2.0. A copy of which is included in LICENSE.txt.
I'm very interested in your experience with PyStore. Please drop me a note with any feedback you have.
The most recent commit recorded on ranaroussi/pystore was 6 months ago, based on the GitHub push timestamp. The repository has 103 forks — one of the better signals of community interest.
How does ranaroussi/pystore compare to other Data projects?
ranaroussi/pystore is tracked by TopGit in the Data category, with 612 GitHub stars and written in Python. Browse the Data topic page on TopGit to compare it against similar projects by stars and activity.
How many stars does ranaroussi/pystore have?
ranaroussi/pystore has 612 GitHub stars — refresh the page for the live number, or check github.com/ranaroussi/pystore. TopGit mirrors GitHub's count but does not claim minute-by-minute accuracy.
What is ranaroussi/pystore?
ranaroussi/pystore (ranaroussi/pystore) is a Python project on GitHub. From the project's own README: Fast data store for Pandas time-series data
What language is ranaroussi/pystore written in?
ranaroussi/pystore is written primarily in Python. GitHub's language field is based on the largest share of bytes in the default branch.
What topics is ranaroussi/pystore associated with?
GitHub's repository topics for ranaroussi/pystore: "dask", "database", "dataframe", "datastore", "pandas", "parquet", "timeseries". TopGit's editorial category is Data.
Why is ranaroussi/pystore categorized under Data?
TopGit places ranaroussi/pystore in the Data category based on its GitHub topics and description (tagged: "dask", "database", "dataframe"). Categories are assigned from real repository metadata, not editorial guesswork.
Read full README in the tab above.
Curious whether pystore is right for you?
Let ChatGPT, Claude, or Perplexity look into it — click below and see what AI actually says about pystore.