pubkey/event-reduce is a data-focused project on GitHub with 755 stars, written primarily in TypeScript. An algorithm to optimize database queries that run multiple times https://pubkey.github.io/event-reduce/
Snapshot summary built from the project's own GitHub metadata — there's no written TopGit review yet. The page will update automatically when a full review is published.
WHY NO REVIEW YET
TopGit writes full reviews for the most-starred, most-requested repositories. This page is a snapshot until then — see the READ ME tab for the original README in full.
Update live query results from change events without re-running the query
Live demo ·
JavaScript implementation ·
Rust implementation ·
Dart implementation ·
Kotlin implementation ·
Swift implementation ·
C# implementation ·
Python implementation ·
Go implementation ·
How it works ·
FAQ
EventReduce is an algorithm for live queries (also called reactive or realtime queries). When a document is inserted, updated or deleted, it takes the old query results and the change event and computes the new results in memory, without querying the database again. It works like a lightweight form of incremental view maintenance for a single query over a single table or collection.
~94% of write events in the browser demo are handled without re-running the query
Up to 12× faster delivery of new query results after a write
Works with any database that gives you a change stream: MongoDB, PostgreSQL, MySQL, IndexedDB, SQLite, in-memory stores...
Proven in production as the query engine behind the live queries of RxDB
Quickstart
npm install event-reduce-js
import { calculateActionName, runAction } from 'event-reduce-js';
// describe your query: SELECT * FROM users WHERE age > 18 ORDER BY age, id LIMIT 10
const queryParams = {
primaryKey: 'id',
sortFields: ['age', 'id'],
limit: 10,
queryMatcher: doc => doc.age > 18,
sortComparator: (a, b) => (a.age - b.age || (a.id < b.id ? -1 : 1)) < 0 ? -1 : 1
};
let results = [{ id: 'a', age: 20 }, { id: 'b', age: 30 }];
// a change event from your database's change stream
const changeEvent = { operation: 'INSERT', id: 'c', doc: { id: 'c', age: 25 }, previous: null };
const action = calculateActionName({ queryParams, changeEvent, previousResults: results });
if (action === 'runFullQueryAgain') {
results = await runQueryOnDatabase(); // EventReduce could not optimize this event
} else {
results = runAction(action, queryParams, changeEvent, results);
// -> [{ id: 'a', age: 20 }, { id: 'c', age: 25 }, { id: 'b', age: 30 }]
}
See the JavaScript implementation for the full API and a MongoDB-style example.
Rust, Dart, Kotlin, Swift, C#, Python and Go implementations that use the same BDD are also available. All implementations run the same shared test suite.
The problem
1. You make a query to the database which returns the result in 100 milliseconds
2. A write event occurs on the database and changes some data
3. To get the new version of the query's results you now have three options:
a. Run the query over the database again which takes another 100 milliseconds
b. Write complex code that somehow merges the incoming event with the old state
c. Use Event-Reduce to calculate the new results on the CPU without disc-IO nearly instant
Efficiency
In the browser demo you can see that for randomly generated events, about 94% of them could be optimized by EventReduce. In real world usage, with non-random events, this can be even higher. For the different implementations in common browser databases, we can observe an up to 12 times faster displaying of new query results after a write occurred.
How it works
EventReduce uses 18 different state functions to 'describe' an event+previousResults combination. A state function is a function that returns a boolean value like isInsert(), wasResultsEmpty(), sortParamsChanged() and so on.
Also there are 16 different action functions. An action function gets the event+previousResults and modifies the results array in a given way like insertFirst(), replaceExisting(), insertAtSortPosition(), doNothing() and so on.
For each of our 2^19 state combinations, we calculate which action function gives the same results that the database would return when the full query is executed again.
From this state-action combinations we create a big truth table that is used to create a binary decision diagram. The BDD is then optimized to call as few state functions as possible to determine the correct action of an incoming event-results combination.
The resulting optimized BDD is then shipped as the EventReduce algorithm and can be used in different programming languages and implementations. The programmer does not need to know about all this optimisation stuff and can directly use three simple functions like shown in the JavaScript implementation
When to use this
Use EventReduce whenever you observe query results and need to keep them up to date, for example for reactive UIs, realtime dashboards, live queries in local-first apps or server-push subscriptions. You can use it to
reduce the latency until a change to the database updates your application
make observing query results more scalable by doing less disk-io
reduce the bandwith when streaming realtime query results from the backend to the client
create a better form of caching where instead of invalidating the cache on write, you update its content
Used by RxDB
RxDB, the local-first JavaScript database, uses EventReduce for its live queries since version 9.0.0, where it is enabled by default.
When an app subscribes to a query, RxDB keeps the results up to date by running each write event of the collection through EventReduce.
Only when EventReduce returns runFullQueryAgain, RxDB queries the storage again.
So most writes update the observed results without an extra read from the storage, whichever storage RxDB runs on.
Before, RxDB used its own QueryChangeDetection, which was made of many handwritten if-else comparisons.
Limitations
EventReduce only works with queries that have a predictable sort-order for any given documents. (you can make any query predicable by adding the primary key as last sort parameter)
EventReduce can be used with relational databases but not on relational queries that run over multiple tables/collections. (you can use views as workarround so that you can query over only one table). In theory Event-Reduce could also be used for relational queries but I did not need this for now. Also the fuzzing that ensures the correctness of the truth table runs for a long time, and having more state functions looks like an NP problem.
Additional Optimization: Result Window Buffer
For a query with a limit or a skip, most runFullQueryAgain results happen when a row has to move into the result window that is not in the previous results:
the row after a full window, or the last skipped row when a row is added in front of the window.
The result window buffer keeps some rows before and after the window in memory, so these rows come from memory instead of a new query.
With the random queries and events of the fuzzing, it raised the share of events that need no full query from about 95 % to about 100 %.
See docs/result-window-buffer.md for how it works, how to use it and the measurements.
Implementations
At the moment there is only the JavaScript implementation that you can use over npm. Pull requests for other languages are welcomed.
Previous Work
Meteor uses a feature called OplogDriver that is limited on queries that do not use skip or sort. Also watch this video to learn how OpLogDriver works.
RxDB used the QueryChangeDetection which works by many handwritten if-else comparisons. RxDB switched to EventReduce since version 9.0.0.
Baqend is creating a database that optimizes for realtime queries. Watch the video Real-Time Databases Explained: Why Meteor, RethinkDB, Parse & Firebase Don't Scale to learn more.
FAQ
Is this incremental view maintenance (IVM)?
In a narrow sense, yes. EventReduce incrementally maintains the result of one query (filter, sort, skip, limit) over one table or collection. Unlike general IVM engines it needs no query planner or dataflow graph and does not run any work on writes. It is a pure function from (old results, change event) to new results that you call whenever you need fresh results.
Is this something like materialized views?
Yes and no. Materialized views solve a similar problem but in a different way with different trade-offs. When you have many users, all subscribing to different queries, you cannot create that many views because they are all recalculated on each write access to the database. EventReduce however has better scalability because it does not affect write performance and the calculation is done when the fresh query results are requested, not beforehand.
Is this something like event sourcing or CQRS?
No, event sourcing is mostly used to calculate a current state by attaching the full event stream to the starting state. This allows for stuff like time travel and so on. EventReduce solves a completely different (performance-) problem and only shares some common keywords like event.
Isn't this optimization already done by database engines?
No. I tested EventReduce with many common databases like MongoDB, MySQL and Postgres. Each of them had better performance with Event-Reduce than just observing the eventstream and running the queries again. If you understand what Event-Reduce exactly does, it becomes clear that this optimization cannot be done by pull-based databases because they have missing information.
Isn't this the same as product XY?
No. EventReduce is not a product, it is not comparable to any database or streaming backend out there. EventReduce is an algorithm with a specific input and output, nothing more, nothing less. You can use EventReduce without having to change your underlying data infrastructure.
How does pubkey/event-reduce compare to other Data projects?
pubkey/event-reduce is tracked by TopGit in the Data category, with 755 GitHub stars and written in TypeScript. Browse the Data topic page on TopGit to compare it against similar projects by stars and activity.
How many stars does pubkey/event-reduce have?
pubkey/event-reduce has 755 GitHub stars — refresh the page for the live number, or check github.com/pubkey/event-reduce. TopGit mirrors GitHub's count but does not claim minute-by-minute accuracy.
Is pubkey/event-reduce open source?
Yes — pubkey/event-reduce ships under the MIT license, which makes its source code freely readable (and, depending on license terms, forkable and reusable). Source: github.com/pubkey/event-reduce.
What else is in the Data space?
pubkey/event-reduce is tracked by TopGit under the Data category, alongside 10 GitHub-tagged topics. Trending and Topics pages list peer repositories of comparable stars and language.
What is pubkey/event-reduce?
pubkey/event-reduce (pubkey/event-reduce) is a TypeScript project on GitHub. From the project's own README: An algorithm to optimize database queries that run multiple times https://pubkey.github.io/event-reduce/
Where can I see pubkey/event-reduce in action?
The project maintains a homepage at https://pubkey.github.io/event-reduce. The README tab on this page also usually contains screenshots and a quickstart.
Where do I read more about pubkey/event-reduce?
This TopGit page is a snapshot — the READ ME tab shows the project's own README content (links stripped, images preserved). The GitHub repository at github.com/pubkey/event-reduce is the definitive source.
Read full README in the tab above.
Curious whether event-reduce is right for you?
Let ChatGPT, Claude, or Perplexity look into it — click below and see what AI actually says about event-reduce.