As an open-source project, systemd/casync has picked up 1.6k stars on GitHub (C). Content-Addressable Data Synchronization Tool
Snapshot summary built from the project's own GitHub metadata — there's no written TopGit review yet. The page will update automatically when a full review is published.
WHY NO REVIEW YET
TopGit writes full reviews for the most-starred, most-requested repositories. This page is a snapshot until then — see the READ ME tab for the original README in full.
A combination of the rsync algorithm and content-addressable storage
An efficient way to store and retrieve multiple related versions of large file systems or directory trees
An efficient way to deliver and update OS, VM, IoT and container images over the Internet in an HTTP and CDN friendly way
An efficient backup system
See the Announcement Blog
Story for a
comprehensive introduction. The medium length explanation goes something like
this:
Encoding: Let's take a large linear data stream, split it into
variable-sized chunks (the size of each being a function of the
chunk's contents), and store these chunks in individual, compressed
files in some directory, each file named after a strong hash value of
its contents, so that the hash value may be used to as key for
retrieving the full chunk data. Let's call this directory a "chunk
store". At the same time, generate a "chunk index" file that lists
these chunk hash values plus their respective chunk sizes in a simple
linear array. The chunking algorithm is supposed to create variable,
but similarly sized chunks from the data stream, and do so in a way
that the same data results in the same chunks even if placed at
varying offsets. For more information see this blog
story.
Decoding: Let's take the chunk index file, and reassemble the large
linear data stream by concatenating the uncompressed chunks retrieved
from the chunk store, keyed by the listed chunk hash values.
As an extra twist, we introduce a well-defined, reproducible,
random-access serialization format for directory trees (think: a more
modern tar), to permit efficient, stable storage of complete directory
trees in the system, simply by serializing them and then passing them
into the encoding step explained above.
Finally, let's put all this on the network: for each image you want to
deliver, generate a chunk index file and place it on an HTTP
server. Do the same with the chunk store, and share it between the
various index files you intend to deliver.
Why bother with all of this? Streams with similar contents will result
in mostly the same chunk files in the chunk store. This means it is
very efficient to store many related versions of a data stream in the
same chunk store, thus minimizing disk usage. Moreover, when
transferring linear data streams chunks already known on the receiving
side can be made use of, thus minimizing network traffic.
Why is this different from rsync or OSTree, or similar tools? Well,
one major difference between casync and those tools is that we
remove file boundaries before chunking things up. This means that
small files are lumped together with their siblings and large files
are chopped into pieces, which permits us to recognize similarities in
files and directories beyond file boundaries, and makes sure our chunk
sizes are pretty evenly distributed, without the file boundaries
affecting them.
The "chunking" algorithm is based on the buzhash rolling hash
function. SHA512/256 is used as a strong hash function to generate digests of the
chunks (alternatively: SHA256). zstd is used to compress the individual chunks
(alternatively xz or gzip).
Is this new? Conceptually, not too much. This uses well-known concepts,
implemented in a variety of other projects, and puts them together in a
moderately new, nice way. That's all. The primary influences are rsync and git,
but there are other systems that use similar algorithms, in particular:
casync uses the Meson build system. To build casync,
install Meson (at least 0.47), as well as the necessary build dependencies
(gcc, libzstd-dev liblzma-dev libacl1-dev libfuse-dev libudev-dev python3-sphinx). Then run:
The most recent commit recorded on systemd/casync was 11 months ago, based on the GitHub push timestamp. The repository has 118 forks — one of the better signals of community interest.
How many stars does systemd/casync have?
systemd/casync has 1.6k GitHub stars — refresh the page for the live number, or check github.com/systemd/casync. TopGit mirrors GitHub's count but does not claim minute-by-minute accuracy.
What language is systemd/casync written in?
systemd/casync is written primarily in C. GitHub's language field is based on the largest share of bytes in the default branch.
What topics is systemd/casync associated with?
GitHub's repository topics for systemd/casync: "archive", "chunking", "delivery", "download", "file-system", "http", "synchronization", "tar", "upload". TopGit's editorial category is open-source.
Where do I read more about systemd/casync?
This TopGit page is a snapshot — the READ ME tab shows the project's own README content (links stripped, images preserved). The GitHub repository at github.com/systemd/casync is the definitive source.
Read full README in the tab above.
Is casync worth your time?
ChatGPT, Claude and Perplexity can all read this page. Ask one of them what it makes of casync.
/casync