Making Atuin sync 32x faster with packfiles

TL;DR: Atuin sync is now 32x faster. We sync many thousands of records in single-digit seconds.

Get the latest Atuin with curl --proto '=https' --tlsv1.2 -LsSf https://setup.atuin.sh | sh

Atuin syncs shell history and other terminal data end-to-end encrypted, meaning the server cannot read, compress, or bundle your data. The client now builds packfiles to work around this constraint. Along with other optimisations, we’re happy to share that Atuin Hub sync is now 32x faster, and for self-hosted users sync is 16x faster!

With the 18.20.0 release, sync has received many improvements. Packfiles collect multiple records and get stored in fast Tigris-backed object storage. Just like regular history entries, packfiles are fully encrypted with the same PASETO-based algorithm we use for records. Additionally, we’ve pipelined record downloads and we’ve also enabled batched SQL inserts. Best of all – we managed to avoid updating the sync protocol version.

This is not just a small performance improvement. This is a paradigm shift in how Atuin does sync.

— lots of numbers on a GPU, August 2026

This blog post is going to go into some more technical details, and if you don’t care, look away!

Packfiles

The median user on Atuin Hub has 7,338 records. For that user, Atuin sync takes about 11.7s. The average user has 26,075 records, meaning their sync takes about 40s – tolerable, but not what we want. Atuin power users, however – the 95th and 99th percentile – experience sync times of about 2.2min and 5.4min, respectively. We set a higher bar:

A sync of 1K records won’t take more than 3s and each additional 10K records won’t add more than 1s to that.

What this is telling us is that Atuin’s sync doesn’t serve its most active users as well as it can – performance degrades as the record store becomes larger. Sync is inherently linear in the number of records, so we can’t change the scaling. But we can reduce the constant!

Atuin’s record store is in many ways similar to a write-ahead log – each new record is an addition to the log. A download sync means downloading each record the client is missing from the server. Atuin 18.19.0 downloads records in pages of 100. Each page contains an array of encrypted records. Unfortunately, for the most part, these pages are uncompressed, and what’s worse, ciphertext is indistinguishable from noise, so it does not compress.

We needed a more clever approach and took some inspiration from Git packfiles (though ours are simpler). The idea behind Atuin’s packfiles is the same: take N plaintext records, bundle them, compress them and then encrypt them.

An astute reader may have already noticed a problem – Atuin servers do not have access to the encryption key! Therefore, the client must be responsible for packing, as it is the only machine which has access to the records and the encryption key necessary to encrypt the packfile.

Packing and uploading

The Atuin daemon now keeps track of a rolling buffer of loose records. As you use Atuin, this buffer will fill up. When it reaches a certain threshold, Atuin daemon will take the rolling buffer and create a new “packfile manifest” record, which encodes the window of records that the packfile contains. Note that no computationally heavy task has occurred here – we just kept track of records and created tiny manifest objects in the database for them.

Whenever the daemon attempts to sync (or you manually invoke atuin sync), our new capabilities system enables Atuin to check whether the server supports packfiles. If that’s the case, Atuin will perform packing – it will find each record the manifest describes, bundle them all together with rmp, compress with zstd and finally encrypt with the existing PASETO v4 scheme.

Packfiles are uploaded out-of-band with the regular records. Servers which support packfiles need to have special endpoints, so special logic and special handling is necessary.

At this time, only Atuin Hub supports packfiles. Self-hosted servers do not have a conforming implementation. We will bring these improvements to the self-hosted server once they stabilise.

Uploading packfiles means requesting the server provide a pre-signed URL the client can PUT.

Downloading and sync

This is the fun part – when performing sync, we first sync packfiles. This means grabbing the packfile manifest records from the server, and then, for each manifest, querying the server for a pre-signed download URL. The client then downloads the packfile from the pre-signed URL, decrypts it, decompresses it, and then splits it up into individual records to store in its own table.

Results

We ran synthetic benchmarks with different decades of record sizes. Between all the fixes, 18.20.0 improves performance by about 30 times for the 100K power-user. Packfiles constitute to about a 2 times improvement there.

These new sync protocols easily meet our thresholds.

Future improvements

We are exploring the idea of “packfiles of packfiles” – a packfile can wrap multiple packfiles, instead of records alone. Currently, Atuin Hub packfiles are all 1000 records wide (and come out to be about 10–100 KB), but for use cases with very high numbers of packfiles, bundling older packfiles into trees of packfiles, enables further compression and latency reduction.

Packfile downloads are pipelined, just as regular record downloads now are, however, a further tail-latency reduction can be achieved by requesting a single pre-signed URL valid for multiple packfiles, reducing the number of round-trips.

Finally, we're exploring the idea of moving away from PASETOv4 towards plain ChaCha20-Poly1305 which lets us encrypt and decrypt packfiles in streaming chunks and drop the PASETO base64 and metadata overhead.

The infrastructure we’ve built enables us to capture a lot more in your history than just the commands you ran. In a future release, Atuin will be able to capture, record and search through your commands’ output, agent sessions and more – made possible by the infrastructure necessary for packfiles.

Other improvements

As part of this release, we’ve done a huge amount of other work too.

Pipelining

Historically, Atuin would sequentially download a page of records, and then insert each record into the database. It would then grab the next page, repeat the inserts, and so on.

A common strategy in improving such algorithms is pipelining. We had realized that if we made just a small adjustment in how we iterated during downloads, we could eagerly request pages from the server, in parallel.

Pipelining alone improved sync speed by about 340%.

This feature is universally available in Atuin 18.20 – all you need to do is atuin update.

Conclusion

We’re really excited about this release – updating Atuin will immediately give you a sync that is 16x times faster. We’re rolling out packfiles incrementally over the next few weeks, so look forward to that!

You can update or install Atuin by running:

curl --proto '=https' --tlsv1.2 -LsSf https://setup.atuin.sh | sh

If you would like to try packfiles today, you can enable them for your account here.

Thanks!
— The Atuin team