A visual explainer
walgit — one binary, one bucket, no leader.
Why hosting git is miserable, how a write-ahead log in object storage changes the economics, and what a Rust git server looks like when the bucket is the repository and every machine is a disposable cache.
Chapter 1
Hosting git is miserable
A packfile is a dense binary blob optimized to be small, not to be read. Every git operation becomes a random walk over gigabytes.
Inside a normal git repository, your code and metadata are compressed into packfiles. Objects are placed to minimize size: they are delta-compressed, out of order, and often physically far from the objects a traversal needs next. A git log, a blame, or a fetch is a DAG walk, and at each step the next pointer depends on the bytes you just read. There is no correlation between logical history and on-disk layout.
On a laptop with the file in page cache, that is fine. Over a network filesystem — NFS, GFS, DRBD — it is catastrophic. Git makes assumptions about filesystem semantics, locking, and tearing that hold on local NVMe and fall apart over the network. The cache is too small, the round trips are too many, and every hop is physical as well as logical.
The design that survived: Spokes
GitHub’s Spokes (and the variants most large hosts use) keeps real repositories on local NVMe so upstream git can do its work. It replicates at the packfile level and keeps every copy in sync with three-phase commit across a fixed replica set.
That consistency is paid for with a database that maps every repository to its machines, leader election, and a fleet of pets — machines whose disks matter, whose corruption matters, and whose placement is tracked like livestock, not cattle. The more replicas you add, the worse push throughput gets, because 3PC waits on the slowest node.
Chapter 2
The insight: WAL in object storage
Cursor’s Continuity makes a write-ahead log in an object store the source of truth, and every on-disk repository a cache. The manifest CAS is the consensus.
Make the bucket the truth. A push is stored as an immutable object in S3 or GCS and becomes visible only when a tiny manifest is rewritten with a compare-and-swap. That CAS is the consensus: no election, no quorum, no primary. Any instance may accept a push; two racing instances cannot both win.
A replica that has never seen a repository reads the log and has it. Reads are consistent without coordination because every read first asks the store whether the manifest changed — a conditional GET, usually a 304 Not Modified. Compaction is done once by whoever holds a lease and published into the log, so other replicas download compacted packs instead of repacking. Because the WAL is the truth, there is complete provenance: every push and every repack, replayable to any point.
A push in Continuity
- Index the pack. The server builds the index, reverse index, bitmap, and commit-graph in a scratch directory.
- Upload the immutable objects. The pack, index, and a log entry go to the bucket as content-addressed objects.
- CAS the manifest. Rewrite the tiny manifest if its version tag still matches. If not, the store returns
412 Precondition Failed; the server re-reads and retries.
The client only sees ok after the manifest has been CASed. The push object was already durable from the moment the upload finished.
Reads: conditional GET first
Every read begins with If-None-Match against the manifest. The store either says 304 Not Modified — serve from the local cache — or returns a new manifest, and the replica applies the new entries. There is no eventual consistency; the bucket is asked every time. The local disk is a cache; memory is a cache; the bucket is the repository.
Chapter 3
walgit: the bucket is the repository
A Rust implementation of Continuity, with the changes needed to run on machines smaller than the repository.
walgit takes the WAL-first architecture as-is and adds what a monorepo on small machines needs. It hosts git with no database, no leader, and no local state that matters. One binary pointed at an S3 or GCS bucket gives you smart HTTP (v0/v2) fetch and push, bundle-uri clones served as static files, Git LFS, a browsing web UI, a JSON API with an SDK, per-repository push policy, and webhooks.
Every machine running walgit is a disposable cache. Add more machines pointed at the same bucket and they serve the same repositories consistently, with nothing to coordinate. Kill them all and you lose warmth, nothing else.
Three mechanisms for repositories larger than the machine
- Remote reader
- When a repository’s packs will never fit on the instance, walgit serves refs and web pages over HTTP range requests into the bucket. It never needs the whole pack locally.
- History pack
- Commits and trees — the metadata needed for history, blame, and traversal — stay local in a history pack, while blobs stay in the bucket. Most operations only need the graph.
- bundle-uri
- Fresh clones and catch-ups are static files the bucket or a CDN hands out. Weekly full bundles, chained dailies and hourlies, cut as a pure function of the WAL. The server only handles the remainder.
Chapter 4
The payoff
A deployment story so simple it is the design: one config, one binary, one push.
[server]
listen = "0.0.0.0:8080"
public_url = "https://git.example.com"
auto_create_on_push = true
[server.auth]
mode = "token"
anonymous_read = false
tokens = [{ principal = "me", token_env = "WALGIT_TOKEN_ME", write = true }]
[store]
backend = "s3"
bucket = "my-walgit"
[store.s3]
endpoint = "https://s3.us-east-1.amazonaws.com"
region = "us-east-1"
# run it
WALGIT_TOKEN_ME=$(openssl rand -hex 24) walgit serve --config walgit.toml
# use it — a push to a new name creates the repository
git -c http.extraHeader="Authorization: Bearer $WALGIT_TOKEN_ME" \
push https://git.example.com/acme/app.git main