Grixel in AI labs

Imagine the postmortem. A training run had four parents. The code that launched it lives in git, and git handles that fine. The dataset revision lives in an object-store path with a date baked into it. The checkpoint it resumed from lives in a bigger object-store path that someone eventually pruned to save money. And the knowledge that shaped it (which eval set was canonical, which preprocessing spec was current) lives in a wiki updated three weeks after the run finished.

Six months later a research lead asks whether run 4471 can be reproduced. The code commit is easy. The dataset path still exists, though nobody can prove the bytes under it are the bytes from March. The checkpoint is gone. The wiki page has been edited nine times since.

The lab and the run are composites, but the pattern gets a nod from everyone I describe it to. Labs version four things with four tools of wildly different quality: code well, datasets by naming convention, checkpoints by bigger naming convention, and the binding knowledge by tribal memory.

I am building Grixel, a version control and data platform, at Unabated Games. It grew out of game development, where the daily workload is the same shape as a lab's: a little code attached to an enormous amount of binary data, plus standards documents everyone is supposed to follow and mostly cannot find. The proposition for a lab is one content-addressed depot where a training run's code, dataset revision, config, and the standards it followed are a single reproducible, access-governed graph. The demo tenant below borrows the hypothetical's name: a research org at /companies/parallax, with a run at runs/run-4471.

#One depot, one catalogue

Before any command lines, the picture. Imagine the whole lab's artifacts in one library. Every version of every book is kept, and shelf space is cheap because two editions that differ by one chapter share the unchanged pages. The catalogue at the front desk is the only authority on what exists. You never photocopy the library to work; you request the three shelves you need. And if you are not cleared for a restricted collection, the catalogue does not show a locked door: those entries are simply absent from your copy of the catalogue.

Technically: Grixel is a server-authoritative depot. One global namespace, with paths like /companies/parallax/datasets/pretrain-v3/. There is no clone; there is checkout of a declared view. The server daemon, grixeld, owns everything (metadata, access rules, chunk storage, atomic changelist commits); the CLI is a thin client. Files are split into content-defined chunks, each hashed with BLAKE3, and only new chunks are stored or transferred.

This is Perforce-shaped rather than Git-shaped, and it matters for a lab even though most labs never ran Perforce. In a Git-shaped system every machine holds a full copy and the server is a hosting convenience; policy lives in a forge layered on top. That model cannot give you path-level invisibility, because the data is already on the laptop. Cross-repo pins it can genuinely approximate: a submodule is a committed SHA pin into another repository. What it lacks is a server that validates the pin, a queryable notion of drift, a mode that advances only past reviewed revisions, and an experience anyone defends. A server-authoritative single namespace turns governed sharing and reproducible pins from per-team glue into structure. Perforce proved the shape at studios pushing terabytes of binary assets; Grixel keeps the shape, replaces the substrate, and adds the layer labs need: governed, pinned context for humans and AI agents.

· · ·

#Three decisions, and their costs

Three design decisions do most of the work in this article, and each has a real cost. I would rather you hear the costs from me.

Local-filesystem chunk store fronted by a caching read proxy, instead of object-storage indirection. Chunks are immutable and named by their hash, so a transparent read proxy can sit between a compute cluster and the depot and cache bytes on the LAN, the way Perforce's proxy has fanned out game builds for decades. The cost: you give up the operational story of S3, its durability, lifecycle policies, and billing. lakeFS rides your existing buckets and inherits all of that on day one; Grixel asks you to run a daemon and own its disks. For hot-path training data the proxy keeps repeated multi-gigabyte reads inside the rack, so I think the trade is right. It is still a trade.

Committed, server-validated lock files, instead of per-machine cache state. A project declares its external dependencies (a dataset elsewhere in the namespace, an eval spec, the lab's data standards) in a committed scope.json; the resolved revisions are written to a committed scope.lock. The server rejects any lock that is not a correct resolution of its scope, so a committed pair is always internally consistent, and reproducibility becomes a property of the depot rather than of any machine's cache. The cost is ceremony: advancing a pin is an explicit, committed operation, never something a sync does silently.

Single-writer metadata. Changelist commits serialize through a single write path, WAL-backed and crash-recoverable. Chunk upload happens outside that critical section; bytes stream in concurrently, and only the metadata commit at the end serializes. That keeps atomicity easy to reason about and the serialized section short, and it still puts a ceiling on metadata write throughput. For changelists the ceiling is irrelevant; humans and CI do not commit thousands of times per second. But it makes the depot the wrong place for high-frequency telemetry. Grixel is the system of record for artifacts and decisions. It does not want your training metrics.

A note on the transcripts: they were captured in local mode, where the CLI asserts a principal with -as <user>; in a team deployment you grixel login once, drop the flag, and identity comes from the server-verified token.

· · ·

#Where chunking wins, and where it finds nothing

Content-defined chunking backs the storage claims, so here are both faces of it, including the one that does not flatter the product.

The win case first: an append-heavy dataset shard. The first submit stores all 96 MiB new; then a week of examples is appended and the shard resubmitted whole:

$ grixel submit -as rin -ref companies/parallax/datasets/pretrain-v3 -m "shard 000: append week-26 examples" pretrain-shard-000.tar
Committed: 85c88f1cb4ccaf0e011d42f6262a49919ab6d53ad7a64bc4bba6074bc4121877
  ref     : companies/parallax/datasets/pretrain-v3
  changes : 1 files, 100.0 MiB
  stored  : 6.5 MiB new, 93.5 MiB deduped
  parent  : 2b14d133e011eff0e8207c6accc3edc79801eff86157187bc51e12228d0e92d9
  author  : rin
  message : shard 000: append week-26 examples

100 MiB submitted, 6.5 MiB stored, and every earlier revision stays checkable-out.

Now the case where you might expect the same result and must not. Adjacent checkpoints of a dense run feel like they should be "mostly the same bytes." They are not: a dense optimizer step perturbs essentially every parameter, the serialized tensors differ everywhere, and chunking finds nothing to share. Two adjacent checkpoints from the demo run:

$ grixel submit -as rin -ref companies/parallax/runs/run-4471 -at checkpoints -m "checkpoint step 8000" step-08000.safetensors
Committed: 71cd8849e6ae6dfa2b165ad9330d968e3b6d08453bb08a93401ebfa6774b10b0
  ref     : companies/parallax/runs/run-4471
  changes : 1 files, 96.0 MiB
  stored  : 96.0 MiB new, 0 B deduped
  parent  : (initial commit)
  author  : rin
  message : checkpoint step 8000

$ grixel submit -as rin -ref companies/parallax/runs/run-4471 -at checkpoints -m "checkpoint step 9000" step-09000.safetensors
Committed: 319e11b6873bfceb2dadc87c03ac717fd722f2b201d358f29d343dd8201ea684
  ref     : companies/parallax/runs/run-4471
  changes : 1 files, 96.0 MiB
  stored  : 96.0 MiB new, 0 B deduped
  parent  : 71cd8849e6ae6dfa2b165ad9330d968e3b6d08453bb08a93401ebfa6774b10b0
  author  : rin
  message : checkpoint step 9000

Zero bytes deduplicated, and the tool says so at submit time. Dense adjacent checkpoints do not dedup, in Grixel or in anything else built on content-defined chunking; if a storage vendor tells you otherwise, ask for this transcript. What versioning buys on those files is retention, addressing, and provenance, at full storage cost per step you keep.

Between those poles sits the warm-start fine-tune, where most of the model is frozen and a slice is retrained. The unchanged tensors dedup against the parent checkpoint:

$ grixel submit -as rin -ref companies/parallax/runs/run-4471 -at finetune -m "warm-start: 11 of 96 MiB retrained" ft-warmstart.safetensors
Committed: e0fa0de37f5fb23b1140280b0e1389714af418c4b37689fca1effa0ab0ee859c
  ref     : companies/parallax/runs/run-4471
  changes : 1 files, 96.0 MiB
  stored  : 29.1 MiB new, 66.9 MiB deduped
  parent  : 319e11b6873bfceb2dadc87c03ac717fd722f2b201d358f29d343dd8201ea684
  author  : rin
  message : warm-start: 11 of 96 MiB retrained

29.1 MiB stored where 11 MiB of tensors changed; chunk boundaries do not line up with tensor boundaries, so the overhead is real and printed. The remaining wins are boring: a re-uploaded identical artifact stores zero new bytes anywhere in the namespace, which makes retries and copies idempotent, and configs and tokenizers are noise in the budget.

On scale: these demo files are around 100 MiB where yours are 6 GB or 600 GB; the mechanism is identical chunk by chunk and unmeasured at your sizes, and the closing section bounds that claim. The same manifests drive delta-pull: syncing a small change to a large file fetches only the new chunks.

#A node pulls only what it needs

A checkout names the subtrees it wants, so an eval node that needs only the config does not pull the checkpoints:

$ grixel checkout -as rin -path 'configs/**' companies/parallax/runs/run-4471 eval-node-ws
Checked out companies/parallax/runs/run-4471
  dir   : ~/demo/eval-node-ws
  base  : 0a897ad88e0b3ba42c3817bea0dae13caffdc2e397abdb17fedf9bd12814ed69
  files : 1
  view  : configs/** (partial checkout)

Edit files, then: grixel changes / grixel diff / grixel submit -m ...

$ du -sh eval-node-ws
 16K    ~/demo/eval-node-ws

The ref holds hundreds of MiB of checkpoints; the workspace on disk is 16K, and grixel sync widens or narrows the view later. It is the difference between "the repo is 2 TB so nobody versions the weights" and versioning the weights.

· · ·

#Reproducibility as a committed file, not a cache

This is the part I care most about for labs, because naming conventions have no equivalent. The run project declares its dependency, pinned to an exact dataset revision (the hash is the appended-shard commit from earlier):

$ cat .grixel/scope.json
{
  "name": "run-4471",
  "owner": "team:pretraining",
  "description": "Run 4471: seasonal-mix pretrain",
  "mappings": [
    {
      "source": "/companies/parallax/datasets/pretrain-v3",
      "mountAt": ".data/pretrain-v3",
      "pin": { "mode": "rev", "rev": "85c88f1cb4ccaf0e011d42f6262a49919ab6d53ad7a64bc4bba6074bc4121877" }
    }
  ]
}

$ grixel scope update -as rin
companies/parallax/runs/run-4471 — resolved 1/1 mapping(s):
  .data/pretrain-v3                <- /companies/parallax/datasets/pretrain-v3 @ 85c88f1c

Committed scope.lock: a022a03146c27869b7687378c61129f95c37a317753a5a3e5829747c86792ed5
  ref     : companies/parallax/runs/run-4471
  author  : rin

The resolved lock is a committed, server-validated file in history. Two consequences follow.

First, checking out the project at any historical commit replays its mappings at the revisions pinned then. Run 4471 pinned the dataset at 85c88f1c; checking out the commit that launched it materializes the dataset at 85c88f1c, today, next year, on a machine that has never seen the project. There is no "whichever version happened to be in the cache." Reproducibility stops being a runbook and becomes a property of the checkout.

Second, freshness is a queryable, shared state. A rev pin reports behind N when the dataset advances N changelists past it; a head-mode pin reports drift-on-head when the source moves, while the workspace keeps the locked snapshot. Advancing a pin is a committed, reviewable act, so "when did we move to the new eval set" has an answer with a hash and an author.

There is a third pin mode: approved. Grixel's review and sign-off system lives in the server, and an approved-mode mapping resolves to the latest revision of the source that has fully cleared its review gate. In the demo depot's other tenant, a mapped standards ref gets a new, unreviewed commit at its head; grixel scope update re-resolves to the same signed-off revision, and moves the pin only after a reviewer signs the new head off. Read "data-cleaning standard" for "brand standards." The MCP integration assembles AI context bundles from these same pinned mappings, so an agent's context advances only past reviewed content, never an unreviewed draft.

That covers three of the four parents from the opening. The code parent closes concretely: grixel vendor pins an external git repository read-only inside a project at a fixed SHA, recording source URL, branch, and commit as import provenance; or, if the training code stays canonical in git, the launcher records the git SHA in the changelist's provenance. Either way the code revision joins the same graph as the dataset pin and the config.

· · ·

#What an unreleased model does not look like

Every lab holds things that must not leak internally: an unreleased model, a restrictively licensed dataset, a partner's data. Grixel's access rule is invisibility. A denied path is not "permission denied"; it is gone, from listings, search, error messages, and AI context. These transcripts come from the demo depot's studio tenant; a hidden unannounced game and a hidden unreleased model are the same mechanism.

Bob, who can read the whole namespace, lists /companies/acme/games and sees two projects, seasonal/ and secret/. Alice, whose permissions cover only the seasonal project, sees one. If she guesses the hidden path anyway, the answer is byte-identical to a path that has never existed:

$ grixel ls -as alice /companies/acme/games/secret
grixel: not found: /companies/acme/games/secret

$ grixel ls -as alice /companies/acme/games/doesnotexist
grixel: not found: /companies/acme/games/doesnotexist

This is enforced structurally: every metadata-returning handler routes through one access-resolution chokepoint, and a model-based leak test asserts a less-privileged view is byte-identical whether a hidden path exists or not. The property extends to everything path-anchored: tickets under a hidden project are hidden, and provenance on a commit you cannot read is absent. An AI assistant authenticated as Alice gets a context bundle in which the secret project has never existed, a far stronger statement than "the assistant was told not to mention it."

#Who, or what, made this change

In a lab, a growing fraction of commits are AI-authored or AI-assisted, and "who changed this" is no longer answerable from a username. Any changelist can carry structured provenance (--tool, --model, --role, a note) alongside the server-verified principal:

$ grixel provenance -as maya 9c80ad050744013231cdcf05e1d5856a866ac1843e05b81c6ee0cdd4107ab498
9c80ad05  (declared by alice, 2026-07-05T23:32:41Z)
    claude-code (claude-opus-4-8) — assisted: drafted from quest brief

Provenance is a side-record keyed by the commit hash, never a field inside the commit, so content hashes stay untouched and an agent can auto-tag its own commits through an environment variable. A change can also capture the AI's working context, the session transcript the assistant worked from, content-addressed, bound to the commit, and optionally encrypted client-side so the server cannot read it. When a compliance review asks "which tool wrote this" and then "what did it know when it did," the depot has both answers.

The work itself is trackable in the same namespace: a jobs system with an admin-defined schema of experiment and bug types, path-anchored so hidden projects hide their tickets, event-sourced so agents and CI edit without clobbering each other, and linked bidirectionally to the changelists that resolved them. The backlog lives with the artifacts it describes, under the same permissions.

#Pipelines, retention, and moving bytes

None of this is worth much to a lab if it is CLI-only. Every command takes --json and emits a frozen, versioned envelope (a schemaVersion field, a stable error-code vocabulary), and --json status carries the mapping freshness state as structured data, so a scheduler can refuse to launch a run whose pins have drifted. Beneath the CLI sits a public, stable gRPC API with first-party Go and Python SDKs, so a training launcher can resolve pins and record the launching commit without shelling out.

The opening featured a pruned checkpoint, and labs prune deliberately, so here is the deletion story. grixel ref rm retires a run's ref, and grixeld gc mark-and-sweeps chunk-store objects unreachable from any live ref. The interlock with reproducibility matters most: commits pinned by a committed scope.lock are protected as GC roots, so retiring an old run cannot break a downstream eval project that still pins its dataset. An offline grixeld verify walks every ref, confirms the objects exist in the chunk store, and scans the op-log for read errors; run it after any unclean shutdown. Backup is a point-in-time archive taken under the writer lock, consistent by construction, and that is its limitation too: stop-the-writer or a filesystem snapshot; continuous replication does not exist. The growth curve at lab scale, chunk and metadata volume over a year of real runs, is unmeasured.

For distribution, the design is the caching read proxy: chunks are immutable and hash-named, so a proxy in the rack serves repeated reads locally and a cluster cold-starting the same dataset revision hits the origin roughly once. The design has a ceiling of its own: three hundred nodes each pulling a 200 GB revision is 60 TB through one proxy's network interface, so the real questions are proxies-per-cluster, cache sizing, and eviction under mixed workloads.

· · ·

#Where lakeFS, DVC, and Git LFS stand

Each of these tools is good at what it was built for.

lakeFS gives you Git-like branch semantics over object storage, per repository. For dataset branching and pipeline isolation inside one repository it is strong, and it inherits S3's scale and durability by riding your existing buckets. What it does not give you is one namespace across repositories: a training project consuming a dataset repo, an eval repo, and a standards repo at coordinated pinned revisions is application-layer stitching you build and maintain. In Grixel that mapping is a first-class, server-governed object with a committed lock and drift detection.

DVC bolts data versioning onto git with pointer files, and the pointers deserve more credit than pitches like this usually give them: .dvc files and dvc.lock are committed hash locks, and dvc pull fetches by hash regardless of any machine's cache state. The real gaps are elsewhere. Nothing enforces pointer/data consistency server-side: a pointer commits fine while the object it names was never pushed, and you find out at pull time. The remote is a bucket whose contents can be lifecycled, garbage-collected, or mutated out from under the committed pointers. Permissions live outside the VCS's own model; S3 IAM prefix conditions are path-shaped, but they are a second permission system, hand-maintained, invisible to the version history, and free to drift from it. And dedup is whole-file: the appended shard above stores 100 MiB in a DVC remote instead of 6.5. Against all that, DVC costs nothing to adopt and composes with infrastructure a lab already runs. That is a real argument.

Git LFS was built for "some big files in a code repo," and it works at that scale. A lab's situation is inverted: the dataset is the repo, and the code is the small thing attached to it. Git LFS at dataset scale means smudge-filter pain, no chunk-level dedup between versions, and a server never meant to be the system of record for petabytes.

The honest scoreboard: lakeFS and DVC are production-proven at labs today, and Grixel is not yet. What Grixel offers that neither can retrofit is the single global namespace with governed cross-project pins, invisibility-grade access control, and the AI context plane, all enforced at one server chokepoint. And the review, jobs, and provenance surfaces are the beginning of a forge inside the VCS. A Git forge's protected branches do block direct pushes; the seam is that branch protection is hosting-layer configuration, in force only where that forge is, while Grixel's gate is a committed policy evaluated by the server that owns the bytes, so it travels with the depot.

#What a lab should demand first

ML infra readers are the most skeptical audience I write for, correctly. The state of the evidence:

The caching read proxy has no published multi-node, multi-gigabyte throughput numbers. The design follows Perforce's proxy, with its decades of production history for this exact fan-out problem, and the single-node path is implemented and tested. But "Perforce-shaped" is an argument, not a measurement. The benchmark a lab should demand is the one sketched above: many nodes, a multi-GB revision, rack-local proxies, with proxy count, cache sizing, and eviction reported. Until I publish it, treat cluster fan-out as design intent.

Petabyte scale is the same category: the system is designed with it in mind and measured far below it. The transcripts above are real and verbatim at around 100 MiB per artifact; your 600 GB checkpoint exercises the same code path and is an extrapolation until measured. Two adjacent gaps matter at lab scale. There is no FUSE mount, so pipelines materialize files through checkout and sync; streaming data loaders and datasets larger than a node's local disk are unaddressed today. And the checkout cost of many-small-files layouts, millions of tiny example files rather than packed shards, is unmeasured; pack your shards.

There is no multi-region replication: one authoritative daemon plus read proxies is the deployment shape, and a lab spanning continents will feel that. The single-writer ceiling means what I said earlier: changelists yes, telemetry no. And the retention machinery has its own unmeasured curve, storage growth under a year of dense checkpoints that, as the transcript shows, do not dedup.

What would settle all of this is one lab running one real project next to its existing stack: dataset and checkpoints in the depot, pins in scope.json, the launcher recording commits through the SDK, gc on a schedule. Hold every claim here to the checkpoint transcript's standard: run it, print it, publish the number, whichever way it comes out.

If you run infrastructure at a lab and any of this contradicts something you learned the hard way, tell me. The checkpoint section owes its current form to one such correction.

Grixel is pre-release. Read the docs, or request early access.