From 4ca4cf95012dd89bf484cc19f1edbf192983704e Mon Sep 17 00:00:00 2001 From: Erik Date: Sun, 5 Jul 2026 23:16:12 +0200 Subject: [PATCH] docs(pipeline): correct 6.6 - EnvCell dedup is REQUIRED (full-bake gate) Full-bake gate baked all 729,888 EnvCells as separate fileId-keyed blobs = 865 GB pak in 52 min, 97 pct near-duplicate cell geometry. The prior 'v1-acceptable duplication' call was wrong by ~100x. Bake-time dedup (compute geomId, extract each unique geometry once, alias all sharing cell-fileIds to one blob offset) is now a v1 requirement; needs no format or reader change and collapses size AND bake time together. MP1b not shipped until the dedup slice lands. Co-Authored-By: Claude Fable 5 --- .../2026-07-05-modern-pipeline-design.md | 35 +++++++++++++------ 1 file changed, 25 insertions(+), 10 deletions(-) diff --git a/docs/superpowers/specs/2026-07-05-modern-pipeline-design.md b/docs/superpowers/specs/2026-07-05-modern-pipeline-design.md index c8978b43..1a5293ef 100644 --- a/docs/superpowers/specs/2026-07-05-modern-pipeline-design.md +++ b/docs/superpowers/specs/2026-07-05-modern-pipeline-design.md @@ -214,19 +214,34 @@ listed follow-up, not v1. class gone; user visual gate (no missing/wrong geometry); legacy decode path deleted from the streaming hot path in the gate commit. -### 6.6 MP1c design note — EnvCell keying (recorded 2026-07-05, MP1b review finding) +### 6.6 EnvCell dedup — REQUIRED (corrected 2026-07-05 after the full-bake gate) The runtime deduplicates interior-cell geometry under a **content hash** (`EnvCellRenderer.GetEnvCellGeomId(environmentId, cellStructure, surfaces)`, -bit 33 set — verbatim WB), NOT under the cell's dat fileId; many cells share -one geometry. The pak keys EnvCell blobs by cell fileId. MP1c's cutover -therefore maps at the existing seam: `RegisterCell` has the concrete cell in -hand when it wants geometry for a geomId — read `pak[cellFileId]`, register -under the geomId, and post-load dedup continues exactly as today. Cost: -cells sharing geometry duplicate blob content on disk (v1-acceptable; -a one-blob-per-unique-content TOC-alias refinement is a future format -option). Baking under geomId keys directly is NOT viable in format v1 — -the hash is a full 64-bit value and does not fit PakKey's 56 usable bits. +bit 33 set — verbatim WB), NOT under the cell's dat fileId; **many cells share +one geometry** (AC dungeons repeat a small palette of tile geometries thousands +of times). + +**The full-bake gate (2026-07-05) proved dedup is mandatory, not optional.** +Baking all 729,888 EnvCells as separate fileId-keyed blobs produced an +**865 GB pak in 52 minutes** — 97% of the blobs were near-duplicate cell +geometry. A prior note here called that duplication "v1-acceptable"; **that was +wrong by ~100×.** Dedup is a v1 requirement. + +**The fix (bake-tool only, no pak format change):** dedup at bake time. +Compute each cell's geomId (cheap — reads the EnvCell/CellStruct header, no mesh +extraction), extract each **unique geometry once**, and emit one TOC entry per +cell-fileId all pointing at the one shared blob offset. The TOC already maps +key→offset with no uniqueness constraint on offsets, and `PakReader` is oblivious +to the sharing (pure offset lookup) — so aliasing needs no format or reader +change. Expected result: ~730K blobs → single-digit-thousand unique blobs, pak +865 GB → a few GB, bake 52 min → minutes (all three collapse from the same fix). + +**MP1c cutover (unchanged):** `RegisterCell` reads `pak[cellFileId]` → the shared +blob → registers under the runtime geomId; post-load dedup continues exactly as +today. Keying blobs by geomId *directly* remains non-viable (the hash exceeds +PakKey's 56 usable bits) — the fileId-key + aliased-offset scheme is why the +format needs no change. ## 7. MP2 — Distance-degrade port (hide-only first cut)