erik/acdream
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

docs(pipeline): correct 6.6 - EnvCell dedup is REQUIRED (full-bake gate)

Browse source 
Full-bake gate baked all 729,888 EnvCells as separate fileId-keyed blobs = 865 GB pak in 52 min, 97 pct near-duplicate cell geometry. The prior 'v1-acceptable duplication' call was wrong by ~100x. Bake-time dedup (compute geomId, extract each unique geometry once, alias all sharing cell-fileIds to one blob offset) is now a v1 requirement; needs no format or reader change and collapses size AND bake time together. MP1b not shipped until the dedup slice lands.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Erik 2026-07-05 23:16:12 +02:00
parent 859cf5ec02
commit 4ca4cf9501
1 changed files with 25 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

35
docs/superpowers/specs/2026-07-05-modern-pipeline-design.md
View file

@ -214,19 +214,34 @@ listed follow-up, not v1.
class gone; user visual gate (no missing/wrong geometry); legacy decode path deleted
from the streaming hot path in the gate commit.
### 6.6 MP1c design note — EnvCell keying (recorded 2026-07-05, MP1b review finding)
### 6.6 EnvCell dedup — REQUIRED (corrected 2026-07-05 after the full-bake gate)
The runtime deduplicates interior-cell geometry under a **content hash**
(`EnvCellRenderer.GetEnvCellGeomId(environmentId, cellStructure, surfaces)`,
bit 33 set — verbatim WB), NOT under the cell's dat fileId; many cells share
one geometry. The pak keys EnvCell blobs by cell fileId. MP1c's cutover
therefore maps at the existing seam: `RegisterCell` has the concrete cell in
hand when it wants geometry for a geomId — read `pak[cellFileId]`, register
under the geomId, and post-load dedup continues exactly as today. Cost:
cells sharing geometry duplicate blob content on disk (v1-acceptable;
a one-blob-per-unique-content TOC-alias refinement is a future format
option). Baking under geomId keys directly is NOT viable in format v1 —
the hash is a full 64-bit value and does not fit PakKey's 56 usable bits.
bit 33 set — verbatim WB), NOT under the cell's dat fileId; **many cells share
one geometry** (AC dungeons repeat a small palette of tile geometries thousands
of times).
**The full-bake gate (2026-07-05) proved dedup is mandatory, not optional.**
Baking all 729,888 EnvCells as separate fileId-keyed blobs produced an
**865 GB pak in 52 minutes** — 97% of the blobs were near-duplicate cell
geometry. A prior note here called that duplication "v1-acceptable"; **that was
wrong by ~100×.** Dedup is a v1 requirement.
**The fix (bake-tool only, no pak format change):** dedup at bake time.
Compute each cell's geomId (cheap — reads the EnvCell/CellStruct header, no mesh
extraction), extract each **unique geometry once**, and emit one TOC entry per
cell-fileId all pointing at the one shared blob offset. The TOC already maps
key→offset with no uniqueness constraint on offsets, and `PakReader` is oblivious
to the sharing (pure offset lookup) — so aliasing needs no format or reader
change. Expected result: ~730K blobs → single-digit-thousand unique blobs, pak
865 GB → a few GB, bake 52 min → minutes (all three collapse from the same fix).
**MP1c cutover (unchanged):** `RegisterCell` reads `pak[cellFileId]` → the shared
blob → registers under the runtime geomId; post-load dedup continues exactly as
today. Keying blobs by geomId *directly* remains non-viable (the hash exceeds
PakKey's 56 usable bits) — the fileId-key + aliased-offset scheme is why the
format needs no change.
## 7. MP2 — Distance-degrade port (hide-only first cut)