From 1560c0513fb4e5e37f812c24ab5baade2eb9a40f Mon Sep 17 00:00:00 2001 From: Erik Date: Fri, 24 Jul 2026 13:16:33 +0200 Subject: [PATCH] docs(perf): plan Slice B EnvCell dedup gate --- ...24-modern-runtime-slice-b-envcell-dedup.md | 136 ++++++++++++++++++ 1 file changed, 136 insertions(+) create mode 100644 docs/plans/2026-07-24-modern-runtime-slice-b-envcell-dedup.md diff --git a/docs/plans/2026-07-24-modern-runtime-slice-b-envcell-dedup.md b/docs/plans/2026-07-24-modern-runtime-slice-b-envcell-dedup.md new file mode 100644 index 00000000..e7d43c14 --- /dev/null +++ b/docs/plans/2026-07-24-modern-runtime-slice-b-envcell-dedup.md @@ -0,0 +1,136 @@ +# Modern Runtime Slice B — EnvCell geometry deduplication and full bake + +**Date:** 2026-07-24 +**Status:** Planned; implementation follows the Slice A diagnostic capture +**Parent:** [`2026-07-24-modern-runtime-architecture.md`](2026-07-24-modern-runtime-architecture.md), Slice B +**Prior implementation:** [`../superpowers/plans/2026-07-05-mp1b-pak-and-bake.md`](../superpowers/plans/2026-07-05-mp1b-pak-and-bake.md) + +## Outcome + +Complete the existing MP1b bake without changing runtime behavior. Every +EnvCell file ID remains an independently addressable pak key, while cells with +the same WorldBuilder geometry identity share one serialized blob and one +physical byte range. The client does not consume the pak in this slice. + +The full gate must replace the failed 865 GB / 52 minute per-cell bake with a +bounded artifact whose size and time scale with unique geometry, not with all +729,888 cell instances. + +## Fixed contracts + +1. `DatCollection` remains the only DAT reader. +2. `MeshExtractor` remains the only mesh/texture interpreter used by both the + live path and the bake. +3. Geometry identity is the already-shipped WorldBuilder calculation: + `environmentId`, then `cellStructure`, then the ordered surface IDs through + the `hash = hash * 31 + value` chain, with bit 33 set. +4. The pure identity helper moves to `AcDream.Core.Rendering.Wb`; App and Bake + call the same implementation. No second or adapted hash is allowed. +5. Every EnvCell TOC key is still + `PakKey.Compose(PakAssetType.EnvCellMesh, fileId)`. +6. Aliases share `offset`, `length`, and `crc32`. The pak format and reader do + not gain an indirection table or a second lookup. +7. The shared payload carries the runtime geometry ID in + `ObjectMeshData.ObjectId`, matching the current live + `PrepareCellStructMeshData(geometryId, ...)` path. +8. A geometry-ID collision whose full source tuple differs fails loudly. It + may never alias silently. +9. Blob and TOC order remain deterministic across worker counts. +10. The destination pak is replaced only after a complete temporary artifact + opens and validates. Cancellation or failure preserves the prior file and + deletes the temporary file. +11. The pak itself and raw bake logs are machine artifacts and are not + committed. A compact, path-free bake report is committed. + +## Implementation checkpoints + +### B1 — One geometry identity seam + +- Extract the pure WorldBuilder identity calculation from the App transaction + builder into `AcDream.Core.Rendering.Wb.EnvCellGeometryIdentity`. +- Retain the existing App entry points as delegates so production behavior and + callers do not change. +- Add Core conformance tests for empty surfaces, ordered surfaces, the bit-33 + namespace, and App/Bake shared values. + +**Gate:** existing App geometry-ID tests and new Core tests pass with the exact +same values. + +### B2 — Pak blob aliases + +- Make `PakWriter` retain each written key's immutable TOC receipt. +- Add `AddAlias(aliasKey, existingKey)`, rejecting a duplicate alias key, + missing source key, alias after finish, and accidental self/duplicate use. +- Emit a normal TOC row with the alias key and the source receipt's physical + range and CRC. +- Keep the reader unchanged except for test-only receipt inspection. + +**Gate:** primary and alias deserialize field-for-field equal, have identical +offset/length/CRC, use one physical blob, preserve sorted TOC order, and isolate +corruption consistently. + +### B3 — Deduplicated bake catalog + +- Enumerate EnvCell file IDs exactly as today. +- Read each EnvCell header once and produce: + - a sorted `fileId -> geometryId` alias stream; + - one full source descriptor per unique geometry ID. +- Validate equal geometry IDs against the complete tuple + `(environmentId, cellStructure, ordered surfaces)`. +- Extract only the unique descriptors through + `MeshExtractor.PrepareCellStructMeshData(geometryId, ...)`. +- Write the first sorted cell key as the physical blob and every remaining + cell key as an alias. +- Keep GfxObj, Setup, and side-staged particle-preload assets in the same + deterministic writer transaction. +- Report enumerated cells, valid aliases, unique geometries, dedup ratio, + physical blobs, failures, wall time, output bytes, and peak working set. + +**Gate:** a filtered real-DAT fixture containing duplicate cells produces more +EnvCell keys than physical EnvCell blobs, and every key reads the same payload +as a live unique-geometry extraction. + +### B4 — Transactional output and compatibility + +- Bake beside the destination under a unique temporary filename. +- Pass cancellation through enumeration and parallel extraction. +- Open the completed temporary pak and validate its header, TOC count, source + DAT iterations, bake-tool version, and every TOC range before replacement. +- Atomically replace the destination on the same volume. +- Add stale-version, corruption, cancellation, partial-write, existing-target + preservation, and temporary-file cleanup tests. + +**Gate:** no failure path damages or replaces a previously valid pak. + +### B5 — Full-scale gate + +- Run Release build and the focused Content/Bake suites. +- Run two filtered bakes with different thread counts and require byte-identical + output. +- Run the complete unfiltered bake against the installed retail DAT files. +- Open the result, validate every TOC entry, sample aliases across multiple + landblocks, and run the existing live-vs-pak field equivalence suite. +- Record counts, dedup ratio, failures, elapsed time, peak working set, and + artifact size in `docs/research/`. +- Delete the generated pak after validation unless it is intentionally retained + outside the repository for the later Slice C cutover. + +**Exit:** the full bake completes in practical time and disk space, all +equivalence and determinism gates are green, duplicate EnvCells demonstrably +share physical offsets, and no runtime code consumes the pak. + +## Verification + +Run: + +```powershell +dotnet test tests/AcDream.Content.Tests/AcDream.Content.Tests.csproj -c Release +dotnet test tests/AcDream.Bake.Tests/AcDream.Bake.Tests.csproj -c Release +dotnet build AcDream.slnx -c Release +dotnet test AcDream.slnx -c Release +``` + +Then run the full bake from the Release binary, capture its compact report, and +validate the output with `PakReader`. No visual gate is required because this +slice changes no runtime path. +