erik/acdream
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1
acdream/docs/plans/2026-07-24-modern-runtime-slice-b-envcell-dedup.md

7.3 KiB
Raw Blame History

Modern Runtime Slice B — EnvCell geometry deduplication and full bake

Date: 2026-07-24
Status: Planned; implementation follows the Slice A diagnostic capture
Parent: 2026-07-24-modern-runtime-architecture.md, Slice B
Prior implementation: ../superpowers/plans/2026-07-05-mp1b-pak-and-bake.md

Outcome

Complete the existing MP1b bake without changing runtime behavior. Every EnvCell file ID remains an independently addressable pak key, while cells with the same WorldBuilder geometry identity share one serialized blob and one physical byte range. The client does not consume the pak in this slice.

The full gate must replace the failed 865 GB / 52 minute per-cell bake with a bounded artifact whose size and time scale with unique geometry, not with all 729,888 cell instances.

Fixed contracts

  1. DatCollection remains the only DAT reader.
  2. MeshExtractor remains the only mesh/texture interpreter used by both the live path and the bake.
  3. Geometry identity is one deterministic hash of the complete typed tuple: environmentId, cellStructure, surface count, then the ordered surface IDs. App and Bake call the same Core implementation.
  4. The pure identity helper lives in AcDream.Core.Rendering.Wb.EnvCellGeometryIdentity. It uses FNV-1a in a dedicated high-bit namespace and retains bit 33 for compatibility with existing synthetic-geometry diagnostics. WorldBuilder's original hash = hash * 31 + value calculation remains as a conformance helper only.
  5. Every EnvCell TOC key is still PakKey.Compose(PakAssetType.EnvCellMesh, fileId).
  6. Aliases share offset, length, and crc32. The pak format and reader do not gain an indirection table or a second lookup.
  7. The shared payload carries the runtime geometry ID in ObjectMeshData.ObjectId, matching the current live PrepareCellStructMeshData(geometryId, ...) path.
  8. A geometry-ID collision whose full source tuple differs fails loudly. It may never alias silently.
  9. Blob and TOC order remain deterministic across worker counts.
  10. The destination pak is replaced only after a complete temporary artifact opens and validates. Cancellation or failure preserves the prior file and deletes the temporary file.
  11. The pak itself and raw bake logs are machine artifacts and are not committed. A compact, path-free bake report is committed.

Full-DAT correction (2026-07-24): the first guarded full-catalog run proved that the original fixed contract was impossible. Installed retail DAT cells 0x00030175 (environment=0x277, structure 0, surfaces 0x013B,0x0034) and 0x01BC0105 (environment=0x276, structure 0, surfaces 0x04FC,0x0034) both produce WorldBuilder ID 0x00000002020E8C13 despite having visibly different geometry. The difference is algebraic: one environment step contributes 31³, exactly canceled by the first-surface difference 0x04FC - 0x013B = 31². The collision guard stopped the bake before writing an artifact. B1 was therefore corrected at the root: the old polynomial is retained and tested as historical evidence, while runtime and bake now share the stronger namespaced identity. The full-catalog collision gate remains mandatory.

Implementation checkpoints

B1 — One geometry identity seam

  • Extract one pure identity calculation from the App transaction builder into AcDream.Core.Rendering.Wb.EnvCellGeometryIdentity.
  • Retain the existing App entry points as delegates so production behavior and callers do not change.
  • Add Core conformance tests for empty surfaces, ordered surfaces, the dedicated namespace, App/Bake shared values, and the installed-DAT collision in the legacy WorldBuilder polynomial.

Gate: existing App geometry-ID tests and new Core tests pass with the exact same values.

B2 — Pak blob aliases

  • Make PakWriter retain each written key's immutable TOC receipt.
  • Add AddAlias(aliasKey, existingKey), rejecting a duplicate alias key, missing source key, alias after finish, and accidental self/duplicate use.
  • Emit a normal TOC row with the alias key and the source receipt's physical range and CRC.
  • Keep the reader unchanged except for test-only receipt inspection.

Gate: primary and alias deserialize field-for-field equal, have identical offset/length/CRC, use one physical blob, preserve sorted TOC order, and isolate corruption consistently.

B3 — Deduplicated bake catalog

  • Enumerate EnvCell file IDs exactly as today.
  • Read each EnvCell header once and produce:
    • a sorted fileId -> geometryId alias stream;
    • one full source descriptor per unique geometry ID.
  • Validate equal geometry IDs against the complete tuple (environmentId, cellStructure, ordered surfaces).
  • Extract only the unique descriptors through MeshExtractor.PrepareCellStructMeshData(geometryId, ...).
  • Write the first sorted cell key as the physical blob and every remaining cell key as an alias.
  • Keep GfxObj, Setup, and side-staged particle-preload assets in the same deterministic writer transaction.
  • Report enumerated cells, valid aliases, unique geometries, dedup ratio, physical blobs, failures, wall time, output bytes, and peak working set.

Gate: a filtered real-DAT fixture containing duplicate cells produces more EnvCell keys than physical EnvCell blobs, and every key reads the same payload as a live unique-geometry extraction.

B4 — Transactional output and compatibility

  • Bake beside the destination under a unique temporary filename.
  • Pass cancellation through enumeration and parallel extraction.
  • Open the completed temporary pak and validate its header, TOC count, source DAT iterations, bake-tool version, and every TOC range before replacement.
  • Atomically replace the destination on the same volume.
  • Add stale-version, corruption, cancellation, partial-write, existing-target preservation, and temporary-file cleanup tests.

Gate: no failure path damages or replaces a previously valid pak.

B5 — Full-scale gate

  • Run Release build and the focused Content/Bake suites.
  • Run two filtered bakes with different thread counts and require byte-identical output.
  • Run the complete unfiltered bake against the installed retail DAT files.
  • Open the result, validate every TOC entry, sample aliases across multiple landblocks, and run the existing live-vs-pak field equivalence suite.
  • Record counts, dedup ratio, failures, elapsed time, peak working set, and artifact size in docs/research/.
  • Delete the generated pak after validation unless it is intentionally retained outside the repository for the later Slice C cutover.

Exit: the full bake completes in practical time and disk space, all equivalence and determinism gates are green, duplicate EnvCells demonstrably share physical offsets, and no runtime code consumes the pak.

Verification

Run:

dotnet test tests/AcDream.Content.Tests/AcDream.Content.Tests.csproj -c Release
dotnet test tests/AcDream.Bake.Tests/AcDream.Bake.Tests.csproj -c Release
dotnet build AcDream.slnx -c Release
dotnet test AcDream.slnx -c Release

Then run the full bake from the Release binary, capture its compact report, and validate the output with PakReader. No visual gate is required because this slice changes no runtime path.