152 lines
7.3 KiB
Markdown
152 lines
7.3 KiB
Markdown
# Modern Runtime Slice B — EnvCell geometry deduplication and full bake
|
|
|
|
**Date:** 2026-07-24
|
|
**Status:** Planned; implementation follows the Slice A diagnostic capture
|
|
**Parent:** [`2026-07-24-modern-runtime-architecture.md`](2026-07-24-modern-runtime-architecture.md), Slice B
|
|
**Prior implementation:** [`../superpowers/plans/2026-07-05-mp1b-pak-and-bake.md`](../superpowers/plans/2026-07-05-mp1b-pak-and-bake.md)
|
|
|
|
## Outcome
|
|
|
|
Complete the existing MP1b bake without changing runtime behavior. Every
|
|
EnvCell file ID remains an independently addressable pak key, while cells with
|
|
the same WorldBuilder geometry identity share one serialized blob and one
|
|
physical byte range. The client does not consume the pak in this slice.
|
|
|
|
The full gate must replace the failed 865 GB / 52 minute per-cell bake with a
|
|
bounded artifact whose size and time scale with unique geometry, not with all
|
|
729,888 cell instances.
|
|
|
|
## Fixed contracts
|
|
|
|
1. `DatCollection` remains the only DAT reader.
|
|
2. `MeshExtractor` remains the only mesh/texture interpreter used by both the
|
|
live path and the bake.
|
|
3. Geometry identity is one deterministic hash of the complete typed tuple:
|
|
`environmentId`, `cellStructure`, surface count, then the ordered surface
|
|
IDs. App and Bake call the same Core implementation.
|
|
4. The pure identity helper lives in
|
|
`AcDream.Core.Rendering.Wb.EnvCellGeometryIdentity`. It uses FNV-1a in a
|
|
dedicated high-bit namespace and retains bit 33 for compatibility with
|
|
existing synthetic-geometry diagnostics. WorldBuilder's original
|
|
`hash = hash * 31 + value` calculation remains as a conformance helper only.
|
|
5. Every EnvCell TOC key is still
|
|
`PakKey.Compose(PakAssetType.EnvCellMesh, fileId)`.
|
|
6. Aliases share `offset`, `length`, and `crc32`. The pak format and reader do
|
|
not gain an indirection table or a second lookup.
|
|
7. The shared payload carries the runtime geometry ID in
|
|
`ObjectMeshData.ObjectId`, matching the current live
|
|
`PrepareCellStructMeshData(geometryId, ...)` path.
|
|
8. A geometry-ID collision whose full source tuple differs fails loudly. It
|
|
may never alias silently.
|
|
9. Blob and TOC order remain deterministic across worker counts.
|
|
10. The destination pak is replaced only after a complete temporary artifact
|
|
opens and validates. Cancellation or failure preserves the prior file and
|
|
deletes the temporary file.
|
|
11. The pak itself and raw bake logs are machine artifacts and are not
|
|
committed. A compact, path-free bake report is committed.
|
|
|
|
**Full-DAT correction (2026-07-24):** the first guarded full-catalog run proved
|
|
that the original fixed contract was impossible. Installed retail DAT cells
|
|
`0x00030175` (`environment=0x277`, structure `0`, surfaces `0x013B,0x0034`)
|
|
and `0x01BC0105` (`environment=0x276`, structure `0`, surfaces
|
|
`0x04FC,0x0034`) both produce WorldBuilder ID `0x00000002020E8C13` despite
|
|
having visibly different geometry. The difference is algebraic:
|
|
one environment step contributes `31³`, exactly canceled by the first-surface
|
|
difference `0x04FC - 0x013B = 31²`. The collision guard stopped the bake before
|
|
writing an artifact. B1 was therefore corrected at the root: the old polynomial
|
|
is retained and tested as historical evidence, while runtime and bake now share
|
|
the stronger namespaced identity. The full-catalog collision gate remains
|
|
mandatory.
|
|
|
|
## Implementation checkpoints
|
|
|
|
### B1 — One geometry identity seam
|
|
|
|
- Extract one pure identity calculation from the App transaction builder into
|
|
`AcDream.Core.Rendering.Wb.EnvCellGeometryIdentity`.
|
|
- Retain the existing App entry points as delegates so production behavior and
|
|
callers do not change.
|
|
- Add Core conformance tests for empty surfaces, ordered surfaces, the
|
|
dedicated namespace, App/Bake shared values, and the installed-DAT collision
|
|
in the legacy WorldBuilder polynomial.
|
|
|
|
**Gate:** existing App geometry-ID tests and new Core tests pass with the exact
|
|
same values.
|
|
|
|
### B2 — Pak blob aliases
|
|
|
|
- Make `PakWriter` retain each written key's immutable TOC receipt.
|
|
- Add `AddAlias(aliasKey, existingKey)`, rejecting a duplicate alias key,
|
|
missing source key, alias after finish, and accidental self/duplicate use.
|
|
- Emit a normal TOC row with the alias key and the source receipt's physical
|
|
range and CRC.
|
|
- Keep the reader unchanged except for test-only receipt inspection.
|
|
|
|
**Gate:** primary and alias deserialize field-for-field equal, have identical
|
|
offset/length/CRC, use one physical blob, preserve sorted TOC order, and isolate
|
|
corruption consistently.
|
|
|
|
### B3 — Deduplicated bake catalog
|
|
|
|
- Enumerate EnvCell file IDs exactly as today.
|
|
- Read each EnvCell header once and produce:
|
|
- a sorted `fileId -> geometryId` alias stream;
|
|
- one full source descriptor per unique geometry ID.
|
|
- Validate equal geometry IDs against the complete tuple
|
|
`(environmentId, cellStructure, ordered surfaces)`.
|
|
- Extract only the unique descriptors through
|
|
`MeshExtractor.PrepareCellStructMeshData(geometryId, ...)`.
|
|
- Write the first sorted cell key as the physical blob and every remaining
|
|
cell key as an alias.
|
|
- Keep GfxObj, Setup, and side-staged particle-preload assets in the same
|
|
deterministic writer transaction.
|
|
- Report enumerated cells, valid aliases, unique geometries, dedup ratio,
|
|
physical blobs, failures, wall time, output bytes, and peak working set.
|
|
|
|
**Gate:** a filtered real-DAT fixture containing duplicate cells produces more
|
|
EnvCell keys than physical EnvCell blobs, and every key reads the same payload
|
|
as a live unique-geometry extraction.
|
|
|
|
### B4 — Transactional output and compatibility
|
|
|
|
- Bake beside the destination under a unique temporary filename.
|
|
- Pass cancellation through enumeration and parallel extraction.
|
|
- Open the completed temporary pak and validate its header, TOC count, source
|
|
DAT iterations, bake-tool version, and every TOC range before replacement.
|
|
- Atomically replace the destination on the same volume.
|
|
- Add stale-version, corruption, cancellation, partial-write, existing-target
|
|
preservation, and temporary-file cleanup tests.
|
|
|
|
**Gate:** no failure path damages or replaces a previously valid pak.
|
|
|
|
### B5 — Full-scale gate
|
|
|
|
- Run Release build and the focused Content/Bake suites.
|
|
- Run two filtered bakes with different thread counts and require byte-identical
|
|
output.
|
|
- Run the complete unfiltered bake against the installed retail DAT files.
|
|
- Open the result, validate every TOC entry, sample aliases across multiple
|
|
landblocks, and run the existing live-vs-pak field equivalence suite.
|
|
- Record counts, dedup ratio, failures, elapsed time, peak working set, and
|
|
artifact size in `docs/research/`.
|
|
- Delete the generated pak after validation unless it is intentionally retained
|
|
outside the repository for the later Slice C cutover.
|
|
|
|
**Exit:** the full bake completes in practical time and disk space, all
|
|
equivalence and determinism gates are green, duplicate EnvCells demonstrably
|
|
share physical offsets, and no runtime code consumes the pak.
|
|
|
|
## Verification
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
dotnet test tests/AcDream.Content.Tests/AcDream.Content.Tests.csproj -c Release
|
|
dotnet test tests/AcDream.Bake.Tests/AcDream.Bake.Tests.csproj -c Release
|
|
dotnet build AcDream.slnx -c Release
|
|
dotnet test AcDream.slnx -c Release
|
|
```
|
|
|
|
Then run the full bake from the Release binary, capture its compact report, and
|
|
validate the output with `PakReader`. No visual gate is required because this
|
|
slice changes no runtime path.
|