docs(perf): plan Slice B EnvCell dedup gate
This commit is contained in:
parent
4285f1dbb1
commit
1560c0513f
1 changed files with 136 additions and 0 deletions
136
docs/plans/2026-07-24-modern-runtime-slice-b-envcell-dedup.md
Normal file
136
docs/plans/2026-07-24-modern-runtime-slice-b-envcell-dedup.md
Normal file
|
|
@ -0,0 +1,136 @@
|
||||||
|
# Modern Runtime Slice B — EnvCell geometry deduplication and full bake
|
||||||
|
|
||||||
|
**Date:** 2026-07-24
|
||||||
|
**Status:** Planned; implementation follows the Slice A diagnostic capture
|
||||||
|
**Parent:** [`2026-07-24-modern-runtime-architecture.md`](2026-07-24-modern-runtime-architecture.md), Slice B
|
||||||
|
**Prior implementation:** [`../superpowers/plans/2026-07-05-mp1b-pak-and-bake.md`](../superpowers/plans/2026-07-05-mp1b-pak-and-bake.md)
|
||||||
|
|
||||||
|
## Outcome
|
||||||
|
|
||||||
|
Complete the existing MP1b bake without changing runtime behavior. Every
|
||||||
|
EnvCell file ID remains an independently addressable pak key, while cells with
|
||||||
|
the same WorldBuilder geometry identity share one serialized blob and one
|
||||||
|
physical byte range. The client does not consume the pak in this slice.
|
||||||
|
|
||||||
|
The full gate must replace the failed 865 GB / 52 minute per-cell bake with a
|
||||||
|
bounded artifact whose size and time scale with unique geometry, not with all
|
||||||
|
729,888 cell instances.
|
||||||
|
|
||||||
|
## Fixed contracts
|
||||||
|
|
||||||
|
1. `DatCollection` remains the only DAT reader.
|
||||||
|
2. `MeshExtractor` remains the only mesh/texture interpreter used by both the
|
||||||
|
live path and the bake.
|
||||||
|
3. Geometry identity is the already-shipped WorldBuilder calculation:
|
||||||
|
`environmentId`, then `cellStructure`, then the ordered surface IDs through
|
||||||
|
the `hash = hash * 31 + value` chain, with bit 33 set.
|
||||||
|
4. The pure identity helper moves to `AcDream.Core.Rendering.Wb`; App and Bake
|
||||||
|
call the same implementation. No second or adapted hash is allowed.
|
||||||
|
5. Every EnvCell TOC key is still
|
||||||
|
`PakKey.Compose(PakAssetType.EnvCellMesh, fileId)`.
|
||||||
|
6. Aliases share `offset`, `length`, and `crc32`. The pak format and reader do
|
||||||
|
not gain an indirection table or a second lookup.
|
||||||
|
7. The shared payload carries the runtime geometry ID in
|
||||||
|
`ObjectMeshData.ObjectId`, matching the current live
|
||||||
|
`PrepareCellStructMeshData(geometryId, ...)` path.
|
||||||
|
8. A geometry-ID collision whose full source tuple differs fails loudly. It
|
||||||
|
may never alias silently.
|
||||||
|
9. Blob and TOC order remain deterministic across worker counts.
|
||||||
|
10. The destination pak is replaced only after a complete temporary artifact
|
||||||
|
opens and validates. Cancellation or failure preserves the prior file and
|
||||||
|
deletes the temporary file.
|
||||||
|
11. The pak itself and raw bake logs are machine artifacts and are not
|
||||||
|
committed. A compact, path-free bake report is committed.
|
||||||
|
|
||||||
|
## Implementation checkpoints
|
||||||
|
|
||||||
|
### B1 — One geometry identity seam
|
||||||
|
|
||||||
|
- Extract the pure WorldBuilder identity calculation from the App transaction
|
||||||
|
builder into `AcDream.Core.Rendering.Wb.EnvCellGeometryIdentity`.
|
||||||
|
- Retain the existing App entry points as delegates so production behavior and
|
||||||
|
callers do not change.
|
||||||
|
- Add Core conformance tests for empty surfaces, ordered surfaces, the bit-33
|
||||||
|
namespace, and App/Bake shared values.
|
||||||
|
|
||||||
|
**Gate:** existing App geometry-ID tests and new Core tests pass with the exact
|
||||||
|
same values.
|
||||||
|
|
||||||
|
### B2 — Pak blob aliases
|
||||||
|
|
||||||
|
- Make `PakWriter` retain each written key's immutable TOC receipt.
|
||||||
|
- Add `AddAlias(aliasKey, existingKey)`, rejecting a duplicate alias key,
|
||||||
|
missing source key, alias after finish, and accidental self/duplicate use.
|
||||||
|
- Emit a normal TOC row with the alias key and the source receipt's physical
|
||||||
|
range and CRC.
|
||||||
|
- Keep the reader unchanged except for test-only receipt inspection.
|
||||||
|
|
||||||
|
**Gate:** primary and alias deserialize field-for-field equal, have identical
|
||||||
|
offset/length/CRC, use one physical blob, preserve sorted TOC order, and isolate
|
||||||
|
corruption consistently.
|
||||||
|
|
||||||
|
### B3 — Deduplicated bake catalog
|
||||||
|
|
||||||
|
- Enumerate EnvCell file IDs exactly as today.
|
||||||
|
- Read each EnvCell header once and produce:
|
||||||
|
- a sorted `fileId -> geometryId` alias stream;
|
||||||
|
- one full source descriptor per unique geometry ID.
|
||||||
|
- Validate equal geometry IDs against the complete tuple
|
||||||
|
`(environmentId, cellStructure, ordered surfaces)`.
|
||||||
|
- Extract only the unique descriptors through
|
||||||
|
`MeshExtractor.PrepareCellStructMeshData(geometryId, ...)`.
|
||||||
|
- Write the first sorted cell key as the physical blob and every remaining
|
||||||
|
cell key as an alias.
|
||||||
|
- Keep GfxObj, Setup, and side-staged particle-preload assets in the same
|
||||||
|
deterministic writer transaction.
|
||||||
|
- Report enumerated cells, valid aliases, unique geometries, dedup ratio,
|
||||||
|
physical blobs, failures, wall time, output bytes, and peak working set.
|
||||||
|
|
||||||
|
**Gate:** a filtered real-DAT fixture containing duplicate cells produces more
|
||||||
|
EnvCell keys than physical EnvCell blobs, and every key reads the same payload
|
||||||
|
as a live unique-geometry extraction.
|
||||||
|
|
||||||
|
### B4 — Transactional output and compatibility
|
||||||
|
|
||||||
|
- Bake beside the destination under a unique temporary filename.
|
||||||
|
- Pass cancellation through enumeration and parallel extraction.
|
||||||
|
- Open the completed temporary pak and validate its header, TOC count, source
|
||||||
|
DAT iterations, bake-tool version, and every TOC range before replacement.
|
||||||
|
- Atomically replace the destination on the same volume.
|
||||||
|
- Add stale-version, corruption, cancellation, partial-write, existing-target
|
||||||
|
preservation, and temporary-file cleanup tests.
|
||||||
|
|
||||||
|
**Gate:** no failure path damages or replaces a previously valid pak.
|
||||||
|
|
||||||
|
### B5 — Full-scale gate
|
||||||
|
|
||||||
|
- Run Release build and the focused Content/Bake suites.
|
||||||
|
- Run two filtered bakes with different thread counts and require byte-identical
|
||||||
|
output.
|
||||||
|
- Run the complete unfiltered bake against the installed retail DAT files.
|
||||||
|
- Open the result, validate every TOC entry, sample aliases across multiple
|
||||||
|
landblocks, and run the existing live-vs-pak field equivalence suite.
|
||||||
|
- Record counts, dedup ratio, failures, elapsed time, peak working set, and
|
||||||
|
artifact size in `docs/research/`.
|
||||||
|
- Delete the generated pak after validation unless it is intentionally retained
|
||||||
|
outside the repository for the later Slice C cutover.
|
||||||
|
|
||||||
|
**Exit:** the full bake completes in practical time and disk space, all
|
||||||
|
equivalence and determinism gates are green, duplicate EnvCells demonstrably
|
||||||
|
share physical offsets, and no runtime code consumes the pak.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
Run:
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
dotnet test tests/AcDream.Content.Tests/AcDream.Content.Tests.csproj -c Release
|
||||||
|
dotnet test tests/AcDream.Bake.Tests/AcDream.Bake.Tests.csproj -c Release
|
||||||
|
dotnet build AcDream.slnx -c Release
|
||||||
|
dotnet test AcDream.slnx -c Release
|
||||||
|
```
|
||||||
|
|
||||||
|
Then run the full bake from the Release binary, capture its compact report, and
|
||||||
|
validate the output with `PakReader`. No visual gate is required because this
|
||||||
|
slice changes no runtime path.
|
||||||
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue