docs(perf): plan Slice B EnvCell dedup gate
This commit is contained in:
parent
4285f1dbb1
commit
1560c0513f
1 changed files with 136 additions and 0 deletions
136
docs/plans/2026-07-24-modern-runtime-slice-b-envcell-dedup.md
Normal file
136
docs/plans/2026-07-24-modern-runtime-slice-b-envcell-dedup.md
Normal file
|
|
@ -0,0 +1,136 @@
|
|||
# Modern Runtime Slice B — EnvCell geometry deduplication and full bake
|
||||
|
||||
**Date:** 2026-07-24
|
||||
**Status:** Planned; implementation follows the Slice A diagnostic capture
|
||||
**Parent:** [`2026-07-24-modern-runtime-architecture.md`](2026-07-24-modern-runtime-architecture.md), Slice B
|
||||
**Prior implementation:** [`../superpowers/plans/2026-07-05-mp1b-pak-and-bake.md`](../superpowers/plans/2026-07-05-mp1b-pak-and-bake.md)
|
||||
|
||||
## Outcome
|
||||
|
||||
Complete the existing MP1b bake without changing runtime behavior. Every
|
||||
EnvCell file ID remains an independently addressable pak key, while cells with
|
||||
the same WorldBuilder geometry identity share one serialized blob and one
|
||||
physical byte range. The client does not consume the pak in this slice.
|
||||
|
||||
The full gate must replace the failed 865 GB / 52 minute per-cell bake with a
|
||||
bounded artifact whose size and time scale with unique geometry, not with all
|
||||
729,888 cell instances.
|
||||
|
||||
## Fixed contracts
|
||||
|
||||
1. `DatCollection` remains the only DAT reader.
|
||||
2. `MeshExtractor` remains the only mesh/texture interpreter used by both the
|
||||
live path and the bake.
|
||||
3. Geometry identity is the already-shipped WorldBuilder calculation:
|
||||
`environmentId`, then `cellStructure`, then the ordered surface IDs through
|
||||
the `hash = hash * 31 + value` chain, with bit 33 set.
|
||||
4. The pure identity helper moves to `AcDream.Core.Rendering.Wb`; App and Bake
|
||||
call the same implementation. No second or adapted hash is allowed.
|
||||
5. Every EnvCell TOC key is still
|
||||
`PakKey.Compose(PakAssetType.EnvCellMesh, fileId)`.
|
||||
6. Aliases share `offset`, `length`, and `crc32`. The pak format and reader do
|
||||
not gain an indirection table or a second lookup.
|
||||
7. The shared payload carries the runtime geometry ID in
|
||||
`ObjectMeshData.ObjectId`, matching the current live
|
||||
`PrepareCellStructMeshData(geometryId, ...)` path.
|
||||
8. A geometry-ID collision whose full source tuple differs fails loudly. It
|
||||
may never alias silently.
|
||||
9. Blob and TOC order remain deterministic across worker counts.
|
||||
10. The destination pak is replaced only after a complete temporary artifact
|
||||
opens and validates. Cancellation or failure preserves the prior file and
|
||||
deletes the temporary file.
|
||||
11. The pak itself and raw bake logs are machine artifacts and are not
|
||||
committed. A compact, path-free bake report is committed.
|
||||
|
||||
## Implementation checkpoints
|
||||
|
||||
### B1 — One geometry identity seam
|
||||
|
||||
- Extract the pure WorldBuilder identity calculation from the App transaction
|
||||
builder into `AcDream.Core.Rendering.Wb.EnvCellGeometryIdentity`.
|
||||
- Retain the existing App entry points as delegates so production behavior and
|
||||
callers do not change.
|
||||
- Add Core conformance tests for empty surfaces, ordered surfaces, the bit-33
|
||||
namespace, and App/Bake shared values.
|
||||
|
||||
**Gate:** existing App geometry-ID tests and new Core tests pass with the exact
|
||||
same values.
|
||||
|
||||
### B2 — Pak blob aliases
|
||||
|
||||
- Make `PakWriter` retain each written key's immutable TOC receipt.
|
||||
- Add `AddAlias(aliasKey, existingKey)`, rejecting a duplicate alias key,
|
||||
missing source key, alias after finish, and accidental self/duplicate use.
|
||||
- Emit a normal TOC row with the alias key and the source receipt's physical
|
||||
range and CRC.
|
||||
- Keep the reader unchanged except for test-only receipt inspection.
|
||||
|
||||
**Gate:** primary and alias deserialize field-for-field equal, have identical
|
||||
offset/length/CRC, use one physical blob, preserve sorted TOC order, and isolate
|
||||
corruption consistently.
|
||||
|
||||
### B3 — Deduplicated bake catalog
|
||||
|
||||
- Enumerate EnvCell file IDs exactly as today.
|
||||
- Read each EnvCell header once and produce:
|
||||
- a sorted `fileId -> geometryId` alias stream;
|
||||
- one full source descriptor per unique geometry ID.
|
||||
- Validate equal geometry IDs against the complete tuple
|
||||
`(environmentId, cellStructure, ordered surfaces)`.
|
||||
- Extract only the unique descriptors through
|
||||
`MeshExtractor.PrepareCellStructMeshData(geometryId, ...)`.
|
||||
- Write the first sorted cell key as the physical blob and every remaining
|
||||
cell key as an alias.
|
||||
- Keep GfxObj, Setup, and side-staged particle-preload assets in the same
|
||||
deterministic writer transaction.
|
||||
- Report enumerated cells, valid aliases, unique geometries, dedup ratio,
|
||||
physical blobs, failures, wall time, output bytes, and peak working set.
|
||||
|
||||
**Gate:** a filtered real-DAT fixture containing duplicate cells produces more
|
||||
EnvCell keys than physical EnvCell blobs, and every key reads the same payload
|
||||
as a live unique-geometry extraction.
|
||||
|
||||
### B4 — Transactional output and compatibility
|
||||
|
||||
- Bake beside the destination under a unique temporary filename.
|
||||
- Pass cancellation through enumeration and parallel extraction.
|
||||
- Open the completed temporary pak and validate its header, TOC count, source
|
||||
DAT iterations, bake-tool version, and every TOC range before replacement.
|
||||
- Atomically replace the destination on the same volume.
|
||||
- Add stale-version, corruption, cancellation, partial-write, existing-target
|
||||
preservation, and temporary-file cleanup tests.
|
||||
|
||||
**Gate:** no failure path damages or replaces a previously valid pak.
|
||||
|
||||
### B5 — Full-scale gate
|
||||
|
||||
- Run Release build and the focused Content/Bake suites.
|
||||
- Run two filtered bakes with different thread counts and require byte-identical
|
||||
output.
|
||||
- Run the complete unfiltered bake against the installed retail DAT files.
|
||||
- Open the result, validate every TOC entry, sample aliases across multiple
|
||||
landblocks, and run the existing live-vs-pak field equivalence suite.
|
||||
- Record counts, dedup ratio, failures, elapsed time, peak working set, and
|
||||
artifact size in `docs/research/`.
|
||||
- Delete the generated pak after validation unless it is intentionally retained
|
||||
outside the repository for the later Slice C cutover.
|
||||
|
||||
**Exit:** the full bake completes in practical time and disk space, all
|
||||
equivalence and determinism gates are green, duplicate EnvCells demonstrably
|
||||
share physical offsets, and no runtime code consumes the pak.
|
||||
|
||||
## Verification
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
dotnet test tests/AcDream.Content.Tests/AcDream.Content.Tests.csproj -c Release
|
||||
dotnet test tests/AcDream.Bake.Tests/AcDream.Bake.Tests.csproj -c Release
|
||||
dotnet build AcDream.slnx -c Release
|
||||
dotnet test AcDream.slnx -c Release
|
||||
```
|
||||
|
||||
Then run the full bake from the Release binary, capture its compact report, and
|
||||
validate the output with `PakReader`. No visual gate is required because this
|
||||
slice changes no runtime path.
|
||||
|
||||
Loading…
Add table
Add a link
Reference in a new issue