feat(content): bake and read flat collision assets

Append strict collision/topology payloads to the existing prepared package so later physics cutover can drop parsed DAT graphs without adding a second mapping or changing traversal behavior. The full 2,232,170-key catalog is deterministic across worker counts, exact-byte aliased, corruption-isolated, and cancellation-safe.
This commit is contained in:
Erik 2026-07-25 15:22:08 +02:00
parent d9300c7854
commit 9cd42417a8
29 changed files with 2868 additions and 63 deletions

View file

@ -173,6 +173,8 @@ src/
TerrainSurface.cs -> triangle-aware terrain contact (done)
BSPQuery.cs -> partial retail BSP dispatcher (active in L.2)
TransitionTypes.cs -> SpherePath / CollisionInfo / transition helpers (active in L.2)
FlatCollisionAssets.cs -> immutable array/index collision schema (Slice I2)
FlatCollisionAssetBuilder.cs -> deterministic DAT graph preparation (Slice I2/I3)
PhysicsDataCache.cs -> GfxObj / Setup / CellStruct collision data (done, active)
ShadowObjectRegistry.cs -> broadphase for nearby physics objects (active)
PhysicsEngine.cs -> ResolveWithTransition active player path

View file

@ -120,8 +120,9 @@ EnvCell, Surface, palette, and texture graphs during portals. The validated
machine-local `acdream.pak` is opened through Content's
`IPreparedAssetSource`; typed GfxObj and EnvCell requests deserialize immutable
`ObjectMeshData` while retaining the existing App worker, staging, render-thread
upload, cache, ownership, and shutdown contracts. The format-1/bake-tool-3
payload persists exact batch translucency so App does not reconstruct a
upload, cache, ownership, and shutdown contracts. The original
format-1/bake-tool-3 render payload persists exact batch translucency so App
does not reconstruct a
`GfxObjMesh` for metadata. Setup activation uses the package TOC as an explicit
type-presence index before reading valid Setup records through the bounded DAT
cache. `DatPreparedAssetSource` and `MeshExtractor` remain explicit
@ -131,6 +132,18 @@ bake/equivalence/UI-Studio tools, not a production fallback. Portal → HighRes
installed-DAT gates are recorded in
`docs/research/2026-07-24-slice-c-prepared-asset-cutover-report.md`.
**Slice I3 prepared collision extension (2026-07-25).** The package remains
format 1 and retains mesh type values 13; bake-tool 4 appends typed GfxObj,
Setup, CellStruct, and EnvCell-topology collision payloads. Core owns the
immutable flat records and deterministic raw-DAT flattener. Content owns the
strict little-endian codec and `IPreparedCollisionSource`.
`PakPreparedAssetSource` implements the render and collision interfaces over
one mmap; it does not create a second DAT reader or mapping. CellStruct
payloads alias only after exact serialized-byte comparison, while each
EnvCell topology remains independently keyed. Production traversal is still
the parsed graph oracle until Slice I6. Full-catalog evidence:
`docs/research/2026-07-25-slice-i3-prepared-collision-package.md`.
**Retail VFX hook compatibility seam (2026-07-14).** Chorizite.DatReaderWriter
2.1.7 models `CreateBlockingParticleHook` as the common hook header only, while
retail inherits the complete `CreateParticleHook` payload. The narrow readers in

View file

@ -1660,7 +1660,7 @@ port in any phase — no separate listing here.
> `git revert ef1d263337997bb030eadb7b8e71d73dc659907a`. Slice H is complete:
> retained UI/frame work, exact bounded light selection, and
> pooled/borrowed/direct network I/O pass 8,292 Release tests / 5 skips and the
> connected lifecycle/reconnect gate. Slice I is active at I3 from
> connected lifecycle/reconnect gate. Slice I is active at I4 from
> [its detailed plan](2026-07-25-modern-runtime-slice-i.md).
**Spec:** `docs/superpowers/specs/2026-07-05-modern-pipeline-design.md` (the
@ -1685,11 +1685,11 @@ hitch and follows.
| MP1a | `AcDream.Content` extraction (GL-free MeshExtractor + boundary records out of App) | ✅ SHIPPED 2026-07-05 — user-gated (renders identical, zero tripwires, perf-neutral); 8 commits `651d041e`..`b0758d77` |
| MP1b | Pak format + `acdream-bake` CLI + mmap `PakReader` + equivalence/full-scale gate | ✅ **SHIPPED 2026-07-24** — 729,888 EnvCell keys → 17,117 unique geometries + 712,771 aliases (42.6×), 751,141 total keys / 38,370 physical blobs, zero failures, 28,192.4 MiB, 81.4 s validated atomic publish; full-DAT gate also corrected a real collision in WB's legacy geometry hash. [Report](../research/2026-07-24-slice-b-full-bake-report.md) |
| **MP-Alloc (safe batch)** | Reuse per-frame buffers: anim pose, particle draw-list, interior partition, animatedIds/drawableCells sets | ✅ SHIPPED + USER-GATED 2026-07-05 — dense-town frame-time spikes 2087ms → 610ms, alloc spikes (3075MB single-frame) eliminated, gen2 GC 511/window → ~0; user confirms FPS steady. 4 commits `b8c05e2b`..`91afea24` |
| MP-Alloc (hard sites) | EnvCell settled-camera rebuild gate + physics `Transition` reuse | 🟡 Physics half SHIPPED 2026-07-25 — retail-shaped reset-complete transition/query scratch is bit-identical and 0 B/resolve across five profiles; immutable flat collision storage is defined and package integration continues in Slice I3. |
| MP-Alloc (hard sites) | EnvCell settled-camera rebuild gate + physics `Transition` reuse | 🟡 Physics half SHIPPED 2026-07-25 — retail-shaped reset-complete transition/query scratch is bit-identical and 0 B/resolve across five profiles; immutable flat collision storage and its complete deterministic prepared package are shipped; traversal differential work continues in Slice I4. |
| MP1c | Streaming cutover to pak + hitch gate | ✅ **SHIPPED 2026-07-24** — typed package-only production source, explicit Setup probes, non-allocating Portal-first lookup, exact translucency metadata, and ordered teardown; capped/uncapped/dense physical routes + user visual gate pass. Uncapped CPU p99 7.825→6.496 ms, GPU p99 3.406→2.706 ms, portal frame max 202.9→39.9 MiB, process allocation 41.8%, zero invalid Setup probes. [Report](../research/2026-07-24-slice-c-prepared-asset-cutover-report.md) |
| MP2 | Retail particle distance/cell-view degradation: DAT-authored range, exact infinite/finite off-view update branches, emitter-first render scan, user-requested 2× default range | ✅ SHIPPED 2026-07-17 — Aerlinthe hotspot `0x32000223` resolves to retail 64m and defaults to 128m; landscape/entity streaming distance unchanged |
| MP3 | Arch ECS render world + delta submission (the 300-FPS lever) | ⚪ — note: does NOT fix the steady-state GC churn (that's MP-Alloc); MP3 is the draw-submission throughput lever |
| MP4 | Zero-alloc frame loop + flat physics data (residual, post-MP-Alloc) | 🟡 ACTIVE — Slice I1 zero-allocation transition scratch and I2 immutable flat collision schema shipped; I3 package serialization/bake is active. |
| MP4 | Zero-alloc frame loop + flat physics data (residual, post-MP-Alloc) | 🟡 ACTIVE — Slice I1 zero-allocation transition scratch, I2 immutable schema, and I3 complete deterministic collision package shipped; I4 flat traversal shadow implementation is active. |
| MP5 | Job-system parallelism | ⚪ stretch, evidence-gated |
---

View file

@ -123,9 +123,13 @@ deterministic immutable physics/containment BSP, polygon, Setup, bounds, and
EnvCell-topology records without cutting traversal over. Synthetic corruption
gates and representative installed-DAT source identity pass bit-for-bit. The
complete I2 Release gate is 3,826 App tests / 3 skips and 8,355 solution tests
/ 5 skips.
/ 5 skips. I3 appends four collision package types and a separate typed
prepared-collision interface over the existing single mmap. Two complete
29,908,271,024-byte bakes contain 2,232,170 keys with zero failures and the
same SHA-256 across different worker counts; cancellation/corruption gates and
8,371 solution tests / 5 skips pass.
Slice H is complete: retained UI/frame work, exact bounded light selection,
and pooled/borrowed/direct network I/O pass. Slice I is active at I3 under
and pooled/borrowed/direct network I/O pass. Slice I is active at I4 under
[`2026-07-25-modern-runtime-slice-i.md`](2026-07-25-modern-runtime-slice-i.md).
The historical exact G4 visual rollback remains
`git revert ef1d263337997bb030eadb7b8e71d73dc659907a`.

View file

@ -2,8 +2,8 @@
**Date:** 2026-07-24
**Status:** Slices AH and I0I2 are complete. Slices FL were explicitly
approved 2026-07-24; Slice I3 is active.
**Status:** Slices AH and I0I3 are complete. Slices FL were explicitly
approved 2026-07-24; Slice I4 is active.
**Scope:** Reconcile and sequence the existing Modern Pipeline (`MP`) and
Linux/headless (`LH`) tracks using the 2026-07-24 connected performance audit.
@ -1200,8 +1200,10 @@ grounded walkable-publication profiles each measure 0 B/resolve, while
fresh-vs-reused output and body state remain bit-identical. I2 now supplies
validated immutable flat collision records: synthetic and installed-DAT
source identity, exact float bits, deterministic pre-order, and corruption
tripwires pass without changing production traversal. I3 package
serialization and complete-catalog baking are the current execution point.
tripwires pass without changing production traversal. I3 now packages all
flat collision/topology records through a shared typed mmap source: two
complete 2,232,170-key bakes are byte-identical with zero failures. I4 flat
traversal shadow implementation is the current execution point.
Evidence:
[`../research/2026-07-25-slice-g5-production-profile.md`](../research/2026-07-25-slice-g5-production-profile.md)
and
@ -1209,6 +1211,8 @@ and
and
[`../research/2026-07-25-slice-i2-flat-collision-schema.md`](../research/2026-07-25-slice-i2-flat-collision-schema.md)
and
[`../research/2026-07-25-slice-i3-prepared-collision-package.md`](../research/2026-07-25-slice-i3-prepared-collision-package.md)
and
[`2026-07-25-modern-runtime-slice-i.md`](2026-07-25-modern-runtime-slice-i.md).
The intended order is therefore:
@ -1222,7 +1226,7 @@ honest metrics + committed baselines (A — exit criteria block C)
-> incremental render scene (F)
-> delta GPU submission (G)
-> residual frame cleanup (H-a, H-b, H-c)
-> flat collision assets (I — CURRENT at I3)
-> flat collision assets (I — CURRENT at I4)
-> presentation-independent runtime (J — §0.3 authorization)
-> Linux/headless/multi-session (K)
-> evidence-gated GPU jobs (L)

View file

@ -1,7 +1,8 @@
# Modern runtime Slice I — flat collision assets and zero-allocation physics
**Status:** I0I2 COMPLETE — retail oracle, zero-allocation transition
scratch, and deterministic immutable collision records fixed; I3 active
**Status:** I0I3 COMPLETE — retail oracle, zero-allocation transition
scratch, deterministic immutable collision records, and complete prepared
package fixed; I4 active
**Parent:** `2026-07-24-modern-runtime-architecture.md`, Slice I
**Purpose:** remove parsed DAT object graphs and steady collision allocations
without changing one retail collision decision, float, comparison, or traversal
@ -176,7 +177,12 @@ Evidence:
Gate: two independent bakes are byte-identical; every collision key required
by the canonical route is present; a failed/cancelled bake cannot replace the
last good pak.
last good pak. **PASSED 2026-07-25:** two complete 29,908,271,024-byte bakes
with different worker counts produced the same SHA-256 and 2,232,170 typed
keys with zero failures. Strict serialization/corruption/cancellation gates,
representative installed-package reads, Release build, and 8,371 solution
tests / 5 skips pass. Production traversal remains graph-only. Evidence:
[`../research/2026-07-25-slice-i3-prepared-collision-package.md`](../research/2026-07-25-slice-i3-prepared-collision-package.md).
### I4 — flat traversal shadow implementation

View file

@ -0,0 +1,97 @@
# Slice I3 — prepared flat-collision package
**Date:** 2026-07-25
**Scope:** Modern Runtime Slice I3 only
**Behavioral boundary:** storage, serialization, bake, and prepared lookup;
production collision traversal remains on the parsed-DAT graph oracle.
## Result
The existing format-1 `acdream.pak` now carries four appended asset types:
| Numeric type | Payload |
|---:|---|
| 4 | GfxObj collision BSP, polygon table, bounding sphere, and visual bounds |
| 5 | Setup cylinders, spheres, and movement dimensions |
| 6 | reusable CellStruct physics BSP, containment BSP, and portal polygons |
| 7 | independently keyed EnvCell portals, visibility, and `seen_outside` |
The existing mesh type values 13 did not change. Bake-tool identity advanced
from 3 to 4; the package binary container remains format 1.
`FlatCollisionAssetSerializer` is a strict little-endian codec. It:
- writes every `float` through `SingleToInt32Bits`;
- retains node, polygon, vertex, Setup-shape, portal, and visible-cell order;
- checks counts before allocation and validates combined byte requirements;
- checks cancellation during large rows;
- rejects wrong payload kinds, unsupported schema versions, non-zero reserved
bits, invalid enum/tree/range contracts, truncation, and trailing bytes;
- remains behind the package entry CRC, so a CRC or structural fault marks
only that typed entry corrupt.
`IPreparedCollisionSource` is separate from the render-only
`IPreparedAssetSource` API. `PakPreparedAssetSource` implements both over the
same `PakReader` and therefore maps the 29.9 GB immutable package once, not
once per subsystem. Render and collision counters are independent.
CellStruct collision geometry is addressed by every EnvCell source key. The
bake emits ordinary TOC aliases within the same source group and across groups
only after comparing the complete serialized payload bytes. Per-cell topology
is always written under its own key and is never inferred from shared render
geometry.
## Complete installed-DAT bake
Two independent full bakes used 16 and 7 workers:
| Metric | Result |
|---|---:|
| GfxObj mesh keys | 15,318 |
| Setup mesh keys | 5,935 |
| EnvCell mesh keys | 729,888 |
| GfxObj collision keys | 15,318 |
| Setup collision keys | 5,935 |
| CellStruct collision keys | 729,888 |
| EnvCell topology keys | 729,888 |
| Total typed keys | 2,232,170 |
| Physical blobs | 781,196 |
| Failures | 0 |
| File bytes | 29,908,271,024 |
| SHA-256 | `FEE8595D512DCB119C98B3D9911CDF9A905C97B624BCB826CEAF8DFD3F67269C` |
Both complete files have identical length and SHA-256. Exact-byte
deduplication produced:
- 9,363 physical GfxObj collision blobs behind 15,318 keys;
- 979 physical Setup collision blobs behind 5,935 keys;
- 2,596 physical CellStruct collision blobs behind 729,888 keys.
The 16-worker bake completed and validated in 107.5 seconds; the 7-worker bake
completed and validated in 80.5 seconds. Peak bake-process memory was
4.434.89 GB working set and 3.814.21 GB private. This is offline extraction
state and is not mapped as committed client memory; the production source
memory-maps the immutable file and pages only accessed ranges.
The validated bake-tool-4 package was installed at the machine-local DAT
location for subsequent Slice I gates.
## Verification
- strict codec round-trip and exact-float-bit tests for all four payloads;
- every truncation of a representative payload rejected;
- wrong-kind, negative/impossible count, trailing-byte, CRC, and valid-CRC
structural-corruption tests;
- concurrent prepared reads and separate render/collision accounting;
- raw-DAT versus current cached-source preparation on representative installed
GfxObjs, Setups, and EnvCells;
- filtered 1/8- and 8/3-worker bake equivalence and exact alias tests;
- complete installed package reads for representative GfxObj, Setup,
CellStruct, and EnvCell topology keys;
- failed validation and timed cancellation preserve the last published file;
- Release build: zero warnings;
- complete Release suite: 8,371 passed / 5 skipped.
No divergence-register row changes: no gameplay decision or traversal route
changed. Slice I4 adds the flat traversal referee while the graph path remains
authoritative.