feat(content): bake and read flat collision assets
Append strict collision/topology payloads to the existing prepared package so later physics cutover can drop parsed DAT graphs without adding a second mapping or changing traversal behavior. The full 2,232,170-key catalog is deterministic across worker counts, exact-byte aliased, corruption-isolated, and cancellation-safe.
This commit is contained in:
parent
d9300c7854
commit
9cd42417a8
29 changed files with 2868 additions and 63 deletions
|
|
@ -173,6 +173,8 @@ src/
|
|||
TerrainSurface.cs -> triangle-aware terrain contact (done)
|
||||
BSPQuery.cs -> partial retail BSP dispatcher (active in L.2)
|
||||
TransitionTypes.cs -> SpherePath / CollisionInfo / transition helpers (active in L.2)
|
||||
FlatCollisionAssets.cs -> immutable array/index collision schema (Slice I2)
|
||||
FlatCollisionAssetBuilder.cs -> deterministic DAT graph preparation (Slice I2/I3)
|
||||
PhysicsDataCache.cs -> GfxObj / Setup / CellStruct collision data (done, active)
|
||||
ShadowObjectRegistry.cs -> broadphase for nearby physics objects (active)
|
||||
PhysicsEngine.cs -> ResolveWithTransition active player path
|
||||
|
|
|
|||
|
|
@ -120,8 +120,9 @@ EnvCell, Surface, palette, and texture graphs during portals. The validated
|
|||
machine-local `acdream.pak` is opened through Content's
|
||||
`IPreparedAssetSource`; typed GfxObj and EnvCell requests deserialize immutable
|
||||
`ObjectMeshData` while retaining the existing App worker, staging, render-thread
|
||||
upload, cache, ownership, and shutdown contracts. The format-1/bake-tool-3
|
||||
payload persists exact batch translucency so App does not reconstruct a
|
||||
upload, cache, ownership, and shutdown contracts. The original
|
||||
format-1/bake-tool-3 render payload persists exact batch translucency so App
|
||||
does not reconstruct a
|
||||
`GfxObjMesh` for metadata. Setup activation uses the package TOC as an explicit
|
||||
type-presence index before reading valid Setup records through the bounded DAT
|
||||
cache. `DatPreparedAssetSource` and `MeshExtractor` remain explicit
|
||||
|
|
@ -131,6 +132,18 @@ bake/equivalence/UI-Studio tools, not a production fallback. Portal → HighRes
|
|||
installed-DAT gates are recorded in
|
||||
`docs/research/2026-07-24-slice-c-prepared-asset-cutover-report.md`.
|
||||
|
||||
**Slice I3 prepared collision extension (2026-07-25).** The package remains
|
||||
format 1 and retains mesh type values 1–3; bake-tool 4 appends typed GfxObj,
|
||||
Setup, CellStruct, and EnvCell-topology collision payloads. Core owns the
|
||||
immutable flat records and deterministic raw-DAT flattener. Content owns the
|
||||
strict little-endian codec and `IPreparedCollisionSource`.
|
||||
`PakPreparedAssetSource` implements the render and collision interfaces over
|
||||
one mmap; it does not create a second DAT reader or mapping. CellStruct
|
||||
payloads alias only after exact serialized-byte comparison, while each
|
||||
EnvCell topology remains independently keyed. Production traversal is still
|
||||
the parsed graph oracle until Slice I6. Full-catalog evidence:
|
||||
`docs/research/2026-07-25-slice-i3-prepared-collision-package.md`.
|
||||
|
||||
**Retail VFX hook compatibility seam (2026-07-14).** Chorizite.DatReaderWriter
|
||||
2.1.7 models `CreateBlockingParticleHook` as the common hook header only, while
|
||||
retail inherits the complete `CreateParticleHook` payload. The narrow readers in
|
||||
|
|
|
|||
|
|
@ -1660,7 +1660,7 @@ port in any phase — no separate listing here.
|
|||
> `git revert ef1d263337997bb030eadb7b8e71d73dc659907a`. Slice H is complete:
|
||||
> retained UI/frame work, exact bounded light selection, and
|
||||
> pooled/borrowed/direct network I/O pass 8,292 Release tests / 5 skips and the
|
||||
> connected lifecycle/reconnect gate. Slice I is active at I3 from
|
||||
> connected lifecycle/reconnect gate. Slice I is active at I4 from
|
||||
> [its detailed plan](2026-07-25-modern-runtime-slice-i.md).
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-07-05-modern-pipeline-design.md` (the
|
||||
|
|
@ -1685,11 +1685,11 @@ hitch and follows.
|
|||
| MP1a | `AcDream.Content` extraction (GL-free MeshExtractor + boundary records out of App) | ✅ SHIPPED 2026-07-05 — user-gated (renders identical, zero tripwires, perf-neutral); 8 commits `651d041e`..`b0758d77` |
|
||||
| MP1b | Pak format + `acdream-bake` CLI + mmap `PakReader` + equivalence/full-scale gate | ✅ **SHIPPED 2026-07-24** — 729,888 EnvCell keys → 17,117 unique geometries + 712,771 aliases (42.6×), 751,141 total keys / 38,370 physical blobs, zero failures, 28,192.4 MiB, 81.4 s validated atomic publish; full-DAT gate also corrected a real collision in WB's legacy geometry hash. [Report](../research/2026-07-24-slice-b-full-bake-report.md) |
|
||||
| **MP-Alloc (safe batch)** | Reuse per-frame buffers: anim pose, particle draw-list, interior partition, animatedIds/drawableCells sets | ✅ SHIPPED + USER-GATED 2026-07-05 — dense-town frame-time spikes 20–87ms → 6–10ms, alloc spikes (30–75MB single-frame) eliminated, gen2 GC 5–11/window → ~0; user confirms FPS steady. 4 commits `b8c05e2b`..`91afea24` |
|
||||
| MP-Alloc (hard sites) | EnvCell settled-camera rebuild gate + physics `Transition` reuse | 🟡 Physics half SHIPPED 2026-07-25 — retail-shaped reset-complete transition/query scratch is bit-identical and 0 B/resolve across five profiles; immutable flat collision storage is defined and package integration continues in Slice I3. |
|
||||
| MP-Alloc (hard sites) | EnvCell settled-camera rebuild gate + physics `Transition` reuse | 🟡 Physics half SHIPPED 2026-07-25 — retail-shaped reset-complete transition/query scratch is bit-identical and 0 B/resolve across five profiles; immutable flat collision storage and its complete deterministic prepared package are shipped; traversal differential work continues in Slice I4. |
|
||||
| MP1c | Streaming cutover to pak + hitch gate | ✅ **SHIPPED 2026-07-24** — typed package-only production source, explicit Setup probes, non-allocating Portal-first lookup, exact translucency metadata, and ordered teardown; capped/uncapped/dense physical routes + user visual gate pass. Uncapped CPU p99 7.825→6.496 ms, GPU p99 3.406→2.706 ms, portal frame max 202.9→39.9 MiB, process allocation −41.8%, zero invalid Setup probes. [Report](../research/2026-07-24-slice-c-prepared-asset-cutover-report.md) |
|
||||
| MP2 | Retail particle distance/cell-view degradation: DAT-authored range, exact infinite/finite off-view update branches, emitter-first render scan, user-requested 2× default range | ✅ SHIPPED 2026-07-17 — Aerlinthe hotspot `0x32000223` resolves to retail 64m and defaults to 128m; landscape/entity streaming distance unchanged |
|
||||
| MP3 | Arch ECS render world + delta submission (the 300-FPS lever) | ⚪ — note: does NOT fix the steady-state GC churn (that's MP-Alloc); MP3 is the draw-submission throughput lever |
|
||||
| MP4 | Zero-alloc frame loop + flat physics data (residual, post-MP-Alloc) | 🟡 ACTIVE — Slice I1 zero-allocation transition scratch and I2 immutable flat collision schema shipped; I3 package serialization/bake is active. |
|
||||
| MP4 | Zero-alloc frame loop + flat physics data (residual, post-MP-Alloc) | 🟡 ACTIVE — Slice I1 zero-allocation transition scratch, I2 immutable schema, and I3 complete deterministic collision package shipped; I4 flat traversal shadow implementation is active. |
|
||||
| MP5 | Job-system parallelism | ⚪ stretch, evidence-gated |
|
||||
|
||||
---
|
||||
|
|
|
|||
|
|
@ -123,9 +123,13 @@ deterministic immutable physics/containment BSP, polygon, Setup, bounds, and
|
|||
EnvCell-topology records without cutting traversal over. Synthetic corruption
|
||||
gates and representative installed-DAT source identity pass bit-for-bit. The
|
||||
complete I2 Release gate is 3,826 App tests / 3 skips and 8,355 solution tests
|
||||
/ 5 skips.
|
||||
/ 5 skips. I3 appends four collision package types and a separate typed
|
||||
prepared-collision interface over the existing single mmap. Two complete
|
||||
29,908,271,024-byte bakes contain 2,232,170 keys with zero failures and the
|
||||
same SHA-256 across different worker counts; cancellation/corruption gates and
|
||||
8,371 solution tests / 5 skips pass.
|
||||
Slice H is complete: retained UI/frame work, exact bounded light selection,
|
||||
and pooled/borrowed/direct network I/O pass. Slice I is active at I3 under
|
||||
and pooled/borrowed/direct network I/O pass. Slice I is active at I4 under
|
||||
[`2026-07-25-modern-runtime-slice-i.md`](2026-07-25-modern-runtime-slice-i.md).
|
||||
The historical exact G4 visual rollback remains
|
||||
`git revert ef1d263337997bb030eadb7b8e71d73dc659907a`.
|
||||
|
|
|
|||
|
|
@ -2,8 +2,8 @@
|
|||
|
||||
**Date:** 2026-07-24
|
||||
|
||||
**Status:** Slices A–H and I0–I2 are complete. Slices F–L were explicitly
|
||||
approved 2026-07-24; Slice I3 is active.
|
||||
**Status:** Slices A–H and I0–I3 are complete. Slices F–L were explicitly
|
||||
approved 2026-07-24; Slice I4 is active.
|
||||
|
||||
**Scope:** Reconcile and sequence the existing Modern Pipeline (`MP`) and
|
||||
Linux/headless (`LH`) tracks using the 2026-07-24 connected performance audit.
|
||||
|
|
@ -1200,8 +1200,10 @@ grounded walkable-publication profiles each measure 0 B/resolve, while
|
|||
fresh-vs-reused output and body state remain bit-identical. I2 now supplies
|
||||
validated immutable flat collision records: synthetic and installed-DAT
|
||||
source identity, exact float bits, deterministic pre-order, and corruption
|
||||
tripwires pass without changing production traversal. I3 package
|
||||
serialization and complete-catalog baking are the current execution point.
|
||||
tripwires pass without changing production traversal. I3 now packages all
|
||||
flat collision/topology records through a shared typed mmap source: two
|
||||
complete 2,232,170-key bakes are byte-identical with zero failures. I4 flat
|
||||
traversal shadow implementation is the current execution point.
|
||||
Evidence:
|
||||
[`../research/2026-07-25-slice-g5-production-profile.md`](../research/2026-07-25-slice-g5-production-profile.md)
|
||||
and
|
||||
|
|
@ -1209,6 +1211,8 @@ and
|
|||
and
|
||||
[`../research/2026-07-25-slice-i2-flat-collision-schema.md`](../research/2026-07-25-slice-i2-flat-collision-schema.md)
|
||||
and
|
||||
[`../research/2026-07-25-slice-i3-prepared-collision-package.md`](../research/2026-07-25-slice-i3-prepared-collision-package.md)
|
||||
and
|
||||
[`2026-07-25-modern-runtime-slice-i.md`](2026-07-25-modern-runtime-slice-i.md).
|
||||
|
||||
The intended order is therefore:
|
||||
|
|
@ -1222,7 +1226,7 @@ honest metrics + committed baselines (A — exit criteria block C)
|
|||
-> incremental render scene (F)
|
||||
-> delta GPU submission (G)
|
||||
-> residual frame cleanup (H-a, H-b, H-c)
|
||||
-> flat collision assets (I — CURRENT at I3)
|
||||
-> flat collision assets (I — CURRENT at I4)
|
||||
-> presentation-independent runtime (J — §0.3 authorization)
|
||||
-> Linux/headless/multi-session (K)
|
||||
-> evidence-gated GPU jobs (L)
|
||||
|
|
|
|||
|
|
@ -1,7 +1,8 @@
|
|||
# Modern runtime Slice I — flat collision assets and zero-allocation physics
|
||||
|
||||
**Status:** I0–I2 COMPLETE — retail oracle, zero-allocation transition
|
||||
scratch, and deterministic immutable collision records fixed; I3 active
|
||||
**Status:** I0–I3 COMPLETE — retail oracle, zero-allocation transition
|
||||
scratch, deterministic immutable collision records, and complete prepared
|
||||
package fixed; I4 active
|
||||
**Parent:** `2026-07-24-modern-runtime-architecture.md`, Slice I
|
||||
**Purpose:** remove parsed DAT object graphs and steady collision allocations
|
||||
without changing one retail collision decision, float, comparison, or traversal
|
||||
|
|
@ -176,7 +177,12 @@ Evidence:
|
|||
|
||||
Gate: two independent bakes are byte-identical; every collision key required
|
||||
by the canonical route is present; a failed/cancelled bake cannot replace the
|
||||
last good pak.
|
||||
last good pak. **PASSED 2026-07-25:** two complete 29,908,271,024-byte bakes
|
||||
with different worker counts produced the same SHA-256 and 2,232,170 typed
|
||||
keys with zero failures. Strict serialization/corruption/cancellation gates,
|
||||
representative installed-package reads, Release build, and 8,371 solution
|
||||
tests / 5 skips pass. Production traversal remains graph-only. Evidence:
|
||||
[`../research/2026-07-25-slice-i3-prepared-collision-package.md`](../research/2026-07-25-slice-i3-prepared-collision-package.md).
|
||||
|
||||
### I4 — flat traversal shadow implementation
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,97 @@
|
|||
# Slice I3 — prepared flat-collision package
|
||||
|
||||
**Date:** 2026-07-25
|
||||
**Scope:** Modern Runtime Slice I3 only
|
||||
**Behavioral boundary:** storage, serialization, bake, and prepared lookup;
|
||||
production collision traversal remains on the parsed-DAT graph oracle.
|
||||
|
||||
## Result
|
||||
|
||||
The existing format-1 `acdream.pak` now carries four appended asset types:
|
||||
|
||||
| Numeric type | Payload |
|
||||
|---:|---|
|
||||
| 4 | GfxObj collision BSP, polygon table, bounding sphere, and visual bounds |
|
||||
| 5 | Setup cylinders, spheres, and movement dimensions |
|
||||
| 6 | reusable CellStruct physics BSP, containment BSP, and portal polygons |
|
||||
| 7 | independently keyed EnvCell portals, visibility, and `seen_outside` |
|
||||
|
||||
The existing mesh type values 1–3 did not change. Bake-tool identity advanced
|
||||
from 3 to 4; the package binary container remains format 1.
|
||||
|
||||
`FlatCollisionAssetSerializer` is a strict little-endian codec. It:
|
||||
|
||||
- writes every `float` through `SingleToInt32Bits`;
|
||||
- retains node, polygon, vertex, Setup-shape, portal, and visible-cell order;
|
||||
- checks counts before allocation and validates combined byte requirements;
|
||||
- checks cancellation during large rows;
|
||||
- rejects wrong payload kinds, unsupported schema versions, non-zero reserved
|
||||
bits, invalid enum/tree/range contracts, truncation, and trailing bytes;
|
||||
- remains behind the package entry CRC, so a CRC or structural fault marks
|
||||
only that typed entry corrupt.
|
||||
|
||||
`IPreparedCollisionSource` is separate from the render-only
|
||||
`IPreparedAssetSource` API. `PakPreparedAssetSource` implements both over the
|
||||
same `PakReader` and therefore maps the 29.9 GB immutable package once, not
|
||||
once per subsystem. Render and collision counters are independent.
|
||||
|
||||
CellStruct collision geometry is addressed by every EnvCell source key. The
|
||||
bake emits ordinary TOC aliases within the same source group and across groups
|
||||
only after comparing the complete serialized payload bytes. Per-cell topology
|
||||
is always written under its own key and is never inferred from shared render
|
||||
geometry.
|
||||
|
||||
## Complete installed-DAT bake
|
||||
|
||||
Two independent full bakes used 16 and 7 workers:
|
||||
|
||||
| Metric | Result |
|
||||
|---|---:|
|
||||
| GfxObj mesh keys | 15,318 |
|
||||
| Setup mesh keys | 5,935 |
|
||||
| EnvCell mesh keys | 729,888 |
|
||||
| GfxObj collision keys | 15,318 |
|
||||
| Setup collision keys | 5,935 |
|
||||
| CellStruct collision keys | 729,888 |
|
||||
| EnvCell topology keys | 729,888 |
|
||||
| Total typed keys | 2,232,170 |
|
||||
| Physical blobs | 781,196 |
|
||||
| Failures | 0 |
|
||||
| File bytes | 29,908,271,024 |
|
||||
| SHA-256 | `FEE8595D512DCB119C98B3D9911CDF9A905C97B624BCB826CEAF8DFD3F67269C` |
|
||||
|
||||
Both complete files have identical length and SHA-256. Exact-byte
|
||||
deduplication produced:
|
||||
|
||||
- 9,363 physical GfxObj collision blobs behind 15,318 keys;
|
||||
- 979 physical Setup collision blobs behind 5,935 keys;
|
||||
- 2,596 physical CellStruct collision blobs behind 729,888 keys.
|
||||
|
||||
The 16-worker bake completed and validated in 107.5 seconds; the 7-worker bake
|
||||
completed and validated in 80.5 seconds. Peak bake-process memory was
|
||||
4.43–4.89 GB working set and 3.81–4.21 GB private. This is offline extraction
|
||||
state and is not mapped as committed client memory; the production source
|
||||
memory-maps the immutable file and pages only accessed ranges.
|
||||
|
||||
The validated bake-tool-4 package was installed at the machine-local DAT
|
||||
location for subsequent Slice I gates.
|
||||
|
||||
## Verification
|
||||
|
||||
- strict codec round-trip and exact-float-bit tests for all four payloads;
|
||||
- every truncation of a representative payload rejected;
|
||||
- wrong-kind, negative/impossible count, trailing-byte, CRC, and valid-CRC
|
||||
structural-corruption tests;
|
||||
- concurrent prepared reads and separate render/collision accounting;
|
||||
- raw-DAT versus current cached-source preparation on representative installed
|
||||
GfxObjs, Setups, and EnvCells;
|
||||
- filtered 1/8- and 8/3-worker bake equivalence and exact alias tests;
|
||||
- complete installed package reads for representative GfxObj, Setup,
|
||||
CellStruct, and EnvCell topology keys;
|
||||
- failed validation and timed cancellation preserve the last published file;
|
||||
- Release build: zero warnings;
|
||||
- complete Release suite: 8,371 passed / 5 skipped.
|
||||
|
||||
No divergence-register row changes: no gameplay decision or traversal route
|
||||
changed. Slice I4 adds the flat traversal referee while the graph path remains
|
||||
authoritative.
|
||||
Loading…
Add table
Add a link
Reference in a new issue