diff --git a/AGENTS.md b/AGENTS.md index 54e0130f..28a1b548 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -163,17 +163,23 @@ fresh-vs-reused results and complete reset/reentrancy gates. Slice I2 is also complete: immutable physics/containment BSP, polygon, Setup, bounds, and EnvCell-topology records preserve source float bits and ordering across synthetic corruption gates and representative installed DATs without changing -production traversal. +production traversal. Slice I3 is complete: bake-tool 4 appends typed GfxObj, +Setup, CellStruct, and EnvCell-topology collision payloads; the collision +interface shares the render package's single mmap. Two complete +29,908,271,024-byte bakes produced 2,232,170 typed keys, zero failures, and the +same SHA-256 across 16 and 7 workers. Corruption, cancellation, installed +package, Release build, and 8,371-test / 5-skip gates pass. Slice H's retained UI/frame, bounded-light, and pooled/borrowed/direct network work is complete; the exact seven-checkpoint connected lifecycle/reconnect gate -passes. Slice I is active at I3 from +passes. Slice I is active at I4 from `docs/plans/2026-07-25-modern-runtime-slice-i.md`. The exact G4 visual rollback is `git revert ef1d263337997bb030eadb7b8e71d73dc659907a`; do not revert G3 or the portal-warmup corrections. Evidence: `docs/research/2026-07-25-slice-h-closeout.md` and `docs/research/2026-07-25-slice-g5-production-profile.md` and `docs/research/2026-07-25-slice-i1-transition-scratch.md` and -`docs/research/2026-07-25-slice-i2-flat-collision-schema.md`. +`docs/research/2026-07-25-slice-i2-flat-collision-schema.md` and +`docs/research/2026-07-25-slice-i3-prepared-collision-package.md`. **Structural prerequisite before new M4 subsystem work:** all eight behavior-preserving `GameWindow` decomposition slices and the automated diff --git a/CLAUDE.md b/CLAUDE.md index 9a5fbe9e..66ac413f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -161,17 +161,23 @@ fresh-vs-reused results and complete reset/reentrancy gates. Slice I2 is also complete: immutable physics/containment BSP, polygon, Setup, bounds, and EnvCell-topology records preserve source float bits and ordering across synthetic corruption gates and representative installed DATs without changing -production traversal. +production traversal. Slice I3 is complete: bake-tool 4 appends typed GfxObj, +Setup, CellStruct, and EnvCell-topology collision payloads; the collision +interface shares the render package's single mmap. Two complete +29,908,271,024-byte bakes produced 2,232,170 typed keys, zero failures, and the +same SHA-256 across 16 and 7 workers. Corruption, cancellation, installed +package, Release build, and 8,371-test / 5-skip gates pass. Slice H's retained UI/frame, bounded-light, and pooled/borrowed/direct network work is complete; the exact seven-checkpoint connected lifecycle/reconnect gate -passes. Slice I is active at I3 from +passes. Slice I is active at I4 from `docs/plans/2026-07-25-modern-runtime-slice-i.md`. The exact G4 visual rollback is `git revert ef1d263337997bb030eadb7b8e71d73dc659907a`; do not revert G3 or the portal-warmup corrections. Evidence: `docs/research/2026-07-25-slice-h-closeout.md` and `docs/research/2026-07-25-slice-g5-production-profile.md` and `docs/research/2026-07-25-slice-i1-transition-scratch.md` and -`docs/research/2026-07-25-slice-i2-flat-collision-schema.md`. +`docs/research/2026-07-25-slice-i2-flat-collision-schema.md` and +`docs/research/2026-07-25-slice-i3-prepared-collision-package.md`. **Structural prerequisite before new M4 subsystem work:** all eight behavior-preserving `GameWindow` decomposition slices and the automated diff --git a/docs/architecture/acdream-architecture.md b/docs/architecture/acdream-architecture.md index 07736112..4bba7034 100644 --- a/docs/architecture/acdream-architecture.md +++ b/docs/architecture/acdream-architecture.md @@ -173,6 +173,8 @@ src/ TerrainSurface.cs -> triangle-aware terrain contact (done) BSPQuery.cs -> partial retail BSP dispatcher (active in L.2) TransitionTypes.cs -> SpherePath / CollisionInfo / transition helpers (active in L.2) + FlatCollisionAssets.cs -> immutable array/index collision schema (Slice I2) + FlatCollisionAssetBuilder.cs -> deterministic DAT graph preparation (Slice I2/I3) PhysicsDataCache.cs -> GfxObj / Setup / CellStruct collision data (done, active) ShadowObjectRegistry.cs -> broadphase for nearby physics objects (active) PhysicsEngine.cs -> ResolveWithTransition active player path diff --git a/docs/architecture/worldbuilder-inventory.md b/docs/architecture/worldbuilder-inventory.md index 71133877..95636b56 100644 --- a/docs/architecture/worldbuilder-inventory.md +++ b/docs/architecture/worldbuilder-inventory.md @@ -120,8 +120,9 @@ EnvCell, Surface, palette, and texture graphs during portals. The validated machine-local `acdream.pak` is opened through Content's `IPreparedAssetSource`; typed GfxObj and EnvCell requests deserialize immutable `ObjectMeshData` while retaining the existing App worker, staging, render-thread -upload, cache, ownership, and shutdown contracts. The format-1/bake-tool-3 -payload persists exact batch translucency so App does not reconstruct a +upload, cache, ownership, and shutdown contracts. The original +format-1/bake-tool-3 render payload persists exact batch translucency so App +does not reconstruct a `GfxObjMesh` for metadata. Setup activation uses the package TOC as an explicit type-presence index before reading valid Setup records through the bounded DAT cache. `DatPreparedAssetSource` and `MeshExtractor` remain explicit @@ -131,6 +132,18 @@ bake/equivalence/UI-Studio tools, not a production fallback. Portal → HighRes installed-DAT gates are recorded in `docs/research/2026-07-24-slice-c-prepared-asset-cutover-report.md`. +**Slice I3 prepared collision extension (2026-07-25).** The package remains +format 1 and retains mesh type values 1–3; bake-tool 4 appends typed GfxObj, +Setup, CellStruct, and EnvCell-topology collision payloads. Core owns the +immutable flat records and deterministic raw-DAT flattener. Content owns the +strict little-endian codec and `IPreparedCollisionSource`. +`PakPreparedAssetSource` implements the render and collision interfaces over +one mmap; it does not create a second DAT reader or mapping. CellStruct +payloads alias only after exact serialized-byte comparison, while each +EnvCell topology remains independently keyed. Production traversal is still +the parsed graph oracle until Slice I6. Full-catalog evidence: +`docs/research/2026-07-25-slice-i3-prepared-collision-package.md`. + **Retail VFX hook compatibility seam (2026-07-14).** Chorizite.DatReaderWriter 2.1.7 models `CreateBlockingParticleHook` as the common hook header only, while retail inherits the complete `CreateParticleHook` payload. The narrow readers in diff --git a/docs/plans/2026-04-11-roadmap.md b/docs/plans/2026-04-11-roadmap.md index 48133478..5184d580 100644 --- a/docs/plans/2026-04-11-roadmap.md +++ b/docs/plans/2026-04-11-roadmap.md @@ -1660,7 +1660,7 @@ port in any phase — no separate listing here. > `git revert ef1d263337997bb030eadb7b8e71d73dc659907a`. Slice H is complete: > retained UI/frame work, exact bounded light selection, and > pooled/borrowed/direct network I/O pass 8,292 Release tests / 5 skips and the -> connected lifecycle/reconnect gate. Slice I is active at I3 from +> connected lifecycle/reconnect gate. Slice I is active at I4 from > [its detailed plan](2026-07-25-modern-runtime-slice-i.md). **Spec:** `docs/superpowers/specs/2026-07-05-modern-pipeline-design.md` (the @@ -1685,11 +1685,11 @@ hitch and follows. | MP1a | `AcDream.Content` extraction (GL-free MeshExtractor + boundary records out of App) | ✅ SHIPPED 2026-07-05 — user-gated (renders identical, zero tripwires, perf-neutral); 8 commits `651d041e`..`b0758d77` | | MP1b | Pak format + `acdream-bake` CLI + mmap `PakReader` + equivalence/full-scale gate | ✅ **SHIPPED 2026-07-24** — 729,888 EnvCell keys → 17,117 unique geometries + 712,771 aliases (42.6×), 751,141 total keys / 38,370 physical blobs, zero failures, 28,192.4 MiB, 81.4 s validated atomic publish; full-DAT gate also corrected a real collision in WB's legacy geometry hash. [Report](../research/2026-07-24-slice-b-full-bake-report.md) | | **MP-Alloc (safe batch)** | Reuse per-frame buffers: anim pose, particle draw-list, interior partition, animatedIds/drawableCells sets | ✅ SHIPPED + USER-GATED 2026-07-05 — dense-town frame-time spikes 20–87ms → 6–10ms, alloc spikes (30–75MB single-frame) eliminated, gen2 GC 5–11/window → ~0; user confirms FPS steady. 4 commits `b8c05e2b`..`91afea24` | -| MP-Alloc (hard sites) | EnvCell settled-camera rebuild gate + physics `Transition` reuse | 🟡 Physics half SHIPPED 2026-07-25 — retail-shaped reset-complete transition/query scratch is bit-identical and 0 B/resolve across five profiles; immutable flat collision storage is defined and package integration continues in Slice I3. | +| MP-Alloc (hard sites) | EnvCell settled-camera rebuild gate + physics `Transition` reuse | 🟡 Physics half SHIPPED 2026-07-25 — retail-shaped reset-complete transition/query scratch is bit-identical and 0 B/resolve across five profiles; immutable flat collision storage and its complete deterministic prepared package are shipped; traversal differential work continues in Slice I4. | | MP1c | Streaming cutover to pak + hitch gate | ✅ **SHIPPED 2026-07-24** — typed package-only production source, explicit Setup probes, non-allocating Portal-first lookup, exact translucency metadata, and ordered teardown; capped/uncapped/dense physical routes + user visual gate pass. Uncapped CPU p99 7.825→6.496 ms, GPU p99 3.406→2.706 ms, portal frame max 202.9→39.9 MiB, process allocation −41.8%, zero invalid Setup probes. [Report](../research/2026-07-24-slice-c-prepared-asset-cutover-report.md) | | MP2 | Retail particle distance/cell-view degradation: DAT-authored range, exact infinite/finite off-view update branches, emitter-first render scan, user-requested 2× default range | ✅ SHIPPED 2026-07-17 — Aerlinthe hotspot `0x32000223` resolves to retail 64m and defaults to 128m; landscape/entity streaming distance unchanged | | MP3 | Arch ECS render world + delta submission (the 300-FPS lever) | ⚪ — note: does NOT fix the steady-state GC churn (that's MP-Alloc); MP3 is the draw-submission throughput lever | -| MP4 | Zero-alloc frame loop + flat physics data (residual, post-MP-Alloc) | 🟡 ACTIVE — Slice I1 zero-allocation transition scratch and I2 immutable flat collision schema shipped; I3 package serialization/bake is active. | +| MP4 | Zero-alloc frame loop + flat physics data (residual, post-MP-Alloc) | 🟡 ACTIVE — Slice I1 zero-allocation transition scratch, I2 immutable schema, and I3 complete deterministic collision package shipped; I4 flat traversal shadow implementation is active. | | MP5 | Job-system parallelism | ⚪ stretch, evidence-gated | --- diff --git a/docs/plans/2026-05-12-milestones.md b/docs/plans/2026-05-12-milestones.md index 8b59484e..085caaad 100644 --- a/docs/plans/2026-05-12-milestones.md +++ b/docs/plans/2026-05-12-milestones.md @@ -123,9 +123,13 @@ deterministic immutable physics/containment BSP, polygon, Setup, bounds, and EnvCell-topology records without cutting traversal over. Synthetic corruption gates and representative installed-DAT source identity pass bit-for-bit. The complete I2 Release gate is 3,826 App tests / 3 skips and 8,355 solution tests -/ 5 skips. +/ 5 skips. I3 appends four collision package types and a separate typed +prepared-collision interface over the existing single mmap. Two complete +29,908,271,024-byte bakes contain 2,232,170 keys with zero failures and the +same SHA-256 across different worker counts; cancellation/corruption gates and +8,371 solution tests / 5 skips pass. Slice H is complete: retained UI/frame work, exact bounded light selection, -and pooled/borrowed/direct network I/O pass. Slice I is active at I3 under +and pooled/borrowed/direct network I/O pass. Slice I is active at I4 under [`2026-07-25-modern-runtime-slice-i.md`](2026-07-25-modern-runtime-slice-i.md). The historical exact G4 visual rollback remains `git revert ef1d263337997bb030eadb7b8e71d73dc659907a`. diff --git a/docs/plans/2026-07-24-modern-runtime-architecture.md b/docs/plans/2026-07-24-modern-runtime-architecture.md index cb5455de..87a1ec2a 100644 --- a/docs/plans/2026-07-24-modern-runtime-architecture.md +++ b/docs/plans/2026-07-24-modern-runtime-architecture.md @@ -2,8 +2,8 @@ **Date:** 2026-07-24 -**Status:** Slices A–H and I0–I2 are complete. Slices F–L were explicitly -approved 2026-07-24; Slice I3 is active. +**Status:** Slices A–H and I0–I3 are complete. Slices F–L were explicitly +approved 2026-07-24; Slice I4 is active. **Scope:** Reconcile and sequence the existing Modern Pipeline (`MP`) and Linux/headless (`LH`) tracks using the 2026-07-24 connected performance audit. @@ -1200,8 +1200,10 @@ grounded walkable-publication profiles each measure 0 B/resolve, while fresh-vs-reused output and body state remain bit-identical. I2 now supplies validated immutable flat collision records: synthetic and installed-DAT source identity, exact float bits, deterministic pre-order, and corruption -tripwires pass without changing production traversal. I3 package -serialization and complete-catalog baking are the current execution point. +tripwires pass without changing production traversal. I3 now packages all +flat collision/topology records through a shared typed mmap source: two +complete 2,232,170-key bakes are byte-identical with zero failures. I4 flat +traversal shadow implementation is the current execution point. Evidence: [`../research/2026-07-25-slice-g5-production-profile.md`](../research/2026-07-25-slice-g5-production-profile.md) and @@ -1209,6 +1211,8 @@ and and [`../research/2026-07-25-slice-i2-flat-collision-schema.md`](../research/2026-07-25-slice-i2-flat-collision-schema.md) and +[`../research/2026-07-25-slice-i3-prepared-collision-package.md`](../research/2026-07-25-slice-i3-prepared-collision-package.md) +and [`2026-07-25-modern-runtime-slice-i.md`](2026-07-25-modern-runtime-slice-i.md). The intended order is therefore: @@ -1222,7 +1226,7 @@ honest metrics + committed baselines (A — exit criteria block C) -> incremental render scene (F) -> delta GPU submission (G) -> residual frame cleanup (H-a, H-b, H-c) - -> flat collision assets (I — CURRENT at I3) + -> flat collision assets (I — CURRENT at I4) -> presentation-independent runtime (J — §0.3 authorization) -> Linux/headless/multi-session (K) -> evidence-gated GPU jobs (L) diff --git a/docs/plans/2026-07-25-modern-runtime-slice-i.md b/docs/plans/2026-07-25-modern-runtime-slice-i.md index e3ac18df..6d61b228 100644 --- a/docs/plans/2026-07-25-modern-runtime-slice-i.md +++ b/docs/plans/2026-07-25-modern-runtime-slice-i.md @@ -1,7 +1,8 @@ # Modern runtime Slice I — flat collision assets and zero-allocation physics -**Status:** I0–I2 COMPLETE — retail oracle, zero-allocation transition -scratch, and deterministic immutable collision records fixed; I3 active +**Status:** I0–I3 COMPLETE — retail oracle, zero-allocation transition +scratch, deterministic immutable collision records, and complete prepared +package fixed; I4 active **Parent:** `2026-07-24-modern-runtime-architecture.md`, Slice I **Purpose:** remove parsed DAT object graphs and steady collision allocations without changing one retail collision decision, float, comparison, or traversal @@ -176,7 +177,12 @@ Evidence: Gate: two independent bakes are byte-identical; every collision key required by the canonical route is present; a failed/cancelled bake cannot replace the -last good pak. +last good pak. **PASSED 2026-07-25:** two complete 29,908,271,024-byte bakes +with different worker counts produced the same SHA-256 and 2,232,170 typed +keys with zero failures. Strict serialization/corruption/cancellation gates, +representative installed-package reads, Release build, and 8,371 solution +tests / 5 skips pass. Production traversal remains graph-only. Evidence: +[`../research/2026-07-25-slice-i3-prepared-collision-package.md`](../research/2026-07-25-slice-i3-prepared-collision-package.md). ### I4 — flat traversal shadow implementation diff --git a/docs/research/2026-07-25-slice-i3-prepared-collision-package.md b/docs/research/2026-07-25-slice-i3-prepared-collision-package.md new file mode 100644 index 00000000..5da806c4 --- /dev/null +++ b/docs/research/2026-07-25-slice-i3-prepared-collision-package.md @@ -0,0 +1,97 @@ +# Slice I3 — prepared flat-collision package + +**Date:** 2026-07-25 +**Scope:** Modern Runtime Slice I3 only +**Behavioral boundary:** storage, serialization, bake, and prepared lookup; +production collision traversal remains on the parsed-DAT graph oracle. + +## Result + +The existing format-1 `acdream.pak` now carries four appended asset types: + +| Numeric type | Payload | +|---:|---| +| 4 | GfxObj collision BSP, polygon table, bounding sphere, and visual bounds | +| 5 | Setup cylinders, spheres, and movement dimensions | +| 6 | reusable CellStruct physics BSP, containment BSP, and portal polygons | +| 7 | independently keyed EnvCell portals, visibility, and `seen_outside` | + +The existing mesh type values 1–3 did not change. Bake-tool identity advanced +from 3 to 4; the package binary container remains format 1. + +`FlatCollisionAssetSerializer` is a strict little-endian codec. It: + +- writes every `float` through `SingleToInt32Bits`; +- retains node, polygon, vertex, Setup-shape, portal, and visible-cell order; +- checks counts before allocation and validates combined byte requirements; +- checks cancellation during large rows; +- rejects wrong payload kinds, unsupported schema versions, non-zero reserved + bits, invalid enum/tree/range contracts, truncation, and trailing bytes; +- remains behind the package entry CRC, so a CRC or structural fault marks + only that typed entry corrupt. + +`IPreparedCollisionSource` is separate from the render-only +`IPreparedAssetSource` API. `PakPreparedAssetSource` implements both over the +same `PakReader` and therefore maps the 29.9 GB immutable package once, not +once per subsystem. Render and collision counters are independent. + +CellStruct collision geometry is addressed by every EnvCell source key. The +bake emits ordinary TOC aliases within the same source group and across groups +only after comparing the complete serialized payload bytes. Per-cell topology +is always written under its own key and is never inferred from shared render +geometry. + +## Complete installed-DAT bake + +Two independent full bakes used 16 and 7 workers: + +| Metric | Result | +|---|---:| +| GfxObj mesh keys | 15,318 | +| Setup mesh keys | 5,935 | +| EnvCell mesh keys | 729,888 | +| GfxObj collision keys | 15,318 | +| Setup collision keys | 5,935 | +| CellStruct collision keys | 729,888 | +| EnvCell topology keys | 729,888 | +| Total typed keys | 2,232,170 | +| Physical blobs | 781,196 | +| Failures | 0 | +| File bytes | 29,908,271,024 | +| SHA-256 | `FEE8595D512DCB119C98B3D9911CDF9A905C97B624BCB826CEAF8DFD3F67269C` | + +Both complete files have identical length and SHA-256. Exact-byte +deduplication produced: + +- 9,363 physical GfxObj collision blobs behind 15,318 keys; +- 979 physical Setup collision blobs behind 5,935 keys; +- 2,596 physical CellStruct collision blobs behind 729,888 keys. + +The 16-worker bake completed and validated in 107.5 seconds; the 7-worker bake +completed and validated in 80.5 seconds. Peak bake-process memory was +4.43–4.89 GB working set and 3.81–4.21 GB private. This is offline extraction +state and is not mapped as committed client memory; the production source +memory-maps the immutable file and pages only accessed ranges. + +The validated bake-tool-4 package was installed at the machine-local DAT +location for subsequent Slice I gates. + +## Verification + +- strict codec round-trip and exact-float-bit tests for all four payloads; +- every truncation of a representative payload rejected; +- wrong-kind, negative/impossible count, trailing-byte, CRC, and valid-CRC + structural-corruption tests; +- concurrent prepared reads and separate render/collision accounting; +- raw-DAT versus current cached-source preparation on representative installed + GfxObjs, Setups, and EnvCells; +- filtered 1/8- and 8/3-worker bake equivalence and exact alias tests; +- complete installed package reads for representative GfxObj, Setup, + CellStruct, and EnvCell topology keys; +- failed validation and timed cancellation preserve the last published file; +- Release build: zero warnings; +- complete Release suite: 8,371 passed / 5 skipped. + +No divergence-register row changes: no gameplay decision or traversal route +changed. Slice I4 adds the flat traversal referee while the graph path remains +authoritative. diff --git a/src/AcDream.Bake/AcDream.Bake.csproj b/src/AcDream.Bake/AcDream.Bake.csproj index 263db624..9b04f741 100644 --- a/src/AcDream.Bake/AcDream.Bake.csproj +++ b/src/AcDream.Bake/AcDream.Bake.csproj @@ -10,6 +10,10 @@ true + + + +