diff --git a/docs/superpowers/specs/2026-07-05-modern-pipeline-design.md b/docs/superpowers/specs/2026-07-05-modern-pipeline-design.md new file mode 100644 index 00000000..9d71a97a --- /dev/null +++ b/docs/superpowers/specs/2026-07-05-modern-pipeline-design.md @@ -0,0 +1,364 @@ +# Design — Modern Pipeline side track (MP): baked assets + ECS render world + zero-alloc frame loop + +**Date:** 2026-07-05 +**Status:** brainstorm-approved; awaiting user spec review +**Origin:** user-commissioned performance side track ("fully modern pipeline, with ECS"). +Explicitly NOT M1.5 work — see §12 for the firewall. +**Phase prefix:** `MP0–MP5` (MP = Modern Pipeline; avoids collision with the existing +P0 conformance-apparatus plan). +**Implementation cadence:** each MP phase gets its own implementation plan +(`docs/superpowers/plans/`) when it starts; this spec is the umbrella design. The +first plan covers MP0 (+ MP1 scaffold) only. + +--- + +## 1. Motivation + measured baseline + +acdream's steady-state dense-town frame rate is ~165 FPS (~6 ms/frame) after the +2026-06-23/24 fixes, versus 300+ FPS in comparable modern titles on the same machine. +Traversal (streaming, portal hops) still hitches. The measured facts this design +builds on (do NOT re-derive; sources in §14): + +| Fact | Evidence | +|---|---| +| GPU is nearly idle: ~0.5 ms/frame real GPU time in dense town | 2026-06-23/24 deep-dive; the "12 ms GPU" was a glFinish artifact | +| The frame is CPU-submission-bound | dense-town attribution report; per-cell batch fix took 75→~165 FPS | +| `_datLock` contention (the 30↔200 swing) is FIXED — lockwait 88 ms p95 → 0.2 µs | `536f1c04` + handoff §1 | +| No distance-degrade: every frustum-visible object draws at full detail at any distance (~1,400 draws / ~24,000 instances) | 2026-06-23 handoff §2; `GfxObjDegradeResolver.cs` "always slot 0" comment | +| Parked CPU levers: `WbDrawDispatcher` per-frame static-scenery rebuild (~2.1 ms), cross-frame dispatcher cache (judged HIGH-risk as a *patch* to the frozen renderer) | feedback_render_perf_measurement.md | +| Load hitches come from runtime dat decode (mesh extraction, texture decode, BSP graph construction) + GC | thread-safety investigation; two-tier streaming architecture | +| Runtime dat reads carry an unbounded second-layer cache (`DatDatabaseWrapper._cache`) stacked on DRW's own | `DatCollectionAdapter.cs:119` | + +The conclusion the whole design follows from: **the language and the GPU are not the +bottleneck; the data layout and the per-frame/at-load CPU work are.** Modernizing means +(a) moving decode work offline, (b) making the renderer process *changes* instead of +re-describing the scene every frame, (c) flattening hot data, (d) removing frame-loop +allocation. + +## 2. Goals & acceptance + +1. **Smoothness (primary):** no frame over ~16 ms during any traversal — town walking + while streaming, portal hops, dungeon transitions. Measured: p99 frame time from the + MP0 profiler across a scripted traversal route, plus user-eyes confirmation. +2. **Throughput:** 300+ FPS sustained in the dense-town worst cases (Fort Tethana, + Arwic, Holtburg), Release build, vsync off, user's machine, standing + panning. +3. **Architecture:** baked asset pipeline (pak), Arch-based ECS render world, + zero-allocation steady-state frame loop. + +**Non-goals (binding):** +- No Rust. Decision recorded: the bottleneck is architectural, not language; a rewrite + discards ~200K lines of verified retail-faithful ports. +- No ECS in the simulation. `MovementManager`, the animation sequencer, physics + transitions, weenies keep their retail-mirroring OO structure — that correspondence + is the project's debugging methodology. +- No retail-visible behavior changes. Everything here is perf-only + (per feedback_render_perf_not_faithfulness_gated) except MP2, which is a *faithful + port* that retires a known divergence. +- Not M1.5 work. #137/#138/A7 continue independently and win all conflicts. + +## 3. Decisions recorded from the brainstorm + +| Decision | Choice | Why | +|---|---|---| +| Language | C#/.NET 10 | Perf gap to Rust for optimized code is ~10–30%; our gap is architectural multiples; 200K LOC of verified ports retained | +| ECS flavor | **Arch framework** (NuGet `Arch`, ~2.1.x) — render world only | User choice (craft/learning value); fastest .NET archetype ECS; contained + swappable behind our interfaces | +| Bake UX | CLI tool (`acdream-bake`) now; client auto-detect + offer later | Iteration speed while the format churns | +| Sequencing | Measure → smooth → throughput | User priority: smoothness first | +| Cutover style | Per-phase gated slices, legacy path deleted at each gate | A8/N.5 history: big-bang cutovers and lingering dual paths both failed here | + +## 4. Architecture overview + +``` +┌────────────────────────────────────────────────────────────────────┐ +│ SIMULATION (unchanged structure) │ +│ retail-shaped OO: physics, MovementManager, sequencer, weenies │ +└──────────────┬─────────────────────────────────────────────────────┘ + │ one seam: transform/lifecycle events (single-owner) +┌──────────────▼─────────────────────────────────────────────────────┐ +│ ECS RENDER WORLD (Arch, AcDream.App) [MP3] │ +│ entities = drawables; components = Transform, MeshRef, Degrade, │ +│ Residency, Flags; systems = TransformSync → Degrade → Visibility │ +│ → Upload(dirty ranges) → Draw(MDI) │ +└──────────────┬─────────────────────────────────────────────────────┘ + │ handles into +┌──────────────▼─────────────────────────────────────────────────────┐ +│ ASSET LAYER (pak) [MP1] │ +│ acdream-bake (offline, uses DRW + existing tested interpretation) │ +│ → pak file → PakReader (mmap, zero-copy, no locks, no decode) │ +└──────────────┬─────────────────────────────────────────────────────┘ + │ persistent-mapped staging, budgeted uploads +┌──────────────▼─────────────────────────────────────────────────────┐ +│ GPU (unchanged N.5 foundation: bindless + MDI, 3 SSBOs) │ +└────────────────────────────────────────────────────────────────────┘ +``` + +The N.5 bindless+MDI foundation is kept — it is already the modern half. This design +replaces what feeds it (per-frame rebuild → deltas) and what feeds *that* (runtime +decode → baked pak). + +## 5. MP0 — Apparatus (honest measurement) + +New `FrameProfiler` (dedicated class in `src/AcDream.App/Diagnostics/`, NOT GameWindow; +CLAUDE.md structure rule 1): + +- **CPU frame time**: swap-to-swap delta, ring buffer, p50/p95/p99 + histogram. +- **GPU frame time**: whole-frame `GL_TIME_ELAPSED` query, **never enabled together + with any glFinish-style per-pass bracketing** (the 2026-06-23 lesson — separate + flags, enforced in code, not convention). +- **Per-stage attribution**: `glQueryCounter` (GL_TIMESTAMP) markers per pass — no + glFinish, no nesting conflict with the frame query. +- **Allocation counter**: `GC.GetAllocatedBytesForCurrentThread()` delta per frame on + the update thread + Gen0/1/2 collection counts. This is MP4's before/after meter. +- Toggles via `RuntimeOptions` + DebugPanel (structure rules 4/5); env + `ACDREAM_FRAME_PROF=1`. **Stays in the tree permanently** (unlike the throwaway + `ACDREAM_FPS_PROF`), because every MP gate reads it. + +**Gate:** a captured baseline report (dense-town steady + scripted traversal route) with +the CPU 6 ms attributed per stage. If the attribution contradicts §1's assumed split, +the MP phase order is re-sequenced and this doc amended before MP1 starts. + +## 6. MP1 — Bake + pak (the smoothness backbone) + +### 6.1 The bake tool + +`acdream-bake` — new project `src/AcDream.Bake/` (console; registered in +`AcDream.slnx`). Reads the user's dats via DRW + **the same interpretation code the +client uses today**, run offline: + +- Mesh extraction (GfxObj/Setup → GPU-layout vertex/index + batch tables) via the + existing ObjectMeshManager build path. +- Texture decode (palette/INDEX16/etc. → RGBA) via the existing TextureHelpers path, + then compressed to BC7 (quality) / BC1 (opaque bulk) with precomputed mip chains. +- BSPs (physics, cell, drawing) flattened to index-based node arrays (blittable). +- Landblock terrain data (heights/attributes) as blittable arrays. +- EnvCells (geometry + portals + surfaces) as blittable records. +- Degrade tables (per-GfxObj slot `ideal_dist`/`max_dist`/ids) — MP2/MP3 read these. +- **Scenery placement**: `SceneryGenerator` evaluated per landblock at bake; the pak + stores final instance lists (deterministic function of dat data, so bake-safe). + +**Extraction prerequisite:** the CPU-side mesh/texture build code currently lives in +`AcDream.App` next to GL code. MP1 moves the GL-free parts into a new +`src/AcDream.Content/` assembly (no GL dependency) that both `AcDream.App` and +`AcDream.Bake` reference. This is a mechanical extraction (move + namespace), not a +rewrite — the same lines keep running. Inventory-doc update required (structure rule 2). + +### 6.2 Pak format (v1) + +Single file `acdream.pak` next to the dats (path via `RuntimeOptions`): + +- Header: magic `ACPK`, format version, **dat iteration stamps** (all four dats) + + bake-tool version. Stale/missing pak at launch → clear error naming the rebake + command (CLI era) → later: auto-offer (per §3). +- TOC: flat sorted array `(assetKey: u64 [type:u8 | fileId:u32 | variant:u24]) → + (offset: u64, length: u32, flags: u32)`; binary-search or hash lookup, O(1)-ish, + loaded once. +- Blobs 64-byte aligned, uncompressed in v1 (mmap + OS page cache + NVMe; LZ4 block + compression is a noted future option, not v1 scope). + +### 6.3 Runtime consumption + +- `PakReader` in `AcDream.Core` (Core stays GL-free; rule 2): one + `MemoryMappedFile`, `GetBlob(key)` → `ReadOnlySpan` into the view; typed + access via `MemoryMarshal.Cast` over blittable structs. **No locks anywhere** — + immutable mapped memory is thread-safe to read by construction. +- Streaming workers stop decoding: a landblock load becomes TOC lookups + handing + spans to the GL upload queue. The decode worker pool shrinks to the residual + runtime-decode set (below). Request-key dedup stays in the streaming layer + (cheap loads still shouldn't be duplicate loads). +- **Retires:** runtime DRW decode on the world *render/streaming* hot path; the + unbounded `DatDatabaseWrapper._cache`. The `PhysicsDatBundle` worker pre-read + **stays through MP1–MP3** (the physics apply consumes parsed DBObj types until MP4 + restructures physics data consumption) and is retired in MP4 when physics reads the + pak's flat arrays directly. + +### 6.4 What stays on runtime DRW (v1 scope control) + +| Stays runtime | Why | +|---|---| +| UI dat assets (sprites, fonts, LayoutDescs — D.2b) | low volume, cold path | +| Audio waves (`DatSoundCache`) | low volume, already cached | +| Character-appearance palette overlays (ObjDesc) | runtime-parameterized per entity; keeps existing decode + cache | +| Motion tables / animations (sequencer) | sim-side, small, load-once | + +`DatCollection` therefore stays in-process, off the hot path. Baking these later is a +listed follow-up, not v1. + +### 6.5 Conformance + gate + +- **Equivalence test:** load a fixture set (Holtburg blocks + a dungeon + a dense-town + block) through the legacy path and the pak path; diff meshes (vertex/index bytes), + textures (pre-compression pixels), BSP structures, scenery instance lists + field-by-field. Golden = legacy path. +- **Gate:** equivalence green; portal-hop / traversal p99 from MP0 shows the hitch + class gone; user visual gate (no missing/wrong geometry); legacy decode path deleted + from the streaming hot path in the gate commit. + +## 7. MP2 — Distance-degrade port (hide-only first cut) + +The already-scoped retail port from the 2026-06-23 handoff §3 (read it + the named +retail sources FULLY before porting — mandatory workflow): + +- Retail anchors: `GfxObjDegradeInfo::get_degrade` (`0x0051e4b0`, pseudo-C :293086), + `CPhysicsPart::UpdateViewerDistance` (`0x0050e030`, :275517), + `get_max_degrade_distance` (`0x0051e2d0`, :292918), `Render_DegradeDistance` + user setting (:3270). +- **Hide-only cut (this phase):** per-entity max-draw-distance resolved at spawn from + the degrade table; distance check in the entity walk; player + near NPCs exempt from + hiding (handoff caution). Full per-slot mesh selection deliberately lands in MP3 + where the ECS DegradeSystem owns it — accepted small double-touch, avoids building + slot-swap plumbing into a dispatcher MP3 replaces. +- Register bookkeeping: the "no distance-degrade" deviation gets its row updated to + "partial (hide faithful; slot selection pending MP3)" in the same commit; the row is + deleted in MP3's gate commit. +- **Gate:** retail side-by-side is the acceptance (same objects visible/hidden at the + same distances, user-confirmed); drawn-instance count before/after recorded from the + MP0 counters (no invented threshold — retail is the oracle); conformance test on the + `get_max_degrade_distance` math vs decomp. + +## 8. MP3 — ECS render world (Arch) + delta submission (the 300 lever) + +### 8.1 The world + +Arch `World` in `AcDream.App` owning every drawable: statics, baked scenery instances, +live entities, cell objects. Components (all blittable structs): + +- `WorldTransform` (position/rotation/scale, cell-relative frame per #145 rules) +- `MeshRef` (pak mesh handle + part index) +- `DegradeState` (current slot, per-slot table handle) +- `Residency` (landblockId, cellId) +- `RenderFlags` (translucent, hidden, player-owned, …) +- `DirtyTag` (tag component; added on change, cleared by Upload) + +### 8.2 Systems (fixed order, update thread in v1) + +1. **TransformSync** — consumes the simulation's transform/lifecycle events through + ONE seam (single-owner-state rule; the sim never writes render state directly). + Static world entities receive a transform exactly once at residency. +2. **Degrade** — retail `get_degrade` banded slot selection (ideal/max band = no + popping/hysteresis by construction), per entity per frame. Completes the MP2 port; + divergence row deleted here. +3. **Visibility** — frustum + cell/PVS gate (Option A "one gate" preserved from the + render digest — visibility computed once, enforced once). +4. **Upload** — iterates `DirtyTag` entities only; writes changed instance records + into the persistent-mapped SSBO mirror (SoA layout matching the N.5 SSBO), budgeted + per frame. +5. **Draw** — builds the MDI command buffer from the visible set and submits. Instance + *data* is GPU-resident and untouched unless dirty; only commands are per-frame. + GPU compute culling is a measured-need stretch inside this phase, not default scope. + +Steady-state per-frame cost becomes: degrade + visibility iteration (linear over +packed arrays) + uploads proportional to *what changed* (a handful of moving +entities), replacing today's full per-frame walk-and-emit (`WalkEntitiesInto` emitting +every MeshRef of every in-frustum entity every frame + the ~2.1 ms scenery rebuild). + +### 8.3 Encoded lessons (from the DO-NOT-RETRY tables — binding on implementation) + +- Cache/dirty reset gates on actual entity change via tracker (the #53 class). +- Visibility volumes derive from drawn data, never synthetic constants (#119/#128). +- Every GL state the draw uses is set, not inherited (render-self-contained rule). +- Player-owned render state written only at the player chokepoint (#131 class). +- H3 (unbounded live-entity growth across hops) is fixed here structurally: residency + eviction drops render entities (render-only eviction; sim state untouched), + re-materialized on re-entry. + +### 8.4 Conformance + gate + +- **Instance-set diff:** probe dumps the drawn (meshId, transform, slot) set from the + legacy dispatcher and the ECS path on identical fixture frames; sets must match + exactly (modulo MP2's intended hides). +- **Screenshot diff:** fixed camera poses (Holtburg outdoor set + dungeon set + + dense-town set), old vs new, pixel-identical requirement. +- Bring-up happens behind a temporary flag; **the gate commit deletes the legacy + dispatcher path** (no lingering dual pipeline — N.5 ship-amendment precedent). +- **Gate:** diffs green; dense-town steady FPS ≥ 300 OR the MP0 profile names the next + dominant cost (which then decides MP4/MP5 emphasis); user visual gate. + +## 9. MP4 — Zero-alloc frame loop + flat physics data + +**Sequencing constraint: starts only after M1.5 #137 (dungeon collision) has landed** +— it touches the same physics data structures. + +- Allocation audit driven by MP0's per-frame counter: identify every steady-state + allocation site (closures, LINQ, params arrays, list growth); fix via structs, + pooled buffers, `Span`, preallocated scratch. +- Physics consumes the pak's flattened BSP arrays: `PhysicsDataCache` swaps the + managed BSP node graph for index-walks over blittable arrays. Same traversal + order, same arithmetic — layout change only. +- **Conformance:** the existing physics test suites + the trajectory replay harness + must pass unchanged (bit-identical trajectories); that harness exists precisely for + this class of change. +- **Gate:** steady-state allocations/frame = 0 (measured); physics suites green; + traversal p99 re-measured. + +## 10. MP5 — Jobs (stretch, evidence-gated) + +Only if the MP0 profile after MP4 still shows the frame short of target: parallelize +the independent stages (Arch has built-in multithreaded query support; visibility and +degrade are embarrassingly parallel once data is flat). **Explicitly skipped with +recorded numbers if targets are already met** — parallelism is a multiplier, not a +goal. + +## 11. Testing strategy (summary) + +| Apparatus | Built in | Used by | +|---|---|---| +| FrameProfiler (CPU/GPU/alloc, percentiles) | MP0 | every gate | +| Pak equivalence diff (mesh/texture/BSP/scenery) | MP1 | MP1 gate, rebakes | +| Instance-set diff probe | MP3 | MP3 gate | +| Screenshot diff harness (fixed poses) | MP3 | MP3 gate, regressions | +| Physics trajectory replay harness | exists | MP4 gate | +| Existing suites (Core ~1568 tests, conformance sweeps) | exist | every phase (green required) | + +`DatConcurrencyStressTests` stays until MP1 retires hot-path DRW, then is scoped to +the residual runtime-DRW set (§6.4). + +## 12. Firewall, process, bookkeeping + +- **Track identity:** phases commit as `feat(pipeline): MP1 — …`. The roadmap gains a + "Modern Pipeline side track" section; the milestones doc gets an explicit + **freeze-exception paragraph** (this track deliberately reopens the frozen + streaming/rendering subsystems under new architecture, user-authorized 2026-07-05). +- **One milestone rule preserved:** M1.5 remains the active milestone; MP work happens + in dedicated side-track sessions in a separate worktree. M1.5 critical path wins + every conflict; MP4 explicitly queues behind #137. +- **Divergence register:** MP2/MP3 update-then-delete the degrade row as described. + MP1/MP3/MP4 are perf-only and must introduce **zero** rows — any behavioral + difference discovered at a gate is a bug to fix, not a deviation to register. +- **Structure rules honored:** no new GameWindow feature bodies (new classes: + `FrameProfiler`, `PakReader`, `AcDream.Bake`, `AcDream.Content`, ECS systems); + env vars through `RuntimeOptions`; tests in matching layer projects; new project + references documented in the inventory doc. +- **Dependency:** `Arch` (NuGet) added to `AcDream.App` only. License verified at + adoption (expected Apache-2.0); if ever abandoned, vendor the source + (WorldBuilder precedent) or swap for hand-rolled SoA behind the same system + interfaces. + +## 13. Risks & mitigations + +| Risk | Mitigation | +|---|---| +| Pak staleness confusion (dat patch → wrong assets) | iteration stamps in header; hard refuse + rebake instruction; auto-offer later | +| ECS cutover regresses rendering (the #53/#119/#128 class) | instance-set + screenshot diffs as hard gates; lessons encoded as binding rules (§8.3); legacy deleted only at gate | +| MP1 extraction (`AcDream.Content`) breaks the mesh path | mechanical move, no rewrite; pak equivalence test catches interpretation drift | +| Degrade double-touch (MP2 hide-only, MP3 slots) | accepted, small; hide plumbing (spawn-time max-dist) carries into MP3 unchanged | +| Arch abandonment / license surprise | verify license before adoption; vendorable; swappable behind system interfaces | +| Side track leaks into M1.5 time or files | separate worktree; MP4 hard-queued behind #137; sessions labeled | +| MP0 profile contradicts the assumed cost split | that's MP0's job — re-sequence and amend this doc before MP1 | +| Baked scenery diverges from runtime generation | scenery placement is a pure function of dat data; equivalence diff covers it | + +## 14. References + +- `docs/research/2026-06-23-fps-distance-degrade-handoff.md` — baseline numbers, degrade + retail anchors, measurement caveats (the §2/§3 this design consumes). +- `claude-memory/feedback_render_perf_measurement.md` — profiler lies + CPU-bound + attribution + parked levers. +- `claude-memory/project_render_pipeline_digest.md`, `project_physics_collision_digest.md` + — DO-NOT-RETRY tables binding on MP3/MP4. +- `docs/superpowers/specs/2026-06-23-datlock-contention-fix-design.md` — the shipped + `_datLock` fix MP1 supersedes (bundle → pak). +- `docs/research/2026-06-09-dat-reader-thread-safety-investigation.md` — DRW read-path + safety + the teardown fix + `DatConcurrencyStressTests`. +- `memory/reference_modern_rendering_pipeline.md` — N.5 SSBO layout the ECS Upload + system mirrors. +- `memory/reference_two_tier_streaming.md` — residency tiers the pak loads plug into. +- Arch ECS: https://github.com/genaray/Arch (NuGet `Arch`).