docs(pipeline): Modern Pipeline (MP) side-track design spec

Umbrella design for the user-commissioned performance side track: MP0 honest frame profiler -> MP1 baked asset pak (acdream-bake CLI, mmap zero-copy reader) -> MP2 retail distance-degrade (hide-only cut) -> MP3 Arch ECS render world + delta submission -> MP4 zero-alloc + flat physics data (queued behind M1.5 #137) -> MP5 jobs (stretch, evidence-gated). Decisions recorded: C# not Rust (bottleneck is architectural, not language); ECS scoped to the render world only (simulation keeps its retail-mirroring OO structure); Arch framework per user choice; bake as CLI now, client auto-detect later; smoothness gated before the 300+ FPS throughput target. Every phase has a numeric + visual gate; legacy paths delete at each gate (no lingering dual pipelines). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 18:56:41 +02:00 · 2026-07-05 18:56:41 +02:00 · 113a59887a
commit 113a59887a
parent 67c3357246
1 changed files with 364 additions and 0 deletions
--- a/docs/superpowers/specs/2026-07-05-modern-pipeline-design.md
+++ b/docs/superpowers/specs/2026-07-05-modern-pipeline-design.md
@ -0,0 +1,364 @@
+# Design — Modern Pipeline side track (MP): baked assets + ECS render world + zero-alloc frame loop
+
+**Date:** 2026-07-05
+**Status:** brainstorm-approved; awaiting user spec review
+**Origin:** user-commissioned performance side track ("fully modern pipeline, with ECS").
+Explicitly NOT M1.5 work — see §12 for the firewall.
+**Phase prefix:** `MP0–MP5` (MP = Modern Pipeline; avoids collision with the existing
+P0 conformance-apparatus plan).
+**Implementation cadence:** each MP phase gets its own implementation plan
+(`docs/superpowers/plans/`) when it starts; this spec is the umbrella design. The
+first plan covers MP0 (+ MP1 scaffold) only.
+
+---
+
+## 1. Motivation + measured baseline
+
+acdream's steady-state dense-town frame rate is ~165 FPS (~6 ms/frame) after the
+2026-06-23/24 fixes, versus 300+ FPS in comparable modern titles on the same machine.
+Traversal (streaming, portal hops) still hitches. The measured facts this design
+builds on (do NOT re-derive; sources in §14):
+
+| Fact | Evidence |
+|---|---|
+| GPU is nearly idle: ~0.5 ms/frame real GPU time in dense town | 2026-06-23/24 deep-dive; the "12 ms GPU" was a glFinish artifact |
+| The frame is CPU-submission-bound | dense-town attribution report; per-cell batch fix took 75→~165 FPS |
+| `_datLock` contention (the 30↔200 swing) is FIXED — lockwait 88 ms p95 → 0.2 µs | `536f1c04` + handoff §1 |
+| No distance-degrade: every frustum-visible object draws at full detail at any distance (~1,400 draws / ~24,000 instances) | 2026-06-23 handoff §2; `GfxObjDegradeResolver.cs` "always slot 0" comment |
+| Parked CPU levers: `WbDrawDispatcher` per-frame static-scenery rebuild (~2.1 ms), cross-frame dispatcher cache (judged HIGH-risk as a *patch* to the frozen renderer) | feedback_render_perf_measurement.md |
+| Load hitches come from runtime dat decode (mesh extraction, texture decode, BSP graph construction) + GC | thread-safety investigation; two-tier streaming architecture |
+| Runtime dat reads carry an unbounded second-layer cache (`DatDatabaseWrapper._cache`) stacked on DRW's own | `DatCollectionAdapter.cs:119` |
+
+The conclusion the whole design follows from: **the language and the GPU are not the
+bottleneck; the data layout and the per-frame/at-load CPU work are.** Modernizing means
+(a) moving decode work offline, (b) making the renderer process *changes* instead of
+re-describing the scene every frame, (c) flattening hot data, (d) removing frame-loop
+allocation.
+
+## 2. Goals & acceptance
+
+1. **Smoothness (primary):** no frame over ~16 ms during any traversal — town walking
+   while streaming, portal hops, dungeon transitions. Measured: p99 frame time from the
+   MP0 profiler across a scripted traversal route, plus user-eyes confirmation.
+2. **Throughput:** 300+ FPS sustained in the dense-town worst cases (Fort Tethana,
+   Arwic, Holtburg), Release build, vsync off, user's machine, standing + panning.
+3. **Architecture:** baked asset pipeline (pak), Arch-based ECS render world,
+   zero-allocation steady-state frame loop.
+
+**Non-goals (binding):**
+- No Rust. Decision recorded: the bottleneck is architectural, not language; a rewrite
+  discards ~200K lines of verified retail-faithful ports.
+- No ECS in the simulation. `MovementManager`, the animation sequencer, physics
+  transitions, weenies keep their retail-mirroring OO structure — that correspondence
+  is the project's debugging methodology.
+- No retail-visible behavior changes. Everything here is perf-only
+  (per feedback_render_perf_not_faithfulness_gated) except MP2, which is a *faithful
+  port* that retires a known divergence.
+- Not M1.5 work. #137/#138/A7 continue independently and win all conflicts.
+
+## 3. Decisions recorded from the brainstorm
+
+| Decision | Choice | Why |
+|---|---|---|
+| Language | C#/.NET 10 | Perf gap to Rust for optimized code is ~10–30%; our gap is architectural multiples; 200K LOC of verified ports retained |
+| ECS flavor | **Arch framework** (NuGet `Arch`, ~2.1.x) — render world only | User choice (craft/learning value); fastest .NET archetype ECS; contained + swappable behind our interfaces |
+| Bake UX | CLI tool (`acdream-bake`) now; client auto-detect + offer later | Iteration speed while the format churns |
+| Sequencing | Measure → smooth → throughput | User priority: smoothness first |
+| Cutover style | Per-phase gated slices, legacy path deleted at each gate | A8/N.5 history: big-bang cutovers and lingering dual paths both failed here |
+
+## 4. Architecture overview
+
+```
+┌────────────────────────────────────────────────────────────────────┐
+│ SIMULATION (unchanged structure)                                   │
+│ retail-shaped OO: physics, MovementManager, sequencer, weenies     │
+└──────────────┬─────────────────────────────────────────────────────┘
+               │ one seam: transform/lifecycle events (single-owner)
+┌──────────────▼─────────────────────────────────────────────────────┐
+│ ECS RENDER WORLD (Arch, AcDream.App)               [MP3]           │
+│ entities = drawables; components = Transform, MeshRef, Degrade,    │
+│ Residency, Flags; systems = TransformSync → Degrade → Visibility   │
+│ → Upload(dirty ranges) → Draw(MDI)                                 │
+└──────────────┬─────────────────────────────────────────────────────┘
+               │ handles into
+┌──────────────▼─────────────────────────────────────────────────────┐
+│ ASSET LAYER (pak)                                  [MP1]           │
+│ acdream-bake (offline, uses DRW + existing tested interpretation)  │
+│ → pak file → PakReader (mmap, zero-copy, no locks, no decode)      │
+└──────────────┬─────────────────────────────────────────────────────┘
+               │ persistent-mapped staging, budgeted uploads
+┌──────────────▼─────────────────────────────────────────────────────┐
+│ GPU (unchanged N.5 foundation: bindless + MDI, 3 SSBOs)            │
+└────────────────────────────────────────────────────────────────────┘
+```
+
+The N.5 bindless+MDI foundation is kept — it is already the modern half. This design
+replaces what feeds it (per-frame rebuild → deltas) and what feeds *that* (runtime
+decode → baked pak).
+
+## 5. MP0 — Apparatus (honest measurement)
+
+New `FrameProfiler` (dedicated class in `src/AcDream.App/Diagnostics/`, NOT GameWindow;
+CLAUDE.md structure rule 1):
+
+- **CPU frame time**: swap-to-swap delta, ring buffer, p50/p95/p99 + histogram.
+- **GPU frame time**: whole-frame `GL_TIME_ELAPSED` query, **never enabled together
+  with any glFinish-style per-pass bracketing** (the 2026-06-23 lesson — separate
+  flags, enforced in code, not convention).
+- **Per-stage attribution**: `glQueryCounter` (GL_TIMESTAMP) markers per pass — no
+  glFinish, no nesting conflict with the frame query.
+- **Allocation counter**: `GC.GetAllocatedBytesForCurrentThread()` delta per frame on
+  the update thread + Gen0/1/2 collection counts. This is MP4's before/after meter.
+- Toggles via `RuntimeOptions` + DebugPanel (structure rules 4/5); env
+  `ACDREAM_FRAME_PROF=1`. **Stays in the tree permanently** (unlike the throwaway
+  `ACDREAM_FPS_PROF`), because every MP gate reads it.
+
+**Gate:** a captured baseline report (dense-town steady + scripted traversal route) with
+the CPU 6 ms attributed per stage. If the attribution contradicts §1's assumed split,
+the MP phase order is re-sequenced and this doc amended before MP1 starts.
+
+## 6. MP1 — Bake + pak (the smoothness backbone)
+
+### 6.1 The bake tool
+
+`acdream-bake` — new project `src/AcDream.Bake/` (console; registered in
+`AcDream.slnx`). Reads the user's dats via DRW + **the same interpretation code the
+client uses today**, run offline:
+
+- Mesh extraction (GfxObj/Setup → GPU-layout vertex/index + batch tables) via the
+  existing ObjectMeshManager build path.
+- Texture decode (palette/INDEX16/etc. → RGBA) via the existing TextureHelpers path,
+  then compressed to BC7 (quality) / BC1 (opaque bulk) with precomputed mip chains.
+- BSPs (physics, cell, drawing) flattened to index-based node arrays (blittable).
+- Landblock terrain data (heights/attributes) as blittable arrays.
+- EnvCells (geometry + portals + surfaces) as blittable records.
+- Degrade tables (per-GfxObj slot `ideal_dist`/`max_dist`/ids) — MP2/MP3 read these.
+- **Scenery placement**: `SceneryGenerator` evaluated per landblock at bake; the pak
+  stores final instance lists (deterministic function of dat data, so bake-safe).
+
+**Extraction prerequisite:** the CPU-side mesh/texture build code currently lives in
+`AcDream.App` next to GL code. MP1 moves the GL-free parts into a new
+`src/AcDream.Content/` assembly (no GL dependency) that both `AcDream.App` and
+`AcDream.Bake` reference. This is a mechanical extraction (move + namespace), not a
+rewrite — the same lines keep running. Inventory-doc update required (structure rule 2).
+
+### 6.2 Pak format (v1)
+
+Single file `acdream.pak` next to the dats (path via `RuntimeOptions`):
+
+- Header: magic `ACPK`, format version, **dat iteration stamps** (all four dats) +
+  bake-tool version. Stale/missing pak at launch → clear error naming the rebake
+  command (CLI era) → later: auto-offer (per §3).
+- TOC: flat sorted array `(assetKey: u64 [type:u8 | fileId:u32 | variant:u24]) →
+  (offset: u64, length: u32, flags: u32)`; binary-search or hash lookup, O(1)-ish,
+  loaded once.
+- Blobs 64-byte aligned, uncompressed in v1 (mmap + OS page cache + NVMe; LZ4 block
+  compression is a noted future option, not v1 scope).
+
+### 6.3 Runtime consumption
+
+- `PakReader` in `AcDream.Core` (Core stays GL-free; rule 2): one
+  `MemoryMappedFile`, `GetBlob(key)` → `ReadOnlySpan<byte>` into the view; typed
+  access via `MemoryMarshal.Cast` over blittable structs. **No locks anywhere** —
+  immutable mapped memory is thread-safe to read by construction.
+- Streaming workers stop decoding: a landblock load becomes TOC lookups + handing
+  spans to the GL upload queue. The decode worker pool shrinks to the residual
+  runtime-decode set (below). Request-key dedup stays in the streaming layer
+  (cheap loads still shouldn't be duplicate loads).
+- **Retires:** runtime DRW decode on the world *render/streaming* hot path; the
+  unbounded `DatDatabaseWrapper._cache`. The `PhysicsDatBundle` worker pre-read
+  **stays through MP1–MP3** (the physics apply consumes parsed DBObj types until MP4
+  restructures physics data consumption) and is retired in MP4 when physics reads the
+  pak's flat arrays directly.
+
+### 6.4 What stays on runtime DRW (v1 scope control)
+
+| Stays runtime | Why |
+|---|---|
+| UI dat assets (sprites, fonts, LayoutDescs — D.2b) | low volume, cold path |
+| Audio waves (`DatSoundCache`) | low volume, already cached |
+| Character-appearance palette overlays (ObjDesc) | runtime-parameterized per entity; keeps existing decode + cache |
+| Motion tables / animations (sequencer) | sim-side, small, load-once |
+
+`DatCollection` therefore stays in-process, off the hot path. Baking these later is a
+listed follow-up, not v1.
+
+### 6.5 Conformance + gate
+
+- **Equivalence test:** load a fixture set (Holtburg blocks + a dungeon + a dense-town
+  block) through the legacy path and the pak path; diff meshes (vertex/index bytes),
+  textures (pre-compression pixels), BSP structures, scenery instance lists
+  field-by-field. Golden = legacy path.
+- **Gate:** equivalence green; portal-hop / traversal p99 from MP0 shows the hitch
+  class gone; user visual gate (no missing/wrong geometry); legacy decode path deleted
+  from the streaming hot path in the gate commit.
+
+## 7. MP2 — Distance-degrade port (hide-only first cut)
+
+The already-scoped retail port from the 2026-06-23 handoff §3 (read it + the named
+retail sources FULLY before porting — mandatory workflow):
+
+- Retail anchors: `GfxObjDegradeInfo::get_degrade` (`0x0051e4b0`, pseudo-C :293086),
+  `CPhysicsPart::UpdateViewerDistance` (`0x0050e030`, :275517),
+  `get_max_degrade_distance` (`0x0051e2d0`, :292918), `Render_DegradeDistance`
+  user setting (:3270).
+- **Hide-only cut (this phase):** per-entity max-draw-distance resolved at spawn from
+  the degrade table; distance check in the entity walk; player + near NPCs exempt from
+  hiding (handoff caution). Full per-slot mesh selection deliberately lands in MP3
+  where the ECS DegradeSystem owns it — accepted small double-touch, avoids building
+  slot-swap plumbing into a dispatcher MP3 replaces.
+- Register bookkeeping: the "no distance-degrade" deviation gets its row updated to
+  "partial (hide faithful; slot selection pending MP3)" in the same commit; the row is
+  deleted in MP3's gate commit.
+- **Gate:** retail side-by-side is the acceptance (same objects visible/hidden at the
+  same distances, user-confirmed); drawn-instance count before/after recorded from the
+  MP0 counters (no invented threshold — retail is the oracle); conformance test on the
+  `get_max_degrade_distance` math vs decomp.
+
+## 8. MP3 — ECS render world (Arch) + delta submission (the 300 lever)
+
+### 8.1 The world
+
+Arch `World` in `AcDream.App` owning every drawable: statics, baked scenery instances,
+live entities, cell objects. Components (all blittable structs):
+
+- `WorldTransform` (position/rotation/scale, cell-relative frame per #145 rules)
+- `MeshRef` (pak mesh handle + part index)
+- `DegradeState` (current slot, per-slot table handle)
+- `Residency` (landblockId, cellId)
+- `RenderFlags` (translucent, hidden, player-owned, …)
+- `DirtyTag` (tag component; added on change, cleared by Upload)
+
+### 8.2 Systems (fixed order, update thread in v1)
+
+1. **TransformSync** — consumes the simulation's transform/lifecycle events through
+   ONE seam (single-owner-state rule; the sim never writes render state directly).
+   Static world entities receive a transform exactly once at residency.
+2. **Degrade** — retail `get_degrade` banded slot selection (ideal/max band = no
+   popping/hysteresis by construction), per entity per frame. Completes the MP2 port;
+   divergence row deleted here.
+3. **Visibility** — frustum + cell/PVS gate (Option A "one gate" preserved from the
+   render digest — visibility computed once, enforced once).
+4. **Upload** — iterates `DirtyTag` entities only; writes changed instance records
+   into the persistent-mapped SSBO mirror (SoA layout matching the N.5 SSBO), budgeted
+   per frame.
+5. **Draw** — builds the MDI command buffer from the visible set and submits. Instance
+   *data* is GPU-resident and untouched unless dirty; only commands are per-frame.
+   GPU compute culling is a measured-need stretch inside this phase, not default scope.
+
+Steady-state per-frame cost becomes: degrade + visibility iteration (linear over
+packed arrays) + uploads proportional to *what changed* (a handful of moving
+entities), replacing today's full per-frame walk-and-emit (`WalkEntitiesInto` emitting
+every MeshRef of every in-frustum entity every frame + the ~2.1 ms scenery rebuild).
+
+### 8.3 Encoded lessons (from the DO-NOT-RETRY tables — binding on implementation)
+
+- Cache/dirty reset gates on actual entity change via tracker (the #53 class).
+- Visibility volumes derive from drawn data, never synthetic constants (#119/#128).
+- Every GL state the draw uses is set, not inherited (render-self-contained rule).
+- Player-owned render state written only at the player chokepoint (#131 class).
+- H3 (unbounded live-entity growth across hops) is fixed here structurally: residency
+  eviction drops render entities (render-only eviction; sim state untouched),
+  re-materialized on re-entry.
+
+### 8.4 Conformance + gate
+
+- **Instance-set diff:** probe dumps the drawn (meshId, transform, slot) set from the
+  legacy dispatcher and the ECS path on identical fixture frames; sets must match
+  exactly (modulo MP2's intended hides).
+- **Screenshot diff:** fixed camera poses (Holtburg outdoor set + dungeon set +
+  dense-town set), old vs new, pixel-identical requirement.
+- Bring-up happens behind a temporary flag; **the gate commit deletes the legacy
+  dispatcher path** (no lingering dual pipeline — N.5 ship-amendment precedent).
+- **Gate:** diffs green; dense-town steady FPS ≥ 300 OR the MP0 profile names the next
+  dominant cost (which then decides MP4/MP5 emphasis); user visual gate.
+
+## 9. MP4 — Zero-alloc frame loop + flat physics data
+
+**Sequencing constraint: starts only after M1.5 #137 (dungeon collision) has landed**
+— it touches the same physics data structures.
+
+- Allocation audit driven by MP0's per-frame counter: identify every steady-state
+  allocation site (closures, LINQ, params arrays, list growth); fix via structs,
+  pooled buffers, `Span<T>`, preallocated scratch.
+- Physics consumes the pak's flattened BSP arrays: `PhysicsDataCache` swaps the
+  managed BSP node graph for index-walks over blittable arrays. Same traversal
+  order, same arithmetic — layout change only.
+- **Conformance:** the existing physics test suites + the trajectory replay harness
+  must pass unchanged (bit-identical trajectories); that harness exists precisely for
+  this class of change.
+- **Gate:** steady-state allocations/frame = 0 (measured); physics suites green;
+  traversal p99 re-measured.
+
+## 10. MP5 — Jobs (stretch, evidence-gated)
+
+Only if the MP0 profile after MP4 still shows the frame short of target: parallelize
+the independent stages (Arch has built-in multithreaded query support; visibility and
+degrade are embarrassingly parallel once data is flat). **Explicitly skipped with
+recorded numbers if targets are already met** — parallelism is a multiplier, not a
+goal.
+
+## 11. Testing strategy (summary)
+
+| Apparatus | Built in | Used by |
+|---|---|---|
+| FrameProfiler (CPU/GPU/alloc, percentiles) | MP0 | every gate |
+| Pak equivalence diff (mesh/texture/BSP/scenery) | MP1 | MP1 gate, rebakes |
+| Instance-set diff probe | MP3 | MP3 gate |
+| Screenshot diff harness (fixed poses) | MP3 | MP3 gate, regressions |
+| Physics trajectory replay harness | exists | MP4 gate |
+| Existing suites (Core ~1568 tests, conformance sweeps) | exist | every phase (green required) |
+
+`DatConcurrencyStressTests` stays until MP1 retires hot-path DRW, then is scoped to
+the residual runtime-DRW set (§6.4).
+
+## 12. Firewall, process, bookkeeping
+
+- **Track identity:** phases commit as `feat(pipeline): MP1 — …`. The roadmap gains a
+  "Modern Pipeline side track" section; the milestones doc gets an explicit
+  **freeze-exception paragraph** (this track deliberately reopens the frozen
+  streaming/rendering subsystems under new architecture, user-authorized 2026-07-05).
+- **One milestone rule preserved:** M1.5 remains the active milestone; MP work happens
+  in dedicated side-track sessions in a separate worktree. M1.5 critical path wins
+  every conflict; MP4 explicitly queues behind #137.
+- **Divergence register:** MP2/MP3 update-then-delete the degrade row as described.
+  MP1/MP3/MP4 are perf-only and must introduce **zero** rows — any behavioral
+  difference discovered at a gate is a bug to fix, not a deviation to register.
+- **Structure rules honored:** no new GameWindow feature bodies (new classes:
+  `FrameProfiler`, `PakReader`, `AcDream.Bake`, `AcDream.Content`, ECS systems);
+  env vars through `RuntimeOptions`; tests in matching layer projects; new project
+  references documented in the inventory doc.
+- **Dependency:** `Arch` (NuGet) added to `AcDream.App` only. License verified at
+  adoption (expected Apache-2.0); if ever abandoned, vendor the source
+  (WorldBuilder precedent) or swap for hand-rolled SoA behind the same system
+  interfaces.
+
+## 13. Risks & mitigations
+
+| Risk | Mitigation |
+|---|---|
+| Pak staleness confusion (dat patch → wrong assets) | iteration stamps in header; hard refuse + rebake instruction; auto-offer later |
+| ECS cutover regresses rendering (the #53/#119/#128 class) | instance-set + screenshot diffs as hard gates; lessons encoded as binding rules (§8.3); legacy deleted only at gate |
+| MP1 extraction (`AcDream.Content`) breaks the mesh path | mechanical move, no rewrite; pak equivalence test catches interpretation drift |
+| Degrade double-touch (MP2 hide-only, MP3 slots) | accepted, small; hide plumbing (spawn-time max-dist) carries into MP3 unchanged |
+| Arch abandonment / license surprise | verify license before adoption; vendorable; swappable behind system interfaces |
+| Side track leaks into M1.5 time or files | separate worktree; MP4 hard-queued behind #137; sessions labeled |
+| MP0 profile contradicts the assumed cost split | that's MP0's job — re-sequence and amend this doc before MP1 |
+| Baked scenery diverges from runtime generation | scenery placement is a pure function of dat data; equivalence diff covers it |
+
+## 14. References
+
+- `docs/research/2026-06-23-fps-distance-degrade-handoff.md` — baseline numbers, degrade
+  retail anchors, measurement caveats (the §2/§3 this design consumes).
+- `claude-memory/feedback_render_perf_measurement.md` — profiler lies + CPU-bound
+  attribution + parked levers.
+- `claude-memory/project_render_pipeline_digest.md`, `project_physics_collision_digest.md`
+  — DO-NOT-RETRY tables binding on MP3/MP4.
+- `docs/superpowers/specs/2026-06-23-datlock-contention-fix-design.md` — the shipped
+  `_datLock` fix MP1 supersedes (bundle → pak).
+- `docs/research/2026-06-09-dat-reader-thread-safety-investigation.md` — DRW read-path
+  safety + the teardown fix + `DatConcurrencyStressTests`.
+- `memory/reference_modern_rendering_pipeline.md` — N.5 SSBO layout the ECS Upload
+  system mirrors.
+- `memory/reference_two_tier_streaming.md` — residency tiers the pak loads plug into.
+- Arch ECS: https://github.com/genaray/Arch (NuGet `Arch`).