acdream/docs/superpowers/specs/2026-07-05-modern-pipeline-design.md

# Design — Modern Pipeline side track (MP): baked assets + ECS render world + zero-alloc frame loop

**Date:** 2026-07-05
**Status:** brainstorm-approved; awaiting user spec review
**Origin:** user-commissioned performance side track ("fully modern pipeline, with ECS").
Explicitly NOT M1.5 work — see §12 for the firewall.
**Phase prefix:** `MP0–MP5` (MP = Modern Pipeline; avoids collision with the existing
P0 conformance-apparatus plan).
**Implementation cadence:** each MP phase gets its own implementation plan
(`docs/superpowers/plans/`) when it starts; this spec is the umbrella design. The
first plan covers MP0 (+ MP1 scaffold) only.

---

## 1. Motivation + measured baseline

acdream's steady-state dense-town frame rate is ~165 FPS (~6 ms/frame) after the
2026-06-23/24 fixes, versus 300+ FPS in comparable modern titles on the same machine.
Traversal (streaming, portal hops) still hitches. The measured facts this design
builds on (do NOT re-derive; sources in §14):

| Fact | Evidence |
|---|---|
| GPU is nearly idle: ~0.5 ms/frame real GPU time in dense town | 2026-06-23/24 deep-dive; the "12 ms GPU" was a glFinish artifact |
| The frame is CPU-submission-bound | dense-town attribution report; per-cell batch fix took 75→~165 FPS |
| `_datLock` contention (the 30↔200 swing) is FIXED — lockwait 88 ms p95 → 0.2 µs | `536f1c04` + handoff §1 |
| No distance-degrade: every frustum-visible object draws at full detail at any distance (~1,400 draws / ~24,000 instances) | 2026-06-23 handoff §2; `GfxObjDegradeResolver.cs` "always slot 0" comment |
| Parked CPU levers: `WbDrawDispatcher` per-frame static-scenery rebuild (~2.1 ms), cross-frame dispatcher cache (judged HIGH-risk as a *patch* to the frozen renderer) | feedback_render_perf_measurement.md |
| Load hitches come from runtime dat decode (mesh extraction, texture decode, BSP graph construction) + GC | thread-safety investigation; two-tier streaming architecture |
| Runtime dat reads carry an unbounded second-layer cache (`DatDatabaseWrapper._cache`) stacked on DRW's own | `DatCollectionAdapter.cs:119` |

The conclusion the whole design follows from: **the language and the GPU are not the
bottleneck; the data layout and the per-frame/at-load CPU work are.** Modernizing means
(a) moving decode work offline, (b) making the renderer process *changes* instead of
re-describing the scene every frame, (c) flattening hot data, (d) removing frame-loop
allocation.

## 2. Goals & acceptance

1. **Smoothness (primary):** no frame over ~16 ms during any traversal — town walking
   while streaming, portal hops, dungeon transitions. Measured: p99 frame time from the
   MP0 profiler across a scripted traversal route, plus user-eyes confirmation.
2. **Throughput:** 300+ FPS sustained in the dense-town worst cases (Fort Tethana,
   Arwic, Holtburg), Release build, vsync off, user's machine, standing + panning.
3. **Architecture:** baked asset pipeline (pak), Arch-based ECS render world,
   zero-allocation steady-state frame loop.

**Non-goals (binding):**
- No Rust. Decision recorded: the bottleneck is architectural, not language; a rewrite
  discards ~200K lines of verified retail-faithful ports.
- No ECS in the simulation. `MovementManager`, the animation sequencer, physics
  transitions, weenies keep their retail-mirroring OO structure — that correspondence
  is the project's debugging methodology.
- No retail-visible behavior changes. Everything here is perf-only
  (per feedback_render_perf_not_faithfulness_gated) except MP2, which is a *faithful
  port* that retires a known divergence.
- Not M1.5 work. #137/#138/A7 continue independently and win all conflicts.

## 3. Decisions recorded from the brainstorm

| Decision | Choice | Why |
|---|---|---|
| Language | C#/.NET 10 | Perf gap to Rust for optimized code is ~10–30%; our gap is architectural multiples; 200K LOC of verified ports retained |
| ECS flavor | **Arch framework** (NuGet `Arch`, ~2.1.x) — render world only | User choice (craft/learning value); fastest .NET archetype ECS; contained + swappable behind our interfaces |
| Bake UX | CLI tool (`acdream-bake`) now; client auto-detect + offer later | Iteration speed while the format churns |
| Sequencing | Measure → smooth → throughput | User priority: smoothness first |
| Cutover style | Per-phase gated slices, legacy path deleted at each gate | A8/N.5 history: big-bang cutovers and lingering dual paths both failed here |

## 4. Architecture overview

```
┌────────────────────────────────────────────────────────────────────┐
│ SIMULATION (unchanged structure)                                   │
│ retail-shaped OO: physics, MovementManager, sequencer, weenies     │
└──────────────┬─────────────────────────────────────────────────────┘
               │ one seam: transform/lifecycle events (single-owner)
┌──────────────▼─────────────────────────────────────────────────────┐
│ ECS RENDER WORLD (Arch, AcDream.App)               [MP3]           │
│ entities = drawables; components = Transform, MeshRef, Degrade,    │
│ Residency, Flags; systems = TransformSync → Degrade → Visibility   │
│ → Upload(dirty ranges) → Draw(MDI)                                 │
└──────────────┬─────────────────────────────────────────────────────┘
               │ handles into
┌──────────────▼─────────────────────────────────────────────────────┐
│ ASSET LAYER (pak)                                  [MP1]           │
│ acdream-bake (offline, uses DRW + existing tested interpretation)  │
│ → pak file → PakReader (mmap, zero-copy, no locks, no decode)      │
└──────────────┬─────────────────────────────────────────────────────┘
               │ persistent-mapped staging, budgeted uploads
┌──────────────▼─────────────────────────────────────────────────────┐
│ GPU (unchanged N.5 foundation: bindless + MDI, 3 SSBOs)            │
└────────────────────────────────────────────────────────────────────┘
```

The N.5 bindless+MDI foundation is kept — it is already the modern half. This design
replaces what feeds it (per-frame rebuild → deltas) and what feeds *that* (runtime
decode → baked pak).

## 5. MP0 — Apparatus (honest measurement)

New `FrameProfiler` (dedicated class in `src/AcDream.App/Diagnostics/`, NOT GameWindow;
CLAUDE.md structure rule 1):

- **CPU frame time**: swap-to-swap delta, ring buffer, p50/p95/p99 + histogram.
- **GPU frame time**: whole-frame `GL_TIME_ELAPSED` query, **never enabled together
  with any glFinish-style per-pass bracketing** (the 2026-06-23 lesson — separate
  flags, enforced in code, not convention).
- **Per-stage attribution**: `glQueryCounter` (GL_TIMESTAMP) markers per pass — no
  glFinish, no nesting conflict with the frame query.
- **Allocation counter**: `GC.GetAllocatedBytesForCurrentThread()` delta per frame on
  the update thread + Gen0/1/2 collection counts. This is MP4's before/after meter.
- Toggles via `RuntimeOptions` + DebugPanel (structure rules 4/5); env
  `ACDREAM_FRAME_PROF=1`. **Stays in the tree permanently** (unlike the throwaway
  `ACDREAM_FPS_PROF`), because every MP gate reads it.

**Gate:** a captured baseline report (dense-town steady + scripted traversal route) with
the CPU 6 ms attributed per stage. If the attribution contradicts §1's assumed split,
the MP phase order is re-sequenced and this doc amended before MP1 starts.

**Gate PASSED 2026-07-05** — see `docs/research/2026-07-05-mp0-baseline.md`.
Verdict: render-side-CPU split CONFIRMED (GPU ≤ ~2.7 ms, stages ≈ 0); steady
medians better than assumed (worst town p50 3.6 ms; Fort Tethana axiom view
not yet re-measured — re-check at the MP2/MP3 gates). AMENDMENT: steady-state
allocation churn (1.5–3 MB/frame → gen2 GC ~1–2/s → all town p99/max
violations) is promoted to a first-class smoothness lever: after MP1, one
bounded allocation-triage session fixes top churn sites OUTSIDE the MP3
rewrite surface; sites inside it wait for MP3; the full zero-alloc pass
remains MP4. Teleport hitch quantified: 211 ms worst frame, 75.7 MB
single-frame allocation — MP1's "before" number.

## 6. MP1 — Bake + pak (the smoothness backbone)

### 6.1 The bake tool

`acdream-bake` — new project `src/AcDream.Bake/` (console; registered in
`AcDream.slnx`). Reads the user's dats via DRW + **the same interpretation code the
client uses today**, run offline:

- Mesh extraction (GfxObj/Setup → GPU-layout vertex/index + batch tables) via the
  existing ObjectMeshManager build path.
- Texture decode (palette/INDEX16/etc. → RGBA) via the existing TextureHelpers path.
  **AMENDED 2026-07-05 (MP1a ground truth):** v1 pak stores **decoded RGBA8 exactly
  as the runtime path produces today** (the atlas consumes RGBA8 via
  `TextureBatchData` — storing BC7/BC1 would change delivered pixels and violate
  this phase's byte-identical conformance gate). BC compression (quarter the disk
  + VRAM, lossy) is an explicit post-conformance option requiring its own visual
  gate and user approval, not v1 scope. Mip chains remain runtime-generated
  (`GenerateMipmaps` flush) in v1 for the same reason.
- BSPs (physics, cell, drawing) flattened to index-based node arrays (blittable).
- Landblock terrain data (heights/attributes) as blittable arrays.
- EnvCells (geometry + portals + surfaces) as blittable records.
- Degrade tables (per-GfxObj slot `ideal_dist`/`max_dist`/ids) — MP2/MP3 read these.
- **Scenery placement**: `SceneryGenerator` evaluated per landblock at bake; the pak
  stores final instance lists (deterministic function of dat data, so bake-safe).

**Extraction prerequisite:** the CPU-side mesh/texture build code currently lives in
`AcDream.App` next to GL code. MP1 moves the GL-free parts into a new
`src/AcDream.Content/` assembly (no GL dependency) that both `AcDream.App` and
`AcDream.Bake` reference. This is a mechanical extraction (move + namespace), not a
rewrite — the same lines keep running. Inventory-doc update required (structure rule 2).

### 6.2 Pak format (v1)

Single file `acdream.pak` next to the dats (path via `RuntimeOptions`):

- Header: magic `ACPK`, format version, **dat iteration stamps** (all four dats) +
  bake-tool version. Stale/missing pak at launch → clear error naming the rebake
  command (CLI era) → later: auto-offer (per §3).
- TOC: flat sorted array `(assetKey: u64 [type:u8 | fileId:u32 | variant:u24]) →
  (offset: u64, length: u32, flags: u32)`; binary-search or hash lookup, O(1)-ish,
  loaded once.
- Blobs 64-byte aligned, uncompressed in v1 (mmap + OS page cache + NVMe; LZ4 block
  compression is a noted future option, not v1 scope).

### 6.3 Runtime consumption

- `PakReader` in **`AcDream.Content`** (AMENDED 2026-07-05: the spec originally said
  Core, but MP1a established the dependency direction Content→Core, and the pak's
  blob shapes are the `ObjectMeshData` family living in Content — a Core-resident
  reader would be circular. Content is GL-free, so the layering intent is
  preserved; Core stays untouched): one `MemoryMappedFile`, `GetBlob(key)` →
  `ReadOnlySpan<byte>` into the view; typed access via `MemoryMarshal.Cast` over
  blittable segments. **No locks anywhere** — immutable mapped memory is
  thread-safe to read by construction.
- Streaming workers stop decoding: a landblock load becomes TOC lookups + handing
  spans to the GL upload queue. The decode worker pool shrinks to the residual
  runtime-decode set (below). Request-key dedup stays in the streaming layer
  (cheap loads still shouldn't be duplicate loads).
- **Retires:** runtime DRW decode on the world *render/streaming* hot path; the
  unbounded `DatDatabaseWrapper._cache`. The `PhysicsDatBundle` worker pre-read
  **stays through MP1–MP3** (the physics apply consumes parsed DBObj types until MP4
  restructures physics data consumption) and is retired in MP4 when physics reads the
  pak's flat arrays directly.

### 6.4 What stays on runtime DRW (v1 scope control)

| Stays runtime | Why |
|---|---|
| UI dat assets (sprites, fonts, LayoutDescs — D.2b) | low volume, cold path |
| Audio waves (`DatSoundCache`) | low volume, already cached |
| Character-appearance palette overlays (ObjDesc) | runtime-parameterized per entity; keeps existing decode + cache |
| Motion tables / animations (sequencer) | sim-side, small, load-once |

`DatCollection` therefore stays in-process, off the hot path. Baking these later is a
listed follow-up, not v1.

### 6.5 Conformance + gate

- **Equivalence test:** load a fixture set (Holtburg blocks + a dungeon + a dense-town
  block) through the legacy path and the pak path; diff meshes (vertex/index bytes),
  textures (pre-compression pixels), BSP structures, scenery instance lists
  field-by-field. Golden = legacy path.
- **Gate:** equivalence green; portal-hop / traversal p99 from MP0 shows the hitch
  class gone; user visual gate (no missing/wrong geometry); legacy decode path deleted
  from the streaming hot path in the gate commit.

## 7. MP2 — Distance-degrade port (hide-only first cut)

The already-scoped retail port from the 2026-06-23 handoff §3 (read it + the named
retail sources FULLY before porting — mandatory workflow):

- Retail anchors: `GfxObjDegradeInfo::get_degrade` (`0x0051e4b0`, pseudo-C :293086),
  `CPhysicsPart::UpdateViewerDistance` (`0x0050e030`, :275517),
  `get_max_degrade_distance` (`0x0051e2d0`, :292918), `Render_DegradeDistance`
  user setting (:3270).
- **Hide-only cut (this phase):** per-entity max-draw-distance resolved at spawn from
  the degrade table; distance check in the entity walk; player + near NPCs exempt from
  hiding (handoff caution). Full per-slot mesh selection deliberately lands in MP3
  where the ECS DegradeSystem owns it — accepted small double-touch, avoids building
  slot-swap plumbing into a dispatcher MP3 replaces.
- Register bookkeeping: the "no distance-degrade" deviation gets its row updated to
  "partial (hide faithful; slot selection pending MP3)" in the same commit; the row is
  deleted in MP3's gate commit.
- **Gate:** retail side-by-side is the acceptance (same objects visible/hidden at the
  same distances, user-confirmed); drawn-instance count before/after recorded from the
  MP0 counters (no invented threshold — retail is the oracle); conformance test on the
  `get_max_degrade_distance` math vs decomp.

## 8. MP3 — ECS render world (Arch) + delta submission (the 300 lever)

### 8.1 The world

Arch `World` in `AcDream.App` owning every drawable: statics, baked scenery instances,
live entities, cell objects. Components (all blittable structs):

- `WorldTransform` (position/rotation/scale, cell-relative frame per #145 rules)
- `MeshRef` (pak mesh handle + part index)
- `DegradeState` (current slot, per-slot table handle)
- `Residency` (landblockId, cellId)
- `RenderFlags` (translucent, hidden, player-owned, …)
- `DirtyTag` (tag component; added on change, cleared by Upload)

### 8.2 Systems (fixed order, update thread in v1)

1. **TransformSync** — consumes the simulation's transform/lifecycle events through
   ONE seam (single-owner-state rule; the sim never writes render state directly).
   Static world entities receive a transform exactly once at residency.
2. **Degrade** — retail `get_degrade` banded slot selection (ideal/max band = no
   popping/hysteresis by construction), per entity per frame. Completes the MP2 port;
   divergence row deleted here.
3. **Visibility** — frustum + cell/PVS gate (Option A "one gate" preserved from the
   render digest — visibility computed once, enforced once).
4. **Upload** — iterates `DirtyTag` entities only; writes changed instance records
   into the persistent-mapped SSBO mirror (SoA layout matching the N.5 SSBO), budgeted
   per frame.
5. **Draw** — builds the MDI command buffer from the visible set and submits. Instance
   *data* is GPU-resident and untouched unless dirty; only commands are per-frame.
   GPU compute culling is a measured-need stretch inside this phase, not default scope.

Steady-state per-frame cost becomes: degrade + visibility iteration (linear over
packed arrays) + uploads proportional to *what changed* (a handful of moving
entities), replacing today's full per-frame walk-and-emit (`WalkEntitiesInto` emitting
every MeshRef of every in-frustum entity every frame + the ~2.1 ms scenery rebuild).

### 8.3 Encoded lessons (from the DO-NOT-RETRY tables — binding on implementation)

- Cache/dirty reset gates on actual entity change via tracker (the #53 class).
- Visibility volumes derive from drawn data, never synthetic constants (#119/#128).
- Every GL state the draw uses is set, not inherited (render-self-contained rule).
- Player-owned render state written only at the player chokepoint (#131 class).
- H3 (unbounded live-entity growth across hops) is fixed here structurally: residency
  eviction drops render entities (render-only eviction; sim state untouched),
  re-materialized on re-entry.

### 8.4 Conformance + gate

- **Instance-set diff:** probe dumps the drawn (meshId, transform, slot) set from the
  legacy dispatcher and the ECS path on identical fixture frames; sets must match
  exactly (modulo MP2's intended hides).
- **Screenshot diff:** fixed camera poses (Holtburg outdoor set + dungeon set +
  dense-town set), old vs new, pixel-identical requirement.
- Bring-up happens behind a temporary flag; **the gate commit deletes the legacy
  dispatcher path** (no lingering dual pipeline — N.5 ship-amendment precedent).
- **Gate:** diffs green; dense-town steady FPS ≥ 300 OR the MP0 profile names the next
  dominant cost (which then decides MP4/MP5 emphasis); user visual gate.

## 9. MP4 — Zero-alloc frame loop + flat physics data

**Sequencing constraint: starts only after M1.5 #137 (dungeon collision) has landed**
— it touches the same physics data structures.

- Allocation audit driven by MP0's per-frame counter: identify every steady-state
  allocation site (closures, LINQ, params arrays, list growth); fix via structs,
  pooled buffers, `Span<T>`, preallocated scratch.
- Physics consumes the pak's flattened BSP arrays: `PhysicsDataCache` swaps the
  managed BSP node graph for index-walks over blittable arrays. Same traversal
  order, same arithmetic — layout change only.
- **Conformance:** the existing physics test suites + the trajectory replay harness
  must pass unchanged (bit-identical trajectories); that harness exists precisely for
  this class of change.
- **Gate:** steady-state allocations/frame = 0 (measured); physics suites green;
  traversal p99 re-measured.

## 10. MP5 — Jobs (stretch, evidence-gated)

Only if the MP0 profile after MP4 still shows the frame short of target: parallelize
the independent stages (Arch has built-in multithreaded query support; visibility and
degrade are embarrassingly parallel once data is flat). **Explicitly skipped with
recorded numbers if targets are already met** — parallelism is a multiplier, not a
goal.

## 11. Testing strategy (summary)

| Apparatus | Built in | Used by |
|---|---|---|
| FrameProfiler (CPU/GPU/alloc, percentiles) | MP0 | every gate |
| Pak equivalence diff (mesh/texture/BSP/scenery) | MP1 | MP1 gate, rebakes |
| Instance-set diff probe | MP3 | MP3 gate |
| Screenshot diff harness (fixed poses) | MP3 | MP3 gate, regressions |
| Physics trajectory replay harness | exists | MP4 gate |
| Existing suites (Core ~1568 tests, conformance sweeps) | exist | every phase (green required) |

`DatConcurrencyStressTests` stays until MP1 retires hot-path DRW, then is scoped to
the residual runtime-DRW set (§6.4).

## 12. Firewall, process, bookkeeping

- **Track identity:** phases commit as `feat(pipeline): MP1 — …`. The roadmap gains a
  "Modern Pipeline side track" section; the milestones doc gets an explicit
  **freeze-exception paragraph** (this track deliberately reopens the frozen
  streaming/rendering subsystems under new architecture, user-authorized 2026-07-05).
- **One milestone rule preserved:** M1.5 remains the active milestone; MP work happens
  in dedicated side-track sessions in a separate worktree. M1.5 critical path wins
  every conflict; MP4 explicitly queues behind #137.
- **Divergence register:** MP2/MP3 update-then-delete the degrade row as described.
  MP1/MP3/MP4 are perf-only and must introduce **zero** rows — any behavioral
  difference discovered at a gate is a bug to fix, not a deviation to register.
- **Structure rules honored:** no new GameWindow feature bodies (new classes:
  `FrameProfiler`, `PakReader`, `AcDream.Bake`, `AcDream.Content`, ECS systems);
  env vars through `RuntimeOptions`; tests in matching layer projects; new project
  references documented in the inventory doc.
- **Dependency:** `Arch` (NuGet) added to `AcDream.App` only. License verified at
  adoption (expected Apache-2.0); if ever abandoned, vendor the source
  (WorldBuilder precedent) or swap for hand-rolled SoA behind the same system
  interfaces.

## 13. Risks & mitigations

| Risk | Mitigation |
|---|---|
| Pak staleness confusion (dat patch → wrong assets) | iteration stamps in header; hard refuse + rebake instruction; auto-offer later |
| ECS cutover regresses rendering (the #53/#119/#128 class) | instance-set + screenshot diffs as hard gates; lessons encoded as binding rules (§8.3); legacy deleted only at gate |
| MP1 extraction (`AcDream.Content`) breaks the mesh path | mechanical move, no rewrite; pak equivalence test catches interpretation drift |
| Degrade double-touch (MP2 hide-only, MP3 slots) | accepted, small; hide plumbing (spawn-time max-dist) carries into MP3 unchanged |
| Arch abandonment / license surprise | verify license before adoption; vendorable; swappable behind system interfaces |
| Side track leaks into M1.5 time or files | separate worktree; MP4 hard-queued behind #137; sessions labeled |
| MP0 profile contradicts the assumed cost split | that's MP0's job — re-sequence and amend this doc before MP1 |
| Baked scenery diverges from runtime generation | scenery placement is a pure function of dat data; equivalence diff covers it |

## 14. References

- `docs/research/2026-06-23-fps-distance-degrade-handoff.md` — baseline numbers, degrade
  retail anchors, measurement caveats (the §2/§3 this design consumes).
- `claude-memory/feedback_render_perf_measurement.md` — profiler lies + CPU-bound
  attribution + parked levers.
- `claude-memory/project_render_pipeline_digest.md`, `project_physics_collision_digest.md`
  — DO-NOT-RETRY tables binding on MP3/MP4.
- `docs/superpowers/specs/2026-06-23-datlock-contention-fix-design.md` — the shipped
  `_datLock` fix MP1 supersedes (bundle → pak).
- `docs/research/2026-06-09-dat-reader-thread-safety-investigation.md` — DRW read-path
  safety + the teardown fix + `DatConcurrencyStressTests`.
- `memory/reference_modern_rendering_pipeline.md` — N.5 SSBO layout the ECS Upload
  system mirrors.
- `memory/reference_two_tier_streaming.md` — residency tiers the pak loads plug into.
- Arch ECS: https://github.com/genaray/Arch (NuGet `Arch`).