acdream/docs/superpowers/specs/2026-07-05-modern-pipeline-design.md

Design — Modern Pipeline side track (MP): baked assets + ECS render world + zero-alloc frame loop

Date: 2026-07-05 Status: brainstorm-approved; awaiting user spec review Origin: user-commissioned performance side track ("fully modern pipeline, with ECS"). Explicitly NOT M1.5 work — see §12 for the firewall. Phase prefix: MP0–MP5 (MP = Modern Pipeline; avoids collision with the existing P0 conformance-apparatus plan). Implementation cadence: each MP phase gets its own implementation plan (docs/superpowers/plans/) when it starts; this spec is the umbrella design. The first plan covers MP0 (+ MP1 scaffold) only.

---

1. Motivation + measured baseline

acdream's steady-state dense-town frame rate is ~165 FPS (~6 ms/frame) after the 2026-06-23/24 fixes, versus 300+ FPS in comparable modern titles on the same machine. Traversal (streaming, portal hops) still hitches. The measured facts this design builds on (do NOT re-derive; sources in §14):

Fact	Evidence
GPU is nearly idle: ~0.5 ms/frame real GPU time in dense town	2026-06-23/24 deep-dive; the "12 ms GPU" was a glFinish artifact
The frame is CPU-submission-bound	dense-town attribution report; per-cell batch fix took 75→~165 FPS
_datLock contention (the 30↔200 swing) is FIXED — lockwait 88 ms p95 → 0.2 µs	536f1c04 + handoff §1
No distance-degrade: every frustum-visible object draws at full detail at any distance (~1,400 draws / ~24,000 instances)	2026-06-23 handoff §2; GfxObjDegradeResolver.cs "always slot 0" comment
Parked CPU levers: WbDrawDispatcher per-frame static-scenery rebuild (~2.1 ms), cross-frame dispatcher cache (judged HIGH-risk as a patch to the frozen renderer)	feedback_render_perf_measurement.md
Load hitches come from runtime dat decode (mesh extraction, texture decode, BSP graph construction) + GC	thread-safety investigation; two-tier streaming architecture
Runtime dat reads carry an unbounded second-layer cache (DatDatabaseWrapper._cache) stacked on DRW's own	DatCollectionAdapter.cs:119

The conclusion the whole design follows from: the language and the GPU are not the bottleneck; the data layout and the per-frame/at-load CPU work are. Modernizing means (a) moving decode work offline, (b) making the renderer process changes instead of re-describing the scene every frame, (c) flattening hot data, (d) removing frame-loop allocation.

2. Goals & acceptance

1. Smoothness (primary): no frame over ~16 ms during any traversal — town walking while streaming, portal hops, dungeon transitions. Measured: p99 frame time from the MP0 profiler across a scripted traversal route, plus user-eyes confirmation.
2. Throughput: 300+ FPS sustained in the dense-town worst cases (Fort Tethana, Arwic, Holtburg), Release build, vsync off, user's machine, standing + panning.
3. Architecture: baked asset pipeline (pak), Arch-based ECS render world, zero-allocation steady-state frame loop.

Non-goals (binding):

• No Rust. Decision recorded: the bottleneck is architectural, not language; a rewrite discards ~200K lines of verified retail-faithful ports.
• No ECS in the simulation. MovementManager, the animation sequencer, physics transitions, weenies keep their retail-mirroring OO structure — that correspondence is the project's debugging methodology.
• No retail-visible behavior changes. Everything here is perf-only (per feedback_render_perf_not_faithfulness_gated) except MP2, which is a faithful port that retires a known divergence.
• Not M1.5 work. #137/#138/A7 continue independently and win all conflicts.

3. Decisions recorded from the brainstorm

Decision	Choice	Why
Language	C#/.NET 10	Perf gap to Rust for optimized code is ~10–30%; our gap is architectural multiples; 200K LOC of verified ports retained
ECS flavor	Arch framework (NuGet Arch, ~2.1.x) — render world only	User choice (craft/learning value); fastest .NET archetype ECS; contained + swappable behind our interfaces
Bake UX	CLI tool (acdream-bake) now; client auto-detect + offer later	Iteration speed while the format churns
Sequencing	Measure → smooth → throughput	User priority: smoothness first
Cutover style	Per-phase gated slices, legacy path deleted at each gate	A8/N.5 history: big-bang cutovers and lingering dual paths both failed here

4. Architecture overview

┌────────────────────────────────────────────────────────────────────┐
│ SIMULATION (unchanged structure)                                   │
│ retail-shaped OO: physics, MovementManager, sequencer, weenies     │
└──────────────┬─────────────────────────────────────────────────────┘
               │ one seam: transform/lifecycle events (single-owner)
┌──────────────▼─────────────────────────────────────────────────────┐
│ ECS RENDER WORLD (Arch, AcDream.App)               [MP3]           │
│ entities = drawables; components = Transform, MeshRef, Degrade,    │
│ Residency, Flags; systems = TransformSync → Degrade → Visibility   │
│ → Upload(dirty ranges) → Draw(MDI)                                 │
└──────────────┬─────────────────────────────────────────────────────┘
               │ handles into
┌──────────────▼─────────────────────────────────────────────────────┐
│ ASSET LAYER (pak)                                  [MP1]           │
│ acdream-bake (offline, uses DRW + existing tested interpretation)  │
│ → pak file → PakReader (mmap, zero-copy, no locks, no decode)      │
└──────────────┬─────────────────────────────────────────────────────┘
               │ persistent-mapped staging, budgeted uploads
┌──────────────▼─────────────────────────────────────────────────────┐
│ GPU (unchanged N.5 foundation: bindless + MDI, 3 SSBOs)            │
└────────────────────────────────────────────────────────────────────┘

The N.5 bindless+MDI foundation is kept — it is already the modern half. This design replaces what feeds it (per-frame rebuild → deltas) and what feeds that (runtime decode → baked pak).

5. MP0 — Apparatus (honest measurement)

New FrameProfiler (dedicated class in src/AcDream.App/Diagnostics/, NOT GameWindow; CLAUDE.md structure rule 1):

• CPU frame time: swap-to-swap delta, ring buffer, p50/p95/p99 + histogram.
• GPU frame time: whole-frame GL_TIME_ELAPSED query, never enabled together with any glFinish-style per-pass bracketing (the 2026-06-23 lesson — separate flags, enforced in code, not convention).
• Per-stage attribution: glQueryCounter (GL_TIMESTAMP) markers per pass — no glFinish, no nesting conflict with the frame query.
• Allocation counter: GC.GetAllocatedBytesForCurrentThread() delta per frame on the update thread + Gen0/1/2 collection counts. This is MP4's before/after meter.
• Toggles via RuntimeOptions + DebugPanel (structure rules 4/5); env ACDREAM_FRAME_PROF=1. Stays in the tree permanently (unlike the throwaway ACDREAM_FPS_PROF), because every MP gate reads it.

Gate: a captured baseline report (dense-town steady + scripted traversal route) with the CPU 6 ms attributed per stage. If the attribution contradicts §1's assumed split, the MP phase order is re-sequenced and this doc amended before MP1 starts.

6. MP1 — Bake + pak (the smoothness backbone)

6.1 The bake tool

acdream-bake — new project src/AcDream.Bake/ (console; registered in AcDream.slnx). Reads the user's dats via DRW + the same interpretation code the client uses today, run offline:

• Mesh extraction (GfxObj/Setup → GPU-layout vertex/index + batch tables) via the existing ObjectMeshManager build path.
• Texture decode (palette/INDEX16/etc. → RGBA) via the existing TextureHelpers path, then compressed to BC7 (quality) / BC1 (opaque bulk) with precomputed mip chains.
• BSPs (physics, cell, drawing) flattened to index-based node arrays (blittable).
• Landblock terrain data (heights/attributes) as blittable arrays.
• EnvCells (geometry + portals + surfaces) as blittable records.
• Degrade tables (per-GfxObj slot ideal_dist/max_dist/ids) — MP2/MP3 read these.
• Scenery placement: SceneryGenerator evaluated per landblock at bake; the pak stores final instance lists (deterministic function of dat data, so bake-safe).

Extraction prerequisite: the CPU-side mesh/texture build code currently lives in AcDream.App next to GL code. MP1 moves the GL-free parts into a new src/AcDream.Content/ assembly (no GL dependency) that both AcDream.App and AcDream.Bake reference. This is a mechanical extraction (move + namespace), not a rewrite — the same lines keep running. Inventory-doc update required (structure rule 2).

6.2 Pak format (v1)

Single file acdream.pak next to the dats (path via RuntimeOptions):

• Header: magic ACPK, format version, dat iteration stamps (all four dats) + bake-tool version. Stale/missing pak at launch → clear error naming the rebake command (CLI era) → later: auto-offer (per §3).
• TOC: flat sorted array (assetKey: u64 [type:u8 | fileId:u32 | variant:u24]) → (offset: u64, length: u32, flags: u32); binary-search or hash lookup, O(1)-ish, loaded once.
• Blobs 64-byte aligned, uncompressed in v1 (mmap + OS page cache + NVMe; LZ4 block compression is a noted future option, not v1 scope).

6.3 Runtime consumption

• PakReader in AcDream.Core (Core stays GL-free; rule 2): one MemoryMappedFile, GetBlob(key) → ReadOnlySpan<byte> into the view; typed access via MemoryMarshal.Cast over blittable structs. No locks anywhere — immutable mapped memory is thread-safe to read by construction.
• Streaming workers stop decoding: a landblock load becomes TOC lookups + handing spans to the GL upload queue. The decode worker pool shrinks to the residual runtime-decode set (below). Request-key dedup stays in the streaming layer (cheap loads still shouldn't be duplicate loads).
• Retires: runtime DRW decode on the world render/streaming hot path; the unbounded DatDatabaseWrapper._cache. The PhysicsDatBundle worker pre-read stays through MP1–MP3 (the physics apply consumes parsed DBObj types until MP4 restructures physics data consumption) and is retired in MP4 when physics reads the pak's flat arrays directly.

6.4 What stays on runtime DRW (v1 scope control)

Stays runtime	Why
UI dat assets (sprites, fonts, LayoutDescs — D.2b)	low volume, cold path
Audio waves (DatSoundCache)	low volume, already cached
Character-appearance palette overlays (ObjDesc)	runtime-parameterized per entity; keeps existing decode + cache
Motion tables / animations (sequencer)	sim-side, small, load-once

DatCollection therefore stays in-process, off the hot path. Baking these later is a listed follow-up, not v1.

6.5 Conformance + gate

• Equivalence test: load a fixture set (Holtburg blocks + a dungeon + a dense-town block) through the legacy path and the pak path; diff meshes (vertex/index bytes), textures (pre-compression pixels), BSP structures, scenery instance lists field-by-field. Golden = legacy path.
• Gate: equivalence green; portal-hop / traversal p99 from MP0 shows the hitch class gone; user visual gate (no missing/wrong geometry); legacy decode path deleted from the streaming hot path in the gate commit.

7. MP2 — Distance-degrade port (hide-only first cut)

The already-scoped retail port from the 2026-06-23 handoff §3 (read it + the named retail sources FULLY before porting — mandatory workflow):

• Retail anchors: GfxObjDegradeInfo::get_degrade (0x0051e4b0, pseudo-C :293086), CPhysicsPart::UpdateViewerDistance (0x0050e030, :275517), get_max_degrade_distance (0x0051e2d0, :292918), Render_DegradeDistance user setting (:3270).
• Hide-only cut (this phase): per-entity max-draw-distance resolved at spawn from the degrade table; distance check in the entity walk; player + near NPCs exempt from hiding (handoff caution). Full per-slot mesh selection deliberately lands in MP3 where the ECS DegradeSystem owns it — accepted small double-touch, avoids building slot-swap plumbing into a dispatcher MP3 replaces.
• Register bookkeeping: the "no distance-degrade" deviation gets its row updated to "partial (hide faithful; slot selection pending MP3)" in the same commit; the row is deleted in MP3's gate commit.
• Gate: retail side-by-side is the acceptance (same objects visible/hidden at the same distances, user-confirmed); drawn-instance count before/after recorded from the MP0 counters (no invented threshold — retail is the oracle); conformance test on the get_max_degrade_distance math vs decomp.

8. MP3 — ECS render world (Arch) + delta submission (the 300 lever)

8.1 The world

Arch World in AcDream.App owning every drawable: statics, baked scenery instances, live entities, cell objects. Components (all blittable structs):

• WorldTransform (position/rotation/scale, cell-relative frame per #145 rules)
• MeshRef (pak mesh handle + part index)
• DegradeState (current slot, per-slot table handle)
• Residency (landblockId, cellId)
• RenderFlags (translucent, hidden, player-owned, …)
• DirtyTag (tag component; added on change, cleared by Upload)

8.2 Systems (fixed order, update thread in v1)

1. TransformSync — consumes the simulation's transform/lifecycle events through ONE seam (single-owner-state rule; the sim never writes render state directly). Static world entities receive a transform exactly once at residency.
2. Degrade — retail get_degrade banded slot selection (ideal/max band = no popping/hysteresis by construction), per entity per frame. Completes the MP2 port; divergence row deleted here.
3. Visibility — frustum + cell/PVS gate (Option A "one gate" preserved from the render digest — visibility computed once, enforced once).
4. Upload — iterates DirtyTag entities only; writes changed instance records into the persistent-mapped SSBO mirror (SoA layout matching the N.5 SSBO), budgeted per frame.
5. Draw — builds the MDI command buffer from the visible set and submits. Instance data is GPU-resident and untouched unless dirty; only commands are per-frame. GPU compute culling is a measured-need stretch inside this phase, not default scope.

Steady-state per-frame cost becomes: degrade + visibility iteration (linear over packed arrays) + uploads proportional to what changed (a handful of moving entities), replacing today's full per-frame walk-and-emit (WalkEntitiesInto emitting every MeshRef of every in-frustum entity every frame + the ~2.1 ms scenery rebuild).

8.3 Encoded lessons (from the DO-NOT-RETRY tables — binding on implementation)

• Cache/dirty reset gates on actual entity change via tracker (the #53 class).
• Visibility volumes derive from drawn data, never synthetic constants (#119/#128).
• Every GL state the draw uses is set, not inherited (render-self-contained rule).
• Player-owned render state written only at the player chokepoint (#131 class).
• H3 (unbounded live-entity growth across hops) is fixed here structurally: residency eviction drops render entities (render-only eviction; sim state untouched), re-materialized on re-entry.

8.4 Conformance + gate

• Instance-set diff: probe dumps the drawn (meshId, transform, slot) set from the legacy dispatcher and the ECS path on identical fixture frames; sets must match exactly (modulo MP2's intended hides).
• Screenshot diff: fixed camera poses (Holtburg outdoor set + dungeon set + dense-town set), old vs new, pixel-identical requirement.
• Bring-up happens behind a temporary flag; the gate commit deletes the legacy dispatcher path (no lingering dual pipeline — N.5 ship-amendment precedent).
• Gate: diffs green; dense-town steady FPS ≥ 300 OR the MP0 profile names the next dominant cost (which then decides MP4/MP5 emphasis); user visual gate.

9. MP4 — Zero-alloc frame loop + flat physics data

Sequencing constraint: starts only after M1.5 #137 (dungeon collision) has landed — it touches the same physics data structures.

• Allocation audit driven by MP0's per-frame counter: identify every steady-state allocation site (closures, LINQ, params arrays, list growth); fix via structs, pooled buffers, Span<T>, preallocated scratch.
• Physics consumes the pak's flattened BSP arrays: PhysicsDataCache swaps the managed BSP node graph for index-walks over blittable arrays. Same traversal order, same arithmetic — layout change only.
• Conformance: the existing physics test suites + the trajectory replay harness must pass unchanged (bit-identical trajectories); that harness exists precisely for this class of change.
• Gate: steady-state allocations/frame = 0 (measured); physics suites green; traversal p99 re-measured.

10. MP5 — Jobs (stretch, evidence-gated)

Only if the MP0 profile after MP4 still shows the frame short of target: parallelize the independent stages (Arch has built-in multithreaded query support; visibility and degrade are embarrassingly parallel once data is flat). Explicitly skipped with recorded numbers if targets are already met — parallelism is a multiplier, not a goal.

11. Testing strategy (summary)

Apparatus	Built in	Used by
FrameProfiler (CPU/GPU/alloc, percentiles)	MP0	every gate
Pak equivalence diff (mesh/texture/BSP/scenery)	MP1	MP1 gate, rebakes
Instance-set diff probe	MP3	MP3 gate
Screenshot diff harness (fixed poses)	MP3	MP3 gate, regressions
Physics trajectory replay harness	exists	MP4 gate
Existing suites (Core ~1568 tests, conformance sweeps)	exist	every phase (green required)

DatConcurrencyStressTests stays until MP1 retires hot-path DRW, then is scoped to the residual runtime-DRW set (§6.4).

12. Firewall, process, bookkeeping

• Track identity: phases commit as feat(pipeline): MP1 — …. The roadmap gains a "Modern Pipeline side track" section; the milestones doc gets an explicit freeze-exception paragraph (this track deliberately reopens the frozen streaming/rendering subsystems under new architecture, user-authorized 2026-07-05).
• One milestone rule preserved: M1.5 remains the active milestone; MP work happens in dedicated side-track sessions in a separate worktree. M1.5 critical path wins every conflict; MP4 explicitly queues behind #137.
• Divergence register: MP2/MP3 update-then-delete the degrade row as described. MP1/MP3/MP4 are perf-only and must introduce zero rows — any behavioral difference discovered at a gate is a bug to fix, not a deviation to register.
• Structure rules honored: no new GameWindow feature bodies (new classes: FrameProfiler, PakReader, AcDream.Bake, AcDream.Content, ECS systems); env vars through RuntimeOptions; tests in matching layer projects; new project references documented in the inventory doc.
• Dependency: Arch (NuGet) added to AcDream.App only. License verified at adoption (expected Apache-2.0); if ever abandoned, vendor the source (WorldBuilder precedent) or swap for hand-rolled SoA behind the same system interfaces.

13. Risks & mitigations

Risk	Mitigation
Pak staleness confusion (dat patch → wrong assets)	iteration stamps in header; hard refuse + rebake instruction; auto-offer later
ECS cutover regresses rendering (the #53/#119/#128 class)	instance-set + screenshot diffs as hard gates; lessons encoded as binding rules (§8.3); legacy deleted only at gate
MP1 extraction (AcDream.Content) breaks the mesh path	mechanical move, no rewrite; pak equivalence test catches interpretation drift
Degrade double-touch (MP2 hide-only, MP3 slots)	accepted, small; hide plumbing (spawn-time max-dist) carries into MP3 unchanged
Arch abandonment / license surprise	verify license before adoption; vendorable; swappable behind system interfaces
Side track leaks into M1.5 time or files	separate worktree; MP4 hard-queued behind #137; sessions labeled
MP0 profile contradicts the assumed cost split	that's MP0's job — re-sequence and amend this doc before MP1
Baked scenery diverges from runtime generation	scenery placement is a pure function of dat data; equivalence diff covers it

14. References

• docs/research/2026-06-23-fps-distance-degrade-handoff.md — baseline numbers, degrade retail anchors, measurement caveats (the §2/§3 this design consumes).
• claude-memory/feedback_render_perf_measurement.md — profiler lies + CPU-bound attribution + parked levers.
• claude-memory/project_render_pipeline_digest.md, project_physics_collision_digest.md — DO-NOT-RETRY tables binding on MP3/MP4.
• docs/superpowers/specs/2026-06-23-datlock-contention-fix-design.md — the shipped _datLock fix MP1 supersedes (bundle → pak).
• docs/research/2026-06-09-dat-reader-thread-safety-investigation.md — DRW read-path safety + the teardown fix + DatConcurrencyStressTests.
• memory/reference_modern_rendering_pipeline.md — N.5 SSBO layout the ECS Upload system mirrors.
• memory/reference_two_tier_streaming.md — residency tiers the pak loads plug into.
• Arch ECS: https://github.com/genaray/Arch (NuGet Arch).