# Report — dense-town FPS attribution (Arwic, post-cells-fix) — 2026-06-23 **Mode:** report-only investigation (no edits, no diagnostic drops). Deliverable is this doc + the chat verdict. Continues the handoff `2026-06-23-dense-town-fps-deepdive-handoff.md`. The cells fix (29→75 fps) is KEPT and confirmed intact. **Method:** 7 parallel read-only subsystem readers (orchestration, punch/seal, particles, portal-vis CPU, alloc/GC, pview-master, apparatus) + independent verification reads of the 6 core files by the lead. Every claim below is cited to `file:line` from source read in full. **No live measurement was taken** — the decisive runs are listed in §6 for the user to drive. --- ## OUTCOME (FINAL — shipped 2026-06-24) Dense Arwic: **75 → ~165 fps** (steady; ~130 at the absolute densest standing-still view — over the 144 target except that one extreme). Two isolated, verified, pixels-identical fixes (render-perf is not faithfulness-gated): 1. **Cell-object draw batching** (`290e731`) — `DrawCellObjectLists` collapsed N per-cell `WbDrawDispatcher.Draw` calls into one cross-cell batched draw (`cellobjects` 3.5 → ~0.4 ms). 2. **Cell-particle consolidation** (`9f51a4d`) — the per-cell `DrawCellParticles` re-walk (O(cells×particles)) collapsed into one union pass; also fixed a latent additive **double-draw** in multi-aperture cells. Throwaway profiling apparatus stripped (`a9d06a6`). Build + full suite green. **The handoff's framing was wrong, and is corrected here:** the dense town was NEVER GPU-bound. The "~12 ms GPU" was a **glFinish self-measurement artifact** (the profiler's own pipeline flushes inflated its number); the true GPU is ~0.5 ms standing still. The frame is **~96 % CPU-bound**, and the cost scales with how many buildings are in view. **Remaining headroom — deliberately NOT pursued (user-approved finalize):** the last ~2.1 ms CPU is `WbDrawDispatcher`'s per-frame static-scenery rebuild (`ls.scenery`); ~2.1 ms GPU is terrain rasterization (`ls.terrain`). Both live in FROZEN, load-bearing subsystems. The only way to fully cut scenery-CPU is a cross-frame cache of the entity dispatcher — judged HIGH risk (regresses ALL entity rendering if mis-keyed; the #53/#119/#128 bug class; thrashes on the eye's rest-jitter) for ~15 fps at one extreme view already over target. The safe micro-opts net only ~0.3 ms. Full analysis: §0b + the dispatcher-investigation workflow. --- ## 0a. MEASURED VERDICT — clean split, `ACDREAM_FPS_PROF=2` (2026-06-23, live Arwic) The hypothesis below was **confirmed by direct measurement.** Added a diagnostic `ACDREAM_FPS_PROF=2` mode (whole-frame `TimeElapsed` query, per-pass `glFinish` DISABLED) and ran it live in Arwic: | view | wall | cpuRender | **gpu** | present(wait) | vsync/msaa | |---|---|---|---|---|---| | facing AWAY from town | 6.7 ms (~149 fps) | 6.6 ms | **0.5 ms** | 0.1 ms | off / 4× | | facing INTO town | ~13 ms (~70 fps) | ~13 ms | **0.5 ms** | 0.1 ms | off / 4× | **The frame is ~96% CPU-render-bound. The GPU renders the entire dense town in 0.5 ms and is idle the rest of the frame.** `cpuRender ≈ wall`, `present ≈ 0.1 ms`, `gpu = 0.5 ms` regardless of view. The handoff's "~12 ms GPU" was **entirely a glFinish-serialization artifact** (§1). Turning the camera into the town doubles `cpuRender` (6.6 → 13 ms) while `gpu` never moves — the cost scales with the number of buildings in frustum, i.e. the per-building CPU work, not pixels. **DEAD for good (GPU is 0.5 ms):** MSAA, fill, overdraw, fragment shaders, far draw distance, any GPU-side lever. An MSAA=0 test is moot. **All fixes target CPU.** The spikes (cpuRender max 30–43 ms → dips to ~25 fps) are consistent with gen-0 GC from the ~5–8 k allocations/frame (§4); confirm with a gen-0 GC counter if needed. --- ## 0b. CPU SUB-PHASE BREAKDOWN — `[CPU-PHASE]`, live Arwic facing town (2026-06-23) Added `[CPU-PHASE]` timers around each `DrawInside` phase (run under `=2`, so draw phases = pure CPU submission). Steady-state, facing town (`update≈0`, `cpuRender` ~8 ms — the absolute varies 8–13 ms with view density; the RANKING is stable): | phase | ms/frame | what | |---|---|---| | **cellobjects** | **3.3–4.5** | `DrawCellObjectLists` — per-cell entity/static draw + per-cell particle pass | | **landscape** | **2.1–2.9** | sky (per-submesh) + terrain + scenery + late dynamics/particles/weather | | **partition** | **0.7–3.2** | `InteriorEntityPartition.Partition` + `ViewconeCuller.Build` | | **dynamics** | **0.6–1.3** | `DrawDynamicsLast` — dynamics draw + dynamics particles | | prepare | 0.04–1.3 | `PrepareRenderBatches` | | shells | 0.07–0.21 | the cells fix (cheap — working) | | flood | 0.08–0.27 | the 48 portal floods | | assemble | 0.03–0.24 | `ClipFrameAssembler` | | portalmask | 0.06–0.09 | the 31 punch/seal fans | **MEASUREMENT OVERTURNED THE STATIC RANKING (§2/§4/§5):** - **Portal floods are NOT the cost** (`flood = 0.1 ms`). **H1's "48 floods" emphasis and Tier C flood-caching are DEAD** — caching saves ~0.1 ms. - **Punch/seal is NOT the cost** (`portalmask = 0.08 ms`). Tier B #5 batching saves nothing. - **The clip-math alloc storm is NOT the steady cost** (flood+assemble ≈ 0.2 ms). It may still drive the *spikes* via GC (separate), but it is not the 8–13 ms steady cost. - **The cells fix works** (`shells = 0.1 ms`). **REAL TARGET — per-cell / per-entity DRAW SUBMISSION:** `cellobjects + landscape + dynamics ≈ 6.5 ms` + `partition ≈ 2 ms`. Common thread: `WbDrawDispatcher.Draw` is called **once per visible cell** in `DrawCellObjectLists` (each orphaning 6 SSBOs via `glBufferData` — `WbDrawDispatcher.cs:1521-1558`), plus per-cell `DrawCellParticles` re-walks, plus un-batched per-submesh sky. Scales with visible cells/entities ⇒ the facing-the-town cost. **Lesson: allocation COUNT ≠ CPU TIME; the per-phase CPU timer is what found the real hotspot.** **Revised fix priority (supersedes §5):** 1. **Batch per-cell entity/static draws** across cells into few `WbDrawDispatcher.Draw` calls (the cell-shell fix, applied to cell OBJECTS) — targets `cellobjects` (~3.5 ms). Pre-cull per-entity by viewcone, then one batched draw. 2. **`WbDrawDispatcher`: persistent SSBOs + `BufferSubData`** instead of 6 `glBufferData` orphans per call — compounds with #1 (helps `cellobjects`/`landscape`/`dynamics`). 3. **Consolidate particle `Draw` calls** (per-cell `DrawCellParticles` re-walks all particles) — targets part of `cellobjects` + `landscape`. 4. **Batch sky submeshes** — targets `landscape`. 5. **Optimize `partition`** (`InteriorEntityPartition.Partition` + `ViewconeCuller.Build`, ~2 ms CPU/frame) — possibly cache/incremental. Spikes (cpuRender max 30–43 ms; correlated `gpu`-max spikes to 300+ ms) coincide with `landscape`/`cellobjects` jumps + GPU upload — likely streaming mesh upload hitches (`_wbMeshAdapter.Tick`) and/or GC. Diagnose separately from the steady cost. --- ## 0. Verdict (TL;DR) — original static prediction, now MEASURED-CONFIRMED above The handoff frames the remaining ~12 ms as "diffuse GPU cost." The static evidence says that frame is **mis-attributed**: the 12 ms is a **glFinish- serialized** number, and the actual GPU *rasterization* is tiny (terrain 0.3, cells 0.2, entities 0.2 ms — the geometry genuinely is cheap, exactly your intuition). The remaining cost is overwhelmingly **CPU-submission / OpenGL driver-overhead / GC**, not GPU fill: - **48 uncached portal floods per frame** (1 root + ~47 buildings), recomputed from scratch every frame even when standing still, with **no caching**. - **~5,000–8,000 short-lived heap allocations per frame** (the Sutherland-Hodgman clip math alone is ~3–5 k) → gen-0 GC → the periodic spikes to 30–40 fps. - **Hundreds of redundant GL state calls per frame**: 31 punch/seal fans each doing a full state set+restore (~450–650 GL calls), `WbDrawDispatcher` orphaning 6 SSBOs via `glBufferData` on every one of its 3–5 calls (~66–110 buffer reallocs), per-submesh sky draws, ~80–128 `glEnable/Disable(ClipDistance)` toggles. - The particle system is **re-walked ~22× per frame** (once per cell), each walk allocating and each `Draw` doing `glBufferData` orphans — draw-count bound, **not fill** (consistent with the resolution-independence observation). **This is the classic OpenGL CPU-bound profile**, and it explains all three of your observations at once: resolution-independent, scales with town density, and a fast GPU can't help. **The single blocking gap:** we cannot prove CPU-vs-GPU today because the whole- frame `TimeElapsed` query and the per-pass `glFinish` are gated on the *same* `ACDREAM_FPS_PROF=1` flag (`FrameProfiler.cs:72-73`). The decisive next step is to decouple them (a ~5-line apparatus change) **or** capture a RenderDoc frame — both give the honest split. --- ## 1. The measurement-trust problem (read this first) `cpuRender` is a wall-clock stopwatch around the *entire* `OnRender` (`FrameProfiler.cs:99,110`), so it **absorbs every glFinish stall**. When `ACDREAM_FPS_PROF=1`: - Each instrumented renderer calls `gl.Finish()` **twice** (before + after): `EnvCellRenderer.cs:845,1050`; `ParticleRenderer.cs:128,181`; `PortalDepthMaskRenderer.cs:214,319`; terrain hooks `GameWindow.cs:10799,10803`. - Counting call frequency: ~62 finishes from punch/seal (31 fans × 2) + ~44 from particles (22 batches × 2) + cells + terrain ≈ **150+ full pipeline flushes per frame.** - Each `glFinish` drains the GPU and blocks the CPU. The GPU then sits **idle** between passes waiting for the CPU to issue the next finish-bracketed batch — and that idle time falls *inside* the whole-frame `TimeElapsed` window. **Consequence:** the `gpu ≈ 12 ms` total is *serialized* GPU time (rasterization + inter-pass idle), not *pipelined* GPU time. The per-pass `[PASS-GPU]` absolutes (particles 3.1, punchseal 2.9) are inflated by their own finishes and are upper bounds, not real costs. **No honest CPU/GPU split exists in the current apparatus.** (Independently re-derived by two of the seven readers.) `vsync` is off by default (`GameWindow.cs:998`), so vsync-quantization is unlikely but must be confirmed from the printed `vsync=` field. --- ## 2. Ranked hypotheses ### H1 (leading) — The frame is CPU-submission / driver-overhead bound, not GPU-fill bound - **For:** attributed GPU rasterization is tiny (terrain 0.31, cells 0.23, entities 0.22 ms — all measured in the handoff). Resolution-independent (resize didn't move FPS). Scales with building count. The CPU side has clear O(buildings) + O(cells) structure issuing hundreds of small state-changing GL calls. The cells fix worked precisely because it collapsed 94 heavy state-setting `Render` submissions into 1 — a *submission* win. - **Against:** not yet measured cleanly (glFinish contamination, §1). The resize test was on a tainted `FAR_RADIUS=4` build (handoff caveat). - **Falsify:** RUN 1 (§6) with glFinish decoupled — if `gpu p50 ≈ wall p50` and `cpuRender ≪ wall`, H1 is **wrong** and it's genuinely GPU-bound. If `cpuRender ≈ wall` and `gpu ≪ wall`, H1 confirmed. ### H2 — Frame spikes (cpuRender p95 ~30 ms, dips to 30–40 fps) are gen-0 GC pauses - **For:** ~5,000–8,000 short-lived heap objects/frame, almost all in the portal-flood + clip + assemble path (§4). At 75 fps that's 375k–600k allocs/s → frequent gen-0 STW collections. Retail uses fixed static scratch buffers; this alloc storm is "purely a .NET port artifact with no retail equivalent." - **Against:** could partly be the streaming/upload hitch (`_wbMeshAdapter.Tick`, `GameWindow.cs:8427`) on cell load. Both can be true. - **Falsify:** watch the .NET gen-0 GC counter / allocation rate during a stutter, or correlate spikes with `dotnet-counters`. If allocs/frame is low, GC isn't it. ### H3 — Particles are draw-count/state bound (~22 fragmented batches), not fill - **For:** `ParticleRenderer.Draw` is called from **~11 call sites**, including **per-cell** `DrawCellParticles` (~31×); each call re-walks the *entire* live particle set via `EnumerateLive` (`ParticleRenderer.cs:196`), sorts it, and each batch does a `glBufferData` orphan (`:278`). Trivial fragment shader, depth-write off. Resolution-independent ⇒ not fill. - **Against:** additive emitters have real overdraw (no depth write); secondary. - **Falsify:** RUN 3 resolution sweep — if particle GPU is flat with resolution, not fill. RenderDoc quad-overdraw overlay. ### H4 — Punch/seal (31 fans) is CPU state-churn, NOT 2.9 ms of GPU - **For:** each `DrawDepthFan` is a depth-only, color-masked, ≤32-vert fan — GPU cost is negligible. The 2.9 ms is 62 glFinish stalls. The *real* cost is the full per-fan state set+restore (`PortalDepthMaskRenderer.cs:229-313`) — ~450–650 GL calls/frame, including a `UseProgram`/`UseProgram(0)` round-trip and a `uViewProjection` re-upload per fan even though all fans share one matrix. - **Against:** none material. - **Falsify:** RUN 2 `[PASS-GPU]` with per-pass `TimeElapsed` (no glFinish) — fan GPU will read ≪ 2.9 ms. RenderDoc per-draw time. ### H5 — `WbDrawDispatcher` SSBO orphaning is a hidden chunk of the "unattributed 5.5 ms" - **For:** every `Draw()` re-uploads 6 SSBOs via `glBufferData(DynamicDraw)` (orphan+realloc, not `BufferSubData`) — `WbDrawDispatcher.cs:1521-1558` — ×3–5 calls/frame = ~66–110 buffer reallocs. Plus per-transparent-cell `EnvCell.Render` repeats all 6 SSBO uploads for a 1-cell instance set (`EnvCellRenderer.cs:1281-1367`). - **Against:** the dispatcher has its own `ACDREAM_WB_DIAG` GPU timer; cross-check it (run separately — it nests illegally with FPS_PROF, `FrameProfiler.cs:31`). - **Falsify:** RenderDoc; or `ACDREAM_WB_DIAG=1` in isolation. --- ## 3. What's ruled out (do not re-chase) - **Distance-degrade / LOD / triangle count** — dead (handoff §3, entity GPU 0.22 ms). - **MSAA / fill / overdraw of opaque geometry** — resolution-independent; re-verify cleanly in RUN 3 but the static evidence agrees (cheap fragment work everywhere). - **Update thread** — `update = 0.1 ms`. - **Terrain per-slice redraw** — the apparatus reader flagged `_terrain.Draw` being inside a per-slice loop (`GameWindow.cs:10795-10803`) as a HIGH suspect, **but the handoff already measured terrain at 1 slice / 0.31 ms.** At *outdoor* Arwic there is one full-screen outside slice, so terrain draws once. The per-slice multiplier only bites for **interior roots with multiple doorway slices** — a separate, non-Arwic concern. **NOT an Arwic FPS lever.** - **The cells fix** — intact and correct (`RetailPViewRenderer.cs:664-701`). Do not touch. --- ## 4. Per-subsystem cost ledger (static, cited) **Portal visibility (CPU + GC — the biggest structural cost)** - 48 BFS floods/frame, **no frame-to-frame caching**, recomputed when stationary (`RetailPViewRenderer.cs:64-83,228-235`; rebuilt `GameWindow.cs:8752`). - Per flood allocates `PortalVisibilityFrame` + 2 `HashSet` + `Dictionary` + `uint[128]` before processing a portal (`PortalVisibilityBuilder.cs:126-227`). - **`PortalProjection` clip math is the #1 GC source: ~3,000–5,000 short-lived `List`/`List`/`ToArray()` per frame** — `ProjectToClip:95`, `ClipToRegion:127`, `ClipHomogeneousEdge:216`, `MergeSubPixelVertices:184`. Retail used a **static two-buffer swap on a fixed vertex array** — so pooling is *more* retail-faithful, not less. - `CellTodoList.Insert` is O(N) (`:1006-1016`); `GetRange` allocates (`:254,583`); `CanonicalKey` allocates a `StringBuilder` per polygon (`PortalView.cs:255`). **ClipFrameAssembler (CPU + GC)** — ~800–1,200 heap objects/frame (4 dicts + 95 lists + 94 `CellView` + 94 `int[]` + per-polygon `Vector4[]`), all discarded each frame (`ClipFrameAssembler.cs:85-219`). No retail counterpart. **Punch/seal (CPU state churn)** — 31 fans × full state set+restore = ~450–650 GL calls/frame; per-fan `UseProgram` round-trip + `uViewProjection` re-upload (`PortalDepthMaskRenderer.cs:229-313`). Retail does **not** restore state per polygon (it leaves state installed for the next poly); our self-contained-state contract is the overhead. The two-pass stencil is an acdream-only #117 addition (retail relied on painter's order). Zero managed alloc inside `DrawDepthFan` (good — `_scratch` field + `stackalloc`). **Particles (CPU draw-count + GC)** — ~11 `Draw` call sites, per-cell `DrawCellParticles` re-walks all live particles ×~31 (`ParticleRenderer.cs:196`); `new List` (`:195`) + `new List(64)` (`:152`) per call; per-batch `glBufferData` orphan (`:278`). Draw-count bound. **EnvCellRenderer (CPU + GC + redundant uploads)** — opaque batched (the fix), but **transparent stays per-cell**: each transparent cell repeats all 6 SSBO uploads for a 1-cell set (`EnvCellRenderer.cs:1281-1367`). `_cellLightSetCache.Clear()` at `:1141` forces **188–282 `int[]` re-allocations/frame** (94 cells × 2–3 passes); the light sets are camera-independent and stable within a frame. `Render` allocates 4 collections per call (`:904,905,927,928`). `CellHasTransparent` is an O(gfx×batch) walk with no cached result. **Orchestration (CPU driver overhead)** — sky drawn per-submesh, no batching (~8–15 `DrawElements` + per-submesh `BlendFunc`/4×`SetFloat`, ×2 passes/frame, `SkyRenderer.cs:219-429`); ~80–128 `glEnable/Disable(ClipDistance0..7)` toggles/frame (`MaxPlanes=8`); `PrepareRenderBatches` runs `Parallel.ForEach` **on the render thread**, blocking `OnRender` (`EnvCellRenderer.cs:642`); `ParseEnvFloat` → `Environment.GetEnvironmentVariable` ×2/frame in the hot path (`GameWindow.cs:8639-8641`); `new[]{entry}` per `DrawEntityBucket` (~50–100/frame, `RetailPViewRenderer.cs:943`); `new[]{NoClipSlice}` per slot-less cell (~125/frame, `:918` — trivially a static readonly array). --- ## 5. Fix leads, ranked by (impact × retail-faithfulness) Brainstorm before any render change (handoff lesson — rushed render changes were reverted). These are *sketches for the implementation session*, not commitments. **Tier A — strictly retail-faithful (matches retail's static-scratch model), high impact, low risk:** 1. **Pool the `PortalProjection` clip buffers** (`ArrayPool`/double-buffer swap). Kills the single largest GC source (~3–5 k allocs/frame). Output contract unchanged. → directly targets H2 spikes. 2. **Pool `ClipFrameAssembler` + `PortalVisibilityBuilder` BFS scratch** (Reset/ Clear instead of `new`). ~1 k allocs/frame gone. 3. **Move `_cellLightSetCache.Clear()` out of `RenderModernMDIInternal`** to once- per-frame. Removes 188–282 `int[]`/frame. Light sets are frame-stable. 4. **Static readonly `NoClipSlice[]`** + pool `DrawEntityBucket`'s `new[]{entry}`. **Tier B — internal GL batching (the cells-fix pattern, no visual change), high impact:** 5. **Batch punch/seal**: set state ONCE before the fan loop, upload `uViewProjection` ONCE, merge all fan vertices into one VBO + draw with offsets. Drop the per-fan `UseProgram(0)`. (Retail also drew per-poly but with cheap D3D state blocks; our GL program-rebind/uniform-upload per fan is the cost.) 6. **Batch transparent EnvCell shells** like opaque (sort instances far→near, one `Render`). Removes N_transparent × 6-SSBO re-uploads. 7. **`WbDrawDispatcher`: persistent SSBOs + `BufferSubData`** instead of `glBufferData(DynamicDraw)` orphan per call. 8. **Consolidate particle `Draw` to ≤3 calls/frame** (one per `RenderPass`), pre-partition emitters by pass, single `BufferSubData` per `Draw`; additive in a separate order-independent group. 9. **Batch sky submeshes** (shared VBO / sort by blend mode). **Tier C — acceleration that diverges from retail (needs explicit brainstorm + a divergence-register row):** 10. **Cache per-building portal floods** keyed on (buildingId, quantized camera pose). Eliminates ~47 of 48 floods/frame when stationary. **But retail recomputes per frame from the BSP walk** — this is a memoization acdream adds, not a faithful match. Pure-function caching is defensible, but it changes the "recompute every frame" structure and must be invalidated on camera move + cell load/unload. Lower priority than Tier A (which kills the GC without diverging). 11. **Move `PrepareRenderBatches` off the render thread** (double-buffer in `OnUpdate`). One-frame visibility latency. --- ## 6. Decisive measurement plan (user-driven; report-only until approved) **Apparatus gap to fix first (one small change, then measure):** add a second flag (e.g. `ACDREAM_FPS_PROF=2` or `ACDREAM_FPS_NOFINISH=1`) that keeps the whole-frame `TimeElapsed` query but **disables the per-pass `glFinish`** — and/or convert the per-pass timers to **non-nested per-pass `TimeElapsed` queries** (the `WbDrawDispatcher` `ACDREAM_WB_DIAG` queries at `WbDrawDispatcher.cs:1642,1670` are the working template — no glFinish, no inflation). This is diagnostic-only. - **RUN 1 — honest split (glFinish off, frame query on), Arwic, stand still 10 s.** `gpu≈wall` ⇒ GPU-bound (go to RUN 4). `cpuRender≈wall, gpu≪wall` ⇒ **CPU-bound (H1)**. `present(wait)≈wall` ⇒ vsync/swap (check `vsync=`). - **RUN 2 — per-pass `[PASS-GPU]` via TimeElapsed (no glFinish).** Honest particles / punchseal / cells / terrain ms + calls/frame. - **RUN 3 — resolution sweep (720p → 1080p) on a CLEAN build.** GPU scales ~2.25× ⇒ fill-bound; flat ⇒ vertex/CPU-bound. (Re-verifies the tainted resize test.) - **RUN 4 — RenderDoc Arwic frame capture.** Exact per-draw GPU, total draw count (>200/frame at 720p = batching problem), quad-overdraw overlay (punch fans over the viewport show as red), per-pass timeline, MSAA-resolve cost. Confirm/refute: H1 ⇐ RUN 1; H3/H4 ⇐ RUN 2/3/4; H5 ⇐ `ACDREAM_WB_DIAG=1` solo + RenderDoc; H2 ⇐ gen-0 GC counter during a stutter. --- ## 7. What this is NOT - **NOT a GPU fill / shading / MSAA problem** — the GPU rasterizes this geometry in a couple of ms; the geometry is cheap (your intuition is correct). - **NOT a terrain per-slice redraw at Arwic** — outdoor = 1 slice, terrain draws once (0.31 ms measured). That lead applies only to multi-slice interior roots. - **NOT a single 24 ms lever** — it's death by a thousand CPU/driver/GC cuts; the win is the *sum* of the Tier A+B fixes. - **NOT fixed by the cells batching alone** — that solved the one big GPU submission sink; the CPU flood + GC + state churn remained underneath. - **The 12 ms `gpu` number is NOT trustworthy as "GPU rasterization time"** — it's glFinish-serialized. Re-measure honestly before drawing conclusions.