acdream/docs/research/2026-06-23-dense-town-fps-attribution-report.md

Report — dense-town FPS attribution (Arwic, post-cells-fix) — 2026-06-23

Mode: report-only investigation (no edits, no diagnostic drops). Deliverable is this doc + the chat verdict. Continues the handoff 2026-06-23-dense-town-fps-deepdive-handoff.md. The cells fix (29→75 fps) is KEPT and confirmed intact.

Method: 7 parallel read-only subsystem readers (orchestration, punch/seal, particles, portal-vis CPU, alloc/GC, pview-master, apparatus) + independent verification reads of the 6 core files by the lead. Every claim below is cited to file:line from source read in full. No live measurement was taken — the decisive runs are listed in §6 for the user to drive.

---

OUTCOME (FINAL — shipped 2026-06-24)

Dense Arwic: 75 → ~165 fps (steady; ~130 at the absolute densest standing-still view — over the 144 target except that one extreme). Two isolated, verified, pixels-identical fixes (render-perf is not faithfulness-gated):

1. Cell-object draw batching (290e731) — DrawCellObjectLists collapsed N per-cell WbDrawDispatcher.Draw calls into one cross-cell batched draw (cellobjects 3.5 → ~0.4 ms).
2. Cell-particle consolidation (9f51a4d) — the per-cell DrawCellParticles re-walk (O(cells×particles)) collapsed into one union pass; also fixed a latent additive double-draw in multi-aperture cells.

Throwaway profiling apparatus stripped (a9d06a6). Build + full suite green.

The handoff's framing was wrong, and is corrected here: the dense town was NEVER GPU-bound. The "~12 ms GPU" was a glFinish self-measurement artifact (the profiler's own pipeline flushes inflated its number); the true GPU is ~0.5 ms standing still. The frame is ~96 % CPU-bound, and the cost scales with how many buildings are in view.

Remaining headroom — deliberately NOT pursued (user-approved finalize): the last ~2.1 ms CPU is WbDrawDispatcher's per-frame static-scenery rebuild (ls.scenery); ~2.1 ms GPU is terrain rasterization (ls.terrain). Both live in FROZEN, load-bearing subsystems. The only way to fully cut scenery-CPU is a cross-frame cache of the entity dispatcher — judged HIGH risk (regresses ALL entity rendering if mis-keyed; the #53/#119/#128 bug class; thrashes on the eye's rest-jitter) for ~15 fps at one extreme view already over target. The safe micro-opts net only ~0.3 ms. Full analysis: §0b + the dispatcher-investigation workflow.

---

0a. MEASURED VERDICT — clean split, ACDREAM_FPS_PROF=2 (2026-06-23, live Arwic)

The hypothesis below was confirmed by direct measurement. Added a diagnostic ACDREAM_FPS_PROF=2 mode (whole-frame TimeElapsed query, per-pass glFinish DISABLED) and ran it live in Arwic:

view	wall	cpuRender	gpu	present(wait)	vsync/msaa
facing AWAY from town	6.7 ms (~149 fps)	6.6 ms	0.5 ms	0.1 ms	off / 4×
facing INTO town	~13 ms (~70 fps)	~13 ms	0.5 ms	0.1 ms	off / 4×

The frame is ~96% CPU-render-bound. The GPU renders the entire dense town in 0.5 ms and is idle the rest of the frame. cpuRender ≈ wall, present ≈ 0.1 ms, gpu = 0.5 ms regardless of view. The handoff's "~12 ms GPU" was entirely a glFinish-serialization artifact (§1). Turning the camera into the town doubles cpuRender (6.6 → 13 ms) while gpu never moves — the cost scales with the number of buildings in frustum, i.e. the per-building CPU work, not pixels.

DEAD for good (GPU is 0.5 ms): MSAA, fill, overdraw, fragment shaders, far draw distance, any GPU-side lever. An MSAA=0 test is moot. All fixes target CPU.

The spikes (cpuRender max 30–43 ms → dips to ~25 fps) are consistent with gen-0 GC from the ~5–8 k allocations/frame (§4); confirm with a gen-0 GC counter if needed.

---

0b. CPU SUB-PHASE BREAKDOWN — [CPU-PHASE], live Arwic facing town (2026-06-23)

Added [CPU-PHASE] timers around each DrawInside phase (run under =2, so draw phases = pure CPU submission). Steady-state, facing town (update≈0, cpuRender ~8 ms — the absolute varies 8–13 ms with view density; the RANKING is stable):

phase	ms/frame	what
cellobjects	3.3–4.5	DrawCellObjectLists — per-cell entity/static draw + per-cell particle pass
landscape	2.1–2.9	sky (per-submesh) + terrain + scenery + late dynamics/particles/weather
partition	0.7–3.2	InteriorEntityPartition.Partition + ViewconeCuller.Build
dynamics	0.6–1.3	DrawDynamicsLast — dynamics draw + dynamics particles
prepare	0.04–1.3	PrepareRenderBatches
shells	0.07–0.21	the cells fix (cheap — working)
flood	0.08–0.27	the 48 portal floods
assemble	0.03–0.24	ClipFrameAssembler
portalmask	0.06–0.09	the 31 punch/seal fans

MEASUREMENT OVERTURNED THE STATIC RANKING (§2/§4/§5):

• Portal floods are NOT the cost (flood = 0.1 ms). H1's "48 floods" emphasis and Tier C flood-caching are DEAD — caching saves ~0.1 ms.
• Punch/seal is NOT the cost (portalmask = 0.08 ms). Tier B #5 batching saves nothing.
• The clip-math alloc storm is NOT the steady cost (flood+assemble ≈ 0.2 ms). It may still drive the spikes via GC (separate), but it is not the 8–13 ms steady cost.
• The cells fix works (shells = 0.1 ms).

REAL TARGET — per-cell / per-entity DRAW SUBMISSION: cellobjects + landscape + dynamics ≈ 6.5 ms + partition ≈ 2 ms. Common thread: WbDrawDispatcher.Draw is called once per visible cell in DrawCellObjectLists (each orphaning 6 SSBOs via glBufferData — WbDrawDispatcher.cs:1521-1558), plus per-cell DrawCellParticles re-walks, plus un-batched per-submesh sky. Scales with visible cells/entities ⇒ the facing-the-town cost. Lesson: allocation COUNT ≠ CPU TIME; the per-phase CPU timer is what found the real hotspot.

Revised fix priority (supersedes §5):

1. Batch per-cell entity/static draws across cells into few WbDrawDispatcher.Draw calls (the cell-shell fix, applied to cell OBJECTS) — targets cellobjects (~3.5 ms). Pre-cull per-entity by viewcone, then one batched draw.
2. WbDrawDispatcher: persistent SSBOs + BufferSubData instead of 6 glBufferData orphans per call — compounds with #1 (helps cellobjects/landscape/dynamics).
3. Consolidate particle Draw calls (per-cell DrawCellParticles re-walks all particles) — targets part of cellobjects + landscape.
4. Batch sky submeshes — targets landscape.
5. Optimize partition (InteriorEntityPartition.Partition + ViewconeCuller.Build, ~2 ms CPU/frame) — possibly cache/incremental.

Spikes (cpuRender max 30–43 ms; correlated gpu-max spikes to 300+ ms) coincide with landscape/cellobjects jumps + GPU upload — likely streaming mesh upload hitches (_wbMeshAdapter.Tick) and/or GC. Diagnose separately from the steady cost.

---

0. Verdict (TL;DR) — original static prediction, now MEASURED-CONFIRMED above

The handoff frames the remaining ~12 ms as "diffuse GPU cost." The static evidence says that frame is mis-attributed: the 12 ms is a glFinish- serialized number, and the actual GPU rasterization is tiny (terrain 0.3, cells 0.2, entities 0.2 ms — the geometry genuinely is cheap, exactly your intuition). The remaining cost is overwhelmingly CPU-submission / OpenGL driver-overhead / GC, not GPU fill:

• 48 uncached portal floods per frame (1 root + ~47 buildings), recomputed from scratch every frame even when standing still, with no caching.
• ~5,000–8,000 short-lived heap allocations per frame (the Sutherland-Hodgman clip math alone is ~3–5 k) → gen-0 GC → the periodic spikes to 30–40 fps.
• Hundreds of redundant GL state calls per frame: 31 punch/seal fans each doing a full state set+restore (~450–650 GL calls), WbDrawDispatcher orphaning 6 SSBOs via glBufferData on every one of its 3–5 calls (~66–110 buffer reallocs), per-submesh sky draws, ~80–128 glEnable/Disable(ClipDistance) toggles.
• The particle system is re-walked ~22× per frame (once per cell), each walk allocating and each Draw doing glBufferData orphans — draw-count bound, not fill (consistent with the resolution-independence observation).

This is the classic OpenGL CPU-bound profile, and it explains all three of your observations at once: resolution-independent, scales with town density, and a fast GPU can't help.

The single blocking gap: we cannot prove CPU-vs-GPU today because the whole- frame TimeElapsed query and the per-pass glFinish are gated on the same ACDREAM_FPS_PROF=1 flag (FrameProfiler.cs:72-73). The decisive next step is to decouple them (a ~5-line apparatus change) or capture a RenderDoc frame — both give the honest split.

---

1. The measurement-trust problem (read this first)

cpuRender is a wall-clock stopwatch around the entire OnRender (FrameProfiler.cs:99,110), so it absorbs every glFinish stall. When ACDREAM_FPS_PROF=1:

• Each instrumented renderer calls gl.Finish() twice (before + after): EnvCellRenderer.cs:845,1050; ParticleRenderer.cs:128,181; PortalDepthMaskRenderer.cs:214,319; terrain hooks GameWindow.cs:10799,10803.
• Counting call frequency: ~62 finishes from punch/seal (31 fans × 2) + ~44 from particles (22 batches × 2) + cells + terrain ≈ 150+ full pipeline flushes per frame.
• Each glFinish drains the GPU and blocks the CPU. The GPU then sits idle between passes waiting for the CPU to issue the next finish-bracketed batch — and that idle time falls inside the whole-frame TimeElapsed window.

Consequence: the gpu ≈ 12 ms total is serialized GPU time (rasterization + inter-pass idle), not pipelined GPU time. The per-pass [PASS-GPU] absolutes (particles 3.1, punchseal 2.9) are inflated by their own finishes and are upper bounds, not real costs. No honest CPU/GPU split exists in the current apparatus. (Independently re-derived by two of the seven readers.)

vsync is off by default (GameWindow.cs:998), so vsync-quantization is unlikely but must be confirmed from the printed vsync= field.

---

2. Ranked hypotheses

H1 (leading) — The frame is CPU-submission / driver-overhead bound, not GPU-fill bound

• For: attributed GPU rasterization is tiny (terrain 0.31, cells 0.23, entities 0.22 ms — all measured in the handoff). Resolution-independent (resize didn't move FPS). Scales with building count. The CPU side has clear O(buildings)
  • O(cells) structure issuing hundreds of small state-changing GL calls. The cells fix worked precisely because it collapsed 94 heavy state-setting Render submissions into 1 — a submission win.
• Against: not yet measured cleanly (glFinish contamination, §1). The resize test was on a tainted FAR_RADIUS=4 build (handoff caveat).
• Falsify: RUN 1 (§6) with glFinish decoupled — if gpu p50 ≈ wall p50 and cpuRender ≪ wall, H1 is wrong and it's genuinely GPU-bound. If cpuRender ≈ wall and gpu ≪ wall, H1 confirmed.

H2 — Frame spikes (cpuRender p95 ~30 ms, dips to 30–40 fps) are gen-0 GC pauses

• For: ~5,000–8,000 short-lived heap objects/frame, almost all in the portal-flood + clip + assemble path (§4). At 75 fps that's 375k–600k allocs/s → frequent gen-0 STW collections. Retail uses fixed static scratch buffers; this alloc storm is "purely a .NET port artifact with no retail equivalent."
• Against: could partly be the streaming/upload hitch (_wbMeshAdapter.Tick, GameWindow.cs:8427) on cell load. Both can be true.
• Falsify: watch the .NET gen-0 GC counter / allocation rate during a stutter, or correlate spikes with dotnet-counters. If allocs/frame is low, GC isn't it.

H3 — Particles are draw-count/state bound (~22 fragmented batches), not fill

• For: ParticleRenderer.Draw is called from ~11 call sites, including per-cell DrawCellParticles (~31×); each call re-walks the entire live particle set via EnumerateLive (ParticleRenderer.cs:196), sorts it, and each batch does a glBufferData orphan (:278). Trivial fragment shader, depth-write off. Resolution-independent ⇒ not fill.
• Against: additive emitters have real overdraw (no depth write); secondary.
• Falsify: RUN 3 resolution sweep — if particle GPU is flat with resolution, not fill. RenderDoc quad-overdraw overlay.

H4 — Punch/seal (31 fans) is CPU state-churn, NOT 2.9 ms of GPU

• For: each DrawDepthFan is a depth-only, color-masked, ≤32-vert fan — GPU cost is negligible. The 2.9 ms is 62 glFinish stalls. The real cost is the full per-fan state set+restore (PortalDepthMaskRenderer.cs:229-313) — ~450–650 GL calls/frame, including a UseProgram/UseProgram(0) round-trip and a uViewProjection re-upload per fan even though all fans share one matrix.
• Against: none material.
• Falsify: RUN 2 [PASS-GPU] with per-pass TimeElapsed (no glFinish) — fan GPU will read ≪ 2.9 ms. RenderDoc per-draw time.

H5 — WbDrawDispatcher SSBO orphaning is a hidden chunk of the "unattributed 5.5 ms"

• For: every Draw() re-uploads 6 SSBOs via glBufferData(DynamicDraw) (orphan+realloc, not BufferSubData) — WbDrawDispatcher.cs:1521-1558 — ×3–5 calls/frame = ~66–110 buffer reallocs. Plus per-transparent-cell EnvCell.Render repeats all 6 SSBO uploads for a 1-cell instance set (EnvCellRenderer.cs:1281-1367).
• Against: the dispatcher has its own ACDREAM_WB_DIAG GPU timer; cross-check it (run separately — it nests illegally with FPS_PROF, FrameProfiler.cs:31).
• Falsify: RenderDoc; or ACDREAM_WB_DIAG=1 in isolation.

---

3. What's ruled out (do not re-chase)

• Distance-degrade / LOD / triangle count — dead (handoff §3, entity GPU 0.22 ms).
• MSAA / fill / overdraw of opaque geometry — resolution-independent; re-verify cleanly in RUN 3 but the static evidence agrees (cheap fragment work everywhere).
• Update thread — update = 0.1 ms.
• Terrain per-slice redraw — the apparatus reader flagged _terrain.Draw being inside a per-slice loop (GameWindow.cs:10795-10803) as a HIGH suspect, but the handoff already measured terrain at 1 slice / 0.31 ms. At outdoor Arwic there is one full-screen outside slice, so terrain draws once. The per-slice multiplier only bites for interior roots with multiple doorway slices — a separate, non-Arwic concern. NOT an Arwic FPS lever.
• The cells fix — intact and correct (RetailPViewRenderer.cs:664-701). Do not touch.

---

4. Per-subsystem cost ledger (static, cited)

Portal visibility (CPU + GC — the biggest structural cost)

• 48 BFS floods/frame, no frame-to-frame caching, recomputed when stationary (RetailPViewRenderer.cs:64-83,228-235; rebuilt GameWindow.cs:8752).
• Per flood allocates PortalVisibilityFrame + 2 HashSet + Dictionary + uint[128] before processing a portal (PortalVisibilityBuilder.cs:126-227).
• PortalProjection clip math is the #1 GC source: ~3,000–5,000 short-lived List<Vector4>/List<Vector2>/ToArray() per frame — ProjectToClip:95, ClipToRegion:127, ClipHomogeneousEdge:216, MergeSubPixelVertices:184. Retail used a static two-buffer swap on a fixed vertex array — so pooling is more retail-faithful, not less.
• CellTodoList.Insert is O(N) (:1006-1016); GetRange allocates (:254,583); CanonicalKey allocates a StringBuilder per polygon (PortalView.cs:255).

ClipFrameAssembler (CPU + GC) — ~800–1,200 heap objects/frame (4 dicts + 95 lists + 94 CellView + 94 int[] + per-polygon Vector4[]), all discarded each frame (ClipFrameAssembler.cs:85-219). No retail counterpart.

Punch/seal (CPU state churn) — 31 fans × full state set+restore = ~450–650 GL calls/frame; per-fan UseProgram round-trip + uViewProjection re-upload (PortalDepthMaskRenderer.cs:229-313). Retail does not restore state per polygon (it leaves state installed for the next poly); our self-contained-state contract is the overhead. The two-pass stencil is an acdream-only #117 addition (retail relied on painter's order). Zero managed alloc inside DrawDepthFan (good — _scratch field + stackalloc).

Particles (CPU draw-count + GC) — ~11 Draw call sites, per-cell DrawCellParticles re-walks all live particles ×~31 (ParticleRenderer.cs:196); new List<ParticleDraw> (:195) + new List<ParticleInstance>(64) (:152) per call; per-batch glBufferData orphan (:278). Draw-count bound.

EnvCellRenderer (CPU + GC + redundant uploads) — opaque batched (the fix), but transparent stays per-cell: each transparent cell repeats all 6 SSBO uploads for a 1-cell set (EnvCellRenderer.cs:1281-1367). _cellLightSetCache.Clear() at :1141 forces 188–282 int[] re-allocations/frame (94 cells × 2–3 passes); the light sets are camera-independent and stable within a frame. Render allocates 4 collections per call (:904,905,927,928). CellHasTransparent is an O(gfx×batch) walk with no cached result.

Orchestration (CPU driver overhead) — sky drawn per-submesh, no batching (~8–15 DrawElements + per-submesh BlendFunc/4×SetFloat, ×2 passes/frame, SkyRenderer.cs:219-429); ~80–128 glEnable/Disable(ClipDistance0..7) toggles/frame (MaxPlanes=8); PrepareRenderBatches runs Parallel.ForEach on the render thread, blocking OnRender (EnvCellRenderer.cs:642); ParseEnvFloat → Environment.GetEnvironmentVariable ×2/frame in the hot path (GameWindow.cs:8639-8641); new[]{entry} per DrawEntityBucket (~50–100/frame, RetailPViewRenderer.cs:943); new[]{NoClipSlice} per slot-less cell (~125/frame, :918 — trivially a static readonly array).

---

5. Fix leads, ranked by (impact × retail-faithfulness)

Brainstorm before any render change (handoff lesson — rushed render changes were reverted). These are sketches for the implementation session, not commitments.

Tier A — strictly retail-faithful (matches retail's static-scratch model), high impact, low risk:

1. Pool the PortalProjection clip buffers (ArrayPool/double-buffer swap). Kills the single largest GC source (~3–5 k allocs/frame). Output contract unchanged. → directly targets H2 spikes.
2. Pool ClipFrameAssembler + PortalVisibilityBuilder BFS scratch (Reset/ Clear instead of new). ~1 k allocs/frame gone.
3. Move _cellLightSetCache.Clear() out of RenderModernMDIInternal to once- per-frame. Removes 188–282 int[]/frame. Light sets are frame-stable.
4. Static readonly NoClipSlice[] + pool DrawEntityBucket's new[]{entry}.

Tier B — internal GL batching (the cells-fix pattern, no visual change), high impact: 5. Batch punch/seal: set state ONCE before the fan loop, upload uViewProjection ONCE, merge all fan vertices into one VBO + draw with offsets. Drop the per-fan UseProgram(0). (Retail also drew per-poly but with cheap D3D state blocks; our GL program-rebind/uniform-upload per fan is the cost.) 6. Batch transparent EnvCell shells like opaque (sort instances far→near, one Render). Removes N_transparent × 6-SSBO re-uploads. 7. WbDrawDispatcher: persistent SSBOs + BufferSubData instead of glBufferData(DynamicDraw) orphan per call. 8. Consolidate particle Draw to ≤3 calls/frame (one per RenderPass), pre-partition emitters by pass, single BufferSubData per Draw; additive in a separate order-independent group. 9. Batch sky submeshes (shared VBO / sort by blend mode).

Tier C — acceleration that diverges from retail (needs explicit brainstorm + a divergence-register row): 10. Cache per-building portal floods keyed on (buildingId, quantized camera pose). Eliminates ~47 of 48 floods/frame when stationary. But retail recomputes per frame from the BSP walk — this is a memoization acdream adds, not a faithful match. Pure-function caching is defensible, but it changes the "recompute every frame" structure and must be invalidated on camera move + cell load/unload. Lower priority than Tier A (which kills the GC without diverging). 11. Move PrepareRenderBatches off the render thread (double-buffer in OnUpdate). One-frame visibility latency.

---

6. Decisive measurement plan (user-driven; report-only until approved)

Apparatus gap to fix first (one small change, then measure): add a second flag (e.g. ACDREAM_FPS_PROF=2 or ACDREAM_FPS_NOFINISH=1) that keeps the whole-frame TimeElapsed query but disables the per-pass glFinish — and/or convert the per-pass timers to non-nested per-pass TimeElapsed queries (the WbDrawDispatcher ACDREAM_WB_DIAG queries at WbDrawDispatcher.cs:1642,1670 are the working template — no glFinish, no inflation). This is diagnostic-only.

• RUN 1 — honest split (glFinish off, frame query on), Arwic, stand still 10 s. gpu≈wall ⇒ GPU-bound (go to RUN 4). cpuRender≈wall, gpu≪wall ⇒ CPU-bound (H1). present(wait)≈wall ⇒ vsync/swap (check vsync=).
• RUN 2 — per-pass [PASS-GPU] via TimeElapsed (no glFinish). Honest particles / punchseal / cells / terrain ms + calls/frame.
• RUN 3 — resolution sweep (720p → 1080p) on a CLEAN build. GPU scales ~2.25× ⇒ fill-bound; flat ⇒ vertex/CPU-bound. (Re-verifies the tainted resize test.)
• RUN 4 — RenderDoc Arwic frame capture. Exact per-draw GPU, total draw count (>200/frame at 720p = batching problem), quad-overdraw overlay (punch fans over the viewport show as red), per-pass timeline, MSAA-resolve cost.

Confirm/refute: H1 ⇐ RUN 1; H3/H4 ⇐ RUN 2/3/4; H5 ⇐ ACDREAM_WB_DIAG=1 solo + RenderDoc; H2 ⇐ gen-0 GC counter during a stutter.

---

7. What this is NOT

• NOT a GPU fill / shading / MSAA problem — the GPU rasterizes this geometry in a couple of ms; the geometry is cheap (your intuition is correct).
• NOT a terrain per-slice redraw at Arwic — outdoor = 1 slice, terrain draws once (0.31 ms measured). That lead applies only to multi-slice interior roots.
• NOT a single 24 ms lever — it's death by a thousand CPU/driver/GC cuts; the win is the sum of the Tier A+B fixes.
• NOT fixed by the cells batching alone — that solved the one big GPU submission sink; the CPU flood + GC + state churn remained underneath.
• The 12 ms gpu number is NOT trustworthy as "GPU rasterization time" — it's glFinish-serialized. Re-measure honestly before drawing conclusions.