acdream/docs/research/2026-06-23-dense-town-fps-deepdive-handoff.md

# Handoff — dense-town FPS deep-dive (push past 75 fps) — 2026-06-23

**Branch:** `claude/thirsty-goldberg-51bb9b` (continue in THIS worktree:
`C:\Users\erikn\source\repos\acdream\.claude\worktrees\thirsty-goldberg-51bb9b`).
**References / named-retail / WorldBuilder** live in the MAIN repo
`C:\Users\erikn\source\repos\acdream\` (read via absolute path; NOT in the worktree).

> **Read with fresh eyes.** This session **shipped a real, verified fix** (EnvCell
> shell-draw batching: dense-town Arwic **29 → 75 fps, 2.6×**) — KEEP IT. But the
> user finds 75 fps in a dense town on modern hardware **unacceptable** ("a modern
> GPU + OpenGL should chew through this") and wants a **deep-dive to find the next
> root cause and push FPS much higher.** This is not "ship it"; this is "keep
> hunting." The remaining cost is **diffuse** (no single big lever left) — so the
> deep-dive needs a *better* instrument than this session's glFinish hack. **Strong
> recommendation: capture a frame with RenderDoc** (the right tool for "where does
> the GPU time actually go, per draw").

---

## 0. TL;DR

1. **SHIPPED + committed (do NOT revert):** EnvCell shell batching. Root cause was
   `EnvCellRenderer.Render` called **~94×/frame** (once per visible/look-in cell ×
   opaque+transparent) = **24.75 ms = 75 % of the GPU frame** at Arwic. Fixed by
   batching the opaque pass into ONE `Render(Opaque, allCells)` + skipping the
   transparent `Render` for opaque-only cells. cells: **94 calls/24.75 ms → 1 call/
   0.37 ms**. Frame **34 ms/29 fps → ~13 ms/75 fps**. Visually verified (no missing
   walls, transparency + look-ins correct).
2. **STILL OPEN (the user's ask):** 75 fps is unacceptable. The remaining ~12–13 ms
   is **diffuse** — particles ~3 ms, aperture punch/seal ~3 ms, ~5.5 ms scattered
   (sky/entities/clear/weather), plus periodic frame **spikes** to 30–40 ms. No
   second 24 ms sink. **Deep-dive to find why a modern GPU needs ~12 ms for a dense
   AC town at 720p, and push FPS much higher.**
3. **The handoff's original theory (distance-degrade / LOD) is DEAD** — refuted by
   measurement (entity-draw GPU is 0.22 ms; it was never triangle/detail-bound).
   Do not re-pursue distance-degrade.

---

## 1. What SHIPPED this session (commits on the branch, newest→oldest)

| SHA | What |
|---|---|
| `376f4e6` | chore(diag): particle + punch/seal glFinish timers [throwaway] |
| `8067d3b` | **perf(pview): batch EnvCell look-in shell opaque pass** (interior roots) |
| `3af7d00` | **perf(pview): batch EnvCell shell opaque + skip empty transparent** (the Arwic win) |
| `f72f7ce` | feat(envcell): `CellHasTransparent` predicate (part of the fix, NOT apparatus) |
| `e27923b` | chore(diag): FPS profiling apparatus [throwaway] |
| `fd9354f` | docs: implementation plan |
| `fcf60d8` | docs: design spec |

(Prior session, KEEP: `536f1c0` `_datLock` fix.)

**The fix (KEEP):** `RetailPViewRenderer.DrawEnvCellShells` + `DrawBuildingLookIns`
no longer call `EnvCellRenderer.Render` per-cell. Opaque shells batch into one
`Render(Opaque, allCells)` (z-buffer order; lighting is per-instance, `CellId`-keyed
SSBO at `EnvCellRenderer.cs` `RenderModernMDIInternal` ~`:1324` → safe to batch).
Transparent stays per-cell far→near but skips opaque-only cells via
`EnvCellRenderer.CellHasTransparent`. Spec/plan: `docs/superpowers/specs|plans/
2026-06-23-envcell-shell-batching-*.md`. **Plan Task 4 (final verify) + Task 5
(strip apparatus) are NOT done** — apparatus kept for the deep-dive.

---

## 2. The apparatus (`ACDREAM_FPS_PROF=1`) — committed, throwaway, STRIP at the end

- **`src/AcDream.App/Rendering/FrameProfiler.cs`** — per-frame split, printed `[FPS-PROF]`
  every ~1 s: `wall` (real frame period, p50/p95/min/max), `update` (OnUpdate CPU),
  `cpuRender` (OnRender CPU), `present(wait)` (= wall − update − cpuRender), `gpu`
  (whole-frame `TimeElapsed` query — **the reliable total**). Plus `[PASS-GPU]` —
  per-renderer **glFinish-bracketed** GPU via `AddRendererGpu(name, ms)`: `cells`,
  `terrain`, `particles`, `punchseal`, each with `ms/frame` + `calls/frame`.
- **Hooks:** `GameWindow.OnRender` BeginFrame/EndFrame + `OnUpdate` `MarkUpdateStart`
  + terrain glFinish (in `DrawRetailPViewLandscapeSlice`); `EnvCellRenderer.Render`
  glFinish try/finally; `ParticleRenderer.Draw` glFinish; `PortalDepthMaskRenderer.
  DrawDepthFan` glFinish. All gated on `FrameProfiler.PassGpuEnabled`.
- **`tests/AcDream.Core.Tests/Conformance/DegradeCoverageProbeTests.cs`** — throwaway
  dat probe (degrade-coverage; was for the dead distance-degrade theory).
- **⚠ glFinish CAVEAT:** the per-renderer glFinish **serializes CPU↔GPU and inflates**
  the per-pass absolutes (and folds GPU wait into `cpuRender`). Use `[PASS-GPU]` for
  **relative** attribution only; trust the whole-frame `gpu` for the total. For
  accurate per-draw GPU, use **RenderDoc** (see §5).
- **STRIP everything** (plan Task 5) when the FPS work fully lands: `FrameProfiler.cs`,
  the GameWindow hooks/fields (`_fpsProf`/`_frameProfiler`/`_msaaSamples`), the terrain
  glFinish, the EnvCell/Particle/PortalDepthMask glFinish brackets (KEEP
  `CellHasTransparent` — it's the fix), `DegradeCoverageProbeTests.cs`, and the
  worktree-root `fps-*.log` / `wbdiag.log` / `degrade-probe.log`.

---

## 3. The investigation — what's RULED OUT (do not re-investigate)

Measured live in Arwic (the dense-town test bed). Frame was 29 fps / 34 ms pre-fix.

| Suspect | Verdict + evidence |
|---|---|
| **Distance-degrade / LOD / triangle count** (the prior handoff's theory) | ❌ DEAD — entity-draw GPU is **0.22 ms**; never detail-bound. (Degrade-coverage data: 92–96 % of scenery HAS degrade tables, but trees hide at 200 m+ and degrade only saves triangles, which aren't the cost.) |
| **MSAA / fill / overdraw / shading** | ❌ **Resolution-independent** — resizing the window tiny did NOT change FPS; MSAA-off (`ACDREAM_MSAA_SAMPLES=0`) didn't help. ⚠ the resize test was run on a `FAR_RADIUS=4` build that had side-effects — **RE-VERIFY resolution-dependence on a CLEAN build in the deep-dive** before fully trusting "not fill". |
| **Update thread** (physics/anim/net/apply) | ❌ `update = 0.1 ms`. |
| **Far terrain draw distance** | ❌ `ACDREAM_FAR_RADIUS=4` didn't help (and slowed things — streaming side-effects; that build's numbers are suspect). |
| **Terrain draw** | ❌ 1 slice / 0.3 ms. |
| **Entity leak** | ❌ modest (`esg≈288`), entity GPU 0.22 ms regardless. |
| **EnvCell per-cell Render (94×)** | ✅ WAS the cause — **FIXED this session.** |

---

## 4. The diffuse remainder — the deep-dive target

Arwic, **after** the cells fix, whole-frame `gpu ≈ 12 ms` (profiler on; ~13 ms /
75 fps profiler-off). glFinish-inflated per-pass (upper bounds):

| Pass | GPU (inflated) | calls/frame | Notes |
|---|---|---|---|
| cells | 0.23 ms | 1 | ✅ fixed |
| terrain | 0.31 ms | 1 | cheap |
| **particles** | ~3.1 ms | **22 batches** | additive blend; overdraw? |
| **punch/seal** (`DrawDepthFan`) | ~2.9 ms | **31 fans** | per-fan full GL-state setup (smells like the cells issue) |
| **UNATTRIBUTED** | **~5.5 ms** | — | sky, entities (0.22), depth clear, weather, MDI-dispatch overhead, + glFinish inflation |

Plus **frame SPIKES** (separate from the steady cost): `cpuRender` p95 ~30 ms,
`present(wait)` p95 ~18 ms → periodic dips to "30 low" fps (stutter). Likely GC or a
periodic streaming/upload hitch. May matter more for *perceived* smoothness.

**Open questions for the deep-dive (ranked):**
1. **Attribute the ~5.5 ms unattributed GPU.** Instrument sky / weather / the depth
   clear / the entity MDI, OR (better) RenderDoc. Is one of them secretly big?
2. **Separate CPU from GPU cleanly.** The glFinish folds GPU into `cpuRender`. Run
   with the per-renderer glFinish **disabled** (whole-frame `gpu` query only) to get
   the honest `cpuRender` vs `gpu` vs `present` split. If `cpuRender` is high, suspect
   the **CPU side**: `PortalVisibilityBuilder` flooding ~47 nearby buildings in
   `MergeNearbyBuildingFloods` (the per-frame flood/walk for a dense town could be
   real CPU). That was never isolated.
3. **particles (~3 ms, 22 batches):** additive overdraw vs draw-count. Reconcile with
   "resolution-independent" (re-verify the resize test cleanly). If overdraw, reducing
   particle fill / count helps; if draw-count, batch the 22.
4. **punch/seal (31 fans):** are we punching/sealing too many apertures for an outdoor
   root? Each `DrawDepthFan` does a full GL-state set/restore (like the per-cell
   Render that we just fixed) — likely **batchable** (set state once, draw N fans).
   Check what retail does (decomp: `DrawPortalPolyInternal`, the seal/punch order).
5. **Frame spikes:** GC (allocations per frame — the pview path allocates
   `new CellView()`/`new Vector4[]` per aperture in `DrawBuildingLookIns`, `new[]`
   tuples per slice) or streaming. A GC/alloc audit of the per-frame render path is
   worthwhile.
6. **The meta-question:** even fully attributed, why ~12 ms GPU for this geometry on
   modern HW? Look for state thrashing / many small draws / the 2-pass translucency /
   redundant per-frame uploads. The user's "something is structurally off" thesis
   still stands for the remaining 12 ms.

---

## 5. RECOMMENDED next-session method

- **RenderDoc frame capture in Arwic** is the right tool (CLAUDE.md: rendering perf →
  RenderDoc). It gives exact per-draw GPU time, draw count, overdraw, and state
  changes WITHOUT the glFinish distortion. This will attribute the 12 ms far better
  than the apparatus. The user is on Windows with a modern GPU. Start here.
- Keep the `[FPS-PROF]`/`[PASS-GPU]` apparatus as a cheap secondary check, but trust
  RenderDoc for absolutes. Consider switching the per-pass timers from glFinish to
  **non-nested per-pass `TimeElapsed` queries** (no glFinish → no inflation; sequential
  passes don't nest) if you want accurate in-engine per-pass numbers without RenderDoc.
- Measurement discipline (this session's lessons): profiler-OFF for the true FPS;
  constant-view (stand still) for stable per-pass reads; re-verify the resize/
  resolution test on a CLEAN build; the user drives the client lifecycle (graceful
  `CloseMainWindow` before any rebuild — the DLL is locked while it runs).

---

## 6. Repro / launch

User: `notan` / `MittSnus81!`. Test bed: **Arwic** surface town (the user can get
there; dense buildings → the cost). Build Release; the user runs + watches FPS.

```powershell
$env:ACDREAM_DAT_DIR="$env:USERPROFILE\Documents\Asheron's Call"; $env:ACDREAM_LIVE="1"
$env:ACDREAM_TEST_HOST="127.0.0.1"; $env:ACDREAM_TEST_PORT="9000"
$env:ACDREAM_TEST_USER="notan"; $env:ACDREAM_TEST_PASS="MittSnus81!"
$env:ACDREAM_FPS_PROF="1"   # [FPS-PROF]+[PASS-GPU]; OMIT for the true FPS the user sees
dotnet run --project src\AcDream.App\AcDream.App.csproj --no-build -c Release 2>&1 | Tee-Object -FilePath "fps.log"
```
Title bar shows `fps | ms` (vsync off by default). `[FPS-PROF]`/`[PASS-GPU]` flush
~1 s. Logs are UTF-16 (Tee-Object) — read via PowerShell `Get-Content | Select-String`.

---

## 7. Do-not-retry / lessons
- The cells fix is correct + verified — do NOT revert. Both phases shipped.
- Distance-degrade is NOT the cause — dead lead, don't re-pursue.
- glFinish per-call inflates per-pass absolutes — relative only; whole-frame `gpu`
  query is the reliable total; RenderDoc for true per-draw.
- The `FAR_RADIUS=4` build had streaming side-effects — its numbers (incl. the resize
  test) are suspect; re-verify on a clean build.
- Don't declare FPS "fixed" on a sub-metric — only on the total frame time the user
  sees + user eyes (the prior session's mistake; honored this session).