acdream/docs/research/2026-06-23-fps-distance-degrade-handoff.md

Handoff — FPS in dense towns = MISSING DISTANCE-DEGRADE (port it) — 2026-06-23

Branch: claude/thirsty-goldberg-51bb9b (continue in THIS worktree: C:\Users\erikn\source\repos\acdream\.claude\worktrees\thirsty-goldberg-51bb9b). Working tree: clean. Everything below is committed on the branch. References / named-retail / WorldBuilder live in the MAIN repo C:\Users\erikn\source\repos\acdream\ (read via absolute path; they are NOT in the worktree).

Read with fresh eyes. This session SHIPPED a real, verified fix (the _datLock contention that caused the 30↔200 swing) — keep it, do not revisit it. But that fix did NOT solve the user's actual pain: sustained ~30 FPS in dense towns (Fort Tethana), and it changes with view direction. That second symptom was then root-caused, end to end, to a different cause: acdream has no distance-based LOD/degrade — it draws every object in the frustum at full detail at any distance. Retail degrades then hides distant objects. The next session's job: port retail's per-frame distance-degrade. The data is already in our dat tables; only the per-frame distance decision is missing.

---

0. TL;DR

1. DONE + committed (do not touch): the _datLock-contention fix. The streaming worker now pre-reads the terrain-apply's dats into a PhysicsDatBundle so ApplyLoadedTerrainLocked makes zero DatCollection calls and its lock(_datLock) is removed. Measured: apply lock-wait 24 ms median / 88 ms p95 → 0.2 µs. This killed the streaming-induced spikes.
2. STILL OPEN (the real FPS pain): sustained ~30 FPS in dense towns, view-direction- dependent ⇒ GPU/render-bound. Root cause verified: no distance LOD/degrade; every frustum-visible object (statics + the huge volume of procedural scenery) is drawn at full detail regardless of distance. ~1,400 draw calls / ~24,000 instances per frame in dense town.
3. THE FIX (next session): port retail CPhysicsPart::UpdateViewerDistance → GfxObjDegradeInfo::get_degrade — per-frame, per-object: pick a lower-detail degrade slot by camera distance, and hide past max distance. Render-pipeline change (objects bake their close-mesh at spawn today; the draw must start carrying degrade variants + selecting per frame). It's a faithful port, not a redesign — but render changes have been reverted here when rushed, so do it via the grep→pseudocode→port→verify workflow with a fresh head.

---

1. What this session SHIPPED — the _datLock fix (committed; DO NOT REVERT)

Symptom it fixed: the wild 30↔200 swing while moving/streaming/portal-hopping.

Root cause (measured + confirmed by the code's own comment): DatCollection is not thread-safe, so a single global _datLock serializes ALL DatCollection.Get<T>. The streaming worker holds that lock for the ENTIRE per-landblock build (BuildLandblockForStreaming, GameWindow.cs:~5910; comment at :~5885 literally predicted this and named the fix). The update thread's ApplyLoadedTerrain also took _datLock (for its own dat reads), so it BLOCKED on the worker — measured 24 ms median / 88 ms p95 of pure lock-wait. A 43 ms apply = a 23-fps frame; idle = 200 fps. That asymmetry was invisible to the title-bar ms (apply runs in OnUpdate, FPS counter is OnRender).

Fix (design A1, spec + plan committed): the worker (already under _datLock) pre-reads the six dats the apply needs into a PhysicsDatBundle carried on LoadedLandblock. The apply reads from the bundle, makes zero Get<T> calls, and its lock(_datLock) is deleted. Safe because _datLock's only cross-thread job is serializing DatCollection; the apply's other writes are update-thread-only (_physicsEngine, ShadowObjects, renderer, _worldState, _buildingRegistries) or already concurrent-safe (PhysicsDataCache is all ConcurrentDictionary).

Commits (newest→oldest, prior HEAD was 92e95be):

SHA	What
b3925f4	chore(diag): [FRAME-DIAG] StreamingController counters (apparatus)
536f1c0	fix(streaming): drop _datLock from the terrain apply (the fix)
81a5605	refactor(apply): read dats from the bundle, not DatCollection
4a99b55	feat(streaming): worker pre-reads the apply's dats into the bundle
3a0e349	feat(streaming): PhysicsDatBundle on LoadedLandblock
5ab5d39	docs: implementation plan
c715e55	docs: design spec

Verified (empirical, full session town-walk + 4 teleports): lockwait = 0.1 µs median / 0.2–0.3 µs p95 on EVERY line (was 24 ms / 88 ms). apply town p95 37 ms → 11 ms. Build + 1568 Core tests green. The lock fix is correct and real. Keep it. (It only addressed the swing, not the dense-town floor — see §2.)

Files the fix touched:

• src/AcDream.Core/World/PhysicsDatBundle.cs (NEW) — the bundle record; .Empty for far tier. Note: Environment aliased to DatEnvironment (collides with System.Environment).
• src/AcDream.Core/World/LoadedLandblock.cs — added PhysicsDatBundle? PhysicsDats = null (trailing default → back-compatible).
• src/AcDream.App/Rendering/GameWindow.cs — BuildPhysicsDatBundle(landblockId, entities) (mirrors the apply's 6 read sites; called from BuildLandblockForStreamingLocked near return); the 6 apply sites rewritten _dats.Get<X>(id) → datBundle.X[id] via TryGetValue; the lock(_datLock) removed from ApplyLoadedTerrain.
• src/AcDream.App/Streaming/StreamingController.cs — only diag counters (apparatus).
• Spec: docs/superpowers/specs/2026-06-23-datlock-contention-fix-design.md.
• Plan: docs/superpowers/plans/2026-06-23-datlock-contention-fix.md (Task 5 — apparatus strip + spec "verified" mark — is NOT done; see §4).

---

2. STILL OPEN — the REAL FPS pain (why Fort Tethana is still ~30 FPS)

User's report (the axiom): "When I portaled in it was high, as soon as I enter the town it drops, also depending on what direction I look." → portal-in over empty terrain = high; turn toward the dense town = drop.

Diagnosis (locked): FPS that changes with view direction can only be the cost of drawing what's in the frustum — i.e. GPU/render-bound. This is NOT the apply, the streaming, or the lock (those are view-independent). The lock fix cannot touch it.

Root cause (VERIFIED against source): there is no distance-based LOD or degrade on entities or scenery. The draw path culls by frustum-AABB only, then draws everything in view at full detail:

• WbDrawDispatcher.WalkEntitiesInto (src/AcDream.App/Rendering/Wb/WbDrawDispatcher.cs:~744-747): per-entity cull is FrustumCuller.IsAabbVisible(...) — no distance check.
• :~779-780: for any in-frustum entity, every MeshRef is emitted as an instance, every frame, regardless of distance.
• src/AcDream.Core/Meshing/GfxObjDegradeResolver.cs resolves each object to its CLOSE-detail mesh (Degrades[0]) and its own comment (~line 57-59) says: "We don't yet have distance-based LOD plumbing, so this resolver always returns slot 0." We ported the degrade DATA, not the per-frame distance DECISION. (It already uses Degrades[i].MaxDist in IsRuntimeHiddenMarker for the #136 "red cone" — so the per-slot distances ARE available to us.)
• Scenery (trees/bushes/rocks) is procedurally generated per landblock (SceneryGenerator), is the BULK of dense-town entity count, and goes through the same no-distance path.
• (Co-factors, lower priority: MSAA 4× on the High preset amplifies fill-rate; 2-pass translucency for foliage. Don't chase these first — the missing distance-degrade is the structural cause.)

Measurement caveat carried forward: the existing [WB-DIAG] gpu_us timer brackets ONLY the entity MDI dispatch (opaque+transparent) — it does NOT measure total GPU frame time (terrain shading, sky, particles, MSAA resolve, fill-rate stalls). So "gpu_us≈400µs" is NOT the frame cost. The next session needs a real total-frame-time / GPU-frame-time meter (swap-to-swap delta, or a glFinish- bracketed full-frame timer) to (a) confirm GPU-bound and (b) prove the fix. Do not trust the existing gpu_us as the frame budget.

---

3. THE FIX (next session) — port retail's per-frame distance-degrade

Retail mechanism (named-retail decomp; addresses verified this session):

• CPhysicsObj::UpdateViewerDistance (0x0050f340 / 0x00510b30) → CPartArray::UpdateViewerDistance → per-part CPhysicsPart::UpdateViewerDistance (0x0050e030, named-retail line ~275517). Per frame, per part: compute distance to viewer, call get_degrade(distance / gfxobj_scale.z, ...).
• GfxObjDegradeInfo::get_degrade (0x0051e4b0, named-retail line ~293086). Algorithm:
  • if degrades_disabled → slot 0.
  • Render::force_level → debug override (skip).
  • effective_dist = |distance| − Render::s_rDegradeDistance (the user's "Degrade Distance" setting, a 0–100 slider; Render_DegradeDistance, named-retail line ~3270-3272).
  • bias = Render::auto_update_deg_mul ? Render::deg_mul : Render::s_rUserSuppliedDegradeBias.
  • walk degrade slots; each slot has ideal_dist and max_dist; threshold per slot = ideal_dist − (ideal_dist − max_dist) * bias. Pick the slot whose band contains effective_dist; past the last band, use the last slot (whose gfxobj_id may be 0 = hidden).
• GfxObjDegradeInfo::get_max_degrade_distance (0x0051e2d0, line ~292918) — the hide-past-this distance.
• CPhysicsPart::Draw (0x0050d7a0) draws gfxobj[deg_level].
• READ THESE FULLY before porting (this handoff only summarizes): get_degrade body at named-retail acclient_2013_pseudo_c.txt:293086+, UpdateViewerDistance at :275517+. Cross-check the degrade-slot struct fields (ideal_dist, max_dist, degrade_mode, gfxobj_id) against docs/research/named-retail/acclient.h and acdream's GfxObjDegradeInfo / GfxObjInfo dat structs (we already expose Degrades[i].Id + Degrades[i].MaxDist; confirm whether ideal_dist is also exposed or must be added).

The render-pipeline change (the hard part — design it first): today an object resolves to Degrades[0] at SPAWN and bakes MeshRefs. Distance-degrade needs the draw to select the slot per frame by camera distance. Options to brainstorm:

• (a) carry ALL degrade-slot GfxObj variants per part on the entity/MeshRef, and have WbDrawDispatcher pick the slot per-frame (true retail; most work; per-frame get_degrade per part).
• (b) hide-only first cut: keep slot-0 mesh, but in WalkEntitiesInto skip emitting instances for entities beyond their get_max_degrade_distance (scaled by the DegradeDistance setting). This is the big, low-risk win (distant scenery/buildings stop drawing) WITHOUT per-frame mesh-swapping. Plumb a per-entity max-draw-distance (read from the degrade table at spawn) + a per-entity camera-distance check in the walk. Recommended phase 1.
• (c) full LOD-mesh selection (slot swap at intermediate distances) as phase 2 on top of (b).

Open design questions for the brainstorm:

• Where to evaluate distance: per-entity (cheap, approximate) vs per-part (retail-exact)? Start per-entity.
• The Render_DegradeDistance setting: pick the retail default; wire through QualitySettings / RuntimeOptions (per CLAUDE.md rule 4). Don't hardcode a magic distance — derive from the degrade table + the setting.
• Hysteresis/popping: retail's bands prevent thrash; a naive per-frame threshold can pop. Use the ideal/max band, not a single cutoff.
• Animated entities currently BYPASS the per-entity cull (they're landblock-tracked). Decide whether they also distance-degrade (retail does, but be careful with the player + nearby NPCs — they should stay slot 0 / never hide).
• Divergence register: the "no distance-degrade" deviation is NOT yet a row in docs/architecture/retail-divergence-register.md. Add the row when the deviation is formally acknowledged, and DELETE it in the commit that lands the port (per the register rules).

Acceptance: user confirms Fort Tethana / Holtburg FPS no longer drops to 30 when facing the dense town. Prove it with the new total-frame-time meter (before/after) AND user eyes. NO "fixed" claim on a sub-metric again (see §6).

---

4. Apparatus — [FRAME-DIAG] (committed; needs strip; build a NEW meter for the next phase)

• [FRAME-DIAG] is committed across GameWindow.cs (fields, OnUpdate flush, the apply timing + cell/bsp/shadow/upload/lockwait sub-spans, the MaybeFlushTerrainDiag print) and StreamingController.cs (the counters). Gated on ACDREAM_WB_DIAG=1, flushes every ~5 s on outdoor frames. Line format: [FRAME-DIAG] apply_us=Xm/Yp95 lockwait=… [cell=… bsp=… shadow=…] (upl=…) entUpl_us=… applies_max/upd=N deferred=N forceReload=N(drop=N) esg=N spawn=N resident=N walked=N
• It measures the APPLY (update) side — now PROVEN fixed (lockwait→0). It does NOT measure the total render frame time, which is what the distance-degrade work needs. Next phase: add a total-frame-time + GPU-frame-time meter (the missing measurement; §2 caveat).
• STRIP all of [FRAME-DIAG] + any new meter when the FPS work fully lands (mirrors 92e95be). Plan Task 5 (apparatus strip + spec "verified" mark) was deliberately NOT done so the before/after stays measurable.

---

5. Verified file:line map (carry forward; lines drift — match by symbol)

Thing	Location
Worker holds _datLock for full build (the old contention)	GameWindow.cs:~5910 BuildLandblockForStreaming; comment :~5885
Apply wrapper (lock now REMOVED)	GameWindow.cs:~6608 ApplyLoadedTerrain
Apply body (now dat-free; reads datBundle)	GameWindow.cs:~6786 ApplyLoadedTerrainLocked
Bundle gather (worker)	GameWindow.cs BuildPhysicsDatBundle
No-distance-cull (the open bug)	WbDrawDispatcher.cs:~744-747 (frustum only), :~779-780 (emit all meshrefs)
Degrade resolver stub ("always slot 0")	GfxObjDegradeResolver.cs:~57-59; IsRuntimeHiddenMarker uses MaxDist
Entity store (drawn set)	GpuWorldState.cs _flatEntities / LandblockEntries / Entities
[WB-DIAG] entity-draw CPU+GPU timer (partial — entity MDI only)	WbDrawDispatcher.cs:~301-329
Retail get_degrade	named-retail acclient_2013_pseudo_c.txt:293086 (0x0051e4b0)
Retail per-part UpdateViewerDistance	named-retail :275517 (0x0050e030)
Retail get_max_degrade_distance	named-retail :292918 (0x0051e2d0)
Retail Render_DegradeDistance setting (0–100)	named-retail :3270-3272, :169441
WorldBuilder reference (no distance LOD either — we must add it)	references/WorldBuilder/.../SceneryRenderManager.cs, VisibilityManager.cs (MAIN repo)

---

6. Repro + capture

Launch (Release; user drives + watches FPS; user manages client lifecycle):

$env:ACDREAM_DAT_DIR="$env:USERPROFILE\Documents\Asheron's Call"; $env:ACDREAM_LIVE="1"
$env:ACDREAM_TEST_HOST="127.0.0.1"; $env:ACDREAM_TEST_PORT="9000"
$env:ACDREAM_TEST_USER="notan"; $env:ACDREAM_TEST_PASS="MittSnus81!"
$env:ACDREAM_WB_DIAG="1"   # turn OFF for the clean FPS number
dotnet run --project src\AcDream.App\AcDream.App.csproj --no-build -c Release 2>&1 | Tee-Object -FilePath "fps.log"

Character +Je (notan / MittSnus81!). Repro: portal in (high FPS) → enter the dense town → look toward the town vs away (FPS drops facing the dense geometry). Fort Tethana / Holtburg are the test beds. Build Release for any FPS read (Debug is not representative). The old probe logs from this session are in the worktree root (frame-diag*.log, fps.log — untracked; ignore/clean).

---

7. DO-NOT-RETRY / lessons (read before you act)

• "lock-wait = 0" is NOT "FPS fixed." This session declared the FPS fixed on the lockwait sub-metric and was wrong — the user was still at 30 in Fort Tethana. Never claim FPS fixed without measuring the TOTAL frame time the user actually sees, plus user-eyes confirmation.
• Verify subagent claims against source. Two Explore agents this session drifted line numbers and one fabricated a "rehydrate replays the player" race that source refuted (SelectGuidsToRehydrate excludes the player). Read the cited code yourself before designing on it.
• Render changes get reverted when rushed/speculative (AP-48 cull deleted _lastSpawnByGuid → missing walls; the sky-fog shader edits). The distance-degrade port is retail-faithful, but do it via grep→pseudocode→port→verify, brainstorm the draw-path approach first, and measure before/after.
• The _datLock fix is correct — do not re-investigate or revert it. lockwait 88ms→0 is proven.
• The existing [WB-DIAG] gpu_us under-reports (entity MDI only) — don't use it as the frame budget.
• DatCollection is not thread-safe; only the worker may Get<T> off the update thread (under _datLock). The apply is now dat-free — keep it that way (the bundle is how it gets dats).

---

8. Other OPEN follow-ups (deferred this session — lower priority than §3)

• H3 — unbounded entity leak / GPU heat-up over hops. _entitiesByServerGuid + _lastSpawnByGuid never evict across teleports (esg/spawn climbed 1→528 over hops; ACE doesn't DeleteObject across teleport). Inflates the in-frustum drawn set, so it COMPOUNDS the §3 cost. Fix = the SAFE render-only eviction (drop far entities' render/GPU state, KEEP _lastSpawnByGuid, re-materialize on re-entry) — NOT the naive AP-48 cull. Pruner: RemoveLiveEntityByServerGuid (GameWindow.cs:~3831, only caller = DeleteObject handler + respawn de-dup). Distance-degrade (§3) may make this less urgent.
• Invisible char on outdoor portal (intermittent). VERIFIED mechanism: ForceReloadWindow (sky-arcs fix 9945d46) tears down the whole window on every outdoor teleport; the player is persistent-rescued (GpuWorldState.RemoveLandblock:~298) → re-injected via DrainRescued/AppendLiveEntity, but lands in _pendingByLandblock (not drawn) until its landblock re-loads. The fade can lift before the player is flushed into the drawn set → invisible. (The Explore agent's "rehydrate replays player" theory was WRONG — player is excluded from rehydrate at LandblockEntityRehydrator.cs:79.) Likely fix: gate the teleport fade-lift on "player is in the drawn set," not just terrain residency. Needs a player-draw- membership probe to pin "fade-too-early" vs "stuck-in-pending."

---

9. References (read FIRST per the relevant thread)

• docs/superpowers/specs/2026-06-23-datlock-contention-fix-design.md — the shipped fix's design.
• docs/research/2026-06-23-fps-gpu-churn-handoff.md (MAIN repo) — the handoff that STARTED this session.
• docs/research/2026-06-22-world-load-fps-rootcause-report.md (MAIN repo) — prior FPS deep-dive.
• named-retail docs/research/named-retail/acclient_2013_pseudo_c.txt — get_degrade (:293086), UpdateViewerDistance (:275517). acclient.h for the degrade structs.
• references/WorldBuilder/ (MAIN repo) — scenery/object render path acdream extracted (no distance LOD; read docs/architecture/worldbuilder-inventory.md first).
• claude-memory/project_render_pipeline_digest.md + project_physics_collision_digest.md (DO-NOT-RETRY tables).
• CLAUDE.md "Development workflow: grep named → decompile → verify → port" — MANDATORY for the distance-degrade port.