From 38e14c6a71cfc7bf7b1bb2b8602eb270ddcc097b Mon Sep 17 00:00:00 2001 From: Erik Date: Fri, 24 Jul 2026 11:38:20 +0200 Subject: [PATCH] docs(perf): audit errata + plan governance/contract amendments (2026-07-24 review) Adversarial review of the 2026-07-24 runtime audit + modern runtime plan (43-agent verification at e627d9a4). Audit: append errata section 13 -- the capture ran with ACDREAM_UNCAPPED_RENDER=1 + ACDREAM_DUMP_MOVE_TRUTH=1 (not "normal capped presentation"; the 32 FPS ceiling was the RDP display path); the 203.6 MiB single-frame figure is frame-thread-only and cannot include worker-side MeshExtractor decode (it is main-thread publication/ retirement); GiB conversions, committed-peak, 65% LOH fragmentation, exception-count provenance, GC-counter semantics, and cache-undersizing corrections. Original sections left untouched per historical-record rules. Plan: new section 0 governance interlocks (Slices F/G resume the deferred 2026-07-05 MP3/ECS decision -> explicit user gate; side-track session discipline; freeze-exception scope for H-network/J; evidence-base corrections; baseline prerequisite blocking Slice C). Slice amendments: A (per-frame history, process-wide alloc, contention stacks, cache counters, screenshot-diff pulled forward, reference config, exit criteria), C (Portal-dat-wins precedence contract, dual-dat + ObjDesc fixtures, cache identity rule, exception-semantics cleanup), D (unify/retune existing eviction rather than rebuild; ResidencyManager concurrency contract), E (gate rescoped to what E owns; release-later != run-later; audio in the immediate detach set; reveal stall behavior with the AP-115 cue; retail teardown anchors), F (referee-first digest, journal coalescing bound, IRenderScene home project, identity enforcement), G (provisional 300 FPS target; retail dynamics viewcone-cull contract), H (split H-a/H-b/H-c with the AP-85 lights pin and receive-side allocation scope), I (verbatim-float bake rule + differential BSP sweep; floats after B), J (teardown protocol, multi-plan track), K (wire-timing parity per TS-33, hours-scale soak). Section 11 gate corrections (percentile populations, anti-gaming process-wide row, provisional budgets, scoped third-visit rule, soak + jitter rows, scheduled harness); section 12 adjudication protocol; section 14 order list now includes Slice I and the user decision gate. Co-Authored-By: Claude Fable 5 --- .../2026-07-24-modern-runtime-architecture.md | 373 +++++++++++++++++- .../2026-07-24-runtime-performance-audit.md | 95 +++++ 2 files changed, 453 insertions(+), 15 deletions(-) diff --git a/docs/plans/2026-07-24-modern-runtime-architecture.md b/docs/plans/2026-07-24-modern-runtime-architecture.md index 76e6e429..56140d93 100644 --- a/docs/plans/2026-07-24-modern-runtime-architecture.md +++ b/docs/plans/2026-07-24-modern-runtime-architecture.md @@ -19,6 +19,51 @@ visibility ports, and all behavior already accepted by the user. --- +## 0. Governance and standing-decision interlocks (2026-07-24 review) + +Added by the 2026-07-24 adversarial review. These interlocks are binding on +every executor of this plan; the audit's correction record is +[`../research/2026-07-24-runtime-performance-audit.md`](../research/2026-07-24-runtime-performance-audit.md) +section 13. + +1. **The ECS deferral stands until the user lifts it.** On 2026-07-05 the + user deferred the ECS/MP3 render-world rewrite ("revisit ONLY if the user + wants a higher raw FPS number and accepts the rewrite risk" — + `claude-memory/project_mp_track_findings.md`), and the follow-up + measurement note concluded MP3 was not warranted for town FPS. Slices A–E + do not touch that deferral. **Slices F and G resume the deferred MP3 + work and MUST NOT start without explicit user approval recorded against + this section**, taken on the post-Slice-E corrected uncapped-local + profile (does partition/submission cost still dominate?). Note the prior + "GPU-bound at ~200 FPS" Sawato reading is itself invalidated by the + `GpuFrameTimer` boundary artifact this plan's Slice A corrects — the + question is genuinely open until re-measured. +2. **Session discipline.** This plan executes in dedicated Track MP/LH + side-track sessions only. The ACTIVE milestone's critical path wins every + conflict; do not begin any slice inside a milestone session. +3. **Freeze-exception scope.** The recorded 2026-07-05 Track MP exception + covers streaming and WB rendering only. Slice H's network-I/O item and + Slice J's gameplay-owner moves touch otherwise-frozen subsystems + (network, chat, input) and each requires its own recorded user + authorization before starting. (The 2026-07-24 receive-thread robustness + fix was separately reviewed and user-approved as a bug fix, not a + rework.) +4. **Evidence-base corrections.** The §1 exception-traffic row previously + cited "4,520 + 1,168" — figures with no derivation in the audit or its + committed artifacts; the verifiable figures are the focused-trace 1,587 + (1,335 invalid DAT probes + 252 receive timeouts). And the audit's + 204 MiB single-frame allocation is a FRAME-THREAD measurement that cannot + include worker-side mesh decode: **Slice E owns the frame-spike symptom; + Slices B/C own the process-wide allocation rate and GC pressure.** Win + attribution must respect that split. +5. **Baseline prerequisite.** No behavior-changing slice (C onward) starts + until the Slice A exit criteria in §10 are met: committed uncapped-local + AND capped-local nine-stop baselines, a pinned dense-town uncapped + baseline, and per-portal allocation/frame timelines, all captured with + the corrected tooling and a committed reference-hardware definition. + +--- + ## 1. Why the architecture should change The complete methodology, attribution, sanitized checkpoint data, and raw- @@ -38,7 +83,7 @@ The 2026-07-24 six-minute, nine-stop connected route established: | Final live GC heap | approximately 194 MiB | The 1–2 GiB process high-water is not an unbounded live-object leak | | Dedicated GPU memory | 633–818 MiB | Texture arrays/caches are bounded but expensive and insufficiently observable | | Revisited locations | entity/emitter/particle counts returned to the same plateaus | Logical lifetime is broadly convergent; optimize rather than replace it | -| Exception traffic | 4,520 invalid DAT probes plus 1,168 receive timeouts | Exceptions are being used as normal control flow | +| Exception traffic | 1,587 focused-trace exceptions (1,335 invalid DAT probes, 252 receive timeouts); full-route totals pending re-derivation (see §0.4) | Exceptions are being used as normal control flow | Allocation attribution names the runtime mesh/texture path (`MeshExtractor.PrepareMeshData` → `PrepareGfxObjMeshData` → @@ -493,6 +538,37 @@ preserving. Each gets a focused implementation plan when it starts. reconcile with active frame time; external GPU engine direction agrees with GL timestamps. +**Amendment (2026-07-24 review).** Slice A additionally includes, and its +gate requires: + +- Per-frame history export (frame index, stage times, frame-thread alloc, + gpu time) — landed post-review via `ACDREAM_FRAME_HISTORY`; route-wide + p50/p95/p99 populations must come from this export. Until it is the gate + population, every percentile gate reads "no 5-second window with p99 > X". +- Process-wide allocation rate per scene (dotnet-counters) alongside the + frame-thread profiler, plus a per-thread split recipe — the frame-thread + gates alone are gameable by moving work to workers (§0.4). +- One contention capture WITH STACKS on the route (the 4,614 contentions are + unattributed; `_datLock` build-vs-hydration and + `ObjectMeshManager._pendingRequests` are the leading candidates, not only + the DAT typed-object lock). +- Cache hit/miss/evict counters in the checkpoint JSON (landed post-review) + so Slice B/C wins are attributable to cold-vs-warm portals. +- LOH size + fragmentation (+POH) checkpoint fields (landed post-review; the + audit's own JSON showed 65% LOH fragmentation, never analyzed). +- Fixed-camera screenshot COMPARISON tooling (pulled forward from Slice G) + with a defined match rule (per-channel tolerance + max differing-pixel + fraction; pinned `/time`, pinned camera, particles pinned-seed or masked). + Slice C's visual gate depends on it existing. +- A committed reference-configuration block (CPU, GPU, driver, OS build, + display path, resolution, refresh, VSync state, re-baseline rule) — the + §11 phrase "reference local hardware" is undefined without it. + +**Exit criterion (blocks Slice C, per §0.5):** committed uncapped-local and +capped-local nine-stop baselines, a pinned dense-town uncapped baseline +(name the town, pin the camera), and per-portal allocation/frame timelines +around each teleport materialization. + ### Slice B — Finish MP1b EnvCell dedup and full bake **Purpose:** Make the existing pak physically usable. @@ -531,6 +607,35 @@ code consumes the pak yet. single-frame allocation and p99 materially improve from the 2026-07-24 baseline; connected nine-stop route and screenshots pass. +**Amendment (2026-07-24 review).** Binding contracts for this slice: + +- **Precedence contract.** The LINQ being replaced is load-bearing content + selection, not just allocation: + `OrderByDescending(r => r.Database == _dats.Portal)` means + **Portal-dat-wins** when an id resolves in more than one database. The + typed non-allocating lookup must encode that precedence explicitly, and + the equivalence fixtures must include at least one dual-dat id (an id + shadowed by `client_highres.dat`) so a precedence regression fails the + gate instead of silently changing pixels. +- **Seam fixtures.** The DAT-equivalence suite must include ObjDesc-composed + fixtures that straddle the bake/runtime split: a drudge, a robed player, + and palette-dyed armor (baked base mesh + runtime appearance overlay). +- **Cache identity rule.** Pak-backed default-palette pixels and + runtime-composed palette-overlaid pixels for the same surface id are + distinct cache identities and must never share a key. +- **Exception semantics.** When the probe storm is removed, also narrow + `ResolveActivation`'s bare `catch { return null; }` (and the sibling + `SequencerFactory`, which today has NO catch at all): catch only the + specific corrupt-data exception types, rethrow `OperationCanceledException`, + log anything unexpected. Otherwise the "no exception storm" gate can pass + while real activation failures stay invisible — the pattern CLAUDE.md's + no-workarounds rule names explicitly. +- **Baseline comparison validity.** "p99 materially improve" compares + against the post-Slice-A uncapped-local baseline, not the capped-RDP + 2026-07-24 capture (whose steady p99 sits at the 31.2 ms display floor). + Portal single-frame allocation is FPS-independent and may compare against + either. + ### Slice D — Typed asset handles and unified residency **Purpose:** Bound and explain CPU/GPU memory. @@ -550,6 +655,31 @@ connected nine-stop route and screenshots pass. **Gate:** same-location third-visit residence plateaus; no stale-generation release; no missing textures; no resource growth after repeated portal loops. +**Amendment (2026-07-24 review).** Two corrections to this slice's scope: + +- **Do not rebuild what exists.** Fence-gated, budgeted eviction already + runs per frame in three of the four systems this slice names: + `ObjectMeshManager` (1 GiB / 50-object LRU, reclaimed every frame), + `CompositeTextureArrayCache` (128/64 MiB budgets, per-frame throttled GL + deletion), and `GlobalMeshBuffer` (capacity shrink with hysteresis). The + 2026-07-24 route never pushed those ceilings, so its "bounded" residency + was bounded-by-route, not policy-exercised. Slice D's job is to UNIFY + accounting and observability over the existing physical caches, add a + forced-pressure eviction test (deliberately exceed the ceilings), and + retune the budgets for multi-session load — plus bring policy to the + items that genuinely have none: deferred-alpha scratch arrays, + `ObjectMeshManager._boundsCache`, and the audio/animation content caches + (audio caches bounded post-review). +- **Concurrency contract before code.** The `ResidencyManager` design in + §5.3 lists states with no transitions. Before this slice starts, its plan + must specify: a transition table including failure/cancellation/corrupt + edges; which fields are single-thread-owned vs shared (policy decisions + on the update thread, physical release render-thread-only, accounting via + atomics or a single-writer journal); the rule that eviction never blocks + the render thread and never runs under a lock the render thread takes; + and that `ResidencyManager` never touches GL names (only the renderer + resource owners row of §4 may). + ### Slice E — Cost-budgeted streaming and retirement **Purpose:** Remove update-thread portal transactions. @@ -563,9 +693,44 @@ release; no missing textures; no resource growth after repeated portal loops. - Preserve FIFO within priority/generation and exact retry receipts. - Connect reveal to the single destination-generation readiness state. -**Gate:** zero `viewport-before-ready`; traversal p99 at or below 16.67 ms target -and maximum below 33.3 ms target on the reference local run; no stranded old -generation, staged upload, collision, effect, or GPU owner. +**Gate (rescoped, 2026-07-24 review):** zero `viewport-before-ready`; no +single-frame publication/retirement transaction exceeds the configured +time/byte/entity budgets; portal-window frame p99 and single-frame +allocation materially improve versus the post-Slice-A baseline; no stranded +old generation, staged upload, collision, effect, or GPU owner. The +program-level 16.67 ms p99 / 33.3 ms maximum targets are §11 gates verified +at Slice H — Slice E cannot own them while the audit's own P2 costs +(whole-world partition, unconditional UI/diagnostic work) are assigned to +Slices G and H. + +**Amendment (2026-07-24 review).** Binding contracts for this slice: + +- **Release-later never means run-later.** Retired-generation owners are + tick-frozen and audio-silenced at the immediate detach edge even when + their memory release is budgeted across later frames. Audio is added to + the immediate set (or bounded at ≤1 frame): a looping emitter on a + retired landblock is user-audible during portal transit, and audio + appears in neither of §6.2's lists today. Retail teardown anchors for the + slice plan: `CLandBlock::Destroy` 0x0052faa0, `destroy_static_objects` + 0x0052fa50, `CObjectMaint::DestroyObjects` 0x00508c30 (retail teardown is + one synchronous transaction; deferral has retail precedent only via the + `AddObjectToBeDestroyed` queue, whose drain is complete-when-run). +- **Reveal stall behavior.** A reveal hold longer than retail's 5-second + fade ceiling keeps the tunnel animating, surfaces the retail "In Portal + Space - Please Wait" cue (AP-115), and emits a diagnostic — never a + silent early reveal. The `IsUnhydratable` loud-failure escape stays. + Retail anchor: `SmartBox::UseTime` 0x00455410 `blocking_for_cells`; the + 1/2/5 s fade thresholds are recorded in + `claude-memory/project_portal_space.md`. +- **All count constants convert.** `MaxCompletionsPerFrame` is not the only + count-shaped budget: `MaxDrainIterations` (the ×64 outer drain ceiling) + and the priority-ring radius are part of the same redesign; and the + uncapped one-frame paths (`BeginFullWindowRetirement`, + `TryAdvanceOriginRecenterPreparation` — the ~600-landblock teleport + retire) are the primary offenders, not the 9-landblock eager ring. +- Deferred loads retained in `_deferredApply` during a burst are a + retained-memory high-water source; the staged pipeline must account for + them under the CPU-bytes budget. ### Slice F — Incremental render scene foundation @@ -583,6 +748,34 @@ generation, staged upload, collision, effect, or GPU owner. **Gate:** shadow scene matches canonical world/resource checkpoints through the nine-stop route with bounded memory and zero authoritative ownership. +**Amendment (2026-07-24 review).** Preconditions and contracts: + +- **User approval required before this slice starts** (§0.1 — F/G resume + the deferred MP3 work). +- **Referee before the match.** The deterministic scene digest is built and + validated against the CURRENT pipeline in steady state as its own + preceding sub-slice, so the comparison tool has a proven track record + before it gates a mirror. During soak/dev runs the digest also runs + continuously through portal windows (every N frames), not only at the + nine checkpoints — checkpoint-only comparison is blind to drift that + self-corrects between stops, which peaks exactly when churn does. +- **Journal discipline.** The delta journal is drained to empty every host + tick while single-threaded (assert on carryover); deltas coalesce + last-writer per (entity, kind) between drains, bounding the journal at + O(live entities) instead of O(mutations); on any future thread + separation it adopts §6.1's bounded-queue discipline with an explicit + no-block rule for the update thread. +- **Interface home.** `IRenderScene`, its components, and the delta journal + types live in `AcDream.App` for Slices F–I. At Slice J the seam becomes + Runtime-emitted ordered entity events (incarnation + transform + flags, + no render vocabulary) that an App-side projection adapter maps to + `IRenderScene` deltas — write Slice F against that end-state so no render + type bakes into a contract Slice J must break. +- **Identity enforcement.** The §4 no-unchecked-cast rule gets a concrete + mechanism here: wrapper structs with a single `.Value` accessor plus a + grep-based architecture test (or analyzer) restricting that accessor to + owning assemblies. + ### Slice G — Render snapshot and delta submission cutover **Purpose:** Make render cost proportional to visible/changed data. @@ -600,6 +793,20 @@ nine-stop route with bounded memory and zero authoritative ownership. 300 FPS or the corrected profile identifies a new dominant stage; steady frame allocation is near zero. +**Amendment (2026-07-24 review).** The 300 FPS / 3.33 ms figure is +provisional until Slice A's corrected pass timing re-measures the dense-town +CPU-vs-GPU split on the local display (the prior "GPU-bound ~200 FPS" +reading was produced by the boundary-spanning `GpuFrameTimer` this plan +corrects, and the 2026-07-24 capture cannot settle it). Bucket queries must +preserve the retail dynamics contract: retail drops out-of-flood dynamics +via the per-dynamic viewcone CULL, not by set membership +(`RetailPViewRenderer.DrawDynamicsLast`) — replacing the whole-world walk +with buckets must not change which dynamics get culled. This slice also +retires the sibling per-frame rebuilds the audit missed: +`EnvCellRenderer.PrepareRenderBatches` (gated post-review) and the +nearby-building gather walk, both of which otherwise keep steady render cost +proportional to resident content. + ### Slice H — Event-driven UI, diagnostics, lights, and frame cleanup **Purpose:** Remove remaining work that scales with uncapped FPS. @@ -617,7 +824,47 @@ allocation is near zero. **Gate:** no steady-state Gen-2 collections; steady allocation target ≤ 4 KiB/frame initially and zero for the core world/render loop; UI and network -behavior tests unchanged. +behavior tests unchanged; process-wide steady allocation rate within the §11 +per-scene target (frame-thread numbers alone are gameable, §0.4); the +program-level traversal frame-time targets from §11 are verified here. + +**Amendment (2026-07-24 review).** This slice splits into three separately +gated pieces: + +- **H-a — UI/diagnostics/frame cleanup.** Includes: `UiText` shaping cache + (appraisal/character-info/effects panels re-run full word-wrap every + visible frame; apply the existing `ChatWindowController.GetTranscriptLines` + revision+width+font cache pattern — it is already proven in-tree), the + cooldown scan's visibility scoping (it is O(all mounted item-list slots) + regardless of open windows — the worst-scoped of the audited UI items), + diagnostics consumer-gating (reuse the AABB results as the partition's + landblock frustum cull rather than deleting them — the same test is + computed and discarded one call away from where it is needed), + `EnvCellRenderer` prepare gating (landed post-review), and the per-frame + scratch items (RetailPViewFrameInput reuse, iterator layers). +- **H-b — Lights (pinned; two-failure regression history, register AP-85).** + The candidate pool remains the resident registry (`_all`); any narrowing + is the existing latency-buffered visible-cell FILTER only; the over-cap + ranking stays dynamics-first + nearest-PLAYER; the only change is + replacing the >MaxGlobalLights overflow `List.Sort` with a top-k partial + select that produces the IDENTICAL selected set. Retail anchors to cite + in the slice plan: `CEnvCell::add_dynamic_lights` 0x0052d410, + `Render::insert_light` 0x0054d1b0, `Render::add_dynamic_light` + 0x0054d420, caps 0x0081ec94/98. Gate adds: selected-set equivalence on + captured Town Network fixtures (463 lights) + fixed-camera screenshot + comparison. Note the audit overstated this item: per-object/per-cell + selection is already an allocation-free top-8 insertion; only the + overflow path sorts. +- **H-c — Network I/O (requires §0.3 authorization).** Scope includes the + receive-side per-packet allocation chain the audit missed (fresh + `UdpClient.Receive` array + `BodyBytes` copy + per-fragment `ToArray` + + per-packet `Packet`/`List`/`Optional` objects), not only the outbound + `ToArray`. Invariants pinned: single outstanding receive, kernel arrival + order preserved into the inbound queue, ack-per-received-packet at decode + acceptance (holtburger pattern), `LinkStatusHolder::OnHeartbeat` + 0x004113D0 last-heard placement unchanged, `PumpOnce` handshake pacing + preserved. (Receive-thread death robustness landed post-review as an + approved bug fix.) ### Slice I — Flat collision assets and residual zero-allocation work @@ -633,6 +880,20 @@ behavior tests unchanged. **Gate:** trajectory and retail conformance suites remain bit-equivalent; navigation/collision fixtures match; portal and steady allocation targets pass. +**Amendment (2026-07-24 review).** Bit-equivalence across the layout change +is achievable (RyuJIT SSE scalar IEEE-754; the x87 concerns in project +memory are decomp-READING lessons, not runtime determinism) — but only +under two conditions the slice plan must state: (a) the bake stores +dat-parsed floats VERBATIM, zero arithmetic transformation (no plane +renormalization, no quantization, no double round-trips); (b) traversal +preserves front/back child visit order and every epsilon comparison. The +existing suites are scenario-bounded, so add an old-vs-flat DIFFERENTIAL +sweep during the shadow phase: mass BSPQuery/point-in-cell/sweep queries +across a large cell sample, outputs compared bit-for-bit between graph and +flat implementations while both coexist. Physics transition pooling stays +behind its identity/lifetime-test condition. This slice floats — it depends +only on Slice B's bake infrastructure, not on F–H. + ### Slice J — Presentation-independent `AcDream.Runtime` **Purpose:** Establish the graphical/headless shared client kernel. @@ -649,6 +910,20 @@ navigation/collision fixtures match; portal and steady allocation targets pass. test connects, enters world, moves, receives inventory/chat/world updates, portals, logs out, reconnects, and tears down without loading App/Silk/OpenAL. +**Amendment (2026-07-24 review).** Requires §0.3 authorization (moves +frozen-subsystem owners). Add a **teardown protocol** subsection to §9.2 +before this slice starts — session reset, logout, and mid-portal disconnect +traverse five new owners, and tests without a specified order mean the +order gets invented per-slice: (1) cancel scheduler generations; (2) drain +or poison journals/queues; (3) unpublish the render scene; (4) release +residency, fence-gated; (5) `ContentStore` last (shared, outlives +sessions). Every new owner registers with the existing +`CompositionAcquisitionScope` lease discipline so disposal order is +structural, not remembered. This slice is a multi-plan track (the +comparable `GameWindow` decomposition took 8 slices for 14K lines), not a +single bisectable slice — plan it as lifetime-group sub-slices with +host-parity tests after each move. + ### Slice K — Linux headless and multi-session host **Purpose:** Deliver efficient automated bots. @@ -667,6 +942,19 @@ portals, logs out, reconnects, and tears down without loading App/Silk/OpenAL. dependency, bounded incremental memory per session, no busy-loop CPU, clean teardown/reconnect, and parity with graphical runtime command/event behavior. +**Amendment (2026-07-24 review).** The parity gate covers packet ORDER but +not inter-packet TIMING — and AutonomousPosition cadence is already a +tracked divergence (register row TS-33, +`claude-memory/project_retail_motion_outbound.md`). Add a wire-timing +assertion: AutonomousPosition inter-packet intervals and +MoveToState-on-change emission points within a stated tolerance band versus +the graphical host on the same scenario; the time-wheel scheduler must +derive wire cadence from the instance clock, never from tick count. The +30-session stress gets an explicit duration measured in HOURS (the +characteristic 30-bot failure is slow per-session creep — the #193 class +leaked ~1 GiB/min and was found by OOM, not by a 6-minute route), with +per-session incremental memory given a numeric ceiling at gate time. + ### Slice L — Linux graphical host and evidence-gated GPU work **Purpose:** Finish platform portability and only then pursue remaining GPU @@ -720,6 +1008,41 @@ Absolute process/VRAM targets are fixed after Slice D produces trustworthy category accounting; until then, the binding rule is plateau plus no regression in pixels/range. +**Amendment (2026-07-24 review).** Gate-table corrections and additions: + +- **Percentile populations.** Until the Slice A per-frame history export is + the population, every p99 row reads "no 5-second window with p99 > X" — + the windowed profiler cannot produce route-wide percentiles. +- **Anti-gaming complement.** New row: process-wide steady allocation rate + per scene (dotnet-counters), alongside the frame-thread rows. The + frame-thread gates alone reward moving garbage to workers, which is this + plan's own remediation shape. +- **Provisional budgets.** "≤ 4 MiB portal" and "≤ 4 KiB steady" are + round-number initial budgets, not derived values; tighten from post-A/C + evidence. "0 B/frame" means alloc p50 = 0 with every nonzero frame + attributable to the profiler/report path (the instrument itself allocates + its 5-second report on the frame thread). +- **Third-visit growth ≤ 5%** is scoped per metric class: same-location + tracked CPU/GPU residence measured against the location's own asset set + (possible after Slice D accounting); entity/emitter/particle counts stay + warnings on a live ACE world; absolute byte deltas reported alongside + percentages. The audit's own data shows legitimate +31.7% Sawato + tracked-GPU growth from route-wide cache warming — the unscoped gate + false-fails on exactly that. +- **New row — long-duration soak:** ≥ 60 minutes looping the route; managed + heap, LOH size + fragmentation, tracked GPU, handles, and threads all + plateau within stated bands. (The canonical route is 359 s; the #193 + leak class is invisible to it.) +- **New row — frame-pacing jitter** (stddev / 99.9th) once the per-frame + export exists. +- **Scheduled harness.** Slice A or H tasks a scheduled local perf harness: + run the committed route against the last committed baseline JSON with the + soak script's relative-limit pattern applied cross-commit, failing + loudly. Manual-and-unscheduled is not a regression net. +- "Reference local hardware" refers to the committed reference-configuration + block required by Slice A; gates against an undefined machine are + unfalsifiable. + --- ## 12. Test and review matrix @@ -752,6 +1075,14 @@ For a replacement cutover: - The gate commit removes the old production path. - No test threshold is loosened to accept a regression. +**Adjudication protocol (2026-07-24 review).** When a cutover comparison +disagrees: (1) any confirmed OLD-path bug fixed during the cutover gets its +divergence-register row or `docs/ISSUES.md` entry in the same commit; +(2) visual differences are adjudicated by the user (visual verification is +the user's role per CLAUDE.md), never self-certified; (3) any threshold +change requires a written rationale in the gate commit — "the new output is +an improvement" carries the same evidence burden as a regression claim. + --- ## 13. Risks and controls @@ -785,17 +1116,29 @@ has been removed and measured. The intended order is therefore: ```text -honest metrics - -> prepared content - -> typed residence - -> cost-budgeted streaming - -> incremental render scene - -> delta GPU submission - -> residual frame cleanup - -> presentation-independent runtime - -> Linux/headless/multi-session - -> evidence-gated GPU jobs +honest metrics + committed baselines (A — exit criteria block C) + -> prepared content (B, C) + -> typed residence (D — unify/retune, not rebuild) + -> cost-budgeted streaming (E) + -> [user decision gate: lift the ECS deferral? §0.1] + -> incremental render scene (F — referee sub-slice first) + -> delta GPU submission (G) + -> residual frame cleanup (H-a, H-b, H-c) + -> flat collision assets (I — floats; depends only on B) + -> presentation-independent runtime (J — §0.3 authorization) + -> Linux/headless/multi-session (K) + -> evidence-gated GPU jobs (L) ``` This order improves the current graphical client first while laying clean boundaries for Linux and approximately 30 automated headless clients later. + +**Execution discipline (2026-07-24 review):** this plan runs in dedicated +Track MP/LH side-track sessions only; the active milestone's critical path +wins every conflict (§0.2). Slice I appears in this list explicitly because +both earlier order summaries omitted it. The 2026-07-24 review also landed +a set of independent fixes ahead of the plan (receive-thread robustness, +bounded audio caches, dead ParticleBatcher chain deletion, CellStruct clone +guard, solid-color texture cache, EnvCellRenderer prepare gating, +measurement tooling) — those are recorded in the relevant slices above as +"landed post-review" and must not be re-done. diff --git a/docs/research/2026-07-24-runtime-performance-audit.md b/docs/research/2026-07-24-runtime-performance-audit.md index 8bddd479..c0bfe3ba 100644 --- a/docs/research/2026-07-24-runtime-performance-audit.md +++ b/docs/research/2026-07-24-runtime-performance-audit.md @@ -15,6 +15,11 @@ behavior were changed during the benchmark. **Architecture response:** [`../plans/2026-07-24-modern-runtime-architecture.md`](../plans/2026-07-24-modern-runtime-architecture.md) +**Errata:** section 13 records the corrections from the 2026-07-24 +adversarial review (test-condition disclosure, single-frame allocation +attribution, unit conversions, LOH fragmentation, exception-count +provenance). Read it before citing this report's §1/§2/§4 numbers. + --- ## 1. Executive result @@ -599,3 +604,93 @@ tools to inspect, and can expose local/session metadata. This report plus the compact JSON and checkpoint CSV preserve the benchmark's method, provenance, measurements, interpretation, and decision-driving evidence in a form another engineer or model can review directly. + +--- + +## 13. Errata — 2026-07-24 adversarial review + +An independent adversarial review (43-agent verification against commit +`e627d9a4`, same date) confirmed every cited code mechanism in this report +(12/12 file:line citations accurate) but found the following statements wrong +or under-qualified. Per the project's historical-record rules the original +sections above are left untouched; this appendix is the correction record. + +1. **Test conditions (§2) — "Normal capped presentation" is wrong.** The + reproduction script this report cites (`tools/run-connected-r6-soak.ps1`) + unconditionally set `ACDREAM_UNCAPPED_RENDER=1`, which disables BOTH VSync + and the software pacer (`FramePacingPolicy.Resolve`). The ~32 FPS ceiling + came solely from the RDP display path, not client pacing. + `ACDREAM_DUMP_MOVE_TRUTH=1` (per-movement console logging) was also active + for the whole capture and undisclosed. Unresolved provenance question: the + CPU trace shows `FramePacingController.CompleteFrame` at 6.44% inclusive, + which should be a near-zero no-op under those flags — either the capture + used a different launcher than documented or the trace row is + misattributed. Post-review the soak script defaults to capped and takes an + explicit `-Uncapped` switch, and emits an `env-disclosure.json` per run. +2. **Single-frame allocation attribution (§1, §9 P0).** The 203.6 MiB + "maximum single-frame allocation" is measured with + `GC.GetAllocatedBytesForCurrentThread()` — frame-thread only. + `MeshExtractor` decode runs exclusively on 4 background `Task.Run` workers + (`ObjectMeshManager.ProcessQueue`), so it structurally cannot appear in + that figure. The single-frame spike belongs to the main-thread + publication/retirement sources (`StreamingController.DrainAndApply` / + `ApplyResult`, retirement, static publication). `MeshExtractor` remains + the top source of the PROCESS-WIDE impacts (allocation rate, Gen-2, GC + pauses — §8's sampled stacks are process-wide). Consequence: Slice E owns + the frame-spike symptom; Slices B/C own the process-wide rate. +3. **Unit conversions (§1).** 1,661.9 MiB = 1.62 GiB (not 1.66) and + 2,269.1 MiB = 2.22 GiB (not 2.27); the printed values divided MiB by 1000. +4. **Committed managed memory (§5).** 830.6 MiB is the FINAL checkpoint, not + the peak: the checkpoint peak was 1,242.1 MiB (caul-return) and the + continuous counter maximum 1,360,900,096 B ≈ 1,298 MiB. +5. **LOH fragmentation (§5).** The companion JSON records LOH size max + 807.5 MB with 524.1 MB fragmentation (~65%) — the largest quantified + contributor to the committed-vs-live gap. It never appears in the prose + above and should anchor the Slice D accounting work. +6. **Exception counts (§8).** This report's own figures are the + focused-trace 1,587 (1,335 `ArgumentOutOfRangeException` + 252 + `SocketException`). The "4,520 + 1,168" figures quoted by the companion + plan document have no derivation here or in the committed artifacts and + must not be cited until re-derived from a fresh capture. +7. **GC counts (§4).** 99/105/38 are per-generation EXCLUSIVE counts from + the .NET 10 `System.Runtime` meter (`dotnet.gc.collections`), under which + gen1 > gen0 is legal; cumulative `GC.CollectionCount` equivalents are + 242/143/38. The "38 Gen-2" headline is correct under either reading. +8. **Cache context (§8, §9 P1).** Four bounded LRU caches already front the + mesh/texture path (GPU render-data 1 GiB / 50 objects; CPU prepared-mesh + 128 MiB / 100 entries; decoded-texture 64 MiB / 128 entries; DAT typed + object 64 MiB / 256 entries ×4 databases). Revisits rebuild because these + are UNDERSIZED relative to the route's working set (~599 tracked + textures), not because no caching mechanism exists. Likewise, texture + atlas arrays have real fence-gated, budgeted eviction (empty-atlas pool + 64 MiB; composite arrays 128/64 MiB) — the route simply never pushed the + ceilings, so "bounded" in §7 was bounded-by-route, not policy-exercised. +9. **Portal-burst list (§4).** The five max-frame values are not attributed + to specific transitions or route order, so warm-vs-cold portal behavior + is not derivable from the committed data. Cache hit/miss/evict counters + were added to the checkpoint JSON post-review to settle this. +10. **Dead-code caveat (§9 P2).** The "fixed 65,536-instance particle + CPU/GPU buffer" belonged to an unreachable code path (`ParticleBatcher`; + never fed by production — the live particle renderer is + `ParticleRenderer` with growable buffers). It was deleted post-review; + its ~7 MiB combined residency was dead weight, and its GPU buffer was + one concrete instance of the §7 tracked-vs-OS gap (never registered with + the GPU tracker). +11. **Soak-script dwell bug (§11).** The script's own comment requires + sampling after the 25-second liveness deadline + (`LiveEntityLivenessController`), but it slept 12 s — return-gate samples + could include un-pruned prior-destination records. Fixed post-review + (26 s). +12. **Reproducibility scope (§12).** The §4 headline table's continuous + CPU/GPU-engine/memory percentiles were produced by manual sampling not + committed to the repo; per-scene steady-allocation figures, four of five + portal-burst values, "~20,000 registry records", "599 tracked textures / + 43–48 atlas arrays", and the 23–30 ms gpu_ms figure likewise have no + surviving committed evidence. They can be reviewed but not independently + regenerated; treat them as reproducible-not-reviewed until the Slice A + tooling recaptures them. + +The full review (verdicts, refutations, and the new findings the audit +missed) and the resulting plan amendments are recorded in +[`../plans/2026-07-24-modern-runtime-architecture.md`](../plans/2026-07-24-modern-runtime-architecture.md) +section 0.