diag(net): #260 outbound/command-gate probe + corrected issue framing

The two-agent investigation refuted #260's as-filed hypotheses: every
UseWithTarget was acked (the J5.2 use gate never latched), and the LOH
leak is bounded sawtooth churn - the real climb is ~2.25 GB of native/
GPU memory (WS 3,261 vs managed 1,015 MiB at wedge). The wedge evidence
also showed why it could hide: the live combat toggle routes through the
generation-gated runtime command seam, and every rejection exit in that
chain (Disposed / StaleGeneration / !IsInWorld at Validate, plus the
operations slot reading IsInWorld=false when unbound) is COMPLETELY
silent - no log, no event.

ACDREAM_PROBE_NET=1 (NetDiagnostics owner, PhysicsDiagnostics pattern)
now arms three probe families, all zero-cost when off:

- [net-out] per reliable send at the SendGameMessage chokepoint: opcode,
  GameAction type+sequence, fragment/packet sequence, managed thread id
  (two tids would prove the cross-thread ISAAC-desync hypothesis alone),
  and state; [net-out-EX] via an exception FILTER that logs without
  catching, so propagation is unchanged.
- [net-tick] 1 Hz cadence from WorldSession.Tick: inbound/s, queue
  depth, budget breaks, worst inter-tick gap (frame-stall witness),
  out/s, acks/s.
- [cmd-gate] every silent runtime-command rejection with expected-vs-
  view generation, lifecycle, and IsInWorld, plus the combat toggle
  result (whose Inactive exit reads a DIFFERENT IsInWorld source).

One walked-portal repro session with this probe distinguishes all
remaining #260 wedge hypotheses. ISSUES.md #260 rewritten to the
corrected two-root framing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Erik 2026-07-29 08:53:58 +02:00
parent ab3da28c34
commit 534bacbc23
4 changed files with 245 additions and 55 deletions

View file

@ -97,14 +97,15 @@ Copy this block when adding a new issue:
---
## #260 — Portal-network wedge: server-round-trip actions die, LOH leaks — REPRODUCED
## #260 — Portal-network wedge: outbound actions die + native/GPU memory climbs
**Status:** OPEN — reproduced live, forensics captured, investigation not started
**Status:** OPEN — investigated 2026-07-29 (two-agent report-only pass over the
captured evidence); fix authorized by the user. **Verdict: two separate roots.**
**Severity:** HIGH (renders the client unplayable after sustained portal use)
**Filed:** 2026-07-29 (instrumented Coldeve session, Vulkan backend)
**Component:** interaction/use transaction gate + inbound stream + managed memory
**Component:** WorldSession outbound boundary + Vulkan resource lifecycle
**Supersedes framing of:** #256 (invisible portal-network objects) and #257
(~1.5 GB working set) — both are almost certainly facets of this.
(~1.5 GB working set) — both are facets of this.
**Reproduction (the first solid one):** on Coldeve (`play.coldeve.ac`), character
Barris, after several portal-network runs returning to the town portal network,
@ -116,36 +117,41 @@ gcdump taken *in the broken state* (15 MB), full session log with
`[cell-transit]`/`[input]`/`[use-target]`/`equipment` traces, the
`dotnet-counters` CSV, and stderr.
**Two symptoms, established from the traces:**
1. **The outbound action pipeline is wedged.** `CombatToggleCombat Press` × 3
produced **zero outbound send** (movement, being client-predicted, still
produces `[cell-transit]`). Signature of the one-request-at-a-time
interaction/use gate (J5.2) latching closed because a `UseWithTarget` never
received its server ack — a burst of `[use-target] … SEND UseWithTarget`
precedes the hang. Every server-round-trip action (portal use, combat mode)
is then silently dropped; local-only actions still work.
2. **A managed leak on the LOH — this one is ours.** `dotnet-counters` at wedge:
**LOH 601,807,472 B (574 MB)**, process WS 3.2 GB — distinct from the flat
~1.9 GB residency *level* the V11 soak plateaued at (that was benign; this
climbs under live walked play). Fingerprint matches the project's prior LOH
leak class (per-event `float[]`/`byte[]` never released). 185 `equipment:
attached … RightHandCombat` re-attach lines but only 1 CreateObject, so it is
re-attach/replay churn on existing entities, not fresh-object accumulation.
**Root 1 — the wedge (cause still open, field narrowed).** The as-filed
hypothesis (J5.2 use gate latched by an un-acked `UseWithTarget`) is **refuted**:
the log shows every `UseWithTarget` received its matching `[use-done]`, all
reveal generations 2→17 reached `event=complete cancelled=False failures=0`,
and `RuntimeCombatModeState.Toggle()`
(`src/AcDream.Runtime/Gameplay/RuntimeCombatModeState.cs:49-93`) never touches
that gate — it gates only on `IsInWorld`. Remaining ranked hypotheses:
(1) frame-thread saturation — inbound drains from an *unbounded*
`_inboundQueue` on a ~4 ms frame budget (`WorldSession.cs:990-1016`), collapsed
by Root 2's memory pressure; (2) outbound ISAAC/sequence desync — a send off
the frame thread desyncs the cipher (`NextGameActionSequence()` is a non-atomic
`++`, `WorldSession.cs:1618`) and the server silently drops every subsequent
packet, which uniquely explains hard-zero effect while client-predicted
movement survives.
**Hypothesis (unproven, for the investigation):** a jammed inbound stream could
both starve the ack the use-gate waits on AND accumulate LOH buffers — one root,
two symptoms. Or two bugs. The gcdump names the LOH type by retainer; the log
timeline names the unacked request. Not confirmed either way.
**Root 2 — the memory climb is NATIVE/GPU, not managed.** The as-filed "LOH
leak" is wrong: LOH is a bounded sawtooth (318→829→318 MiB; churn, not
retention) and live managed heap at wedge was only 333 MB. The monotonic climb
is **working set 1,295→3,261 MiB (~180 MB/min)**; at peak, WS 3,261 vs managed
committed 1,015 MiB leaves **~2.25 GB unaccounted native/GPU memory**. Prime
suspect: Vulkan device resources minted per equipment re-attach (185187
`equipment: attached … RightHandCombat` lines vs 1 CreateObject) and per vfx
setup, never released.
**Why the scripted gates missed it:** the V11 portal-churn soak used `/teleloc`,
which enters the transit state machine by a different door than a *walked* portal
transit — exactly the gap the walked-play session was designed to probe.
**Next (investigation, user-approved fix only):** read the gcdump retainer graph
per `reference_memory_leak_toolchain.md`; reconstruct the use-gate timeline from
the log to find the unacked request; determine shared-vs-separate root. Do NOT
patch the symptom (a gate timeout would be the classic forbidden workaround) —
find why the ack never lands.
**Next (authorized):** (a) instrument the `WorldSession` outbound boundary —
opcode+sequence per send, swallowed exceptions, `IsInWorld`, per-frame dt,
inbound-queue depth — and reproduce with walked portal play; at a dead combat
toggle, whether the send reaches the wire and whether the sequence still
advances distinguishes the remaining hypotheses. (b) audit the
attach→GPU-resource path for create-without-release. Do NOT patch the symptom
(a gate timeout or retry loop is the classic forbidden workaround).
---