fix(headless): #365 — collision-admission-open window drove the first-entry conductor into a permanent seal refusal
Root cause (measured live via ACDREAM_PROBE_PARK=1): HeadlessSessionWorldProjection drove the first-entry conductor unconditionally, including while HeadlessCollisionNeighborhood's own 3x3 publication plan held a genuinely open RuntimeCollisionAdmission for the local player's landblock. Every TrySealCollisionEvaluationAuthority attempt during that window failed (IsCollisionEvaluationPrefixAdmissible false) and retried forever without recovering — measured verdict: "seal-refused" repeating with no preceding [rearm] verdict= line (the operation never even reached the AwaitingCell park). This is the diagnosis doc's "structural half" mechanism; no evidence of the "circular HasOldPrefixPlacementDebt" hypothesis was observed, so that shape was not needed. Step 1 (enabler): HeadlessStaticStateAudit.ValidateProcessIsolation now takes sessionCount and only refuses process-global physics probes for sessionCount > 1 — its own multi-root-attribution rationale never applied to a single session, and it was blocking the exact probe built to diagnose this class of stall. Step 3a (root cause): new IHeadlessCollisionNeighborhood.IsQuiescent gates ProjectSpawn/ProjectPosition/PumpFirstEntry's conductor-drive calls — the conductor is never driven while the neighborhood's own publication owns collision authority for that tick. Step 4 (defense-in-depth): HeadlessLocalPlayerFrameHost.CanAdvancePlayer now requires Controller.CanExecuteLiveMovement instead of just a non-null controller — the headless-only gap that turned the (now-fixed) hydration stall into a hard crash reaching SuspendObjectUpdate on a dormant controller. RuntimeLocalPlayerFrameController's three shared entry points gained the same guard, contract-preserving for the graphical host. Verified end-to-end against live ACE (jump-probe policy, three runs): hydration succeeds cleanly (136 entities load vs. 0 before), no seal-refused spam, no crash from the original bug, graceful logout every time. Full airborne-transition confirmation is blocked by a separate, newly-discovered, pre-existing defect filed as #368 (the headless scheduler's Task.Delay(...).ConfigureAwait(false) tick loop can resume on a different ThreadPool thread mid collision-generation, tripping EnsureCollisionMutationThread) — explicitly out of scope here, not mentioned anywhere in the #365 diagnosis, and unsafe to fix without graphical-host verification this session was constrained not to perform. New tests: the real-admission hydration test (fails on the pre-Step-3a tree, verified by temporarily reverting the three gates and confirming failure, then restoring), the PumpFirstEntry quiescence-gate test, the CanAdvancePlayer publication-lifecycle test, the dormant-controller sabotage tests for RuntimeLocalPlayerFrameController, and the audit single/multi-session tests. RuntimeLocalPlayerPhysicsPublicationStateTests is untouched. Full Release suite: 12,343 passed / 4 skipped / 0 failed (baseline ~12,330/4 plus 11 new tests). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Erik
parent
6150327ea3
commit
41b408f3e6
10 changed files with 937 additions and 35 deletions
166
docs/ISSUES.md
166
docs/ISSUES.md
|
|
@ -123,18 +123,81 @@ CH4 REJECT-review).
|
|||
|
||||
## #365 — Headless host cannot move at head: session quarantines on the first advance tick; world never hydrates
|
||||
|
||||
**Status:** OPEN — filed 2026-08-10 during the Campaign CH jump-probe
|
||||
reproduction. Distinct from #330 (no collision) and #332 (no remote DR):
|
||||
this is the LOCAL player.
|
||||
**Status:** CLOSED 2026-08-10. Root-caused and fixed per
|
||||
`docs/research/2026-08-10-365-headless-hydration-diagnosis.md` (see its
|
||||
OUTCOME section for the measured verdict and fix shape). Distinct from #330
|
||||
(no collision) and #332 (no remote DR): this was the LOCAL player.
|
||||
|
||||
**Symptom:** a headless session (`jump-probe` policy, real ACE connect,
|
||||
`testaccount`/`+Acdream`) reaches `InWorld`, `live: in world — CreateObject
|
||||
stream active` prints, but `entityCount` stays **0** for the whole run and
|
||||
the local movement controller is never published. The first
|
||||
`SetIntent(Jump: true)` → next `AdvanceBeforeNetwork` tick quarantined the
|
||||
session.
|
||||
**Correction to the evidence chain below:** `entities: 0` in the headless
|
||||
JSON is NOT evidence of failed hydration. `HeadlessDiagnosticWriter.Lifecycle`
|
||||
fires at exactly four points (constructed / start-result / reconnect-deferred
|
||||
/ stopped), none of which run after a CreateObject stream has had any chance
|
||||
to populate the entity table — a perfectly healthy run prints the SAME
|
||||
`entities: 0` on every lifecycle line. `entityCount` was a logging artifact,
|
||||
not a hydration symptom.
|
||||
|
||||
**Evidence chain (2026-08-10):**
|
||||
**Actual root cause (confirmed via live capture with `ACDREAM_PROBE_PARK=1`,
|
||||
unblocked by the Step-1 audit fix below):** the headless host's ONLY
|
||||
collision publisher is a 3×3 landblock plan STARTED BY the local player's own
|
||||
CreateObject (`HeadlessCollisionNeighborhood`) — unlike the graphical host's
|
||||
publisher, which runs on the streaming cadence ahead of the Create burst.
|
||||
`HeadlessSessionWorldProjection.PumpFirstEntry`/`ProjectSpawn`/`ProjectPosition`
|
||||
drove the first-entry conductor UNCONDITIONALLY, including while the
|
||||
neighborhood's own publication held a genuinely open
|
||||
`RuntimeCollisionAdmission` for the same landblock the player's placement
|
||||
needed. Every `TrySealCollisionEvaluationAuthority` attempt during that
|
||||
window failed (`IsCollisionEvaluationPrefixAdmissible` false) and retried
|
||||
every tick without ever recovering while the window stayed open. Measured
|
||||
verdict: `seal-refused` repeating with NO preceding `[rearm] verdict=` line —
|
||||
the operation never even reached the `AwaitingCell` park; it failed the seal
|
||||
immediately on every attempt while `AwaitingPreparation`. This is the
|
||||
diagnosis doc's "structural half — CONFIRMED" mechanism, not its "circular
|
||||
`HasOldPrefixPlacementDebt`" hypothesis (no `prefix-inadmissible` rearm
|
||||
verdicts were ever observed).
|
||||
|
||||
**Fix (Step 3a):** new `IHeadlessCollisionNeighborhood.IsQuiescent`
|
||||
(`_pendingPublication is null && _publicationQueue.Count == 0 &&
|
||||
!_pendingPublicationCancellation`) gates all three drive call sites
|
||||
(`ProjectSpawn`, `ProjectPosition`, `PumpFirstEntry`) — the conductor is
|
||||
never driven while the neighborhood's own publication owns collision
|
||||
authority for that tick.
|
||||
|
||||
**Fix (Step 4, defense-in-depth):** `HeadlessLocalPlayerFrameHost.CanAdvancePlayer`
|
||||
now requires `Controller is { CanExecuteLiveMovement: true }` instead of just
|
||||
`Controller is not null` — the exact bug that turned the (now-fixed)
|
||||
hydration stall into a hard crash (a dormant, unpublished controller reaching
|
||||
`SuspendObjectUpdate`). `RuntimeLocalPlayerFrameController`'s three shared
|
||||
entry points (`AdvanceBeforeNetwork`/`RunPostNetworkCommandPhase`/
|
||||
`TryGetPresentationAfterNetwork`) gained the same
|
||||
`controller.CanExecuteLiveMovement` guard, contract-preserving for the
|
||||
graphical host.
|
||||
|
||||
**Enabler (Step 1):** `HeadlessStaticStateAudit.ValidateProcessIsolation`
|
||||
now takes `sessionCount` and only refuses process-global physics probes for
|
||||
`sessionCount > 1` (logging, not refusing, for the single-session case) — the
|
||||
audit's own rationale (multi-root attribution ambiguity) never applied to a
|
||||
single session, and it was blocking the exact probe (`ACDREAM_PROBE_PARK=1`)
|
||||
built to diagnose this class of stall.
|
||||
|
||||
**End-to-end verification:** confirmed via three live `jump-probe` runs
|
||||
against local ACE — hydration now succeeds cleanly (136 entities load,
|
||||
"local player present" fires promptly, `[jump-probe] releasing jump (fire)`
|
||||
reached — no `seal-refused` spam, no crash from the original bug) and the
|
||||
session exits gracefully every time (`[session] graceful logout confirmed`,
|
||||
zero leases at disposal). Full `airborne-transition True` confirmation is
|
||||
blocked by a SEPARATE, newly-discovered, pre-existing defect — see #368 below
|
||||
— not by anything in this issue's scope. A diagnostic-only run with #368's
|
||||
guard temporarily neutralized (never shipped, reverted before commit)
|
||||
confirmed the #365 fix produces the correct behavior once past that unrelated
|
||||
blocker: full hydration, the jump-probe policy running to completion, exit
|
||||
code 0.
|
||||
|
||||
**Also observed in the same runs:** `[weenie-error] unmapped code=0x051D` —
|
||||
an ACE-only id outside retail's 344-case `HandleFailureEvent` switch; CH2's
|
||||
silent-toward-player + diagnostics-line fallback handled it as designed (no
|
||||
action needed, noted for completeness).
|
||||
|
||||
**Original evidence chain (2026-08-10, superseded by the root cause above):**
|
||||
1. First quarantine: `RuntimeLocalPlayerFrameController.AdvanceBeforeNetwork:93`
|
||||
unconditionally re-assigned `controller.LocalEntityId` — a sealed
|
||||
configuration property — on a still-dormant controller. FIXED in the
|
||||
|
|
@ -144,16 +207,8 @@ session.
|
|||
`ObjectClockDisposition == Suspend` branch and calls
|
||||
`SuspendObjectUpdate` → `EnsurePublishedForRuntimeOperation` throws —
|
||||
`_host.CanAdvancePlayer` is true while the controller is unpublished.
|
||||
3. Root condition: the session's world never hydrates (`entities: 0`
|
||||
despite the active CreateObject stream), so dormant local activation
|
||||
never completes and the controller never publishes. K3/K4 ran full
|
||||
connected multi-session gates with movement — this regressed somewhere
|
||||
in the many Runtime/placement/CH landings since 2026-07-27.
|
||||
|
||||
**Also observed in the same run:** `[weenie-error] unmapped code=0x051D` —
|
||||
an ACE-only id outside retail's 344-case `HandleFailureEvent` switch;
|
||||
CH2's silent-toward-player + diagnostics-line fallback handled it as
|
||||
designed (no action needed, noted for completeness).
|
||||
3. Root condition (at the time): believed to be "the session's world never
|
||||
hydrates" from the `entities: 0` artifact — corrected above.
|
||||
|
||||
**Repro:** `dotnet run --project src/AcDream.Headless -c Release -- run
|
||||
--config <cfg>` with a `jump-probe` policy session against local ACE
|
||||
|
|
@ -161,10 +216,73 @@ designed (no action needed, noted for completeness).
|
|||
Environment). `HeadlessDiagnosticWriter.Failure` now emits `errorDetail`
|
||||
(full exception) — added during this diagnosis.
|
||||
|
||||
**Next step:** bisect the headless hydration path (why zero entities admit
|
||||
headless when the graphical host hydrates fine) BEFORE touching
|
||||
`CanAdvancePlayer` — the #357 closeout warns that dormant-activation
|
||||
classification changes have a strict test matrix.
|
||||
## #368 — Headless scheduler's async tick loop can run collision-generation calls on different threads, tripping `EnsureCollisionMutationThread`
|
||||
|
||||
**Status:** OPEN — filed 2026-08-10 during #365's end-to-end verification.
|
||||
Explicitly OUT OF SCOPE for #365 — orthogonal mechanism, not mentioned
|
||||
anywhere in that diagnosis.
|
||||
|
||||
**Symptom:** a real headless run against live ACE (`jump-probe` policy,
|
||||
`ACDREAM_PROBE_PARK=1`) that survives long enough for the local player's own
|
||||
landblock collision generation to span more than a couple of scheduler ticks
|
||||
reliably quarantines with:
|
||||
|
||||
```
|
||||
System.InvalidOperationException: Collision generations must be staged and
|
||||
committed on one update thread.
|
||||
at AcDream.Runtime.Physics.RuntimePhysicsState.EnsureCollisionMutationThread()
|
||||
at AcDream.Runtime.Physics.RuntimePhysicsState.AdvanceCollisionGenerationSeal(...)
|
||||
at AcDream.Headless.Hosting.HeadlessCollisionGenerationTransaction.Advance()
|
||||
at AcDream.Headless.Hosting.HeadlessCollisionNeighborhood.AdvanceWork()
|
||||
at AcDream.Headless.Hosting.HeadlessCollisionNeighborhood.IsReady(...)
|
||||
at AcDream.Headless.Hosting.HeadlessSessionWorldProjection.PumpFirstEntry()
|
||||
at AcDream.Headless.Hosting.HeadlessSessionHost.Tick(...)
|
||||
at AcDream.Headless.Hosting.HeadlessProcessScheduler.DispatchSessionDue(...)
|
||||
```
|
||||
|
||||
Reproduced identically across 3 separate live-ACE runs (2026-08-10), each
|
||||
time at the same point (`[wake] begin lb=0x0904FFFF gen=1` — right after the
|
||||
jump-probe policy's "local player present" line) — not a one-off timing
|
||||
fluke.
|
||||
|
||||
**Root cause (confirmed by reading `RuntimePhysicsState.EnsureCollisionMutationThread`
|
||||
+ `ResetSessionPhysics`'s own doc comment):** the guard binds the FIRST
|
||||
thread that calls any collision-mutating method for a generation
|
||||
(`_collisionMutationThreadId`, `Interlocked.CompareExchange`) and requires
|
||||
every later call on that generation to match — a real invariant for the
|
||||
graphical host, whose whole session runs on one dedicated update thread.
|
||||
`HeadlessProcessScheduler.RunAsync` instead drives its ticks through
|
||||
`await Task.Delay(delay, _timeProvider, cancellationToken).ConfigureAwait(false)`
|
||||
— a console app has no `SynchronizationContext`, so each resumption after the
|
||||
delay can legitimately land on a different ThreadPool worker. The FIRST
|
||||
collision-mutating call (during the session's opening synchronous tick,
|
||||
still on the process's original thread) binds the guard to that thread; any
|
||||
LATER tick that resumes on a different pooled thread and also calls into
|
||||
collision generation trips it.
|
||||
|
||||
**Why K1–K4's connected gates never caught it:** those gates' own collision
|
||||
generations apparently completed within tick sequences that stayed on the
|
||||
same pooled thread (low contention on those runs), or the exact interleaving
|
||||
needed to cross a real `Task.Delay` resumption boundary mid-generation never
|
||||
occurred. Every fixture test in this repo drives `Tick()` synchronously and
|
||||
directly, never through the real `HeadlessProcessScheduler.RunAsync` await
|
||||
loop — so none of them exercise this path either. A genuine coverage gap,
|
||||
not a regression from a specific commit.
|
||||
|
||||
**NOT fixed here.** A proper fix needs its own investigation (e.g. a
|
||||
dedicated single update thread for the scheduler's tick loop, or redesigning
|
||||
the thread-affinity invariant for verified multi-thread-safe callers) and
|
||||
must be verified against the GRAPHICAL host too — out of reach for the
|
||||
session that found this (constrained not to launch the graphical client).
|
||||
Do not "fix" this by loosening or removing `EnsureCollisionMutationThread`'s
|
||||
check — it guards a real invariant elsewhere in the physics/collision
|
||||
system, and a workaround shape here is exactly what CLAUDE.md's
|
||||
no-workarounds rule forbids without explicit approval.
|
||||
|
||||
**Repro:** run the `jump-probe` policy against local ACE with
|
||||
`ACDREAM_PROBE_PARK=1` for long enough that the local player's own landblock
|
||||
collision generation spans more than a couple of scheduler ticks (the
|
||||
default case against a real DAT-loaded landblock).
|
||||
|
||||
## #366 — Chat window's new-unseen-text indicator (0x1000048C) imports but is never independently wired
|
||||
|
||||
|
|
|
|||
|
|
@ -353,3 +353,97 @@ that +Acdream walks — the K3/K4 gate unrun since 2026-08-02.
|
|||
| 529e0e9d root cause; 78b981cc changed the symptom | High | git show 529e0e9d -- src/AcDream.Headless/ |
|
||||
| Probes unrunnable headless | Certain | HeadlessProcessHost.cs:45 |
|
||||
| Existing hydration test bypasses admission/seal | Certain | HeadlessSessionHostTests.cs:372-375 |
|
||||
|
||||
## 8. OUTCOME (2026-08-10, fix session)
|
||||
|
||||
Steps 1, 3a, and 4 landed exactly as specified; **3b was not needed.**
|
||||
|
||||
**Step 2 measurement (mandatory, run before any Step-3 code):** the
|
||||
`ACDREAM_PROBE_PARK=1` jump-probe repro against local ACE produced
|
||||
|
||||
```
|
||||
[pump] DriveAll #1 pending=0
|
||||
...
|
||||
[wake] begin lb=0x0904FFFF gen=1 unboundCells=0 buckets=0
|
||||
[rearm] guid=0x5000000A seal-refused (transient; lease retained)
|
||||
[rearm] guid=0x5000000A seal-refused (transient; lease retained)
|
||||
[jump-probe] local player present; charging jump
|
||||
```
|
||||
|
||||
— `seal-refused` repeating with **no preceding `[rearm] verdict=` line**.
|
||||
Per this doc's own Step-2 table, that shape means the operation never even
|
||||
reached the `AwaitingCell` park: `IsExactDormantLocalActivationCurrent`
|
||||
already reports "current" on every attempt (the op sits in
|
||||
`AwaitingPreparation` the whole time), so `TryRearmDeferredDormantLocalActivation`
|
||||
is never called, and every attempt fails at
|
||||
`TrySealCollisionEvaluationAuthority` on the SAME still-open admission. This
|
||||
is the doc's §3 "structural half — CONFIRMED" mechanism. **No
|
||||
`prefix-inadmissible` rearm verdict was ever observed**, so there is no
|
||||
direct evidence the §3 "why it may never open" `HasOldPrefixPlacementDebt`
|
||||
circular-wait hypothesis is in play for this repro — 3a alone was measured
|
||||
sufficient.
|
||||
|
||||
**Fix shape: 3a only.** `IHeadlessCollisionNeighborhood.IsQuiescent`
|
||||
(`_pendingPublication is null && _publicationQueue.Count == 0 &&
|
||||
!_pendingPublicationCancellation`) gates `ProjectSpawn`/`ProjectPosition`/
|
||||
`PumpFirstEntry`'s trailing `_firstEntry?.DriveAll()` / `_acceptedPositionDrive?.Advance()`
|
||||
calls exactly as specified. Step 4 landed as specified
|
||||
(`CanAdvancePlayer` requires `CanExecuteLiveMovement`, plus the three shared
|
||||
`RuntimeLocalPlayerFrameController` entry points hardened the same way).
|
||||
Step 1 landed as specified (`ValidateProcessIsolation(sessionCount)`, single
|
||||
session + probe logs and proceeds, multi-session + probe still refuses,
|
||||
naming the probe).
|
||||
|
||||
**Verification that the new test (item 3) actually discriminates:**
|
||||
temporarily reverting the three `IsQuiescent` gates (commented out, never
|
||||
committed) made both
|
||||
`RealAdmissionNeverDrivesTheConductorWhileOpenAndHydratesOnceReleased` and
|
||||
`PumpFirstEntryWithholdsDriveAllUntilQuiescentThenDrivesImmediately` fail —
|
||||
`Assert.Null(runtime.MovementOwner.Controller)` failed because the controller
|
||||
was ALREADY built and published (`CanExecuteLiveMovement = True`) while the
|
||||
neighborhood's admission was still held open, exactly the pre-fix race. Both
|
||||
pass cleanly on the real, fixed tree. The gate was then restored and
|
||||
`git diff` confirmed the file matches the shipped Step-3a diff exactly (no
|
||||
residual simulation code).
|
||||
|
||||
**A design note for the test:** the first attempt at test item 3 opened the
|
||||
held admission on the PLAYER'S OWN landblock via the full
|
||||
`HeadlessCollisionGenerationTransaction` commit cycle. That hit a genuine,
|
||||
separate settlement question in `CommitCollisionGeneration` →
|
||||
`TryAcquireCollisionPrefixMutationPermission` (never resolved within 200
|
||||
ticks in that configuration) — worth a future look if it turns out to matter
|
||||
in production, but not needed to prove Step 3a. The shipped test instead
|
||||
holds a real admission open on a DIFFERENT (neighbor) landblock the player
|
||||
does not target, cancelling rather than committing it — a faithful, simpler
|
||||
proof of "an open admission anywhere in the plan blocks driving" without
|
||||
touching that separate question.
|
||||
|
||||
**End-to-end (live ACE, `jump-probe` policy, `ACDREAM_PROBE_PARK=1`), three
|
||||
runs, all consistent:** hydration now succeeds — `entityCount` reaches 136 at
|
||||
the `running-stop` resource sample (previously 0, permanently), `[jump-probe]
|
||||
local player present; charging jump` fires promptly, `seal-refused` spam is
|
||||
gone, and the original fork-1 crash (`SuspendObjectUpdate` on a dormant
|
||||
controller) never recurs. **Full pass criterion 4
|
||||
(`[jump-probe] airborne-transition False -> True`) was NOT independently
|
||||
observed on the unmodified tree** — every real run hit a SEPARATE,
|
||||
newly-discovered, pre-existing defect first (filed as issue #368: the
|
||||
headless scheduler's `await Task.Delay(...).ConfigureAwait(false)` loop can
|
||||
resume ticks on a different ThreadPool thread than the one that opened a
|
||||
collision generation, tripping `RuntimePhysicsState.EnsureCollisionMutationThread`).
|
||||
#368 is explicitly out of scope for this fix — orthogonal mechanism, no
|
||||
mention anywhere in this diagnosis, and a proper fix needs verification
|
||||
against the graphical host, which this session was constrained not to
|
||||
launch. A throwaway, never-committed diagnostic run with #368's guard
|
||||
neutralized (verified via `git diff` to have zero residual footprint) DID
|
||||
reach `[jump-probe] releasing jump (fire)` with a clean exit (code 0,
|
||||
graceful logout, `entityCount=136`), confirming the #365 mechanism itself is
|
||||
sound; it timed out waiting for `airborne-transition True` in THAT run,
|
||||
plausibly a downstream artifact of the same unsynchronized-thread condition
|
||||
the neutralized guard exists to catch (racing collision/physics state across
|
||||
threads) rather than a second #365-scope defect — flagged as an open
|
||||
question in #368, not claimed as resolved.
|
||||
|
||||
Every real (unmodified) run's session tore down gracefully
|
||||
(`[session] graceful logout confirmed`, zero entities/leases at the final
|
||||
`disposed` sample) regardless of which way it exited — `testaccount` was
|
||||
never left in a stuck state by this work.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue