docs: #365 Opus diagnosis — three layered defects, fix plan C->measure->B->A

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-10 10:41:26 +02:00 · 2026-08-10 10:41:26 +02:00 · 6150327ea3
commit 6150327ea3
parent 98de4f5ab3
1 changed files with 355 additions and 0 deletions
--- a/docs/research/2026-08-10-365-headless-hydration-diagnosis.md
+++ b/docs/research/2026-08-10-365-headless-hydration-diagnosis.md
@ -0,0 +1,355 @@
+# #365 headless hydration — Opus diagnosis (2026-08-10, read-only pass)
+
+Verbatim agent output. The fix agent works solely from this document.
+
+## 0. Executive summary
+
+Three distinct defects layered on top of each other. Only the middle one is
+the hydration root cause.
+
+| # | Layer | Verdict |
+|---|---|---|
+| **A** | `HeadlessLocalPlayerFrameHost.CanAdvancePlayer` accepts a **dormant (unpublished)** controller; the graphical host does not. This is the crash. | **CONFIRMED by source.** Real host-asymmetry bug. |
+| **B** | The headless collision neighborhood publishes collision **as a reaction to the local-player Create**, interleaved with the Create burst. Every `TrySealCollisionEvaluationAuthority` during that window is structurally refused, and (hypothesis) the mutual `HasOldPrefixPlacementDebt` ↔ seal dependency between remote first-entry operations and the landblock commit closes a circular wait that never opens. This is why the controller never publishes. | **Mechanism CONFIRMED; the "never opens" closure is HIGH-CONFIDENCE HYPOTHESIS.** One existing probe answers it in one line. |
+| **C** | `HeadlessStaticStateAudit` unconditionally refuses to start when **any** `PhysicsDiagnostics.Probe*`/`Dump*` flag is set — including single-session runs. The exact probe built to diagnose this stall class (`ACDREAM_PROBE_PARK=1`) cannot be run headless at all. | **CONFIRMED.** Why #365 arrived with no `[rearm]`/`[pump]` evidence. |
+
+**Fix C first (two-line predicate change), then use it to confirm B, then fix
+B, then fix A.** Fixing A alone is the forbidden workaround shape: it converts
+a hard crash into a silent never-moves bot.
+
+## 1. Correction to the ISSUES.md evidence chain
+
+**`entities: 0` in the headless JSON is NOT evidence of failed hydration.**
+`HeadlessDiagnosticWriter.Lifecycle` (`HeadlessDiagnosticWriter.cs:22-45`) is
+called at exactly four points (`HeadlessSessionHost.cs:289, 463, 551, 588`):
+`constructed` (pre-connect), `start-result` (inside `StartLive`, BEFORE the
+first `Runtime.Session.Tick()` drains a packet), `reconnect-deferred`, and
+`stopped` (AFTER teardown emptied the directory). A perfectly healthy run also
+prints `entities: 0` on every lifecycle line. `live: in world — CreateObject
+stream active` (`LiveSessionController.cs:593`) prints before any CreateObject
+routes. The periodic `resources` sampler sums real counts but defaults to a
+30 s period the quarantined run never reached.
+
+**Do NOT hunt for "CreateObjects never admit."** The record almost certainly
+IS registered: `RuntimeLocalPlayerPhysicsPublicationState.Prepare`
+dereferences `record.Key!.Value.LocalEntityId` (`:214`), `CanPrepare` requires
+`_entities.IsCurrent(record)` (`:1052`), and the crash proves a controller was
+constructed. The `entityCount` datum is an artifact.
+
+## 2. Q1 — inbound route and the fork points
+
+Headless inbound route: `WorldSession.EntitySpawned` → `LiveSessionEventRouter`
+→ `RuntimeLiveEntitySessionController.OnSpawned` (`:129`) →
+`RegisterEntityWithInitialResidence` (`:147-152`) → `ApplyAcceptedSpawn`
+(`:158`) → `HeadlessSessionWorldProjection.ProjectSpawn` (`:627`): local player
+→ `_collision.CenterOn(position.LandblockId)` (`:644-648`) then
+`_firstEntry.DriveAll()` (`:666`) → `RuntimeFirstEntryDriveController.DriveOne`
+(`:200`) → `RuntimeLocalPlayerFirstEntryState.Advance` (`:247`).
+
+Per-tick pump (`HeadlessSessionHost.Tick:333-362`): `Clock.Advance` →
+`AdvanceBeforeNetwork` → `Session.Tick()` → `PumpFirstEntry()` →
+`PumpPortalCompletion()` → `RetryPending()` → `RunPostNetworkCommandPhase()` →
+`policy.Tick`.
+
+**No presentation-gated admission step, no missing subscription, no
+reveal/streaming gate on the headless admission path** — C3c (`529e0e9d`)
+unified them. The fork is downstream:
+
+### Fork point 1 (the crash) — `HeadlessLocalPlayerFrameHost.cs:42-44`
+
+```csharp
+public bool CanAdvancePlayer =>
+    _runtime.Session.IsInWorld
+    && _runtime.MovementOwner.Controller is not null;
+```
+
+Graphical host: `LocalPlayerFrameRuntime.cs:24-25` → `CanPresentPlayer` →
+`IsPlayerMode`, entered only when `PlayerModeAutoEntry.cs:100`
+`Controller is { IsRuntimePublished: true }` (the C3c-F2 fix).
+`Controller` becomes non-null at publication Commit
+(`RuntimeLocalPlayerPhysicsPublicationState.Commit:472`) in
+`RuntimeOwnedDormant`; `RuntimePublished` only arrives inside
+`TryApplyDormantLocalActivationFinalCommit` (`RuntimeSetPositionState.cs:2823`
+`controller.ActivateRuntimePublication()`). So headless runs
+`RuntimeLocalPlayerFrameController.AdvanceBeforeNetwork` (`:75`) against a
+dormant controller: `:93` LocalEntityId write (quarantine #1, neutralized by
+`ab82347d`) and `:96-99` Suspend branch → `SuspendObjectUpdate` →
+`EnsurePublishedForRuntimeOperation` throw (quarantine #2, current).
+
+### Fork point 2 (the hydration stall) — `HeadlessSessionWorldProjection.cs:471-548`
+
+`HeadlessCollisionNeighborhood.AdvanceWork`, called from `PumpFirstEntry`
+(`:736-742`) and `ProjectSpawn` (`:644-648`). The headless host's ONLY
+collision publisher is a 3×3 landblock plan **built and started by the local
+player's own CreateObject**. The graphical publisher
+(`LandblockPhysicsPublisher.cs`) runs on the streaming cadence, independent of
+and ahead of the Create burst. That timing difference is the whole asymmetry.
+
+## 3. Q2 — the never-satisfied precondition
+
+### The stall state
+
+Conductor parked at `Stage.PublicationCommitted` in
+`RuntimeLocalPlayerFirstEntryState.AdvanceCore` (`:437-506`), cycling
+`EvaluateActivation` → `AwaitingActivation` forever. `MovementOwner.Controller`
+non-null (Commit ran) but `IsRuntimeOwnedDormant`; `record.PhysicsBody` set,
+`body.InWorld == false` (`Publication.Prepare:358-359`); `record.FullCellId
+== 0` → `ObjectClockDisposition` (`:78-96`) returns `Suspend`;
+`RuntimeFirstEntryDriveController.DriveOne` hits `:264-270` and returns every
+tick. 1:1 match for every symptom.
+
+### The precondition
+
+`EvaluateActivation` (`:496-541`) fails because
+`RuntimeSetPositionState.TryEvaluateDormantLocalActivation` (`:1989-2094`)
+returns false at the seal (`:2061` → `:2088`). `TrySealCollisionEvaluationAuthority`
+(`RuntimePhysicsState.cs:2411-2469`) requires for EVERY prefix the placement's
+ring search queried:
+
+```csharp
+// RuntimePhysicsState.cs:2365-2371
+internal bool IsCollisionEvaluationPrefixAdmissible(uint exactCellId)
+{
+    uint landblockId = CanonicalLandblock(exactCellId);
+    return landblockId != 0u
+        && !_collisionAdmissions.ContainsKey(landblockId)
+        && !SetPosition.IsCollisionPrefixQuiescing(landblockId);
+}
+```
+
+### Why headless cannot satisfy it (structural half — CONFIRMED)
+
+`AdvanceWork` drains a 9-entry plan one commit poll per call, yielding on
+every non-completed `CommitCollisionGeneration` (`:540-543`,
+`HeadlessCollisionGenerationTransaction.Advance:142-156`). While it holds
+`_pendingPublication`, that landblock's `RuntimeCollisionAdmission` IS
+registered, and once `CommitCollisionGeneration` opens
+`BeginCollisionPrefixQuiescence` (`RuntimePhysicsState.cs:1866-1889`) the
+prefix IS quiescing — both admissibility terms false for the duration.
+`PumpFirstEntry` calls `DriveAll()` immediately after `IsReady` (`:739-741`),
+so the conductor evaluates exactly while the admission is open. Same in
+`ProjectSpawn` (`:648` → `:666`). The ring search touches the center + its
+neighbours — every one in the 3×3 plan. The graphical host has the identical
+seal but long admission-free windows. Same class #357 named; headless is
+strictly worse because publication is STARTED by the Create.
+
+### Why it may never open (HIGH-CONFIDENCE HYPOTHESIS — must be confirmed)
+
+Circular wait between the collision commit and remote first-entry operations:
+1. Committing landblock L requires `TryAcquireCollisionPrefixMutationPermission`
+   (`RuntimeSetPositionState.cs:925-972`) → requires
+   `!HasOldPrefixPlacementDebt(current)`.
+2. `HasOldPrefixPlacementDebt` (`:4076-4100`) walks every `_operations` entry,
+   skipping only `WakeableLostCell || DormantLocalActivation`, true if the
+   operation's command/result/mover-preparation accepted position or its
+   record's collision residency touches L.
+3. `RuntimeInitialCreateResidenceState.Own` (`:750-780`) opens an exclusive
+   authored placement operation for EVERY remote Create whose route
+   `PerformsSetPosition`. Those sit in `AwaitingPreparation`, not exempt
+   (they gain `WakeableLostCell` only after successfully submitting and
+   parking; `0934a121` gives them a real body in prefix L, which also trips
+   `IsAffectedCollisionResident`).
+4. A remote's own placement submission seals against the same prefix —
+   refused while L is admitted/quiescing — so it never reaches the park that
+   would exempt it.
+
+→ L never commits; remotes never park; the player's dormant activation never
+wakes. The local player IS exempt from step 2 from publication-Commit onward
+(`PrepareDormantLocalActivationOwnership` sets
+`operation.DormantLocalActivation = true`, `RuntimeSetPositionState.cs:1526`)
+— exactly why the local controller gets built and then freezes.
+
+Other blocking terms checked by the same probe:
+`HasPendingProjectionThrough(ProjectionBarrierSequence)` (`:4059-4062`) — note
+`HeadlessRuntimePlacementProjectionSink.TryApply` deliberately returns FALSE
+(leaves at FIFO head) for Place/Withdraw of an entity still holding an
+initial-create residence (`HeadlessRuntimePlacementProjectionSink.cs:63-81`) —
+and `HasCollisionDispatchDebt` (`:4064-4074`).
+
+### The one line that settles it
+
+`TryRearmDeferredDormantLocalActivation` prints a purpose-built verdict
+enumerating every term (`RuntimeSetPositionState.cs:~2200-2225`), behind
+`PhysicsDiagnostics.ProbeParkEnabled`:
+
+```
+[rearm] guid=... verdict=<not-current | stage=X | not-dormant | not-wakeable
+        | gen-not-ready | proj-seq | gen-mismatch(a!=b) | spawn-not-ready
+        | prefix-inadmissible | OK>
+[rearm] guid=... seal-refused (transient; lease retained)
+[pump]  DriveAll #N pending=K
+[wake]  begin lb=0x... gen=... unboundCells=... buckets=...
+```
+
+And it is unrunnable headless — defect C.
+
+## 4. Q3 — when it broke
+
+| Commit | Date | Headless impact |
+|---|---|---|
+| **`529e0e9d` — C3c production placement cutover (routes 1+8)** | 2026-08-02 | **Prime culprit.** Rewrote the headless local-player path wholesale; deleted `SynchronizeLocalPlayer`/`CreateController`/`ApplySetupStepHeights`; controller now only publishable through the residence→publication→activation→seal chain. Gate list: unit tests + a GRAPHICAL connected gate. **No connected headless gate.** |
+| `175ad6b0` | 2026-08-02 | Same seam, same window. |
+| `9966b531`/`2e8e09ac`/`e0f96a55` | 08-03→05 | Touched HeadlessSessionWorldProjection; graphical-gated only. |
+| `6921a027` C5a, `3aab05b0` #280 | 08-05/06 | Comment-only / shape — not causal. |
+| **`78b981cc` — #357** | 2026-08-08 | **Symptom-shape culprit.** Seal failure reclassified terminal→DeferredCell: pre-#357 headless failed SILENTLY (controller discarded, no crash); post-#357 the dormant controller persists → crash. Correct fix graphically; DO NOT revert. |
+| `ab82347d` | 2026-08-10 | Fixed quarantine #1; filed #365. |
+
+Coverage gap: the one headless hydration test
+(`HeadlessSessionHostTests.WorldProjectionHydratesCanonicalMovementAndTeleportState`,
+`:347-475`) uses a `FixtureCollisionNeighborhood` fake and pre-commits
+collision by calling `SetPosition.BeginCollisionGeneration`/
+`CommitCollisionGeneration` DIRECTLY (`:372-375`) — bypassing admission
+registry, seal, and quiescence. **The production `HeadlessCollisionNeighborhood`
+has never been exercised against the first-entry conductor in any test.**
+
+## 5. Q4 — Suspend / CanAdvancePlayer contract
+
+- **`Suspend` is correct** for a cell-less pre-hydration player (retail's
+  parent/cell-less/Frozen early gate, `PlayerMovementController.cs:1036-1041`).
+  The crash is calling ANY live-movement op on an unpublished controller —
+  the Advance branch would throw identically.
+- **`CanAdvancePlayer` must require publication.** Three confirmations: the
+  graphical host gates on `IsRuntimePublished` (C3c-F2,
+  `PlayerModeAutoEntry.cs:100`); #356 (`972c7ab3`) established
+  `CanExecuteLiveMovement` (`PlayerMovementController.cs:797-805`) as the
+  lifecycle-caller idiom ("including before the controller is published during
+  login… neither is an error"; `MouseLookController.cs:205` is the consumer);
+  and `IsActivationOwnershipEnvelopeCurrent`
+  (`RuntimeLocalPlayerPhysicsPublicationState.cs:900-928`) asserts the dormant
+  controller is untouched across activation — advancing it is semantically
+  wrong, not merely fatal. No dormant-activation classification changes needed;
+  the #357 test matrix stays untouched.
+- There is NO #357 closeout doc in docs/research; its record is commit
+  `78b981cc`'s message + source comments (`RuntimeSetPositionState.cs:2073-2088`,
+  `RuntimeLocalPlayerPhysicsPublicationState.cs:513-534`).
+
+## 6. Q5 — fix plan (ORDERING IS LOAD-BEARING)
+
+### Step 1 (enabler, ~10 lines) — make the diagnosis runnable headless
+`HeadlessStaticStateAudit.cs:15` + call site `HeadlessProcessHost.cs:45`: the
+audit's stated rationale is multi-root isolation; give `ValidateProcessIsolation`
+a `sessionCount` parameter and skip the refusal for `sessionCount == 1`
+(emit a diagnostic Message naming enabled probes). Tests: single-session +
+probe ⇒ starts (and logs); two sessions + probe ⇒ still throws naming the
+probe. Not a workaround — a correctness fix to a guard whose rationale does
+not hold for its condition.
+
+### Step 2 (measurement, no code) — confirm which term is stuck
+Run the repro with `ACDREAM_PROBE_PARK=1`; read `[rearm] verdict=`:
+
+| verdict | Meaning | Fix target |
+|---|---|---|
+| `prefix-inadmissible` persisting | structural claim — admission/quiescence never clears | Step 3 |
+| `gen-not-ready` / `gen-mismatch` | neighborhood commits a generation the lease isn't parked against | narrower fix in HeadlessCollisionNeighborhood |
+| `spawn-not-ready` | 3×3 plan never published the destination cell | BuildPublicationPlan / CreatePublication |
+| `proj-seq` | unacknowledged placement projection — the projection-sink residence gate wedge | drain rule in RuntimeFirstEntryDriveController |
+| `seal-refused` repeating, no verdict line | operation never re-parks; abort in pre-park evaluation | TryEvaluateDormantLocalActivation |
+
+Also capture `[pump] DriveAll #N pending=K`. **This step is mandatory** (the
+C4 closeout's "inferring a fact you can observe" finding applies exactly).
+
+### Step 3 (root cause) — headless collision publication quiescent-before-drive
+Do NOT relax the seal, add a retry budget, or special-case headless inside
+Runtime. The host's publication cadence violates the (correct) precondition.
+
+**3a.** Never drive the conductor while the neighborhood holds an open
+admission or in-flight quiescence. Add read-only
+`IHeadlessCollisionNeighborhood.IsQuiescent`
+(`_pendingPublication is null && _publicationQueue.Count == 0 &&
+!_pendingPublicationCancellation`); gate `PumpFirstEntry` (and
+`ProjectSpawn:666`, `ProjectPosition:690`):
+
+```csharp
+if (!_collision.IsQuiescent)
+    return;                       // publication owns the authority this tick
+_firstEntry?.DriveAll();
+_acceptedPositionDrive?.Advance();
+```
+
+**3b.** If Step 2 shows `HasOldPrefixPlacementDebt` (remote first-entry
+operations) is the blocker, 3a alone won't close it: hoist `CenterOn` off
+`ProjectSpawn` onto the accepted local-player position observed at
+`RegisterEntityCore`'s `Physics.ObserveLocalPlayerCreate` seam, publish the
+login window to completion BEFORE the first Create is projected, hold
+conductor driving until `IsQuiescent` — restoring the graphical ordering
+(world published, then entities placed).
+
+**Explicitly rejected shapes** (workarounds): retry budget on
+AwaitingActivation; forcing `IsCollisionEvaluationPrefixAdmissible` to ignore
+the headless admission; nulling the controller on stall.
+
+### Step 4 (crash guard, AFTER step 3) — `CanAdvancePlayer` requires publication
+`HeadlessLocalPlayerFrameHost.cs:42-44`:
+
+```csharp
+public bool CanAdvancePlayer =>
+    _runtime.Session.IsInWorld
+    && _runtime.MovementOwner.Controller is { CanExecuteLiveMovement: true };
+```
+
+Use the PUBLIC `CanExecuteLiveMovement` (#356 idiom), not internal
+IsRuntimePublished. Optionally harden the shared owner:
+`RuntimeLocalPlayerFrameController.AdvanceBeforeNetwork/RunPostNetworkCommandPhase/
+TryGetPresentationAfterNetwork` (`:75, :135, :166`) add
+`|| !controller.CanExecuteLiveMovement` to the null checks (contract-preserving
+for the graphical host).
+
+### Test list
+1. Runtime: `RuntimeLocalPlayerFrameControllerTests` — host reporting
+   CanAdvancePlayer:true with a DORMANT controller must not throw in either
+   Suspend or Advance branch (sabotage-verify both directions).
+2. `RuntimeLocalPlayerPhysicsPublicationStateTests` — assert UNCHANGED (the
+   seven #357 DeferredCell tests + two terminal tests). If any needs editing,
+   the fix is wrong.
+3. **The missing headless test**: sibling of
+   `WorldProjectionHydratesCanonicalMovementAndTeleportState` driving the REAL
+   `HeadlessCollisionNeighborhood` (or a faithful fake registering a
+   RuntimeCollisionAdmission + opening prefix quiescence across ticks);
+   asserts first-entry reaches Completed and
+   `MovementOwner.Controller.IsRuntimePublished` within a bounded tick count.
+   **Must FAIL on the current tree** — the acceptance criterion.
+4. `PumpFirstEntry` does not call DriveAll while non-quiescent; calls it on
+   the first tick after quiescence.
+5. `CanAdvancePlayer`: false dormant, true published, false retired.
+6. Audit: single-session+probe allowed (logged); multi-session+probe refused.
+7. Keep `HeadlessCollisionNeighborhoodServiceWindowTests`,
+   `HeadlessSessionEventRouteRetryPendingTests` green.
+
+### End-to-end verification recipe
+Config on disk (content-bearing, so the content-less branch at
+`RuntimeLiveEntitySessionController.cs:147` is NOT in play):
+`<scratchpad>/jump-probe-config.json` (testaccount/+Acdream, policy
+jump-probe, credential env ACDREAM_HEADLESS_PASS).
+
+```powershell
+$env:ACDREAM_HEADLESS_PASS = "testpassword"
+$env:ACDREAM_PROBE_PARK    = "1"      # requires Step 1
+dotnet run --project src\AcDream.Headless -c Release -- run --config <cfg> 2>&1 |
+  Tee-Object -FilePath headless-365.log
+```
+
+Cautions: testaccount must not be held by a running graphical client;
+terminate with Ctrl-C/SIGINT (never Stop-Process — graceful Stop() clears the
+ACE session in ~3-5 s; a hard kill costs ~3 min of exit-29).
+
+Pass criteria: (1) no `kind:"failure"`; (2) `[jump-probe] local player
+present; charging jump`; (3) `[pump]` reaches pending=0 and `[rearm]
+verdict=OK` (or clean first-pass seal); (4) `[jump-probe] airborne-transition
+False -> True` — the bot actually moves (the real acceptance test); (5)
+graceful exit, converged final `disposed` sample. Second gate: run the
+`observer-movement` policy ~60 s and watch from the graphical/retail client
+that +Acdream walks — the K3/K4 gate unrun since 2026-08-02.
+
+## 7. Confidence summary
+
+| Claim | Confidence | Falsify cheaply |
+|---|---|---|
+| Crash = dormant controller reaching SuspendObjectUpdate via headless CanAdvancePlayer | Certain | — |
+| Graphical gates on IsRuntimePublished; headless doesn't | Certain | PlayerModeAutoEntry.cs:100 |
+| entities:0 is a logging artifact | Certain | HeadlessDiagnosticWriter call sites |
+| Conductor parked at PublicationCommitted/AwaitingActivation | Very high | [pump] probe |
+| Blocked precondition = IsCollisionEvaluationPrefixAdmissible in the seal | High | [rearm] verdict= |
+| Circular HasOldPrefixPlacementDebt ↔ seal wait makes it permanent | Hypothesis, well-supported | [rearm] verdict= + log on TryAcquireCollisionPrefixMutationPermission's four early returns |
+| 529e0e9d root cause; 78b981cc changed the symptom | High | git show 529e0e9d -- src/AcDream.Headless/ |
+| Probes unrunnable headless | Certain | HeadlessProcessHost.cs:45 |
+| Existing hydration test bypasses admission/seal | Certain | HeadlessSessionHostTests.cs:372-375 |