diff --git a/docs/research/2026-08-10-365-headless-hydration-diagnosis.md b/docs/research/2026-08-10-365-headless-hydration-diagnosis.md new file mode 100644 index 00000000..9cf6c099 --- /dev/null +++ b/docs/research/2026-08-10-365-headless-hydration-diagnosis.md @@ -0,0 +1,355 @@ +# #365 headless hydration — Opus diagnosis (2026-08-10, read-only pass) + +Verbatim agent output. The fix agent works solely from this document. + +## 0. Executive summary + +Three distinct defects layered on top of each other. Only the middle one is +the hydration root cause. + +| # | Layer | Verdict | +|---|---|---| +| **A** | `HeadlessLocalPlayerFrameHost.CanAdvancePlayer` accepts a **dormant (unpublished)** controller; the graphical host does not. This is the crash. | **CONFIRMED by source.** Real host-asymmetry bug. | +| **B** | The headless collision neighborhood publishes collision **as a reaction to the local-player Create**, interleaved with the Create burst. Every `TrySealCollisionEvaluationAuthority` during that window is structurally refused, and (hypothesis) the mutual `HasOldPrefixPlacementDebt` ↔ seal dependency between remote first-entry operations and the landblock commit closes a circular wait that never opens. This is why the controller never publishes. | **Mechanism CONFIRMED; the "never opens" closure is HIGH-CONFIDENCE HYPOTHESIS.** One existing probe answers it in one line. | +| **C** | `HeadlessStaticStateAudit` unconditionally refuses to start when **any** `PhysicsDiagnostics.Probe*`/`Dump*` flag is set — including single-session runs. The exact probe built to diagnose this stall class (`ACDREAM_PROBE_PARK=1`) cannot be run headless at all. | **CONFIRMED.** Why #365 arrived with no `[rearm]`/`[pump]` evidence. | + +**Fix C first (two-line predicate change), then use it to confirm B, then fix +B, then fix A.** Fixing A alone is the forbidden workaround shape: it converts +a hard crash into a silent never-moves bot. + +## 1. Correction to the ISSUES.md evidence chain + +**`entities: 0` in the headless JSON is NOT evidence of failed hydration.** +`HeadlessDiagnosticWriter.Lifecycle` (`HeadlessDiagnosticWriter.cs:22-45`) is +called at exactly four points (`HeadlessSessionHost.cs:289, 463, 551, 588`): +`constructed` (pre-connect), `start-result` (inside `StartLive`, BEFORE the +first `Runtime.Session.Tick()` drains a packet), `reconnect-deferred`, and +`stopped` (AFTER teardown emptied the directory). A perfectly healthy run also +prints `entities: 0` on every lifecycle line. `live: in world — CreateObject +stream active` (`LiveSessionController.cs:593`) prints before any CreateObject +routes. The periodic `resources` sampler sums real counts but defaults to a +30 s period the quarantined run never reached. + +**Do NOT hunt for "CreateObjects never admit."** The record almost certainly +IS registered: `RuntimeLocalPlayerPhysicsPublicationState.Prepare` +dereferences `record.Key!.Value.LocalEntityId` (`:214`), `CanPrepare` requires +`_entities.IsCurrent(record)` (`:1052`), and the crash proves a controller was +constructed. The `entityCount` datum is an artifact. + +## 2. Q1 — inbound route and the fork points + +Headless inbound route: `WorldSession.EntitySpawned` → `LiveSessionEventRouter` +→ `RuntimeLiveEntitySessionController.OnSpawned` (`:129`) → +`RegisterEntityWithInitialResidence` (`:147-152`) → `ApplyAcceptedSpawn` +(`:158`) → `HeadlessSessionWorldProjection.ProjectSpawn` (`:627`): local player +→ `_collision.CenterOn(position.LandblockId)` (`:644-648`) then +`_firstEntry.DriveAll()` (`:666`) → `RuntimeFirstEntryDriveController.DriveOne` +(`:200`) → `RuntimeLocalPlayerFirstEntryState.Advance` (`:247`). + +Per-tick pump (`HeadlessSessionHost.Tick:333-362`): `Clock.Advance` → +`AdvanceBeforeNetwork` → `Session.Tick()` → `PumpFirstEntry()` → +`PumpPortalCompletion()` → `RetryPending()` → `RunPostNetworkCommandPhase()` → +`policy.Tick`. + +**No presentation-gated admission step, no missing subscription, no +reveal/streaming gate on the headless admission path** — C3c (`529e0e9d`) +unified them. The fork is downstream: + +### Fork point 1 (the crash) — `HeadlessLocalPlayerFrameHost.cs:42-44` + +```csharp +public bool CanAdvancePlayer => + _runtime.Session.IsInWorld + && _runtime.MovementOwner.Controller is not null; +``` + +Graphical host: `LocalPlayerFrameRuntime.cs:24-25` → `CanPresentPlayer` → +`IsPlayerMode`, entered only when `PlayerModeAutoEntry.cs:100` +`Controller is { IsRuntimePublished: true }` (the C3c-F2 fix). +`Controller` becomes non-null at publication Commit +(`RuntimeLocalPlayerPhysicsPublicationState.Commit:472`) in +`RuntimeOwnedDormant`; `RuntimePublished` only arrives inside +`TryApplyDormantLocalActivationFinalCommit` (`RuntimeSetPositionState.cs:2823` +`controller.ActivateRuntimePublication()`). So headless runs +`RuntimeLocalPlayerFrameController.AdvanceBeforeNetwork` (`:75`) against a +dormant controller: `:93` LocalEntityId write (quarantine #1, neutralized by +`ab82347d`) and `:96-99` Suspend branch → `SuspendObjectUpdate` → +`EnsurePublishedForRuntimeOperation` throw (quarantine #2, current). + +### Fork point 2 (the hydration stall) — `HeadlessSessionWorldProjection.cs:471-548` + +`HeadlessCollisionNeighborhood.AdvanceWork`, called from `PumpFirstEntry` +(`:736-742`) and `ProjectSpawn` (`:644-648`). The headless host's ONLY +collision publisher is a 3×3 landblock plan **built and started by the local +player's own CreateObject**. The graphical publisher +(`LandblockPhysicsPublisher.cs`) runs on the streaming cadence, independent of +and ahead of the Create burst. That timing difference is the whole asymmetry. + +## 3. Q2 — the never-satisfied precondition + +### The stall state + +Conductor parked at `Stage.PublicationCommitted` in +`RuntimeLocalPlayerFirstEntryState.AdvanceCore` (`:437-506`), cycling +`EvaluateActivation` → `AwaitingActivation` forever. `MovementOwner.Controller` +non-null (Commit ran) but `IsRuntimeOwnedDormant`; `record.PhysicsBody` set, +`body.InWorld == false` (`Publication.Prepare:358-359`); `record.FullCellId +== 0` → `ObjectClockDisposition` (`:78-96`) returns `Suspend`; +`RuntimeFirstEntryDriveController.DriveOne` hits `:264-270` and returns every +tick. 1:1 match for every symptom. + +### The precondition + +`EvaluateActivation` (`:496-541`) fails because +`RuntimeSetPositionState.TryEvaluateDormantLocalActivation` (`:1989-2094`) +returns false at the seal (`:2061` → `:2088`). `TrySealCollisionEvaluationAuthority` +(`RuntimePhysicsState.cs:2411-2469`) requires for EVERY prefix the placement's +ring search queried: + +```csharp +// RuntimePhysicsState.cs:2365-2371 +internal bool IsCollisionEvaluationPrefixAdmissible(uint exactCellId) +{ + uint landblockId = CanonicalLandblock(exactCellId); + return landblockId != 0u + && !_collisionAdmissions.ContainsKey(landblockId) + && !SetPosition.IsCollisionPrefixQuiescing(landblockId); +} +``` + +### Why headless cannot satisfy it (structural half — CONFIRMED) + +`AdvanceWork` drains a 9-entry plan one commit poll per call, yielding on +every non-completed `CommitCollisionGeneration` (`:540-543`, +`HeadlessCollisionGenerationTransaction.Advance:142-156`). While it holds +`_pendingPublication`, that landblock's `RuntimeCollisionAdmission` IS +registered, and once `CommitCollisionGeneration` opens +`BeginCollisionPrefixQuiescence` (`RuntimePhysicsState.cs:1866-1889`) the +prefix IS quiescing — both admissibility terms false for the duration. +`PumpFirstEntry` calls `DriveAll()` immediately after `IsReady` (`:739-741`), +so the conductor evaluates exactly while the admission is open. Same in +`ProjectSpawn` (`:648` → `:666`). The ring search touches the center + its +neighbours — every one in the 3×3 plan. The graphical host has the identical +seal but long admission-free windows. Same class #357 named; headless is +strictly worse because publication is STARTED by the Create. + +### Why it may never open (HIGH-CONFIDENCE HYPOTHESIS — must be confirmed) + +Circular wait between the collision commit and remote first-entry operations: +1. Committing landblock L requires `TryAcquireCollisionPrefixMutationPermission` + (`RuntimeSetPositionState.cs:925-972`) → requires + `!HasOldPrefixPlacementDebt(current)`. +2. `HasOldPrefixPlacementDebt` (`:4076-4100`) walks every `_operations` entry, + skipping only `WakeableLostCell || DormantLocalActivation`, true if the + operation's command/result/mover-preparation accepted position or its + record's collision residency touches L. +3. `RuntimeInitialCreateResidenceState.Own` (`:750-780`) opens an exclusive + authored placement operation for EVERY remote Create whose route + `PerformsSetPosition`. Those sit in `AwaitingPreparation`, not exempt + (they gain `WakeableLostCell` only after successfully submitting and + parking; `0934a121` gives them a real body in prefix L, which also trips + `IsAffectedCollisionResident`). +4. A remote's own placement submission seals against the same prefix — + refused while L is admitted/quiescing — so it never reaches the park that + would exempt it. + +→ L never commits; remotes never park; the player's dormant activation never +wakes. The local player IS exempt from step 2 from publication-Commit onward +(`PrepareDormantLocalActivationOwnership` sets +`operation.DormantLocalActivation = true`, `RuntimeSetPositionState.cs:1526`) +— exactly why the local controller gets built and then freezes. + +Other blocking terms checked by the same probe: +`HasPendingProjectionThrough(ProjectionBarrierSequence)` (`:4059-4062`) — note +`HeadlessRuntimePlacementProjectionSink.TryApply` deliberately returns FALSE +(leaves at FIFO head) for Place/Withdraw of an entity still holding an +initial-create residence (`HeadlessRuntimePlacementProjectionSink.cs:63-81`) — +and `HasCollisionDispatchDebt` (`:4064-4074`). + +### The one line that settles it + +`TryRearmDeferredDormantLocalActivation` prints a purpose-built verdict +enumerating every term (`RuntimeSetPositionState.cs:~2200-2225`), behind +`PhysicsDiagnostics.ProbeParkEnabled`: + +``` +[rearm] guid=... verdict= +[rearm] guid=... seal-refused (transient; lease retained) +[pump] DriveAll #N pending=K +[wake] begin lb=0x... gen=... unboundCells=... buckets=... +``` + +And it is unrunnable headless — defect C. + +## 4. Q3 — when it broke + +| Commit | Date | Headless impact | +|---|---|---| +| **`529e0e9d` — C3c production placement cutover (routes 1+8)** | 2026-08-02 | **Prime culprit.** Rewrote the headless local-player path wholesale; deleted `SynchronizeLocalPlayer`/`CreateController`/`ApplySetupStepHeights`; controller now only publishable through the residence→publication→activation→seal chain. Gate list: unit tests + a GRAPHICAL connected gate. **No connected headless gate.** | +| `175ad6b0` | 2026-08-02 | Same seam, same window. | +| `9966b531`/`2e8e09ac`/`e0f96a55` | 08-03→05 | Touched HeadlessSessionWorldProjection; graphical-gated only. | +| `6921a027` C5a, `3aab05b0` #280 | 08-05/06 | Comment-only / shape — not causal. | +| **`78b981cc` — #357** | 2026-08-08 | **Symptom-shape culprit.** Seal failure reclassified terminal→DeferredCell: pre-#357 headless failed SILENTLY (controller discarded, no crash); post-#357 the dormant controller persists → crash. Correct fix graphically; DO NOT revert. | +| `ab82347d` | 2026-08-10 | Fixed quarantine #1; filed #365. | + +Coverage gap: the one headless hydration test +(`HeadlessSessionHostTests.WorldProjectionHydratesCanonicalMovementAndTeleportState`, +`:347-475`) uses a `FixtureCollisionNeighborhood` fake and pre-commits +collision by calling `SetPosition.BeginCollisionGeneration`/ +`CommitCollisionGeneration` DIRECTLY (`:372-375`) — bypassing admission +registry, seal, and quiescence. **The production `HeadlessCollisionNeighborhood` +has never been exercised against the first-entry conductor in any test.** + +## 5. Q4 — Suspend / CanAdvancePlayer contract + +- **`Suspend` is correct** for a cell-less pre-hydration player (retail's + parent/cell-less/Frozen early gate, `PlayerMovementController.cs:1036-1041`). + The crash is calling ANY live-movement op on an unpublished controller — + the Advance branch would throw identically. +- **`CanAdvancePlayer` must require publication.** Three confirmations: the + graphical host gates on `IsRuntimePublished` (C3c-F2, + `PlayerModeAutoEntry.cs:100`); #356 (`972c7ab3`) established + `CanExecuteLiveMovement` (`PlayerMovementController.cs:797-805`) as the + lifecycle-caller idiom ("including before the controller is published during + login… neither is an error"; `MouseLookController.cs:205` is the consumer); + and `IsActivationOwnershipEnvelopeCurrent` + (`RuntimeLocalPlayerPhysicsPublicationState.cs:900-928`) asserts the dormant + controller is untouched across activation — advancing it is semantically + wrong, not merely fatal. No dormant-activation classification changes needed; + the #357 test matrix stays untouched. +- There is NO #357 closeout doc in docs/research; its record is commit + `78b981cc`'s message + source comments (`RuntimeSetPositionState.cs:2073-2088`, + `RuntimeLocalPlayerPhysicsPublicationState.cs:513-534`). + +## 6. Q5 — fix plan (ORDERING IS LOAD-BEARING) + +### Step 1 (enabler, ~10 lines) — make the diagnosis runnable headless +`HeadlessStaticStateAudit.cs:15` + call site `HeadlessProcessHost.cs:45`: the +audit's stated rationale is multi-root isolation; give `ValidateProcessIsolation` +a `sessionCount` parameter and skip the refusal for `sessionCount == 1` +(emit a diagnostic Message naming enabled probes). Tests: single-session + +probe ⇒ starts (and logs); two sessions + probe ⇒ still throws naming the +probe. Not a workaround — a correctness fix to a guard whose rationale does +not hold for its condition. + +### Step 2 (measurement, no code) — confirm which term is stuck +Run the repro with `ACDREAM_PROBE_PARK=1`; read `[rearm] verdict=`: + +| verdict | Meaning | Fix target | +|---|---|---| +| `prefix-inadmissible` persisting | structural claim — admission/quiescence never clears | Step 3 | +| `gen-not-ready` / `gen-mismatch` | neighborhood commits a generation the lease isn't parked against | narrower fix in HeadlessCollisionNeighborhood | +| `spawn-not-ready` | 3×3 plan never published the destination cell | BuildPublicationPlan / CreatePublication | +| `proj-seq` | unacknowledged placement projection — the projection-sink residence gate wedge | drain rule in RuntimeFirstEntryDriveController | +| `seal-refused` repeating, no verdict line | operation never re-parks; abort in pre-park evaluation | TryEvaluateDormantLocalActivation | + +Also capture `[pump] DriveAll #N pending=K`. **This step is mandatory** (the +C4 closeout's "inferring a fact you can observe" finding applies exactly). + +### Step 3 (root cause) — headless collision publication quiescent-before-drive +Do NOT relax the seal, add a retry budget, or special-case headless inside +Runtime. The host's publication cadence violates the (correct) precondition. + +**3a.** Never drive the conductor while the neighborhood holds an open +admission or in-flight quiescence. Add read-only +`IHeadlessCollisionNeighborhood.IsQuiescent` +(`_pendingPublication is null && _publicationQueue.Count == 0 && +!_pendingPublicationCancellation`); gate `PumpFirstEntry` (and +`ProjectSpawn:666`, `ProjectPosition:690`): + +```csharp +if (!_collision.IsQuiescent) + return; // publication owns the authority this tick +_firstEntry?.DriveAll(); +_acceptedPositionDrive?.Advance(); +``` + +**3b.** If Step 2 shows `HasOldPrefixPlacementDebt` (remote first-entry +operations) is the blocker, 3a alone won't close it: hoist `CenterOn` off +`ProjectSpawn` onto the accepted local-player position observed at +`RegisterEntityCore`'s `Physics.ObserveLocalPlayerCreate` seam, publish the +login window to completion BEFORE the first Create is projected, hold +conductor driving until `IsQuiescent` — restoring the graphical ordering +(world published, then entities placed). + +**Explicitly rejected shapes** (workarounds): retry budget on +AwaitingActivation; forcing `IsCollisionEvaluationPrefixAdmissible` to ignore +the headless admission; nulling the controller on stall. + +### Step 4 (crash guard, AFTER step 3) — `CanAdvancePlayer` requires publication +`HeadlessLocalPlayerFrameHost.cs:42-44`: + +```csharp +public bool CanAdvancePlayer => + _runtime.Session.IsInWorld + && _runtime.MovementOwner.Controller is { CanExecuteLiveMovement: true }; +``` + +Use the PUBLIC `CanExecuteLiveMovement` (#356 idiom), not internal +IsRuntimePublished. Optionally harden the shared owner: +`RuntimeLocalPlayerFrameController.AdvanceBeforeNetwork/RunPostNetworkCommandPhase/ +TryGetPresentationAfterNetwork` (`:75, :135, :166`) add +`|| !controller.CanExecuteLiveMovement` to the null checks (contract-preserving +for the graphical host). + +### Test list +1. Runtime: `RuntimeLocalPlayerFrameControllerTests` — host reporting + CanAdvancePlayer:true with a DORMANT controller must not throw in either + Suspend or Advance branch (sabotage-verify both directions). +2. `RuntimeLocalPlayerPhysicsPublicationStateTests` — assert UNCHANGED (the + seven #357 DeferredCell tests + two terminal tests). If any needs editing, + the fix is wrong. +3. **The missing headless test**: sibling of + `WorldProjectionHydratesCanonicalMovementAndTeleportState` driving the REAL + `HeadlessCollisionNeighborhood` (or a faithful fake registering a + RuntimeCollisionAdmission + opening prefix quiescence across ticks); + asserts first-entry reaches Completed and + `MovementOwner.Controller.IsRuntimePublished` within a bounded tick count. + **Must FAIL on the current tree** — the acceptance criterion. +4. `PumpFirstEntry` does not call DriveAll while non-quiescent; calls it on + the first tick after quiescence. +5. `CanAdvancePlayer`: false dormant, true published, false retired. +6. Audit: single-session+probe allowed (logged); multi-session+probe refused. +7. Keep `HeadlessCollisionNeighborhoodServiceWindowTests`, + `HeadlessSessionEventRouteRetryPendingTests` green. + +### End-to-end verification recipe +Config on disk (content-bearing, so the content-less branch at +`RuntimeLiveEntitySessionController.cs:147` is NOT in play): +`/jump-probe-config.json` (testaccount/+Acdream, policy +jump-probe, credential env ACDREAM_HEADLESS_PASS). + +```powershell +$env:ACDREAM_HEADLESS_PASS = "testpassword" +$env:ACDREAM_PROBE_PARK = "1" # requires Step 1 +dotnet run --project src\AcDream.Headless -c Release -- run --config 2>&1 | + Tee-Object -FilePath headless-365.log +``` + +Cautions: testaccount must not be held by a running graphical client; +terminate with Ctrl-C/SIGINT (never Stop-Process — graceful Stop() clears the +ACE session in ~3-5 s; a hard kill costs ~3 min of exit-29). + +Pass criteria: (1) no `kind:"failure"`; (2) `[jump-probe] local player +present; charging jump`; (3) `[pump]` reaches pending=0 and `[rearm] +verdict=OK` (or clean first-pass seal); (4) `[jump-probe] airborne-transition +False -> True` — the bot actually moves (the real acceptance test); (5) +graceful exit, converged final `disposed` sample. Second gate: run the +`observer-movement` policy ~60 s and watch from the graphical/retail client +that +Acdream walks — the K3/K4 gate unrun since 2026-08-02. + +## 7. Confidence summary + +| Claim | Confidence | Falsify cheaply | +|---|---|---| +| Crash = dormant controller reaching SuspendObjectUpdate via headless CanAdvancePlayer | Certain | — | +| Graphical gates on IsRuntimePublished; headless doesn't | Certain | PlayerModeAutoEntry.cs:100 | +| entities:0 is a logging artifact | Certain | HeadlessDiagnosticWriter call sites | +| Conductor parked at PublicationCommitted/AwaitingActivation | Very high | [pump] probe | +| Blocked precondition = IsCollisionEvaluationPrefixAdmissible in the seal | High | [rearm] verdict= | +| Circular HasOldPrefixPlacementDebt ↔ seal wait makes it permanent | Hypothesis, well-supported | [rearm] verdict= + log on TryAcquireCollisionPrefixMutationPermission's four early returns | +| 529e0e9d root cause; 78b981cc changed the symptom | High | git show 529e0e9d -- src/AcDream.Headless/ | +| Probes unrunnable headless | Certain | HeadlessProcessHost.cs:45 | +| Existing hydration test bypasses admission/seal | Certain | HeadlessSessionHostTests.cs:372-375 |