#368's entry now records the fix mechanism (dedicated acdream-headless-update thread owning Start + every scheduler turn; synchronous TimeProvider-timer scheduler loop; guard untouched, zero shared Runtime changes) and the 3/3 live-ACE verification vs the 3/3 pre-fix quarantines. The #365 entry and diagnosis doc get dated pointers: their open question is answered — the airborne residual persists with threads provably single, refuting the unsynchronized-thread hypothesis — and is split off as #370 with the evidence and starting points. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
462 lines
26 KiB
Markdown
462 lines
26 KiB
Markdown
# #365 headless hydration — Opus diagnosis (2026-08-10, read-only pass)
|
||
|
||
Verbatim agent output. The fix agent works solely from this document.
|
||
|
||
## 0. Executive summary
|
||
|
||
Three distinct defects layered on top of each other. Only the middle one is
|
||
the hydration root cause.
|
||
|
||
| # | Layer | Verdict |
|
||
|---|---|---|
|
||
| **A** | `HeadlessLocalPlayerFrameHost.CanAdvancePlayer` accepts a **dormant (unpublished)** controller; the graphical host does not. This is the crash. | **CONFIRMED by source.** Real host-asymmetry bug. |
|
||
| **B** | The headless collision neighborhood publishes collision **as a reaction to the local-player Create**, interleaved with the Create burst. Every `TrySealCollisionEvaluationAuthority` during that window is structurally refused, and (hypothesis) the mutual `HasOldPrefixPlacementDebt` ↔ seal dependency between remote first-entry operations and the landblock commit closes a circular wait that never opens. This is why the controller never publishes. | **Mechanism CONFIRMED; the "never opens" closure is HIGH-CONFIDENCE HYPOTHESIS.** One existing probe answers it in one line. |
|
||
| **C** | `HeadlessStaticStateAudit` unconditionally refuses to start when **any** `PhysicsDiagnostics.Probe*`/`Dump*` flag is set — including single-session runs. The exact probe built to diagnose this stall class (`ACDREAM_PROBE_PARK=1`) cannot be run headless at all. | **CONFIRMED.** Why #365 arrived with no `[rearm]`/`[pump]` evidence. |
|
||
|
||
**Fix C first (two-line predicate change), then use it to confirm B, then fix
|
||
B, then fix A.** Fixing A alone is the forbidden workaround shape: it converts
|
||
a hard crash into a silent never-moves bot.
|
||
|
||
## 1. Correction to the ISSUES.md evidence chain
|
||
|
||
**`entities: 0` in the headless JSON is NOT evidence of failed hydration.**
|
||
`HeadlessDiagnosticWriter.Lifecycle` (`HeadlessDiagnosticWriter.cs:22-45`) is
|
||
called at exactly four points (`HeadlessSessionHost.cs:289, 463, 551, 588`):
|
||
`constructed` (pre-connect), `start-result` (inside `StartLive`, BEFORE the
|
||
first `Runtime.Session.Tick()` drains a packet), `reconnect-deferred`, and
|
||
`stopped` (AFTER teardown emptied the directory). A perfectly healthy run also
|
||
prints `entities: 0` on every lifecycle line. `live: in world — CreateObject
|
||
stream active` (`LiveSessionController.cs:593`) prints before any CreateObject
|
||
routes. The periodic `resources` sampler sums real counts but defaults to a
|
||
30 s period the quarantined run never reached.
|
||
|
||
**Do NOT hunt for "CreateObjects never admit."** The record almost certainly
|
||
IS registered: `RuntimeLocalPlayerPhysicsPublicationState.Prepare`
|
||
dereferences `record.Key!.Value.LocalEntityId` (`:214`), `CanPrepare` requires
|
||
`_entities.IsCurrent(record)` (`:1052`), and the crash proves a controller was
|
||
constructed. The `entityCount` datum is an artifact.
|
||
|
||
## 2. Q1 — inbound route and the fork points
|
||
|
||
Headless inbound route: `WorldSession.EntitySpawned` → `LiveSessionEventRouter`
|
||
→ `RuntimeLiveEntitySessionController.OnSpawned` (`:129`) →
|
||
`RegisterEntityWithInitialResidence` (`:147-152`) → `ApplyAcceptedSpawn`
|
||
(`:158`) → `HeadlessSessionWorldProjection.ProjectSpawn` (`:627`): local player
|
||
→ `_collision.CenterOn(position.LandblockId)` (`:644-648`) then
|
||
`_firstEntry.DriveAll()` (`:666`) → `RuntimeFirstEntryDriveController.DriveOne`
|
||
(`:200`) → `RuntimeLocalPlayerFirstEntryState.Advance` (`:247`).
|
||
|
||
Per-tick pump (`HeadlessSessionHost.Tick:333-362`): `Clock.Advance` →
|
||
`AdvanceBeforeNetwork` → `Session.Tick()` → `PumpFirstEntry()` →
|
||
`PumpPortalCompletion()` → `RetryPending()` → `RunPostNetworkCommandPhase()` →
|
||
`policy.Tick`.
|
||
|
||
**No presentation-gated admission step, no missing subscription, no
|
||
reveal/streaming gate on the headless admission path** — C3c (`529e0e9d`)
|
||
unified them. The fork is downstream:
|
||
|
||
### Fork point 1 (the crash) — `HeadlessLocalPlayerFrameHost.cs:42-44`
|
||
|
||
```csharp
|
||
public bool CanAdvancePlayer =>
|
||
_runtime.Session.IsInWorld
|
||
&& _runtime.MovementOwner.Controller is not null;
|
||
```
|
||
|
||
Graphical host: `LocalPlayerFrameRuntime.cs:24-25` → `CanPresentPlayer` →
|
||
`IsPlayerMode`, entered only when `PlayerModeAutoEntry.cs:100`
|
||
`Controller is { IsRuntimePublished: true }` (the C3c-F2 fix).
|
||
`Controller` becomes non-null at publication Commit
|
||
(`RuntimeLocalPlayerPhysicsPublicationState.Commit:472`) in
|
||
`RuntimeOwnedDormant`; `RuntimePublished` only arrives inside
|
||
`TryApplyDormantLocalActivationFinalCommit` (`RuntimeSetPositionState.cs:2823`
|
||
`controller.ActivateRuntimePublication()`). So headless runs
|
||
`RuntimeLocalPlayerFrameController.AdvanceBeforeNetwork` (`:75`) against a
|
||
dormant controller: `:93` LocalEntityId write (quarantine #1, neutralized by
|
||
`ab82347d`) and `:96-99` Suspend branch → `SuspendObjectUpdate` →
|
||
`EnsurePublishedForRuntimeOperation` throw (quarantine #2, current).
|
||
|
||
### Fork point 2 (the hydration stall) — `HeadlessSessionWorldProjection.cs:471-548`
|
||
|
||
`HeadlessCollisionNeighborhood.AdvanceWork`, called from `PumpFirstEntry`
|
||
(`:736-742`) and `ProjectSpawn` (`:644-648`). The headless host's ONLY
|
||
collision publisher is a 3×3 landblock plan **built and started by the local
|
||
player's own CreateObject**. The graphical publisher
|
||
(`LandblockPhysicsPublisher.cs`) runs on the streaming cadence, independent of
|
||
and ahead of the Create burst. That timing difference is the whole asymmetry.
|
||
|
||
## 3. Q2 — the never-satisfied precondition
|
||
|
||
### The stall state
|
||
|
||
Conductor parked at `Stage.PublicationCommitted` in
|
||
`RuntimeLocalPlayerFirstEntryState.AdvanceCore` (`:437-506`), cycling
|
||
`EvaluateActivation` → `AwaitingActivation` forever. `MovementOwner.Controller`
|
||
non-null (Commit ran) but `IsRuntimeOwnedDormant`; `record.PhysicsBody` set,
|
||
`body.InWorld == false` (`Publication.Prepare:358-359`); `record.FullCellId
|
||
== 0` → `ObjectClockDisposition` (`:78-96`) returns `Suspend`;
|
||
`RuntimeFirstEntryDriveController.DriveOne` hits `:264-270` and returns every
|
||
tick. 1:1 match for every symptom.
|
||
|
||
### The precondition
|
||
|
||
`EvaluateActivation` (`:496-541`) fails because
|
||
`RuntimeSetPositionState.TryEvaluateDormantLocalActivation` (`:1989-2094`)
|
||
returns false at the seal (`:2061` → `:2088`). `TrySealCollisionEvaluationAuthority`
|
||
(`RuntimePhysicsState.cs:2411-2469`) requires for EVERY prefix the placement's
|
||
ring search queried:
|
||
|
||
```csharp
|
||
// RuntimePhysicsState.cs:2365-2371
|
||
internal bool IsCollisionEvaluationPrefixAdmissible(uint exactCellId)
|
||
{
|
||
uint landblockId = CanonicalLandblock(exactCellId);
|
||
return landblockId != 0u
|
||
&& !_collisionAdmissions.ContainsKey(landblockId)
|
||
&& !SetPosition.IsCollisionPrefixQuiescing(landblockId);
|
||
}
|
||
```
|
||
|
||
### Why headless cannot satisfy it (structural half — CONFIRMED)
|
||
|
||
`AdvanceWork` drains a 9-entry plan one commit poll per call, yielding on
|
||
every non-completed `CommitCollisionGeneration` (`:540-543`,
|
||
`HeadlessCollisionGenerationTransaction.Advance:142-156`). While it holds
|
||
`_pendingPublication`, that landblock's `RuntimeCollisionAdmission` IS
|
||
registered, and once `CommitCollisionGeneration` opens
|
||
`BeginCollisionPrefixQuiescence` (`RuntimePhysicsState.cs:1866-1889`) the
|
||
prefix IS quiescing — both admissibility terms false for the duration.
|
||
`PumpFirstEntry` calls `DriveAll()` immediately after `IsReady` (`:739-741`),
|
||
so the conductor evaluates exactly while the admission is open. Same in
|
||
`ProjectSpawn` (`:648` → `:666`). The ring search touches the center + its
|
||
neighbours — every one in the 3×3 plan. The graphical host has the identical
|
||
seal but long admission-free windows. Same class #357 named; headless is
|
||
strictly worse because publication is STARTED by the Create.
|
||
|
||
### Why it may never open (HIGH-CONFIDENCE HYPOTHESIS — must be confirmed)
|
||
|
||
Circular wait between the collision commit and remote first-entry operations:
|
||
1. Committing landblock L requires `TryAcquireCollisionPrefixMutationPermission`
|
||
(`RuntimeSetPositionState.cs:925-972`) → requires
|
||
`!HasOldPrefixPlacementDebt(current)`.
|
||
2. `HasOldPrefixPlacementDebt` (`:4076-4100`) walks every `_operations` entry,
|
||
skipping only `WakeableLostCell || DormantLocalActivation`, true if the
|
||
operation's command/result/mover-preparation accepted position or its
|
||
record's collision residency touches L.
|
||
3. `RuntimeInitialCreateResidenceState.Own` (`:750-780`) opens an exclusive
|
||
authored placement operation for EVERY remote Create whose route
|
||
`PerformsSetPosition`. Those sit in `AwaitingPreparation`, not exempt
|
||
(they gain `WakeableLostCell` only after successfully submitting and
|
||
parking; `0934a121` gives them a real body in prefix L, which also trips
|
||
`IsAffectedCollisionResident`).
|
||
4. A remote's own placement submission seals against the same prefix —
|
||
refused while L is admitted/quiescing — so it never reaches the park that
|
||
would exempt it.
|
||
|
||
→ L never commits; remotes never park; the player's dormant activation never
|
||
wakes. The local player IS exempt from step 2 from publication-Commit onward
|
||
(`PrepareDormantLocalActivationOwnership` sets
|
||
`operation.DormantLocalActivation = true`, `RuntimeSetPositionState.cs:1526`)
|
||
— exactly why the local controller gets built and then freezes.
|
||
|
||
Other blocking terms checked by the same probe:
|
||
`HasPendingProjectionThrough(ProjectionBarrierSequence)` (`:4059-4062`) — note
|
||
`HeadlessRuntimePlacementProjectionSink.TryApply` deliberately returns FALSE
|
||
(leaves at FIFO head) for Place/Withdraw of an entity still holding an
|
||
initial-create residence (`HeadlessRuntimePlacementProjectionSink.cs:63-81`) —
|
||
and `HasCollisionDispatchDebt` (`:4064-4074`).
|
||
|
||
### The one line that settles it
|
||
|
||
`TryRearmDeferredDormantLocalActivation` prints a purpose-built verdict
|
||
enumerating every term (`RuntimeSetPositionState.cs:~2200-2225`), behind
|
||
`PhysicsDiagnostics.ProbeParkEnabled`:
|
||
|
||
```
|
||
[rearm] guid=... verdict=<not-current | stage=X | not-dormant | not-wakeable
|
||
| gen-not-ready | proj-seq | gen-mismatch(a!=b) | spawn-not-ready
|
||
| prefix-inadmissible | OK>
|
||
[rearm] guid=... seal-refused (transient; lease retained)
|
||
[pump] DriveAll #N pending=K
|
||
[wake] begin lb=0x... gen=... unboundCells=... buckets=...
|
||
```
|
||
|
||
And it is unrunnable headless — defect C.
|
||
|
||
## 4. Q3 — when it broke
|
||
|
||
| Commit | Date | Headless impact |
|
||
|---|---|---|
|
||
| **`529e0e9d` — C3c production placement cutover (routes 1+8)** | 2026-08-02 | **Prime culprit.** Rewrote the headless local-player path wholesale; deleted `SynchronizeLocalPlayer`/`CreateController`/`ApplySetupStepHeights`; controller now only publishable through the residence→publication→activation→seal chain. Gate list: unit tests + a GRAPHICAL connected gate. **No connected headless gate.** |
|
||
| `175ad6b0` | 2026-08-02 | Same seam, same window. |
|
||
| `9966b531`/`2e8e09ac`/`e0f96a55` | 08-03→05 | Touched HeadlessSessionWorldProjection; graphical-gated only. |
|
||
| `6921a027` C5a, `3aab05b0` #280 | 08-05/06 | Comment-only / shape — not causal. |
|
||
| **`78b981cc` — #357** | 2026-08-08 | **Symptom-shape culprit.** Seal failure reclassified terminal→DeferredCell: pre-#357 headless failed SILENTLY (controller discarded, no crash); post-#357 the dormant controller persists → crash. Correct fix graphically; DO NOT revert. |
|
||
| `ab82347d` | 2026-08-10 | Fixed quarantine #1; filed #365. |
|
||
|
||
Coverage gap: the one headless hydration test
|
||
(`HeadlessSessionHostTests.WorldProjectionHydratesCanonicalMovementAndTeleportState`,
|
||
`:347-475`) uses a `FixtureCollisionNeighborhood` fake and pre-commits
|
||
collision by calling `SetPosition.BeginCollisionGeneration`/
|
||
`CommitCollisionGeneration` DIRECTLY (`:372-375`) — bypassing admission
|
||
registry, seal, and quiescence. **The production `HeadlessCollisionNeighborhood`
|
||
has never been exercised against the first-entry conductor in any test.**
|
||
|
||
## 5. Q4 — Suspend / CanAdvancePlayer contract
|
||
|
||
- **`Suspend` is correct** for a cell-less pre-hydration player (retail's
|
||
parent/cell-less/Frozen early gate, `PlayerMovementController.cs:1036-1041`).
|
||
The crash is calling ANY live-movement op on an unpublished controller —
|
||
the Advance branch would throw identically.
|
||
- **`CanAdvancePlayer` must require publication.** Three confirmations: the
|
||
graphical host gates on `IsRuntimePublished` (C3c-F2,
|
||
`PlayerModeAutoEntry.cs:100`); #356 (`972c7ab3`) established
|
||
`CanExecuteLiveMovement` (`PlayerMovementController.cs:797-805`) as the
|
||
lifecycle-caller idiom ("including before the controller is published during
|
||
login… neither is an error"; `MouseLookController.cs:205` is the consumer);
|
||
and `IsActivationOwnershipEnvelopeCurrent`
|
||
(`RuntimeLocalPlayerPhysicsPublicationState.cs:900-928`) asserts the dormant
|
||
controller is untouched across activation — advancing it is semantically
|
||
wrong, not merely fatal. No dormant-activation classification changes needed;
|
||
the #357 test matrix stays untouched.
|
||
- There is NO #357 closeout doc in docs/research; its record is commit
|
||
`78b981cc`'s message + source comments (`RuntimeSetPositionState.cs:2073-2088`,
|
||
`RuntimeLocalPlayerPhysicsPublicationState.cs:513-534`).
|
||
|
||
## 6. Q5 — fix plan (ORDERING IS LOAD-BEARING)
|
||
|
||
### Step 1 (enabler, ~10 lines) — make the diagnosis runnable headless
|
||
`HeadlessStaticStateAudit.cs:15` + call site `HeadlessProcessHost.cs:45`: the
|
||
audit's stated rationale is multi-root isolation; give `ValidateProcessIsolation`
|
||
a `sessionCount` parameter and skip the refusal for `sessionCount == 1`
|
||
(emit a diagnostic Message naming enabled probes). Tests: single-session +
|
||
probe ⇒ starts (and logs); two sessions + probe ⇒ still throws naming the
|
||
probe. Not a workaround — a correctness fix to a guard whose rationale does
|
||
not hold for its condition.
|
||
|
||
### Step 2 (measurement, no code) — confirm which term is stuck
|
||
Run the repro with `ACDREAM_PROBE_PARK=1`; read `[rearm] verdict=`:
|
||
|
||
| verdict | Meaning | Fix target |
|
||
|---|---|---|
|
||
| `prefix-inadmissible` persisting | structural claim — admission/quiescence never clears | Step 3 |
|
||
| `gen-not-ready` / `gen-mismatch` | neighborhood commits a generation the lease isn't parked against | narrower fix in HeadlessCollisionNeighborhood |
|
||
| `spawn-not-ready` | 3×3 plan never published the destination cell | BuildPublicationPlan / CreatePublication |
|
||
| `proj-seq` | unacknowledged placement projection — the projection-sink residence gate wedge | drain rule in RuntimeFirstEntryDriveController |
|
||
| `seal-refused` repeating, no verdict line | operation never re-parks; abort in pre-park evaluation | TryEvaluateDormantLocalActivation |
|
||
|
||
Also capture `[pump] DriveAll #N pending=K`. **This step is mandatory** (the
|
||
C4 closeout's "inferring a fact you can observe" finding applies exactly).
|
||
|
||
### Step 3 (root cause) — headless collision publication quiescent-before-drive
|
||
Do NOT relax the seal, add a retry budget, or special-case headless inside
|
||
Runtime. The host's publication cadence violates the (correct) precondition.
|
||
|
||
**3a.** Never drive the conductor while the neighborhood holds an open
|
||
admission or in-flight quiescence. Add read-only
|
||
`IHeadlessCollisionNeighborhood.IsQuiescent`
|
||
(`_pendingPublication is null && _publicationQueue.Count == 0 &&
|
||
!_pendingPublicationCancellation`); gate `PumpFirstEntry` (and
|
||
`ProjectSpawn:666`, `ProjectPosition:690`):
|
||
|
||
```csharp
|
||
if (!_collision.IsQuiescent)
|
||
return; // publication owns the authority this tick
|
||
_firstEntry?.DriveAll();
|
||
_acceptedPositionDrive?.Advance();
|
||
```
|
||
|
||
**3b.** If Step 2 shows `HasOldPrefixPlacementDebt` (remote first-entry
|
||
operations) is the blocker, 3a alone won't close it: hoist `CenterOn` off
|
||
`ProjectSpawn` onto the accepted local-player position observed at
|
||
`RegisterEntityCore`'s `Physics.ObserveLocalPlayerCreate` seam, publish the
|
||
login window to completion BEFORE the first Create is projected, hold
|
||
conductor driving until `IsQuiescent` — restoring the graphical ordering
|
||
(world published, then entities placed).
|
||
|
||
**Explicitly rejected shapes** (workarounds): retry budget on
|
||
AwaitingActivation; forcing `IsCollisionEvaluationPrefixAdmissible` to ignore
|
||
the headless admission; nulling the controller on stall.
|
||
|
||
### Step 4 (crash guard, AFTER step 3) — `CanAdvancePlayer` requires publication
|
||
`HeadlessLocalPlayerFrameHost.cs:42-44`:
|
||
|
||
```csharp
|
||
public bool CanAdvancePlayer =>
|
||
_runtime.Session.IsInWorld
|
||
&& _runtime.MovementOwner.Controller is { CanExecuteLiveMovement: true };
|
||
```
|
||
|
||
Use the PUBLIC `CanExecuteLiveMovement` (#356 idiom), not internal
|
||
IsRuntimePublished. Optionally harden the shared owner:
|
||
`RuntimeLocalPlayerFrameController.AdvanceBeforeNetwork/RunPostNetworkCommandPhase/
|
||
TryGetPresentationAfterNetwork` (`:75, :135, :166`) add
|
||
`|| !controller.CanExecuteLiveMovement` to the null checks (contract-preserving
|
||
for the graphical host).
|
||
|
||
### Test list
|
||
1. Runtime: `RuntimeLocalPlayerFrameControllerTests` — host reporting
|
||
CanAdvancePlayer:true with a DORMANT controller must not throw in either
|
||
Suspend or Advance branch (sabotage-verify both directions).
|
||
2. `RuntimeLocalPlayerPhysicsPublicationStateTests` — assert UNCHANGED (the
|
||
seven #357 DeferredCell tests + two terminal tests). If any needs editing,
|
||
the fix is wrong.
|
||
3. **The missing headless test**: sibling of
|
||
`WorldProjectionHydratesCanonicalMovementAndTeleportState` driving the REAL
|
||
`HeadlessCollisionNeighborhood` (or a faithful fake registering a
|
||
RuntimeCollisionAdmission + opening prefix quiescence across ticks);
|
||
asserts first-entry reaches Completed and
|
||
`MovementOwner.Controller.IsRuntimePublished` within a bounded tick count.
|
||
**Must FAIL on the current tree** — the acceptance criterion.
|
||
4. `PumpFirstEntry` does not call DriveAll while non-quiescent; calls it on
|
||
the first tick after quiescence.
|
||
5. `CanAdvancePlayer`: false dormant, true published, false retired.
|
||
6. Audit: single-session+probe allowed (logged); multi-session+probe refused.
|
||
7. Keep `HeadlessCollisionNeighborhoodServiceWindowTests`,
|
||
`HeadlessSessionEventRouteRetryPendingTests` green.
|
||
|
||
### End-to-end verification recipe
|
||
Config on disk (content-bearing, so the content-less branch at
|
||
`RuntimeLiveEntitySessionController.cs:147` is NOT in play):
|
||
`<scratchpad>/jump-probe-config.json` (testaccount/+Acdream, policy
|
||
jump-probe, credential env ACDREAM_HEADLESS_PASS).
|
||
|
||
```powershell
|
||
$env:ACDREAM_HEADLESS_PASS = "testpassword"
|
||
$env:ACDREAM_PROBE_PARK = "1" # requires Step 1
|
||
dotnet run --project src\AcDream.Headless -c Release -- run --config <cfg> 2>&1 |
|
||
Tee-Object -FilePath headless-365.log
|
||
```
|
||
|
||
Cautions: testaccount must not be held by a running graphical client;
|
||
terminate with Ctrl-C/SIGINT (never Stop-Process — graceful Stop() clears the
|
||
ACE session in ~3-5 s; a hard kill costs ~3 min of exit-29).
|
||
|
||
Pass criteria: (1) no `kind:"failure"`; (2) `[jump-probe] local player
|
||
present; charging jump`; (3) `[pump]` reaches pending=0 and `[rearm]
|
||
verdict=OK` (or clean first-pass seal); (4) `[jump-probe] airborne-transition
|
||
False -> True` — the bot actually moves (the real acceptance test); (5)
|
||
graceful exit, converged final `disposed` sample. Second gate: run the
|
||
`observer-movement` policy ~60 s and watch from the graphical/retail client
|
||
that +Acdream walks — the K3/K4 gate unrun since 2026-08-02.
|
||
|
||
## 7. Confidence summary
|
||
|
||
| Claim | Confidence | Falsify cheaply |
|
||
|---|---|---|
|
||
| Crash = dormant controller reaching SuspendObjectUpdate via headless CanAdvancePlayer | Certain | — |
|
||
| Graphical gates on IsRuntimePublished; headless doesn't | Certain | PlayerModeAutoEntry.cs:100 |
|
||
| entities:0 is a logging artifact | Certain | HeadlessDiagnosticWriter call sites |
|
||
| Conductor parked at PublicationCommitted/AwaitingActivation | Very high | [pump] probe |
|
||
| Blocked precondition = IsCollisionEvaluationPrefixAdmissible in the seal | High | [rearm] verdict= |
|
||
| Circular HasOldPrefixPlacementDebt ↔ seal wait makes it permanent | Hypothesis, well-supported | [rearm] verdict= + log on TryAcquireCollisionPrefixMutationPermission's four early returns |
|
||
| 529e0e9d root cause; 78b981cc changed the symptom | High | git show 529e0e9d -- src/AcDream.Headless/ |
|
||
| Probes unrunnable headless | Certain | HeadlessProcessHost.cs:45 |
|
||
| Existing hydration test bypasses admission/seal | Certain | HeadlessSessionHostTests.cs:372-375 |
|
||
|
||
## 8. OUTCOME (2026-08-10, fix session)
|
||
|
||
Steps 1, 3a, and 4 landed exactly as specified; **3b was not needed.**
|
||
|
||
**Step 2 measurement (mandatory, run before any Step-3 code):** the
|
||
`ACDREAM_PROBE_PARK=1` jump-probe repro against local ACE produced
|
||
|
||
```
|
||
[pump] DriveAll #1 pending=0
|
||
...
|
||
[wake] begin lb=0x0904FFFF gen=1 unboundCells=0 buckets=0
|
||
[rearm] guid=0x5000000A seal-refused (transient; lease retained)
|
||
[rearm] guid=0x5000000A seal-refused (transient; lease retained)
|
||
[jump-probe] local player present; charging jump
|
||
```
|
||
|
||
— `seal-refused` repeating with **no preceding `[rearm] verdict=` line**.
|
||
Per this doc's own Step-2 table, that shape means the operation never even
|
||
reached the `AwaitingCell` park: `IsExactDormantLocalActivationCurrent`
|
||
already reports "current" on every attempt (the op sits in
|
||
`AwaitingPreparation` the whole time), so `TryRearmDeferredDormantLocalActivation`
|
||
is never called, and every attempt fails at
|
||
`TrySealCollisionEvaluationAuthority` on the SAME still-open admission. This
|
||
is the doc's §3 "structural half — CONFIRMED" mechanism. **No
|
||
`prefix-inadmissible` rearm verdict was ever observed**, so there is no
|
||
direct evidence the §3 "why it may never open" `HasOldPrefixPlacementDebt`
|
||
circular-wait hypothesis is in play for this repro — 3a alone was measured
|
||
sufficient.
|
||
|
||
**Fix shape: 3a only.** `IHeadlessCollisionNeighborhood.IsQuiescent`
|
||
(`_pendingPublication is null && _publicationQueue.Count == 0 &&
|
||
!_pendingPublicationCancellation`) gates `ProjectSpawn`/`ProjectPosition`/
|
||
`PumpFirstEntry`'s trailing `_firstEntry?.DriveAll()` / `_acceptedPositionDrive?.Advance()`
|
||
calls exactly as specified. Step 4 landed as specified
|
||
(`CanAdvancePlayer` requires `CanExecuteLiveMovement`, plus the three shared
|
||
`RuntimeLocalPlayerFrameController` entry points hardened the same way).
|
||
Step 1 landed as specified (`ValidateProcessIsolation(sessionCount)`, single
|
||
session + probe logs and proceeds, multi-session + probe still refuses,
|
||
naming the probe).
|
||
|
||
**Verification that the new test (item 3) actually discriminates:**
|
||
temporarily reverting the three `IsQuiescent` gates (commented out, never
|
||
committed) made both
|
||
`RealAdmissionNeverDrivesTheConductorWhileOpenAndHydratesOnceReleased` and
|
||
`PumpFirstEntryWithholdsDriveAllUntilQuiescentThenDrivesImmediately` fail —
|
||
`Assert.Null(runtime.MovementOwner.Controller)` failed because the controller
|
||
was ALREADY built and published (`CanExecuteLiveMovement = True`) while the
|
||
neighborhood's admission was still held open, exactly the pre-fix race. Both
|
||
pass cleanly on the real, fixed tree. The gate was then restored and
|
||
`git diff` confirmed the file matches the shipped Step-3a diff exactly (no
|
||
residual simulation code).
|
||
|
||
**A design note for the test:** the first attempt at test item 3 opened the
|
||
held admission on the PLAYER'S OWN landblock via the full
|
||
`HeadlessCollisionGenerationTransaction` commit cycle. That hit a genuine,
|
||
separate settlement question in `CommitCollisionGeneration` →
|
||
`TryAcquireCollisionPrefixMutationPermission` (never resolved within 200
|
||
ticks in that configuration) — worth a future look if it turns out to matter
|
||
in production, but not needed to prove Step 3a. The shipped test instead
|
||
holds a real admission open on a DIFFERENT (neighbor) landblock the player
|
||
does not target, cancelling rather than committing it — a faithful, simpler
|
||
proof of "an open admission anywhere in the plan blocks driving" without
|
||
touching that separate question.
|
||
|
||
**End-to-end (live ACE, `jump-probe` policy, `ACDREAM_PROBE_PARK=1`), three
|
||
runs, all consistent:** hydration now succeeds — `entityCount` reaches 136 at
|
||
the `running-stop` resource sample (previously 0, permanently), `[jump-probe]
|
||
local player present; charging jump` fires promptly, `seal-refused` spam is
|
||
gone, and the original fork-1 crash (`SuspendObjectUpdate` on a dormant
|
||
controller) never recurs. **Full pass criterion 4
|
||
(`[jump-probe] airborne-transition False -> True`) was NOT independently
|
||
observed on the unmodified tree** — every real run hit a SEPARATE,
|
||
newly-discovered, pre-existing defect first (filed as issue #368: the
|
||
headless scheduler's `await Task.Delay(...).ConfigureAwait(false)` loop can
|
||
resume ticks on a different ThreadPool thread than the one that opened a
|
||
collision generation, tripping `RuntimePhysicsState.EnsureCollisionMutationThread`).
|
||
#368 is explicitly out of scope for this fix — orthogonal mechanism, no
|
||
mention anywhere in this diagnosis, and a proper fix needs verification
|
||
against the graphical host, which this session was constrained not to
|
||
launch. A throwaway, never-committed diagnostic run with #368's guard
|
||
neutralized (verified via `git diff` to have zero residual footprint) DID
|
||
reach `[jump-probe] releasing jump (fire)` with a clean exit (code 0,
|
||
graceful logout, `entityCount=136`), confirming the #365 mechanism itself is
|
||
sound; it timed out waiting for `airborne-transition True` in THAT run,
|
||
plausibly a downstream artifact of the same unsynchronized-thread condition
|
||
the neutralized guard exists to catch (racing collision/physics state across
|
||
threads) rather than a second #365-scope defect — flagged as an open
|
||
question in #368, not claimed as resolved.
|
||
|
||
Every real (unmodified) run's session tore down gracefully
|
||
(`[session] graceful logout confirmed`, zero entities/leases at the final
|
||
`disposed` sample) regardless of which way it exited — `testaccount` was
|
||
never left in a stuck state by this work.
|
||
|
||
## 9. ADDENDUM (2026-08-10, #368 fix session)
|
||
|
||
#368 is CLOSED at `b7f59923`: one dedicated headless update thread now owns
|
||
Start, every scheduler turn, and the post-loop captures; the scheduler loop
|
||
is synchronous with TimeProvider-timer event waits (no `Task.Delay`
|
||
resumption migration; zero shared Runtime changes). Three live jump-probe
|
||
runs on the fixed tree each crossed `[wake] begin gen=1` cleanly with
|
||
204–205 hydrated entities and graceful exits. The §8 open question is now
|
||
answered: the `airborne-transition True` timeout PERSISTS 3/3 with threading
|
||
provably single, so the "downstream artifact of the unsynchronized-thread
|
||
condition" hypothesis is refuted — the residual is a distinct pre-existing
|
||
defect, filed as #370.
|