acdream/docs/research/2026-08-04-invisible-recalled-remote-diagnosis.md

Invisible recalled remote — diagnosis

Date: 2026-08-04 HEAD: 204d0ae0 Mode: REPORT-ONLY. No source or test edits were made. Subject: remote player 0x50000001 recalls into the observer's location and is absent from both the 3-D world and the radar, while remaining logically alive (chat and spellcasting visible, physics ticking, equipment attached).

Bottom line: the failing stage is pinned to a narrow set — something that the per-packet prologue rebucket does not restore. Two hypotheses survive every piece of evidence. They are separated by two cheap, decisive tests named in §6. 7f1c1f5a is not exonerated: it introduced a concrete mechanism that produces exactly this signature. Do not act before running §6.

---

1. Evidence held

For guid 0x50000001:

Signal	Count	Meaning
[remote-slide-tick]	71	body ticking in a remote physics workset
[remote-slide-enq]	9	accepted Positions reached interpolation enqueue
[remote-slide-up]	9	accepted Positions reached the App routing tail
[remote-slide-vec]	1	a 0xF74E VectorUpdate accepted
equipment attach	1	child 0x800046D9 → RightHand, before any tick or UP

User follow-ups:

• Never recovers — not on remote movement, not on observer walk-away-and-back.
• Did not reproduce on a second recall — intermittent.
• Ordering inversion vs. a working entity in the same session:

entity	first [remote-slide-tick]	first [remote-slide-up]	order
0x5000000F (visible)	t=93313875	t=93313843	UP 32 ms before first tick
0x50000001 (invisible)	t=93507234	t=93508546	UP 1312 ms after first tick

Both firstUpAtEntry=True; distances 46.4 m (failing) / 28.97 m (working) — both inside the 96 m far threshold at observation time.

---

2. The one thing the log proves outright

LiveEntityNetworkUpdateController.OnPosition spans lines 1459–2617 (verified by brace-depth scan; single method). Within it:

• src/AcDream.App/Physics/LiveEntityNetworkUpdateController.cs:1768 — if (!_liveEntities!.RebucketLiveEntity(update.Guid, p.LandblockId) || …) → early return at :1781.
• src/AcDream.App/Physics/LiveEntityNetworkUpdateController.cs:2476 — ApplyRemoteContactRouting(…), sole caller of src/AcDream.Runtime/Physics/RuntimeRemoteSteadyStatePosition.cs:187 (LogRemoteSlideEnqueue → [remote-slide-enq]).

:2476 is downstream of :1768. Nine [remote-slide-enq] lines therefore prove RebucketLiveEntity returned true nine times.

Each of those nine calls (src/AcDream.App/World/LiveEntityRuntime.cs:801-977):

• did not take the C3c residence refusal at :806-834,
• found record.WorldEntity non-null (:803-805),
• set record.IsSpatiallyProjected = true (:844),
• re-placed the GPU bucket via _spatial.RebucketLiveEntity (:868) — which is remove-then-place (src/AcDream.App/Streaming/GpuWorldState.cs:1132-1133) and therefore recovers a removed or pending projection,
• recomputed IsSpatiallyVisible (:894-895),
• re-fired visibility observers (:949-959),
• succeeded at CommitRebucket (:904-914).

2.1 What this eliminates

• Materialization never ran / null WorldEntity. Eliminated (:803-805).
• C3c initial-create residence gate stuck active. Eliminated (:806-834).
• Draw bucket simply never installed / permanently lost (#184 class). Eliminated — it is reinstalled on every packet.
• _pendingByLandblock strand (#168 class). Eliminated as a permanent state: GpuWorldState.RebucketLiveEntity's fast path only short-circuits when current.IsLoaded is true (GpuWorldState.cs:1106-1111), so a pending entity is re-placed every packet and promotes as soon as its landblock is loaded.
• Unrestored park withdrawal of canonical residency. FullCellId is recommitted by CommitRebucket on every packet.

2.2 Correction to an earlier draft of this document

An earlier draft additionally claimed the 71 [remote-slide-tick] lines proved IsSpatiallyVisible == true via HasSpatialRuntimeProjection (src/AcDream.App/World/LiveEntityRuntime.cs:3171-3176). That claim is withdrawn. [remote-slide-tick] is emitted from the Runtime-side src/AcDream.Runtime/Physics/RuntimeRemotePhysicsUpdater.cs:726, whose workset RuntimePhysicsState.CopySpatialRemotesTo (src/AcDream.Runtime/Physics/RuntimePhysicsState.cs:1314-1332) filters on IsSpatialRoot(record) — not on the App-side IsSpatiallyProjected / IsSpatiallyVisible. The App-side filtered copy (LiveEntityRuntime.cs:2666-2685) is used by src/AcDream.App/Physics/RemotePhysicsUpdater.cs:79, which is TickHiddenEntities (declared :59) — a hidden-only loop whose :84 guard (record.FinalPhysicsState & PhysicsStateFlags.Hidden) == 0 → continue skips non-hidden entities.

Resolved on re-verification: the emit site RuntimeRemotePhysicsUpdater.cs:726 falls inside internal bool Tick (declared :61), not internal bool TickHidden (declared :878). App-side, TickHiddenEntities reaches the hidden path via TickHidden (:121 → :251 → :279 _runtime.TickHidden). So the 71 tick lines came from the ORDINARY remote path, not the hidden-only loop.

Consequence: H1 loses the "existing log is self-confirming" corroboration that an earlier draft claimed. Whether it also refutes H1 depends on whether the ordinary loop filters Hidden — NOT ESTABLISHED; the ordinary DR tick driver was not located (it is not in LiveEntityNetworkUpdateController, LiveEntityMotionRuntimeController, or LiveEntityOrdinaryPhysicsUpdater's searched surface). If the ordinary loop has no Hidden filter, H1 stands intact; if it skips Hidden entities the way TickHiddenEntities:84 skips non-hidden ones, H1 is refuted outright and H2 becomes the sole surviving hypothesis. This is a five-minute source question and should be the first action of the next session — it may remove a whole hypothesis before any client is launched.

---

3. Constraint on the answer

The fault must be something that:

1. survives a successful RebucketLiveEntity on every packet,
2. suppresses both world-render and radar, and
3. leaves physics/interpolation, chat, casting, and equipment working.

Two candidates satisfy all three.

---

4. H1 — PhysicsStateFlags.Hidden latched on

src/AcDream.App/World/LiveEntityRuntime.cs:3357-3376:

PhysicsStateFlags state = record.FinalPhysicsState;
bool residenceVisible = …;
entity.IsDrawVisible = residenceVisible
    && (state & (PhysicsStateFlags.NoDraw | PhysicsStateFlags.Hidden)) == 0;

bool interactionVisible = record.IsSpatiallyVisible
    && record.ProjectionKind is LiveEntityProjectionKind.World
    && (state & PhysicsStateFlags.Hidden) == 0;

Hidden = 0x00004000 (src/AcDream.Core/Physics/PhysicsBody.cs:52).

Consumer	Gate	file:line
World render	entity.IsDrawVisible	src/AcDream.App/Rendering/Wb/WbDrawDispatcher.cs:1159-1164
Radar blip	_projections.SetVisible → _visible	src/AcDream.App/UI/Layout/RadarSnapshotProvider.cs:116-120 → LiveEntityRuntime.cs:1859 → LiveEntityProjectionStore.cs:139-152

Survives the rebucket: RebucketLiveEntity calls RefreshPresentation (:896), which faithfully re-publishes a FinalPhysicsState that still carries Hidden. Presentation is not stale — it is correctly rendering a wrong flag. Only a fresh accepted SetState clears it, and ACE does not resend one.

Recall is the one flow where a player is legitimately hidden then un-hidden, so a lost un-hide is scenario-appropriate. Two candidate drop sites:

1. Queued behind the initial-create residence and lost. LiveEntityNetworkUpdateController.cs:1429-1438 states the accepted SetState is "queued behind the initial residence". The executor has six arms that publish nothing (src/AcDream.Runtime/Entities/RuntimeInitialCreateContinuationExecutor.cs:1015, :1022, :1028, :1045, :1048, :1108) versus one Released arm reaching PublishExecutorCompletion (:1092). Its BecameHidden handling is at :2315, :2347. The 1.3 s tick-before-first-UP inversion is direct evidence this window was open ~40 physics quanta for the failing entity.
2. Edge-trigger against a wrong previousState. src/AcDream.Core/Physics/RetailPhysicsStateTransition.cs:49-55 computes hiddenChanged from previousState ^ requestedState. ConstructorState (:37-41) deliberately excludes Hidden. A mismatched previousState makes the clearing edge read as "no change" and skips it, latching Hidden.

Corroboration withdrawn. An earlier draft argued the 71 tick lines came from TickHiddenEntities and were therefore self-confirming for H1. §2.2 shows they came from the ordinary Tick path instead. H1 now rests entirely on the elimination argument in §3 plus the §6.1 walk-through test.

---

5. H2 — park Withdraw tears down presentation state the rebucket never restores

This is a 7f1c1f5a regression mechanism.

7f1c1f5a is the first commit that lets an ordinary remote UpdatePosition open a canonical SetPosition operation (LiveEntityNetworkUpdateController.cs:1083-1088 → src/AcDream.Runtime/Session/RuntimeRemotePlacementDriveController.cs:760-784). Before it, the only remote placement path was RemoteTeleportController behind the remotePlacementRequired gate.

When that placement parks (src/AcDream.Runtime/Physics/RuntimeSetPositionState.cs:2997, :3053, :3086 → ParkDeferred:4418), it publishes a Withdraw projection synchronously (:4541-4545 → src/AcDream.Runtime/Physics/RuntimePlacementProjectionSubscription.cs:136).

The production sink's withdrawal — src/AcDream.App/World/RuntimePlacementPresentationSink.cs:195-224 — does:

for (…) _visibilitySinks[i](record, false);   // :202-207
_worldState.RemoveById(entity.Id);            // :209
_worldEvents.ForgetEntity(entity.Id);         // :212
_effectPoses.Remove(entity.Id);               // :215
_clearSelectionForUnavailableEntity(…);       // :222

Its mirror image TryPublishPlace (:162-193) is the only thing that re-adds _worldState.Add (:168), _worldEvents.UpsertCurrent (:171), and _effectPoses.PublishMeshRefs (:174).

RestoreParkWithdrawal (RuntimeSetPositionState.cs:3503-3552) restores InWorld, the object clock, FullCellId, and the spatial root — and by its own documented design restores none of the render side (:3494-3501). Critically, its SetFullCell is a plain field write (src/AcDream.Runtime/Entities/RuntimeEntityDirectory.cs:340-347) that bypasses CommitCanonicalCell, so the CellCommitted → RebucketLiveEntity recovery at LiveEntityRuntime.cs:3316 never fires. TryAdoptWireCellAfterRouting (LiveEntityNetworkUpdateController.cs:1193-1194) suppresses the one remaining same-packet re-commit on the NPC arm.

The per-packet prologue rebucket restores IsSpatiallyProjected, the GPU bucket, IsSpatiallyVisible, and the visibility sinks — but not _worldState, _worldEvents, or _effectPoses.PublishMeshRefs. If mesh refs are among the casualties, WbDrawDispatcher.cs:1163 (if (entity.MeshRefs.Count == 0) continue;) skips the entity permanently, which is exactly "never recovers".

Fit: intermittent (only when a placement parks), never recovers (only a Place receipt or re-materialization restores it), physics healthy (Runtime state fully restored), enq lines present (rebucket succeeds).

NOT ESTABLISHED: whether _effectPoses.Remove actually clears entity.MeshRefs, and whether the radar's candidate source (GpuWorldState._loadedLiveByLandblock, restored by the rebucket) or its _visible gate is affected. If neither kills the radar, H2 explains render-only and H1 remains the better fit for the combined symptom.

---

6. Discriminating tests — run these first, they are cheap and decisive

6.1 Walk into the invisible player (free, no code)

src/AcDream.App/Physics/LiveEntityShadowPublisher.cs:46-55 gates collision-shadow publication on (record.FinalPhysicsState & PhysicsStateFlags.Hidden) == 0.

H1 predicts the observer walks THROUGH the invisible player. H2 predicts it is solid (Runtime spatial root and shadow rows intact after the next placement).

Also: NoDraw (0x20) suppresses render only — radar's interactionVisible does not test it. Since radar is also dead, if H1 holds the flag is Hidden.

6.2 ACDREAM_PROBE_PARK=1 (already shipped by 7f1c1f5a)

Emits [park] guid=… cause=… eligible=… captured=… (RuntimeSetPositionState.cs:4466-4475) and [park-restore] guid=… residency=… (:3545-3551).

A [park] line for the failing guid ⇒ H2 live and 7f1c1f5a implicated. No [park] lines anywhere in the session ⇒ 7f1c1f5a exonerated outright.

These two tests together resolve the regression question definitively.

---

7. Regression verdict

Not settled by code reading alone — §6.2 settles it. Current position:

7.1 204d0ae0 — no mechanism found

Its src/ files are LiveEntityNetworkUpdateController.cs, InterpolationManager.cs, MotionTableDispatchSink.cs, PhysicsDiagnostics.cs, RuntimeRemotePhysicsUpdater.cs, RuntimeRemoteSteadyStatePosition.cs. The commit primitives it routes through are pure body-state functions (src/AcDream.Core/Physics/PhysicsObjUpdate.cs:153-179, :181-185, :209-248) — no cell, no shadow registry, no WorldEntity, no bucket. It neither added nor removed a publication.

Residual, flagged not dismissed: it deleted the per-tick forge of Contact|OnWalkable and the per-tick velocity zero and made gravity persistent, so a freshly created contact-free remote now settles and moves during the pre-Position window where it was previously pinned motionless. It also moved Airborne derivation inside if (resolveResult.Ok && candidateMoved) (RuntimeRemotePhysicsUpdater.cs:536-537, :683) and added a new LeaveGround re-entrancy early-exit (:634-642). These can plausibly widen a pre-existing window without creating it. NOT ESTABLISHED.

7.2 7f1c1f5a — NOT exonerated

The prior review's specific warning ("losing the rebucket leaves a remote without a draw bucket") is refuted: the prologue rebucket at :1768 runs for every arm, before routing, and its diff hunk in RuntimeEntityObjectLifetime.cs is comment-only. TryAdoptWireCellAfterRouting suppresses only remote.CellId on the FarSnapPlacement arm, which the 46.4 m failing entity did not take at observation time.

But §5 is a real, different mechanism that this commit introduced, and it produces precisely the reported signature. Note the recall shape fits: the remote was far away before recalling, so its pre-recall packets were ≥96 m and did take the far arm, where a park is reachable.

7.3 The honest caveat

Intermittency makes "it worked before" weak evidence. A pre-existing race that earlier gates never sampled remains fully consistent with everything observed — that is exactly what H1 would be. Do not conclude regression from the timeline alone; conclude it from §6.2.

---

8. Capture plan for the next run

8.1 Why new instrumentation is needed

Verified by grep with zero hits in the owning files:

Capability	Existing probe?
Draw-bucket publication (RebucketLiveEntity/CommitRebucket)	NONE
Render residency (IsLiveEntityProjectionResident/IsLiveEntityVisible)	NONE
Radar registration	NONE
Wire create/despawn (0xF745/0xF747)	NONE — ACDREAM_DUMP_OPCODES fires only for unhandled opcodes (WorldSession.cs:1988-1995); both are handled at :1687-1700
Materialization	partial — ACDREAM_DUMP_LIVE_SPAWNS (RuntimeOptions.cs:111, DatLiveEntityProjectionMaterializer.cs:149) logs entry + two DROP paths, no success line
Physics state / Hidden	[setstate] exists (LiveEntityNetworkUpdateController.cs:1454-1456) but is gated on ACDREAM_PROBE_BUILDING, an unrelated heavy BSP flag
Parks	ACDREAM_PROBE_PARK — exists and is exactly right for H2

ACDREAM_PROBE_ENT is hard-wired to the local player's guid and cannot target a remote.

8.2 Required properties

Always-on for the whole entity lifetime — the failure is rare and must not require luck. Every line must carry the server guid: several render probes key on WorldEntity.Id, a LocalEntityId from RuntimeEntityDirectory.FirstLocalEntityId = 1_000_000, structurally unrelated to 0x50000001 and not greppable by guid.

8.3 Proposed probe ACDREAM_PROBE_VISGATE=1

One new diagnostic owner (Code Structure Rule 5):

1. [vis-state] — every FinalPhysicsState mutation: guid, prevState, requestedState, finalState, hiddenTransition, stateSeq, instSeq, accepted, droppedReason. Must fire on dropped and queued updates naming the reason. Settles H1.
2. [vis-present] — every RefreshPresentation (LiveEntityRuntime.cs:3357), changed-values-only: guid, finalState, residenceVisible, isSpatiallyProjected, isSpatiallyVisible, projectionKind, fullCellId, meshRefCount, isDrawVisible, interactionVisible. meshRefCount is the H2 discriminator.
3. [vis-publish] — every TryPublishPlace / TryPublishWithdrawal (RuntimePlacementPresentationSink.cs:162, :195): guid, kind(place|withdraw), worldState, worldEvents, effectPoses, sinkCount. Settles H2 directly.
4. [vis-residence] — initial-create residence lifecycle including every non-publishing executor arm (RuntimeInitialCreateContinuationExecutor.cs:1015, :1022, :1028, :1045, :1048, :1108). Measures the 1.3 s window.
5. [vis-create] — accepted CreateObject/DeleteObject: guid, opcode, instSeq, wireState, wireCell.

Also re-gate the existing [setstate] line off ACDREAM_PROBE_BUILDING onto the new flag so it is usable without the BSP dump.

8.4 Protocol

1. Release build, ACDREAM_PROBE_PARK=1 ACDREAM_PROBE_VISGATE=1 ACDREAM_PROBE_REMOTE_SLIDE=<guid> (that family takes a per-guid allow-list, PhysicsDiagnostics.cs:397-404).
2. Two clients; recall in/out until one fails.
3. On failure, before anything else: walk into the invisible player (§6.1).
4. Grep the failing guid. Verdict is mechanical:
   • [vis-state] finalState retains 0x4000 → H1; droppedReason + [vis-residence] name the drop site.
   • [park] present and [vis-publish] kind=withdraw with no later place, and [vis-present] meshRefCount=0 → H2; 7f1c1f5a implicated.
   • Neither → both refuted; re-scope to the render candidate stream.

---

9. Proposed fix

Needs §6 first. The failing stage is pinned (§2, §3); which of the two mechanisms fires is not, and they need different fixes. Guessing between them is what the workflow forbids.

Shape, per branch:

• If H1: fix the lost state update at its drop site. If the residence queue loses it, the non-Released executor arms must replay the queued state continuation or refuse the residence — adjacent to open #310, which shares the "executor arm that never completes" shape. If the edge-trigger is at fault, fix the caller supplying previousState; do not change RetailPhysicsStateTransitions.Apply — it is a faithful port of CPhysicsObj::set_state @0x00514DD0 / set_hidden @0x00514C60.
• If H2: the asymmetry between TryPublishWithdrawal and TryPublishPlace is the bug. Either the park's Withdraw must not tear down presentation state the restore cannot rebuild, or RestoreParkWithdrawal must route its SetFullCell through CommitCanonicalCell so the existing CellCommitted → RebucketLiveEntity recovery (LiveEntityRuntime.cs:3316) fires. Blast radius: Runtime placement + the App presentation sink. Rollback for the introducing commit is git revert 7f1c1f5aa6cd7842726d2edd909564d620eb587f, but that also reverts C4 route 4b-2 wholesale — prefer the targeted fix.

Forbidden either way (CLAUDE.md, and the digests' DO-NOT-RETRY tables): no "re-publish if invisible for N ms" guard, no periodic re-assert of FinalPhysicsState, no retry loop, no timer. "Never recovers" is a symptom of the lost update, not a defect to paper over.

A register row (docs/architecture/retail-divergence-register.md) and a new docs/ISSUES.md entry are required. This is not covered by #309 — §2.1 rules out its canonical-residency mechanism.

---

10. NOT ESTABLISHED

1. Whether the ordinary remote DR loop filters Hidden. The tick lines are confirmed to come from the ordinary Tick path (§2.2), but the loop that drives it was not located. If it skips Hidden entities, H1 is refuted outright and H2 is the sole hypothesis. Highest-value check; five minutes of source reading; do it before launching a client.
2. Whether Hidden is set at all. H1 is an elimination argument, not an observation. Settled by §6.1.
3. Whether a park occurred for this guid. Settled by §6.2.
4. Whether _effectPoses.Remove clears entity.MeshRefs, i.e. whether H2 can kill render permanently.
5. Whether ACE sends Hidden→un-Hidden across a recall for a remote observer. Not checked against references/ACE/ (absent from this worktree; present in the parent repo). One grep before the run.
6. Whether 204d0ae0 widened the window (§7.1).
7. Whether LiveRenderProjectionJournal (src/AcDream.App/Rendering/Scene/LiveRenderProjectionJournal.cs:141) drives a shipping draw call or is a shadow scene.

Incidental findings (not the bug, worth filing)

• Stale comments: src/AcDream.App/World/LiveEntityRuntime.cs:1195 and src/AcDream.App/World/RuntimePlacementPresentationSink.cs:74 both assert PublishExecutorCompletion has zero production callers. src/AcDream.Runtime/Entities/RuntimeInitialCreateContinuationExecutor.cs:1092 is one. Pre-C3c-flip and actively misleading to this investigation.
• TryAdoptWireCellAfterRouting's justification is incomplete. Its comment (LiveEntityNetworkUpdateController.cs:1160-1185) argues the suppressed write "would have written the value that is already there". That holds for Refused/Contention/RejectedPreparation/NotApplicable, but not after a park, because WithdrawCanonical zeroes record.FullCellId (RuntimeSetPositionState.cs:5082) and CommitCanonicalCell early-returns only on equality (RuntimePhysicsState.cs:2145-2146).
• RuntimeRemotePhysicsUpdater's acknowledgeProjection seam is inert — every production call site passes null.

---

11. DO-NOT-RETRY compliance

Checked against claude-memory/project_render_pipeline_digest.md and claude-memory/project_physics_collision_digest.md:

• No symptom-site guard, retry, timer, or settle period is proposed.
• The #168 _pendingByLandblock / RelocateEntity family (feedback_streaming_residence_race) is positively excluded by §2.1 — the recovery path exists and provably ran every packet.
• feedback_no_placeholder_racing_real_data (#192) honoured: the proposal fixes the lost update / asymmetric teardown, not a correlated lifecycle proxy.
• No change proposed to AP-87's threshold, InterpolationManager's node_fail_counter snap-to-tail, or the calc_friction/jump chains.
• feedback_probe_identity_attribution: every proposed probe line carries the server guid — which is why §8.2 rejects the WorldEntity.Id-keyed render probes.
• feedback_verify_subagent_claims_against_source: every load-bearing claim here was read at the cited file:line; §2.2 records one claim withdrawn on re-verification.