acdream/docs/research/2026-08-02-collision-throughput-handoff/design-note.md

# O(changed) collision clone — design note

**Phase:** research + design only. No production edits, nothing staged, no probes left
behind. Worktree `C:\Users\erikn\.codex\worktrees\af5e\acdream`, branch
`codex/port-claude-agents`, HEAD `c52ce14a`.

**Problem:** the collision-generation staging clone is O(resident world) per landblock
publication, so loading an N-landblock ring costs O(N²). The far ring never converges.
C3c made it user-visible (late monster pop-in, extended/stuck portal space, portal-exit
pop-in, failing nine-stop soak) but did not cause it.

---

## (a) What the one-leaf-per-step invariant actually protects

### It is a frame-time bound. Nothing else.

The whole-world copy did not arrive with `6b28ff99`. It arrived one commit earlier, in
`be94bc9b` "fix(physics): activate collision generations atomically" (2026-07-31), as a
**synchronous** copy performed in a single call at admission:

```csharp
// be94bc9b, PhysicsEngine.CreateCollisionStagingCopy
foreach ((uint id, LandblockPhysics landblock) in _landblocks)
    staging._landblocks[id] = landblock;
staging.ShadowObjects.CopyCollisionStateFrom(ShadowObjects, stagingCache);
```

`6b28ff99` "make collision activation starvation-free" replaced that with
`CollisionStagingBuilder` (`src/AcDream.Core/Physics/PhysicsEngine.cs:785-941`), which
performs the *same* copy chopped into single leaves across frames. The retired AD-6 row
states the purpose verbatim
(`docs/architecture/retail-divergence-register.md:113`):

> "Admission captures the active root in O(1); a stable landblock/owner slot suffix
> materializes non-target leaves incrementally, **so resident-world size cannot become a
> synchronous clone spike**."

The committed test says the same thing three ways
(`tests/AcDream.Runtime.Tests/Physics/RuntimePhysicsStateTests.cs:921-991`):

| Assertion | line | What it pins |
|---|---|---|
| `Assert.InRange(admissionAllocation, 1L, 128L*1024L)` | :973 | admission allocates a constant |
| `Assert.Equal(0, prepared.Engine.LandblockCount)` | :974 | admission copies no resident landblock |
| `Assert.InRange(step.WorkUnits, 0, 1)` | :983 | **the copy is chopped to one leaf per host step** |
| `Assert.True(advances > residentLandblocks)` | :988 | it really walked the resident world |
| `Assert.Equal(residentLandblocks, prepared.Engine.LandblockCount)` | :989 | the draft ends up holding the whole world |

So the invariant protects **hitch avoidance**: a dense resident world must not produce
one long synchronous copy inside a single update step. It is a *scheduling* property
asserted as a *mechanism*, which is why the batching lever tripped it.

### What it does NOT protect

- **Not concurrent-reader isolation.** That is `CollisionWorldStateSlot.TransferTo`'s
  single `Volatile.Write` (`src/AcDream.Core/Physics/CollisionWorldState.cs:66-84`).
  And a full threading audit of every writer and reader of `CollisionWorldState` found
  **no concurrent reader or writer exists**: `GameWindow` runs one Silk.NET loop thread;
  `UpdateFrameOrchestrator.Tick` runs `_streaming.Tick()` → `DrainAndApply` and then the
  live/physics/camera phases strictly sequentially on that thread; the only background
  workers (`LandblockStreamer` worker thread, `EnvCellRenderer` `Parallel.ForEach`,
  `ObjectMeshManager` `Task.Run`) never touch `PhysicsDataCache` / `CellGraph` /
  `ShadowObjectRegistry` / `PhysicsEngine` — grep of `LandblockBuildFactory.cs` and
  `LandblockMesh.cs` for those types returns zero hits. Every one of
  `BeginCollisionAdmission` (:2040), `PrepareCollisionGeneration` (:2092),
  `AdvanceCollisionGenerationPreparation` (:2123), `StageCollisionAssets` (:2218),
  `AdvanceCollisionGenerationSeal` (:2298), `CommitCollisionGeneration` (:2382),
  `CancelCollisionGeneration` (:2139) passes through
  `RuntimePhysicsState.EnsureCollisionMutationThread` (:2857-2869). The
  `ConcurrentDictionary` choices are load-bearing only for *single-threaded*
  mutate-while-enumerating (`PhysicsDataCache.cs:958-984`, and the seal cursor holding a
  live enumerator across frames at :1081-1160) — the cross-thread rationale in the
  `CellGraph.cs:17` and `PhysicsDataCache.cs:14-20` doc comments is **stale after
  6b28ff99**.
- **Not admission fairness.** That is the separate journal/coalescing machinery
  (`RuntimePhysicsState.cs:475-529`, research doc step 5) — the other half of what
  "starvation-free" meant. It is orthogonal to the leaf metering and stays.

### What the pre-6b28ff99 mechanism did

`be94bc9b`'s commit was a **delta apply**, not a root swap:

```csharp
// be94bc9b, PhysicsEngine.CommitLandblockReplacement — deleted by 6b28ff99
DataCache.CommitLandblockReplacement(replacement.DataCache);   // O(changed)
_landblocks[replacement.LandblockId] = replacement.Landblock;
ShadowObjects.CommitLandblockReplacement(replacement.Shadows);
```

`6b28ff99` replaced those three lines with
`stagingCache.CollisionWorld.TransferTo(activeCache.CollisionWorld)`
(`PhysicsEngine.cs:304-321`). **That is the change that made the clone load-bearing.**
Before it, the clone was a build sandbox; after it, the clone *is* the world that gets
published, so every leaf not cloned is a leaf deleted from the world.

Before `be94bc9b` the client mutated the active maps in place across many frames — the
genuinely non-equivalent state the research doc describes ("the active `PhysicsDataCache`,
`CellGraph`, `PhysicsEngine`, buildings, static shadows, and retained-object refloods
changed at different cursors", `docs/research/2026-07-31-atomic-collision-generation.md:12-16`).
**The atomicity requirement is "the multi-frame build must not be observable", not "the
whole world must be swapped".** A delta applied inside one synchronous update-thread call
satisfies it.

### The cost is worse than F4 measured

F4 attributed median 19,736 / p90 32,135 / max 38,021 leaves and median 3.64 ms per
publication to the staging clone. The **seal** does the same walk again: the replacement
builder holds live enumerators over four `_staging` maps *and* four `_active` maps
(`PhysicsDataCache.cs:1081, 1092, 1103, 1114, 1125, 1136, 1147, 1158`) plus two in
`CellGraph.cs:184, 203`, using `CapturePrefixOne` / `CaptureRemovalOne` — a full scan of
each map to find O(target) keys. Real per-publication cost is therefore roughly **2–3×
resident world**, not 1×. **Fixing only the clone leaves O(N²) in the seal.** Any design
that does not also scope the removal capture is not a fix.

---

## (b) Candidate designs

### D1 — Structural sharing (persistent/immutable `CollisionWorldState`)

Replace the ~20 mutable maps with persistent maps (HAMT / `ImmutableDictionary`) so a
staging clone shares unchanged subtrees and copies only the changed path.

- **Blast radius:** every read and write site of every map in `CollisionWorldState`,
  `PhysicsDataCache`, `CellGraph`, `ShadowObjectRegistry`, `PhysicsEngine`.
- **Invariant changes:** none semantically; the root swap survives unchanged, so the
  atomicity story is untouched.
- **Throughput:** admission O(1), clone O(1), commit O(changed · log N). Excellent on the
  copy axis.
- **Why rejected:** it pays for the copy with the *query*. A HAMT probe is several times a
  `Dictionary` probe and allocates on write; the resolver runs thousands of these per
  frame at 30 Hz. Slice I's entire thesis is flat, integer-indexed, zero-allocation
  collision (`docs/plans/2026-07-25-modern-runtime-slice-i.md`; I1 measured 0 B/resolve).
  D1 optimizes the rare operation at the expense of the hot one and fights the I-series
  architecture head-on.

### D1b — Landblock-sliced root (per-prefix immutable slice + small map)

Regroup the root so each landblock's cells / flat cells / EnvCells / buildings / terrain /
outdoor cells / `LandblockPhysics` live in one immutable `LandblockCollisionSlice`, and
the root becomes `Dictionary<prefix, slice>` (~625 entries). Commit = one dictionary
write per prefix.

- **Blast radius:** every keyed read becomes mask + two probes; the seal's removal scans
  collapse to "old slice vs new slice". The **shadow registry does not partition** —
  `ShadowEntityCells`, `ShadowEntityShapes`, `ShadowEntityRegistrations`,
  `ShadowOwnerVersions` are owner-keyed and owners legitimately span prefixes (that is
  the whole retained-owner problem), so the shadow half needs a separate mechanism.
- **Invariant changes:** the atomic unit becomes the slice; the root swap disappears.
- **Throughput:** O(changed) by construction, and atomic even for a hypothetical
  concurrent reader.
- **Verdict:** this is the right answer *if* concurrent readers existed. They do not.
  Keep it on the shelf as the migration target should the runtime ever go multi-threaded;
  do not pay its refactor cost now.

### D2 — Per-landblock atomic unit: restore the delta apply *(recommended)*

`CommitLandblockReplacement` drains the **already-existing**
`PhysicsEngine.LandblockReplacementApplyCursor` (`PhysicsEngine.cs:568-777`) against the
**active** root inside one synchronous call, instead of `TransferTo`. The staging root
becomes empty-at-admission (target content only); `CollisionStagingBuilder` phases 0–8 are
deleted.

The delta record already exists and is already tested: `PreparedPhysicsDataCacheLandblock`
(`PhysicsDataCache.cs:1255-1270`) is exactly lists of key/value pairs to install and lists
of ids to remove, all target-scoped. The apply cursor already handles removals, installs,
terrain, the 0x40 synthesized outdoor cells, the landblock itself, and yields the reflood
owner ids to the caller at phase 12 (`PhysicsEngine.cs:702-712`). Today it is used to
rebase a committed peer delta into a *later draft*; pointing its `destination` at the
active engine is a constructor argument, not new machinery.

**What breaks, honestly:**

1. *Readers mid-query* — nothing. Single-threaded, evidenced above. A drained cursor
   inside one call is indivisible with respect to every reader that exists.
2. *Re-entrancy* — real, and the audit flagged it: `OwnerMutated` /
   `OwnerPrefixMembershipChanged` (`ShadowObjectRegistry.cs:86-87`) can fire mid-delta.
   Precedent already exists: the commit brackets itself with
   `_suppressCollisionOwnerJournal = true` (`RuntimePhysicsState.cs:2491-2501`). Extend
   that bracket to cover the whole apply.
3. *Cross-frame enumerators* — the seal holds live enumerators over the **active** maps
   across frames (`PhysicsDataCache.cs:1092, 1136, 1158`). A delta apply now mutates the
   maps those enumerators walk. `ConcurrentDictionary` will not throw, but the observed
   set is unspecified. **O1 below removes those enumerators entirely**, which is why O1
   must land first.
4. *The retirement machinery* — `LandblockRetirementCursor` (`PhysicsEngine.cs:348-...`)
   currently retires from an off-side draft. Same cursor, destination becomes the active
   root, still drained in one call.
5. *The reflood context* — the seal currently computes retained-owner refloods against a
   full staging world. With an empty staging root that context is gone, so the reflood
   moves to the commit call, against the now-current active world. **That is precisely
   retail**: `CObjCell::init_objects` (0x0052B420) → `CPhysicsObj::recalc_cross_cells`
   (0x00515A30), already the retail anchor cited on the AD-6 row.
6. *The peer-rebase / journal apparatus* — with no snapshot there is nothing to rebase.
   `EnqueueCommittedRebase` (`RuntimePhysicsState.cs:563-578, 2502-2507`) and most of the
   journal become dead. Delete them in the same slice; do not leave dead invariants
   guarding a deleted mechanism.

- **Throughput:** per publication ≈ target payload (~70–200 leaves at the measured
  ~184 ns/leaf) + the owners touching the target, versus today's ~2–3 × 20,000. Roughly
  **300× less work per publication**, and — decisively — **independent of resident-world
  size**, so total ring load goes O(N²) → O(N). At the failing run's numbers that is
  ~13.7 M leaf copies for a 625-landblock ring down to ~44 K.

### D3 — Adjacency-scoped clone (the tempting middle ground) — **rejected as unsafe**

Copy only leaves in the target's 3×3 landblock neighbourhood. One predicate change in
`CopyOneOutsideTarget` (`PhysicsEngine.cs:949-961`); clone drops ~20,000 → ~630 and
becomes O(1) in world size.

Rejected for a structural reason worth stating plainly: **while commit is a whole-root
transfer, "clone less" means "delete more."** Anything not copied into the draft is absent
from the root that replaces the world. A partial clone is therefore a silent world-erasure
bug, not a perf tuning knob. Only after commit becomes a delta does bounded context become
safe — at which point D2 has already removed the need for it. It also leaves the seal's
O(world) scans untouched, so O(N²) survives regardless.

---

## (c) Recommendation

**Take D2, in three landable slices, with O1 first.**

Rationale in one line: the delta-apply commit path is not a new invention — it is the
mechanism that shipped in `be94bc9b` and was deleted by `6b28ff99` to buy an atomicity
guarantee against concurrent readers that do not exist; restoring it makes the cost
O(changed) by construction and moves the client *toward* retail's `init_objects` shape,
not away from it.

### Invariant-test replacement

Delete from `DenseResidentWorldAdmissionIsConstantAndMaterializesOneLeafPerStep`
(`tests/AcDream.Runtime.Tests/Physics/RuntimePhysicsStateTests.cs:921-991`) the three
assertions that pin the clone itself — `:983` `Assert.InRange(step.WorkUnits, 0, 1)`,
`:988` `Assert.True(advances > residentLandblocks)`, `:989`
`Assert.Equal(residentLandblocks, prepared.Engine.LandblockCount)`. They assert the exact
mechanism being removed.

Replace with `CollisionPreparationCostIsIndependentOfResidentWorldSize` — a strictly
stronger invariant, because it pins the *property* (bounded, world-size-independent work)
rather than a mechanism:

```
Run the full admission → preparation → seal → commit sequence twice,
at residentLandblocks = 32 and residentLandblocks = 256.

Assert total preparation advances(32) == total preparation advances(256)   // O(changed)
Assert total seal WorkUnits(32)      == total seal WorkUnits(256)          // closes the seal scans
Assert every step.WorkUnits <= K                                           // K = retained per-step bound
Assert admissionAllocation in [1, 128 KiB]                                 // kept from :973
Assert prepared.Engine.LandblockCount == 0 after preparation completes     // stronger than :974:
                                                                           // the draft now holds ONLY the target
```

Add two more:

- `CommitAppliesOneLandblockDeltaInASingleCall` — the engine-mutating
  `CommitCollisionGeneration` call drains the apply cursor to `Completed` before it
  returns; the active world holds no target-prefix content before it and the complete
  target after it, with no observable intermediate.
- `CommitTimeRefloodMatchesPrecomputedReflood` — for a fixed scenario, the owner set and
  each owner's resulting cross-cell set after a commit-time reflood are **equal** to what
  the pre-change staged reflood produced. This is the proof that D2 is a scheduling
  change and not a semantics change, and it is the test that makes the perf framing in
  (d) legitimate.

**Keep unchanged:** every `Assert.InRange(seal.WorkUnits, 0, 1)` at `:558, :667, :747,
:848, :1797, :2020, :2318, :2428, :3033` — the seal stays metered; the zero-managed-byte
commit assertions (the delta lists are built during seal, so the apply must still be
allocation-free); and `CommittedPreparationRevokesItsStagingCollisionRoot` (`:1664`) in
spirit — the staging root must still be revoked after commit, it simply no longer becomes
the active root.

### Migration plan

| Slice | Change | Gate |
|---|---|---|
| **O1** | Per-prefix installed-key ledger in `CollisionWorldState`, maintained by the install/remove paths. Rewrite the seal's ten full-map scans (`PhysicsDataCache.cs:1081-1160`, `CellGraph.cs:184, 203`) to enumerate that set. Removes the cross-frame active-map enumerators. **Behaviour-identical; a pure win that lands alone.** | existing suites green + the new seal-independence assertion |
| **O2** | `PhysicsEngine.CommitLandblockReplacement` drains `LandblockReplacementApplyCursor` against the active root instead of `TransferTo`. Extend the `_suppressCollisionOwnerJournal` bracket over the whole apply. Retirement cursor destination → active root. | focused Runtime physics suite + connected lifecycle gate |
| **O3** | Empty staging root: delete `CollisionStagingBuilder` phases 0–8. Move retained-owner reflood into the commit call (retail `init_objects` → `recalc_cross_cells`). Delete the now-dead peer-rebase/journal paths and their tests. | full ladder below |

### Gate ladder (O3 closeout)

1. **Focused:** Runtime physics collision-generation suite, App
   `LandblockPhysicsPublisherTests`, Headless `HeadlessSessionHostTests`.
2. **Complete Release solution:** baseline to match or beat is **10,808 passed / 0 failed
   / 4 skips** (`-m:1`, `ACDREAM_PAK_PATH`).
3. **Connected lifecycle/reconnect gate:** signature must hold — `Passed=true`,
   `Failures=[]`, both sessions `ExitCode=0`, zero render-shadow mismatches, zero pending
   deltas, graceful exits.
4. **Nine-stop soak must reach `Passed: true` with `Failures: []`.** The failing run is
   `logs/connected-r6-soak-20260802-143157.report.json` (37 failures, `Passed: false`);
   the passing baseline is `logs/connected-r6-soak-20260727-004942.report.json`
   (`Passed: true`, commit `a9a822f2`). Concrete acceptance, per checkpoint:

   | Key | Failing run | Required |
   |---|---|---|
   | `resources.streamingWork.deferredCompletions` | 92–501 at 8/9 stops | `0` at all 9 |
   | `resources.streamingWork.farBacklog` | same values | `0` at all 9 |
   | `resources.streamingWork.pendingPublications` | `1` at 8/9 | `0` at all 9 |
   | `resources.streamingWork.deferredAdoptedCpuBytes` | 1.5–8.5 MB | `0` at all 9 |
   | `resources.streamingWork.oldestDeferredAgeMilliseconds` | 37,764–69,728 | `0` |
   | `resources.loadedLandblocks` | 124–533 | **625** at the eight outdoor stops |
   | `reveal.waitCueShown` | `true` at 6/9 | `false` at all 9 — *this is the user-reported "extended/stuck portal space"* |
   | `streamingWork.lifetimeFrameOverrunCount` | 1,706 | materially lower |
   | `streamingWork.maximumOperationStage` | `"publication-index-physics"` | must no longer name this stage |

   **`aerlinthe` (sequence 4) is the control, not a target.** It is the one indoor
   destination and the one stop that is already clean in the failing run (374/173 vs the
   baseline's 374/176, every streaming counter `0`) — precisely because an indoor
   destination streams few landblocks, so O(N²) never bites. It must stay clean; do not
   expect it to reach 625.

5. **Frame time must not regress — and must recover.** Route-level `cpuUs` from
   `frame-history-summary.json`, microseconds:

   | | p50 | p95 | p99 | p999 |
   |---|---|---|---|---|
   | baseline `20260727` | 9,730 | 41,262 | 44,875 | 63,474 |
   | failing `20260802` | 17,001 | 47,434 | 57,061 | 102,876 |

   Gate on the sharper per-checkpoint window numbers: **`checkpointWindows[].metrics.cpuUs`
   p99 within +10 % of the `20260727` baseline at every stop.** The worst offenders are
   `caul-plateau` 101,408 → target ≈ 47,821; `caul-return` 103,309 → ≈ 46,938;
   `caul-baseline` 108,148 → ≈ 43,664; `sawato-baseline` 49,748 → ≈ 10,592. Frame count
   should recover toward the baseline's 35,492 frames / 498.5 s from the failing run's
   25,965 / 583.9 s.

6. **Do NOT gate on these — retest only after convergence.** `trackedGpuBytes` (62 MB
   failing vs 474 MB baseline), `meshRenderData` 588 vs 607, `meshEstimatedBytes`
   233.8 MB vs 268.6 MB, and the inverted CPU mesh-cache hit ratio
   (3,666 hits / 5,689 misses vs 20,825 / 6,845) are all far-ring-never-converged
   artifacts of the same mechanism. F4 already reached this conclusion; a leak
   investigation before convergence is restored will chase a ghost.

### Register / plan bookkeeping (same commit as the code)

- **`docs/architecture/retail-divergence-register.md:113`** — the retired AD-6 row
  describes the deleted mechanism verbatim ("one shared off-side `CollisionWorldState`",
  "one zero-managed-byte volatile root transfer", the journal, the peer rebases). A
  retired row still documents what shipped; leaving it describing a deleted clone is
  exactly the out-of-sync failure the register rules forbid. Rewrite it to the delta-apply
  mechanism. The row's retail anchor is already
  `CObjCell::init_objects` → `recalc_cross_cells` (0x0052b420 / 0x00515a30) — the new
  mechanism is **closer** to that anchor, so no new deviation row is created.
- **Judgment call for the implementer, do not assume:** the *original* AD-6 deviation was
  "Per-LANDBLOCK shadow re-flood on hydration vs retail per-CELL `recalc_cross_cells`"
  (`be94bc9b` register diff). If O3's commit-time reflood is again per-landblock rather
  than per-cell, decide explicitly whether AD-6 must be un-retired or a successor row
  added, and record the decision. Flagged, not decided here.
- **`docs/research/2026-07-31-atomic-collision-generation.md`** — steps 2, 3, 5, 6, 7, 8
  and most of the "Deterministic evidence" list describe the clone / journal / rebase.
  Rewrite in the same commit.
- **`memory/project_collision_port.md`** — the 37-line block `6b28ff99` added is now wrong.
- **`claude-memory/project_physics_collision_digest.md`** — add two DO-NOT-RETRY entries:
  (1) *"Do not re-introduce a whole-world staging clone. The atomic unit is the landblock
  delta applied in one update-thread call; the runtime is single-threaded and the root
  swap buys nothing."* (2) *"Batching N leaves per staging step is not a fix — measured
  1.8× at N=256, not convergence, and it trips the committed invariant test."*
- **`docs/ISSUES.md`** — F4 established this regression is not in the C3c diff, so it
  needs its own issue id (the C3c smoke-test commit `c52ce14a` filed #279 for a different
  finding). File it, and reference it from the O1/O2/O3 commit messages.
- **Rollback:** each slice lands as one commit with its own recorded `git revert` SHA,
  per the Modern Runtime convention.

---

## (d) Perf-work framing

**This is modern-runtime infrastructure, not retail-scoped behaviour work.** The delta
apply installs the *identical* `PreparedPhysicsDataCacheLandblock` content that
`TransferTo` publishes today — the same cells, flat cells, EnvCell topology, buildings,
terrain, synthesized outdoor cells, landblock, and owner set. Only the path by which that
content reaches the active root changes, and only the amount of work done to get there.
Collision results, contact planes, walkable polygons, membership, and therefore game feel
are bit-identical.

The project's render-perf-not-faithfulness-gated rule
(`claude-memory/feedback_render_perf_not_faithfulness_gated.md`) applies: throughput work
that is pixel- and feel-identical does not need a retail-behaviour gate. But because this
is collision, the acceptance bar is still the connected gates plus the user's visual pass
— green unit tests prove nothing about a streaming convergence bug.

Two guards keep the framing honest:

1. **The direction of travel is toward retail, not away.** Retail hydrates a cell
   synchronously in `CObjCell::init_objects` and refloods the objects associated with it
   via `recalc_cross_cells`. A per-landblock delta applied in one update-thread call is
   the streaming-shaped version of exactly that. The whole-world clone was the adaptation;
   removing it retires an adaptation rather than adding one.
2. **The one thing that could change feel is reflood timing** — owners near the target
   re-flooding at commit rather than from a pre-computed staged set.
   `CommitTimeRefloodMatchesPrecomputedReflood` (above) is the specific test that turns
   that from an assumption into evidence. If that test cannot be made to pass, the perf
   framing is void and the slice needs a behaviour gate.

**Explicitly not a workaround.** Per the no-workarounds rule, note what this is *not*: no
suppression flag, no grace period, no budget loosening, no early-return guard at the
symptom. The root cause is an algorithm that is quadratic in resident-world size, and the
fix is to make it linear by restoring the per-landblock atomic unit the mechanism had
before `6b28ff99`.

### Measure before and after

A stripped-after probe should count, per publication: (clone leaves, seal leaves, apply
leaves) and wall-clock for each. F4 measured only the clone (median 19,736 / p90 32,135 /
max 38,021 leaves, median 3.64 ms, 1,584 preparations = 8.53 s CPU in one 4-minute capped
session). The seal was never measured and D2 must beat both. Expected after O3: clone
leaves 0, seal leaves ≈ target payload, apply leaves ≈ target payload, total per
publication well under 100 µs and flat as the ring fills.