perf #418: parallelize landblock builds across a striped worker pool
Login publishes the 25x25 window at a flat 32 blocks/s (~27 s in the
tunnel). The reveal-timing probe A/B (695a27b4) showed the consumer
budget env ceilings change nothing, which was read as producer-limited:
one "acdream.streaming.worker" thread, ~31 ms/block. This replaces the
single worker with min(ProcessorCount-2, 8) workers, floor 1.
Design: striped/affinity dispatch. Each worker owns one unbounded lane
channel plus its own high/low priority queues; jobs route to
lane = ((id >> 16) * 2654435761) % N (the low word of a landblock id is
constant, so the id is mixed before reduction). Striping was chosen
over a shared queue + in-flight conflict tracker because it preserves
the per-landblock contract structurally rather than by bookkeeping:
every job for one id lives on one lane, so per-id enqueue order IS
execution and completion-arrival order, and the same-landblock
supersede rules (PromoteToNear removes queued LoadFar/Unload) keep
seeing every queued job for that id. Contract, point by point:
- Per-landblock ordering: same id -> same lane -> serial FIFO.
- ClearLoads: broadcast to every lane inside the same _inboxGate lock
that serializes enqueues, so any load enqueued before
ClearPendingLoads() returns sits ahead of its lane's ClearLoads copy
in that lane's FIFO and is dropped at read time, exactly like the
single-thread path. Already-dequeued builds still complete (now up
to one per worker instead of one total); StreamingController's
SweepCollapsed already unloads those uniformly.
- Priority: per-lane high/low split unchanged. Cross-lane, priority is
not globally ordered (a lane cannot run another lane's job), which
the contract permits; near-tier jobs hash-spread across lanes and
are preferred within each.
- Outbox: SingleWriter flipped to false; nothing assumed single-writer
(PublishResult already used TryWrite + an Interlocked backlog, and
the consumer's peek->read head-stability holds because only the
single reader ever moves the head). Cross-landblock arrival order
was verified arbitrary-tolerant before relying on it:
StreamingController.AdmitCompletions classifies each result
independently into per-priority FIFOs (generation staleness +
per-landblock retirement blocking); per-landblock arrival order is
preserved by striping.
- Crash surface: per-worker. The first real crash publishes
WorkerCrashed (prefixed "worker N:" in pools > 1), sets
_workerFailure, completes every lane, and cancels the pool (a crash
still ends all processing, as before); siblings that merely observe
the closed lanes (ChannelClosedException) exit quietly instead of
reporting spurious crashes; the outbox completes only when the LAST
worker exits so no in-flight completions are dropped.
- Disposal: joins every worker under the same _disposeGate; Start
stays idempotent and dispose-serialized.
Thread-safety audit of the production build closures
(SessionPlayerComposition), per shared object:
- DatCollection (every read in LandblockBuildFactory.BuildLocked:
LandblockLoader.Load, SceneryGenerator.Generate, SetupMesh.Flatten,
CellMesh.Build, GfxObjBounds.Get, GfxObjDegradeResolver): NOT
thread-safe; already serialized under the shared _datLock, which
BuildLocked holds for the whole read transaction. Unchanged; the
probe run measured hold 0-13 ms / wait <= 12 ms during the login
window, so the lock is not the new bottleneck and the build was NOT
serialized beyond it.
- PakPreparedAssetSource / PakReader (BuildPreparedCollisionClosure,
outside the lock): immutable TOC array + read-only
MemoryMappedViewAccessor random-access reads + ConcurrentDictionary
verdict caches - safe for N concurrent readers (Slice I3 design;
the headless SharedPreparedCollisionCache wrapper is fully
lock-protected).
- LandblockMesh.Build (outside the lock): pure math over the dat
record + the composition-time height table + the immutable
TerrainBlendingContext record; the shared SurfaceCache is a
ConcurrentDictionary and BuildSurface is deterministic, so its
lookup-or-build race is last-write-wins-benign (the code already
documented exactly this).
- PhysicsDiagnostics probe statics: read-only bools + thread-safe
Console writes.
MEASURED OUTCOME (gate 4): the timing acceptance did NOT pass, and per
the task contract that is reported, not tuned around. With 8 workers
on this 16-core machine all 625 builds complete in ~203 ms
(ACDREAM_PROBE_TELEPORT BUILD lines t=3475390..3475593) - the producer
is off the critical path - but loaded= still advances at exactly
+32/1000 ms and SUMMARY totalMs measured 27395 and 27503 across two
runs (baseline 26728). The 32/s pacer is in the consumer
admission/publication path and is not governed by the
StreamingWorkBudgetOptions env ceilings. #418 stays IN-PROGRESS on the
consumer side; see docs/ISSUES.md for the evidence chain.
Tests: per-landblock ordering under 4-worker contention, cross-lane
ClearLoads drop, per-lane near-before-far preference, pool-of-1 serial
equivalence, disposal joining every worker, lane-spread guard, and
worker-count validation (LandblockStreamerPoolTests). Two existing
tests asserted a GLOBAL cross-landblock execution order - a serial
implementation detail, not the contract - and now pin workerCount: 1
with justification comments (LoadNear_OvertakesQueuedFarLoads,
TwoQueuedLoads_RetainTheirDistinctOriginAndGeneration).
Gates: Release build 0 errors; App suite 5575 passed / 3 skipped
(5568 + 7 new); Runtime suite 1756/0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
695a27b48a
commit
39967e78bd
5 changed files with 643 additions and 59 deletions
|
|
@ -24,6 +24,37 @@ What does NOT go here:
|
|||
- Every session: scan OPEN issues at start; promote/close anything we touched during the session before ending.
|
||||
- Promoting to a Phase: mark as `DONE (promoted to Phase X)` + commit SHA where the Phase entry landed.
|
||||
|
||||
## #418 — Login world load takes ~27 s: publication advances at a flat 32 blocks/s
|
||||
|
||||
**Status:** IN-PROGRESS 2026-08-17 — producer half landed (this commit's
|
||||
striped `LandblockStreamer` worker pool); the pacer measurably remains on the
|
||||
consumer side. **Symptom:** login holds the portal tunnel ~27 s while the
|
||||
25×25 window (625 landblocks) drips in at exactly 32 blocks/s
|
||||
(`ACDREAM_PROBE_REVEAL_TIMING=1`, probe `695a27b4`; baseline
|
||||
totalMs=26728), then render/composites/collision/gate/materialization all
|
||||
flip ready in the same millisecond. **Evidence chain:** an A/B with every
|
||||
`StreamingWorkBudgetOptions` env ceiling cranked 25–64x changed nothing →
|
||||
read as producer-limited (ONE `acdream.streaming.worker` thread,
|
||||
~31 ms/block serial). This commit parallelized the producer
|
||||
(min(cores−2, 8) striped workers, per-landblock ordering preserved) and
|
||||
**disproved that reading**: with 8 workers ALL 625 builds complete in
|
||||
~203 ms of wall clock (`ACDREAM_PROBE_TELEPORT=1` BUILD lines
|
||||
t=3475390→3475593, dat-lock waited ≤ 12 ms, held 0–13 ms), yet `loaded=`
|
||||
still advances at exactly +32/1000 ms and totalMs measured 27395 / 27503
|
||||
across two runs of the new binary. **Hypothesis:** the 32/s cadence lives in
|
||||
the update-thread admission/publication path (`StreamingController` meter →
|
||||
`LandblockPresentationPipeline`), is frame-quantized (an exactly-integer
|
||||
per-second rate held for 14+ consecutive seconds — N update ticks per
|
||||
landblock at a stable tick rate, e.g. 2 ticks × 64 Hz), and is NOT governed
|
||||
by the budget env ceilings (the original A/B and this change now agree on
|
||||
that). **Next step:** instrument per-frame meter operations/yields + the
|
||||
tunnel frame rate, find which operation stage eats the ensured-progress
|
||||
floor, then lift the actual limiter. The producer pool stays: it takes the
|
||||
builds off the critical path (23 s → 0.2 s) and is quality-neutral
|
||||
(per-landblock ordering, ClearLoads, priority, and disposal semantics
|
||||
preserved; a pool of 1 reproduces the old serial behavior,
|
||||
regression-tested in `LandblockStreamerPoolTests`).
|
||||
|
||||
## #417 — World ambience keeps playing (and re-firing) on the character-select screen after the in-world logoff
|
||||
|
||||
**Status:** ✅ FIXED 2026-08-17 (logout-audio round; fix + tests in the same
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue