The transport row said to cap xUnit.MaxParallelThreads; serializing that
assembly actually REGRESSED Windows (1000 passed in 7 s -> 999/1000 in 17 s)
and the real fix was widening the virtual-clock harness's wall-clock patience.
The Avalonia row said to serialize via xunit.runner.json; that does not fix it
either — proven twice, including with a compiled-in attribute so delivery could
not be blamed. It needs a real desktop and is now Lane=Manual.
Both rows now record what was tried and disproved, which is the part worth
keeping.
docs/ci-and-releases.md documents what happens on a push to main, why Gitea
rather than GitHub (billing-blocked, private repo, and Forgejo ships no hosted
runners), both runners and their prerequisites, the release/manifest layout,
and how to verify a release with the Lane=Live install test.
Its landmine table is the part worth keeping: every row cost a red pipeline —
Node for JS actions, setup-dotnet unmirrored on data.forgejo.org, the
zombie-task timeout caused by run-release-gate.ps1 redirecting child output,
an en-SE runner locale breaking 40 tests on decimal commas, DAT tests missing
their InstalledDat lane tag, parallel-load timing flakes, and the Avalonia
compositor threading failure that must NOT be 'fixed' by de-async-ing the test.
Also records the culture finding: config, parsing and the wire are all
invariant-safe, so US and European installs behave identically; only
diagnostic strings follow the current culture.
Cross-linked from docs/README.md and release-gate.md, which keeps ownership of
the local bounded gate.
Both attempts removed from the branch by user direction after live
gates (attempt 1 culled the interior — the 'fixed' exit was a
nothing-drawn false positive; attempt 2's luminosity port produced no
visible change, theory-vs-plumbing unresolved). Patches preserved in
the session scratchpad. Next attempt starts with apparatus: tunnel
freeze probe, RenderDoc capture, ACViewer oracle, retail side-by-side.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
While a destination reservation hides the world behind the authored
tunnel, the streaming frame meter now runs a hold-widened profile
(StreamingWorkBudget.WidenForDestinationHold): the time ceiling rises
from the authored 2 ms to an absolute 8 ms default
(ACDREAM_STREAM_WORK_HOLD_DEST_MS is a measurement-only override), every
count/byte dimension scales by the same factor so elapsed time stays the
authoritative guard (the measured binder is Time at both ceilings), and
the reserve fraction is re-derived (0.75 -> 0.9375) so the
NON-destination lane's absolute per-frame caps are unchanged. The
widening keys off the existing BeginDestinationReservation/
EndDestinationReservation bracket only, is derived per-Tick from the
CURRENT budget (mid-hold quality swaps compose), and a frame with no
reservation uses the authored budget verbatim (test-pinned). Portal
holds ride the same bracket as login holds by construction - intended,
and pinned by a kind-parity test through the real coordinator plus a
live @telepoi portal hold (kind=portal gate-ready 3589 ms).
Why: issue #418's next-hypothesis (1). Measured result: the ~5 s
publication drip collapsed to ~2 s (loaded 625/625 at ~3.0 s, tunnel at
64-66 fps), the portal-hold gate-ready fell to ~3.6 s - and login
gate-ready/total stayed at 8.4-8.8 s / 12.6-12.7 s, exposing the real
remaining pacer: the login-cold render-thread upload/registration
barrier behind GpuWorldState.IsRenderReady, which ran concurrently under
the old drip. Full attribution appended to docs/ISSUES.md #418; no
divergence-register row (the streamed result and reveal gate are
byte-identical; only the scheduling rate during a hidden hold changed).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The user's benchmark (retail ~6 s at a smaller window without our
complete-at-reveal guarantee) is now the issue's done bar. Next levers
in order: hold-scoped destination-lane budget widening (predicted
gate-ready ~2-3 s), then upload/composite overlap; retail's authored
tunnel exit is not a tuning target. Miss-branch closeout per the
session goal: acceptance measured 12689/12734 ms (> 12000), attribution
complete, no blind tuning.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase-1 measurement (new [publish-timing]/[stream-tick] probe surfaces,
ACDREAM_PROBE_REVEAL_TIMING=1) refuted the ~31 ms-per-admission
hypothesis: the hold runs at ~64 fps with the streaming tick at ~32 Hz,
the whole 625-block window costs only ~500 ms of publication CPU (far
blocks ~0.17 ms, near 2-43 ms), and steady state showed ZERO meter
yields with ~0.22 ms of the 2 ms budget used - yet exactly one block
published per tick against a ~400-deep completion queue.
The real limiter: Runtime's collision-generation activation is a
deliberate two-poll transaction (the first
TryAcquireCollisionPrefixMutationPermission poll parks residents and
refuses by design), and LandblockPresentationPipeline.Advance's metered
arm returned Completed=false on ANY nonterminal commit, which
DrainAndApply treats as end-of-frame. One landblock per 32 Hz tick =
the flat 32/s, with the authored budget ~90% idle.
Fix: the metered arm now uses the same Runtime-owned gate the unmetered
arm and the synchronous CompletePublication API always used
(CanContinueMutationSynchronously). The second poll runs in the same
frame under the same meter, so the unchanged 2 ms elapsed-time ceiling
is now genuinely the authoritative per-frame bound; with any real debt
(live residents parked mid-game, pending withdrawals, dispatch backlog)
publication defers to the next frame exactly as before. No budget
values change, no reveal-gate/readiness change, and the streamed result
is byte-identical - only the frame scheduling of identical operations.
Measured A/B (this binary, two runs): totalMs 12689 / 12734 vs baseline
26728/27395/27503; loaded slope 32/s -> bursts of 100-360/s, 625/625 in
~6-7 s vs ~23 s. The remaining ~12.7 s floor is fully attributed in
docs/ISSUES.md: ~8 s of real budgeted readiness work plus retail's
authored tunnel exit (TunnelContinue 2-5 s + two 1 s fades, golden
constants), so the <12 s acceptance needs a lead decision on the
hold-time budget, not another hidden limiter.
New regression pin:
MeteredLoaded_NonterminalCommitWithoutDebt_CompletesInOneMeteredAdvance.
Gates: Release build 0 errors; App tests 5576/3 skips/0 failed;
Runtime tests 1756/0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The identical flat 32/s across the serial and pooled binaries points at a
single ~31 ms indivisible per-landblock admission on the update thread
(one admission per frame under the 2 ms floor, frame stretched to ~31 ms,
~32 fps x 1/frame = 32/s); the old serial builder producing at the same
~31 ms/block masked it. Predicts ~31 ms tunnel frame time; fix shape is
splitting/off-threading the publication cost, not budgets (disproved
twice). Verify the frame-time prediction first next session.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Login publishes the 25x25 window at a flat 32 blocks/s (~27 s in the
tunnel). The reveal-timing probe A/B (695a27b4) showed the consumer
budget env ceilings change nothing, which was read as producer-limited:
one "acdream.streaming.worker" thread, ~31 ms/block. This replaces the
single worker with min(ProcessorCount-2, 8) workers, floor 1.
Design: striped/affinity dispatch. Each worker owns one unbounded lane
channel plus its own high/low priority queues; jobs route to
lane = ((id >> 16) * 2654435761) % N (the low word of a landblock id is
constant, so the id is mixed before reduction). Striping was chosen
over a shared queue + in-flight conflict tracker because it preserves
the per-landblock contract structurally rather than by bookkeeping:
every job for one id lives on one lane, so per-id enqueue order IS
execution and completion-arrival order, and the same-landblock
supersede rules (PromoteToNear removes queued LoadFar/Unload) keep
seeing every queued job for that id. Contract, point by point:
- Per-landblock ordering: same id -> same lane -> serial FIFO.
- ClearLoads: broadcast to every lane inside the same _inboxGate lock
that serializes enqueues, so any load enqueued before
ClearPendingLoads() returns sits ahead of its lane's ClearLoads copy
in that lane's FIFO and is dropped at read time, exactly like the
single-thread path. Already-dequeued builds still complete (now up
to one per worker instead of one total); StreamingController's
SweepCollapsed already unloads those uniformly.
- Priority: per-lane high/low split unchanged. Cross-lane, priority is
not globally ordered (a lane cannot run another lane's job), which
the contract permits; near-tier jobs hash-spread across lanes and
are preferred within each.
- Outbox: SingleWriter flipped to false; nothing assumed single-writer
(PublishResult already used TryWrite + an Interlocked backlog, and
the consumer's peek->read head-stability holds because only the
single reader ever moves the head). Cross-landblock arrival order
was verified arbitrary-tolerant before relying on it:
StreamingController.AdmitCompletions classifies each result
independently into per-priority FIFOs (generation staleness +
per-landblock retirement blocking); per-landblock arrival order is
preserved by striping.
- Crash surface: per-worker. The first real crash publishes
WorkerCrashed (prefixed "worker N:" in pools > 1), sets
_workerFailure, completes every lane, and cancels the pool (a crash
still ends all processing, as before); siblings that merely observe
the closed lanes (ChannelClosedException) exit quietly instead of
reporting spurious crashes; the outbox completes only when the LAST
worker exits so no in-flight completions are dropped.
- Disposal: joins every worker under the same _disposeGate; Start
stays idempotent and dispose-serialized.
Thread-safety audit of the production build closures
(SessionPlayerComposition), per shared object:
- DatCollection (every read in LandblockBuildFactory.BuildLocked:
LandblockLoader.Load, SceneryGenerator.Generate, SetupMesh.Flatten,
CellMesh.Build, GfxObjBounds.Get, GfxObjDegradeResolver): NOT
thread-safe; already serialized under the shared _datLock, which
BuildLocked holds for the whole read transaction. Unchanged; the
probe run measured hold 0-13 ms / wait <= 12 ms during the login
window, so the lock is not the new bottleneck and the build was NOT
serialized beyond it.
- PakPreparedAssetSource / PakReader (BuildPreparedCollisionClosure,
outside the lock): immutable TOC array + read-only
MemoryMappedViewAccessor random-access reads + ConcurrentDictionary
verdict caches - safe for N concurrent readers (Slice I3 design;
the headless SharedPreparedCollisionCache wrapper is fully
lock-protected).
- LandblockMesh.Build (outside the lock): pure math over the dat
record + the composition-time height table + the immutable
TerrainBlendingContext record; the shared SurfaceCache is a
ConcurrentDictionary and BuildSurface is deterministic, so its
lookup-or-build race is last-write-wins-benign (the code already
documented exactly this).
- PhysicsDiagnostics probe statics: read-only bools + thread-safe
Console writes.
MEASURED OUTCOME (gate 4): the timing acceptance did NOT pass, and per
the task contract that is reported, not tuned around. With 8 workers
on this 16-core machine all 625 builds complete in ~203 ms
(ACDREAM_PROBE_TELEPORT BUILD lines t=3475390..3475593) - the producer
is off the critical path - but loaded= still advances at exactly
+32/1000 ms and SUMMARY totalMs measured 27395 and 27503 across two
runs (baseline 26728). The 32/s pacer is in the consumer
admission/publication path and is not governed by the
StreamingWorkBudgetOptions env ceilings. #418 stays IN-PROGRESS on the
consumer side; see docs/ISSUES.md for the evidence chain.
Tests: per-landblock ordering under 4-worker contention, cross-lane
ClearLoads drop, per-lane near-before-far preference, pool-of-1 serial
equivalence, disposal joining every worker, lane-spread guard, and
worker-count validation (LandblockStreamerPoolTests). Two existing
tests asserted a GLOBAL cross-landblock execution order - a serial
implementation detail, not the contract - and now pin workerCount: 1
with justification comments (LoadNear_OvertakesQueuedFarLoads,
TwoQueuedLoads_RetainTheirDistinctOriginAndGeneration).
Gates: Release build 0 errors; App suite 5575 passed / 3 skipped
(5568 + 7 new); Runtime suite 1756/0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The character-session reset manifest had no audio step: retail's logoff
destroys the world's sound sources with the world, but our OpenAL world
pool and ambient scheduler are process-lifetime — the continuous ambient
beds played on at character select and the scheduler kept RE-FIRING
deadlines against the stale listener (Suspend/StopAll had zero callers;
WorldGenerationQuiescence only cycles around teleport-style generation
replaces).
New WorldAudioSessionGate: the reset manifest's 'world audio' step stops
all sixteen world-pool voices (SuspendWorldAudio) and drops every ambient
deadline (StopAll); the pool reopens at the entered-world edge through the
new default-null LiveSessionEnteredWorldBindings.ResumeWorldAudio binding,
invoked first in ApplyEnteredWorld. The ambient soundscape needs no
explicit resume — the next objcell observation rebuilds it exactly as a
cell change always did. Covers logout, reconnect, and full stop uniformly.
UI-pool sounds (interface bank, portal cues) untouched by design.
App tests 5568/3 skips, Runtime 1756/0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
#416 (char-select roster highlight never cleared on hover-leave): three
decomp-grounded mechanisms replace the media-keyed _availableStates
approximation.
- UIElement_Button::UpdateState_ @0x00471CF0: the button machine commits
ONLY states authored on the button's OWN ElementDesc (AccessStateDesc
gate); unauthored requests no-op, preserving custom semantic states.
- UIElement::SetState @0x00464E70: an unauthored state id is coerced to
state 0 (the unnamed base state) and committed — ported into
UiDatElement.TrySetRetailState with the base-descriptor PassToChildren
cascade arm.
- The SetState media rule @0x004651c0: a committed state replaces the
playing media ONLY when its media array is non-empty. UiButton now keeps
per-face-segment media states under that rule (segments model retail's
PassToChildren children), and LayoutImporter records the raw MediaCount
including the File=0 draw-nothing images the drawable filter drops —
the roster bar children's base state is exactly such an image, and it is
what clears the bar.
The row template truth (probe, installed DAT): the row authors EMPTY
Normal/rollover/Highlight descriptors with PassToChildren; the three bar
children author rollover/Highlight media, NO Normal state, and a File=0
base image. An empty-media Normal_pressed still never blanks a Normal-art
button (the media rule keeps the previous art — the exact behavior the
old gate approximated), and the Appearance spins' property-only Highlight
now genuinely commits: label recolors, arrow art lingers — the retail
split AP-222 approximated with a requested-keyed label hack, now retired.
Live-verified at char select: hover +alex shows the grey bar, moving off
clears it, the selected row keeps its amber bar.
#415 (probe wait world-* verbs dead): the filed snapshot-reset diagnosis
was wrong — the automation bridge simply never bound without
ACDREAM_AUTOMATION_ARTIFACT_DIR. A facts-only
WorldRevealFactsAutomationRuntime now binds whenever the retained UI
exists; checkpoint/screenshot verbs still require the artifact directory
and now report that instead of a generic timeout.
App tests 5568/3 skips, Runtime 1756/0, UI.Abstractions 926/0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
