acdream/tests/AcDream.App.Tests/Streaming/LandblockBuildOriginTests.cs
Erik 39967e78bd perf #418: parallelize landblock builds across a striped worker pool
Login publishes the 25x25 window at a flat 32 blocks/s (~27 s in the
tunnel). The reveal-timing probe A/B (695a27b4) showed the consumer
budget env ceilings change nothing, which was read as producer-limited:
one "acdream.streaming.worker" thread, ~31 ms/block. This replaces the
single worker with min(ProcessorCount-2, 8) workers, floor 1.

Design: striped/affinity dispatch. Each worker owns one unbounded lane
channel plus its own high/low priority queues; jobs route to
lane = ((id >> 16) * 2654435761) % N (the low word of a landblock id is
constant, so the id is mixed before reduction). Striping was chosen
over a shared queue + in-flight conflict tracker because it preserves
the per-landblock contract structurally rather than by bookkeeping:
every job for one id lives on one lane, so per-id enqueue order IS
execution and completion-arrival order, and the same-landblock
supersede rules (PromoteToNear removes queued LoadFar/Unload) keep
seeing every queued job for that id. Contract, point by point:

- Per-landblock ordering: same id -> same lane -> serial FIFO.
- ClearLoads: broadcast to every lane inside the same _inboxGate lock
  that serializes enqueues, so any load enqueued before
  ClearPendingLoads() returns sits ahead of its lane's ClearLoads copy
  in that lane's FIFO and is dropped at read time, exactly like the
  single-thread path. Already-dequeued builds still complete (now up
  to one per worker instead of one total); StreamingController's
  SweepCollapsed already unloads those uniformly.
- Priority: per-lane high/low split unchanged. Cross-lane, priority is
  not globally ordered (a lane cannot run another lane's job), which
  the contract permits; near-tier jobs hash-spread across lanes and
  are preferred within each.
- Outbox: SingleWriter flipped to false; nothing assumed single-writer
  (PublishResult already used TryWrite + an Interlocked backlog, and
  the consumer's peek->read head-stability holds because only the
  single reader ever moves the head). Cross-landblock arrival order
  was verified arbitrary-tolerant before relying on it:
  StreamingController.AdmitCompletions classifies each result
  independently into per-priority FIFOs (generation staleness +
  per-landblock retirement blocking); per-landblock arrival order is
  preserved by striping.
- Crash surface: per-worker. The first real crash publishes
  WorkerCrashed (prefixed "worker N:" in pools > 1), sets
  _workerFailure, completes every lane, and cancels the pool (a crash
  still ends all processing, as before); siblings that merely observe
  the closed lanes (ChannelClosedException) exit quietly instead of
  reporting spurious crashes; the outbox completes only when the LAST
  worker exits so no in-flight completions are dropped.
- Disposal: joins every worker under the same _disposeGate; Start
  stays idempotent and dispose-serialized.

Thread-safety audit of the production build closures
(SessionPlayerComposition), per shared object:

- DatCollection (every read in LandblockBuildFactory.BuildLocked:
  LandblockLoader.Load, SceneryGenerator.Generate, SetupMesh.Flatten,
  CellMesh.Build, GfxObjBounds.Get, GfxObjDegradeResolver): NOT
  thread-safe; already serialized under the shared _datLock, which
  BuildLocked holds for the whole read transaction. Unchanged; the
  probe run measured hold 0-13 ms / wait <= 12 ms during the login
  window, so the lock is not the new bottleneck and the build was NOT
  serialized beyond it.
- PakPreparedAssetSource / PakReader (BuildPreparedCollisionClosure,
  outside the lock): immutable TOC array + read-only
  MemoryMappedViewAccessor random-access reads + ConcurrentDictionary
  verdict caches - safe for N concurrent readers (Slice I3 design;
  the headless SharedPreparedCollisionCache wrapper is fully
  lock-protected).
- LandblockMesh.Build (outside the lock): pure math over the dat
  record + the composition-time height table + the immutable
  TerrainBlendingContext record; the shared SurfaceCache is a
  ConcurrentDictionary and BuildSurface is deterministic, so its
  lookup-or-build race is last-write-wins-benign (the code already
  documented exactly this).
- PhysicsDiagnostics probe statics: read-only bools + thread-safe
  Console writes.

MEASURED OUTCOME (gate 4): the timing acceptance did NOT pass, and per
the task contract that is reported, not tuned around. With 8 workers
on this 16-core machine all 625 builds complete in ~203 ms
(ACDREAM_PROBE_TELEPORT BUILD lines t=3475390..3475593) - the producer
is off the critical path - but loaded= still advances at exactly
+32/1000 ms and SUMMARY totalMs measured 27395 and 27503 across two
runs (baseline 26728). The 32/s pacer is in the consumer
admission/publication path and is not governed by the
StreamingWorkBudgetOptions env ceilings. #418 stays IN-PROGRESS on the
consumer side; see docs/ISSUES.md for the evidence chain.

Tests: per-landblock ordering under 4-worker contention, cross-lane
ClearLoads drop, per-lane near-before-far preference, pool-of-1 serial
equivalence, disposal joining every worker, lane-spread guard, and
worker-count validation (LandblockStreamerPoolTests). Two existing
tests asserted a GLOBAL cross-landblock execution order - a serial
implementation detail, not the contract - and now pin workerCount: 1
with justification comments (LoadNear_OvertakesQueuedFarLoads,
TwoQueuedLoads_RetainTheirDistinctOriginAndGeneration).

Gates: Release build 0 errors; App suite 5575 passed / 3 skipped
(5568 + 7 new); Runtime suite 1756/0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-17 19:34:20 +02:00

473 lines
18 KiB
C#

using AcDream.App.Rendering.Wb;
using AcDream.App.Streaming;
using AcDream.Core.Terrain;
using AcDream.Core.World;
using DatReaderWriter.DBObjs;
namespace AcDream.App.Tests.Streaming;
public sealed class LandblockBuildOriginTests
{
private const int SpinTimeoutMs = 2000;
private const int SpinStepMs = 10;
[Fact]
public void ExplicitMapZeroOrigin_IsDistinctFromUnspecifiedCompatibilityOrigin()
{
var mapZero = new LandblockBuildOrigin(0, 0);
Assert.True(mapZero.IsSpecified);
Assert.False(default(LandblockBuildOrigin).IsSpecified);
Assert.NotEqual(default, mapZero);
}
[Fact]
public async Task RequestAwareLoad_CarriesExplicitMapZeroOriginThroughFactoryAndCompletion()
{
const uint landblockId = 0x0000FFFFu;
var capturedOrigin = new LandblockBuildOrigin(0, 0);
LandblockBuildRequest? observedRequest = null;
using var streamer = LandblockStreamer.CreateForRequests(
loadLandblock: request =>
{
observedRequest = request;
return EmptyBuild(request.LandblockId, request.Origin);
},
buildMeshOrNull: (_, _) => EmptyMesh());
streamer.EnqueueLoad(new LandblockBuildRequest(
landblockId,
LandblockStreamJobKind.LoadNear,
Generation: 77,
capturedOrigin));
streamer.Start();
var loaded = Assert.IsType<LandblockStreamResult.Loaded>(
await DrainFirstAsync(streamer));
Assert.Equal(
new LandblockBuildRequest(
landblockId,
LandblockStreamJobKind.LoadNear,
77,
capturedOrigin),
observedRequest);
Assert.Equal(capturedOrigin, loaded.Build.Origin);
Assert.Equal(77ul, loaded.Generation);
}
[Fact]
public async Task RequestAwareLoad_WhenFactoryReturnsUnspecifiedOrigin_FailsAtMapZero()
{
using var streamer = LandblockStreamer.CreateForRequests(
loadLandblock: request => EmptyBuild(request.LandblockId, default));
streamer.EnqueueLoad(new LandblockBuildRequest(
0x0000FFFFu,
LandblockStreamJobKind.LoadNear,
Generation: 78,
new LandblockBuildOrigin(0, 0)));
streamer.Start();
var failed = Assert.IsType<LandblockStreamResult.Failed>(
await DrainFirstAsync(streamer));
Assert.Contains("origin", failed.Error, StringComparison.OrdinalIgnoreCase);
Assert.Equal(78ul, failed.Generation);
}
[Fact]
public void CompatibilityLoader_RejectsEvenExplicitMapZeroOriginBeforeQueueing()
{
using var streamer = new LandblockStreamer(loadLandblock: _ => null);
Assert.Throws<InvalidOperationException>(() => streamer.EnqueueLoad(
new LandblockBuildRequest(
0x0000FFFFu,
LandblockStreamJobKind.LoadNear,
Generation: 79,
new LandblockBuildOrigin(0, 0))));
}
[Fact]
public void RequestAwareLoader_RejectsOriginlessCompatibilityEnqueue()
{
using var streamer = LandblockStreamer.CreateForRequests(
loadLandblock: request => EmptyBuild(request.LandblockId, request.Origin));
Assert.Throws<InvalidOperationException>(() =>
streamer.EnqueueLoad(0xA9B4FFFFu));
}
[Fact]
public void RequestAwareLoader_RejectsDirectRequestWithUnspecifiedOrigin()
{
using var streamer = LandblockStreamer.CreateForRequests(
loadLandblock: request => EmptyBuild(request.LandblockId, request.Origin));
Assert.Throws<InvalidOperationException>(() => streamer.EnqueueLoad(
new LandblockBuildRequest(
0xA9B4FFFFu,
LandblockStreamJobKind.LoadNear,
Generation: 80,
Origin: default)));
}
[Fact]
public async Task TwoQueuedLoads_RetainTheirDistinctOriginAndGeneration()
{
var observed = new List<LandblockBuildRequest>();
var firstOrigin = new LandblockBuildOrigin(0xA9, 0xB4);
var secondOrigin = new LandblockBuildOrigin(0x71, 0xEC);
// workerCount: 1 — the ordered `observed`/completion asserts span two
// DIFFERENT landblock ids; only the serial degenerate pool (#418)
// guarantees that global order (per-id order is the pool contract).
using var streamer = LandblockStreamer.CreateForRequests(
loadLandblock: request =>
{
observed.Add(request);
return EmptyBuild(request.LandblockId, request.Origin);
},
buildMeshOrNull: (_, _) => EmptyMesh(),
workerCount: 1);
streamer.EnqueueLoad(new LandblockBuildRequest(
0xA9B4FFFFu,
LandblockStreamJobKind.LoadNear,
Generation: 80,
firstOrigin));
streamer.EnqueueLoad(new LandblockBuildRequest(
0x71ECFFFFu,
LandblockStreamJobKind.LoadNear,
Generation: 81,
secondOrigin));
streamer.Start();
IReadOnlyList<LandblockStreamResult> completions =
await DrainCountAsync(streamer, 2);
Assert.Equal([firstOrigin, secondOrigin], observed.Select(request => request.Origin));
Assert.Equal([80ul, 81ul], completions.Select(result => result.Generation));
Assert.Equal(
[firstOrigin, secondOrigin],
completions
.Cast<LandblockStreamResult.Loaded>()
.Select(result => result.Build.Origin));
}
[Fact]
public async Task QueuedPromotion_SupersedesOldFarWithItsOwnOriginAndGeneration()
{
var observed = new List<LandblockBuildRequest>();
var oldOrigin = new LandblockBuildOrigin(0xA9, 0xB4);
var newOrigin = new LandblockBuildOrigin(0x71, 0xEC);
const uint landblockId = 0x71ECFFFFu;
using var streamer = LandblockStreamer.CreateForRequests(
loadLandblock: request =>
{
observed.Add(request);
return EmptyBuild(request.LandblockId, request.Origin);
},
buildMeshOrNull: (_, _) => EmptyMesh());
streamer.EnqueueLoad(new LandblockBuildRequest(
landblockId,
LandblockStreamJobKind.LoadFar,
Generation: 82,
oldOrigin));
streamer.EnqueueLoad(new LandblockBuildRequest(
landblockId,
LandblockStreamJobKind.PromoteToNear,
Generation: 83,
newOrigin));
streamer.Start();
var promoted = Assert.IsType<LandblockStreamResult.Promoted>(
await DrainFirstAsync(streamer));
Assert.Equal([new LandblockBuildRequest(
landblockId,
LandblockStreamJobKind.PromoteToNear,
83,
newOrigin)], observed);
Assert.Equal(newOrigin, promoted.Build.Origin);
Assert.Equal(83ul, promoted.Generation);
}
[Fact]
public async Task FarLoad_StripsEnvCellsAndPhysicsEvenWhenEntityListIsAlreadyEmpty()
{
const uint landblockId = 0xA9B4FFFFu;
var origin = new LandblockBuildOrigin(0xA9, 0xB4);
var physics = new PhysicsDatBundle(
new LandBlockInfo(),
new Dictionary<uint, EnvCell>(),
new Dictionary<uint, DatReaderWriter.DBObjs.Environment>(),
new Dictionary<uint, Setup>(),
new Dictionary<uint, GfxObj>());
var envCells = new EnvCellLandblockBuild(
landblockId,
Array.Empty<AcDream.App.Rendering.LoadedCell>(),
Array.Empty<EnvCellShellPlacement>());
using var streamer = LandblockStreamer.CreateForRequests(
loadLandblock: request => new LandblockBuild(
new LoadedLandblock(
request.LandblockId,
new LandBlock(),
Array.Empty<WorldEntity>(),
physics),
envCells,
request.Origin),
buildMeshOrNull: (_, _) => EmptyMesh());
streamer.EnqueueLoad(new LandblockBuildRequest(
landblockId,
LandblockStreamJobKind.LoadFar,
Generation: 84,
origin));
streamer.Start();
#if DEBUG
// The near-payload tripwire is config-divergent by design ("fail loud
// in Debug builds and strip in Release" — LandblockStreamer.HandleJob):
// in Debug the Debug.Assert fires and the VSTest host translates it
// into a thrown DebugAssertException, which the worker's catch folds
// into a Failed completion (#351).
var failed = Assert.IsType<LandblockStreamResult.Failed>(
await DrainFirstAsync(streamer));
Assert.Contains(
"Far-tier factory returned Near payload",
failed.Error,
StringComparison.Ordinal);
Assert.Equal(84ul, failed.Generation);
#else
var loaded = Assert.IsType<LandblockStreamResult.Loaded>(
await DrainFirstAsync(streamer));
Assert.Equal(LandblockStreamTier.Far, loaded.Tier);
Assert.Empty(loaded.Landblock.Entities);
Assert.Same(PhysicsDatBundle.Empty, loaded.Landblock.PhysicsDats);
Assert.Null(loaded.Build.EnvCells);
Assert.Equal(origin, loaded.Build.Origin);
#endif
}
[Fact]
public void BuildFactoryAndRenderPublisher_UseCapturedOriginWithoutGameWindowFacade()
{
string root = FindRepoRoot();
string gameWindowSource = File.ReadAllText(Path.Combine(
root,
"src",
"AcDream.App",
"Rendering",
"GameWindow.cs"));
string livePresentationSource = File.ReadAllText(Path.Combine(
root,
"src",
"AcDream.App",
"Composition",
"LivePresentationComposition.cs"));
string buildSource = File.ReadAllText(Path.Combine(
root,
"src",
"AcDream.App",
"Streaming",
"LandblockBuildFactory.cs"));
string renderPublisherSource = File.ReadAllText(Path.Combine(
root,
"src",
"AcDream.App",
"Streaming",
"LandblockRenderPublisher.cs"));
string recenterSource = File.ReadAllText(Path.Combine(
root,
"src",
"AcDream.App",
"Streaming",
"StreamingOriginRecenterCoordinator.cs"));
Assert.Contains("request.Origin", buildSource, StringComparison.Ordinal);
Assert.DoesNotContain("_liveCenterX", buildSource, StringComparison.Ordinal);
Assert.DoesNotContain("_liveCenterY", buildSource, StringComparison.Ordinal);
Assert.DoesNotContain(
"BuildLandblockForStreaming",
gameWindowSource,
StringComparison.Ordinal);
Assert.DoesNotContain(
"BuildSceneryEntitiesForStreaming",
gameWindowSource,
StringComparison.Ordinal);
Assert.DoesNotContain(
"BuildInteriorEntitiesForStreaming",
gameWindowSource,
StringComparison.Ordinal);
Assert.DoesNotContain(
"BuildPhysicsDatBundle",
gameWindowSource,
StringComparison.Ordinal);
Assert.DoesNotContain(
"ApplyLoadedTerrain",
gameWindowSource,
StringComparison.Ordinal);
Assert.DoesNotContain(
"PublishLandblockStaticLightingBeforeCollision",
gameWindowSource,
StringComparison.Ordinal);
Assert.DoesNotContain(
"_landblockPhysicsPublisher!.RemoveLandblock",
gameWindowSource,
StringComparison.Ordinal);
Assert.DoesNotContain(
"_landblockRenderPublisher",
gameWindowSource,
StringComparison.Ordinal);
Assert.DoesNotContain(
"_landblockPhysicsPublisher",
gameWindowSource,
StringComparison.Ordinal);
Assert.DoesNotContain(
"_landblockStaticPresentationPublisher",
gameWindowSource,
StringComparison.Ordinal);
Assert.Contains(
"_landblockPresentationPipeline",
gameWindowSource,
StringComparison.Ordinal);
Assert.Contains(
"new LandblockRenderPublisher(",
livePresentationSource,
StringComparison.Ordinal);
Assert.Contains(
"new LandblockPhysicsPublisher(",
livePresentationSource,
StringComparison.Ordinal);
Assert.Contains(
"new LandblockStaticPresentationPublisher(",
livePresentationSource,
StringComparison.Ordinal);
Assert.DoesNotContain("applyTerrain:", gameWindowSource, StringComparison.Ordinal);
Assert.DoesNotContain("demoteNearLayer:", gameWindowSource, StringComparison.Ordinal);
Assert.DoesNotContain("retirementCoordinator:", gameWindowSource, StringComparison.Ordinal);
Assert.DoesNotContain(
"_liveWorldOrigin.Recenter(lbX, lbY)",
gameWindowSource,
StringComparison.Ordinal);
int retirementBarrier = recenterSource.IndexOf(
"IsOriginRecenterRetirementComplete()",
StringComparison.Ordinal);
int originCommit = recenterSource.IndexOf(
"_origin.Recenter(",
StringComparison.Ordinal);
int destinationCommit = recenterSource.IndexOf(
"_streaming.TryCommitOriginRecenter(",
StringComparison.Ordinal);
Assert.True(retirementBarrier >= 0);
Assert.True(originCommit > retirementBarrier);
Assert.True(destinationCommit > originCommit);
Assert.Contains("ComputeOrigin(landblockId, build.Origin)", renderPublisherSource, StringComparison.Ordinal);
Assert.DoesNotContain("_liveCenterX", renderPublisherSource, StringComparison.Ordinal);
Assert.DoesNotContain("_liveCenterY", renderPublisherSource, StringComparison.Ordinal);
}
[Fact]
public void GameWindowShutdownKeepsStreamerAliveUntilSessionResetConverges()
{
string source = File.ReadAllText(Path.Combine(
FindRepoRoot(),
"src",
"AcDream.App",
"Rendering",
"GameWindowLifetime.cs"));
int sessionStage = source.IndexOf(
"new ResourceShutdownStage(\"host and session barriers\"",
StringComparison.Ordinal);
Assert.True(sessionStage >= 0);
int sessionOperation = source.IndexOf(
"Hard(\"game runtime session\", ingress.Runtime.StopSession)",
sessionStage,
StringComparison.Ordinal);
Assert.True(sessionOperation > sessionStage);
int dependentStage = source.IndexOf(
"new ResourceShutdownStage(\"session dependents\"",
sessionOperation,
StringComparison.Ordinal);
Assert.True(dependentStage > sessionOperation);
int streamerDispose = source.IndexOf(
"Hard(\"streamer\", () => live.Streamer?.Dispose())",
dependentStage,
StringComparison.Ordinal);
Assert.True(streamerDispose > dependentStage);
string runtime = File.ReadAllText(Path.Combine(
FindRepoRoot(),
"src",
"AcDream.Runtime",
"GameRuntime.cs"));
int helper = runtime.IndexOf(
"public void StopSession()",
StringComparison.Ordinal);
Assert.True(helper >= 0);
int sessionDispose = runtime.IndexOf(
"Session.Dispose();",
helper,
StringComparison.Ordinal);
Assert.True(sessionDispose > helper);
int disposalCompletionBarrier = runtime.IndexOf(
"if (!Session.IsDisposalComplete)",
sessionDispose,
StringComparison.Ordinal);
Assert.True(disposalCompletionBarrier > sessionDispose);
Assert.Contains(
"The Runtime session shutdown was deferred by a re-entrant callback.",
runtime[disposalCompletionBarrier..],
StringComparison.Ordinal);
}
private static LandblockBuild EmptyBuild(uint landblockId, LandblockBuildOrigin origin) =>
new(
new LoadedLandblock(
landblockId,
new LandBlock(),
Array.Empty<WorldEntity>()),
Origin: origin);
private static LandblockMeshData EmptyMesh() =>
new(Array.Empty<TerrainVertex>(), Array.Empty<uint>());
private static async Task<LandblockStreamResult> DrainFirstAsync(
LandblockStreamer streamer) =>
(await DrainCountAsync(streamer, 1))[0];
private static async Task<IReadOnlyList<LandblockStreamResult>> DrainCountAsync(
LandblockStreamer streamer,
int count)
{
var results = new List<LandblockStreamResult>(count);
for (int i = 0; i < SpinTimeoutMs / SpinStepMs && results.Count < count; i++)
{
results.AddRange(streamer.DrainCompletions(count - results.Count));
if (results.Count < count)
await Task.Delay(SpinStepMs);
}
Assert.Equal(count, results.Count);
return results;
}
private static string FindRepoRoot()
{
string? dir = AppContext.BaseDirectory;
while (dir is not null)
{
if (File.Exists(Path.Combine(dir, "AcDream.slnx")))
return dir;
dir = Directory.GetParent(dir)?.FullName;
}
throw new DirectoryNotFoundException("Could not locate AcDream.slnx.");
}
}