erik/acdream
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

fix(pipeline): MP1b review - bake CLI determinism + bounded memory

Browse source 
Adversarially-verified review findings 1 and 10:

(1) The bake pipeline no longer accumulates every decoded ObjectMeshData
in one ConcurrentBag before writing (multi-GB OOM risk on the full
bake), and no longer writes blobs in thread-completion order (which
violated the plan's "bakes must be byte-reproducible run-to-run").
New shape in BakeRunner: build the FULL id list, sort by PakKey, chunk
into 512-id batches; Parallel.ForEach WITHIN each batch; sort each
batch's results by key and AddBlob sequentially; release the batch.
Batches are contiguous key ranges, so the blob region lands in global
key order regardless of thread scheduling, and peak memory is one
batch's output. Side-staged particle-preload meshes drain per batch
into a key-deduped map (first instance wins — per-id extraction output
is deterministic, so instance choice cannot affect bytes) and are
written after all batches, sorted by key, skipping keys already
written.

Program.cs is now a thin arg-parsing shell over the public BakeRunner
so the new dat-gated byte-reproducibility test can drive the REAL
pipeline: tests/AcDream.Bake.Tests (new project, rule 6; registered in
slnx; no Silk.NET in its resolved dependency graph — verified) bakes
the same 9-id mixed fixture twice with DIFFERENT thread counts (8 vs
3 — thread scheduling was the nondeterminism source) and asserts the
two pak files are byte-identical. Ran for real against the dats on
this machine: green.

(10) The isSetup argument for EnvCell extraction now matches at both
call sites (BakeRunner and PakEquivalenceTests both pass false) and is
documented at each: the runtime's own request sites
(WbMeshAdapter.IncrementRefCount / EnsureLoaded) pass isSetup: false
for every MeshRef id including cell-geometry ids. The parameter is
currently dead in MeshExtractor.PrepareMeshData (dispatch is on the
resolved dat type), but two disagreeing call sites were a latent trap.

Header FormatVersion is no longer set by the bake (PakWriter stamps it
per review finding 2, previous commit).
This commit is contained in:
Erik 2026-07-05 22:14:31 +02:00
parent 84d1956d84
commit 86e0dc4655
6 changed files with 371 additions and 208 deletions
Download patch file Download diff file Expand all files Collapse all files

6
tests/AcDream.Content.Tests/PakEquivalenceTests.cs
View file

@ -59,7 +59,11 @@ public sealed class PakEquivalenceTests {
var work = new List<(PakAssetType Type, uint FileId, ulong ExtractorId, bool IsSetup)>();
work.AddRange(gfxObjIds.Select(id => (PakAssetType.GfxObjMesh, id, (ulong)id, false)));
work.AddRange(setupIds.Select(id => (PakAssetType.SetupMesh, id, (ulong)id, true)));
work.AddRange(envCellIds.Select(id => (PakAssetType.EnvCellMesh, id, id | 0x1_0000_0000UL, true)));
// isSetup: false for EnvCell — matches the runtime's own request sites
// (WbMeshAdapter.IncrementRefCount/EnsureLoaded pass isSetup: false for
// every MeshRef id, incl. cell-geometry ids) and BakeRunner's call site.
// (Review finding 10 — the two call sites previously disagreed.)
work.AddRange(envCellIds.Select(id => (PakAssetType.EnvCellMesh, id, id | 0x1_0000_0000UL, false)));
// ---- LIVE extraction (golden) ----
var golden = new Dictionary<(PakAssetType, uint), ObjectMeshData>();