erik/acdream
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 3

fix(pipeline): MP1b review - bake CLI determinism + bounded memory

Browse source 
Adversarially-verified review findings 1 and 10:

(1) The bake pipeline no longer accumulates every decoded ObjectMeshData
in one ConcurrentBag before writing (multi-GB OOM risk on the full
bake), and no longer writes blobs in thread-completion order (which
violated the plan's "bakes must be byte-reproducible run-to-run").
New shape in BakeRunner: build the FULL id list, sort by PakKey, chunk
into 512-id batches; Parallel.ForEach WITHIN each batch; sort each
batch's results by key and AddBlob sequentially; release the batch.
Batches are contiguous key ranges, so the blob region lands in global
key order regardless of thread scheduling, and peak memory is one
batch's output. Side-staged particle-preload meshes drain per batch
into a key-deduped map (first instance wins — per-id extraction output
is deterministic, so instance choice cannot affect bytes) and are
written after all batches, sorted by key, skipping keys already
written.

Program.cs is now a thin arg-parsing shell over the public BakeRunner
so the new dat-gated byte-reproducibility test can drive the REAL
pipeline: tests/AcDream.Bake.Tests (new project, rule 6; registered in
slnx; no Silk.NET in its resolved dependency graph — verified) bakes
the same 9-id mixed fixture twice with DIFFERENT thread counts (8 vs
3 — thread scheduling was the nondeterminism source) and asserts the
two pak files are byte-identical. Ran for real against the dats on
this machine: green.

(10) The isSetup argument for EnvCell extraction now matches at both
call sites (BakeRunner and PakEquivalenceTests both pass false) and is
documented at each: the runtime's own request sites
(WbMeshAdapter.IncrementRefCount / EnsureLoaded) pass isSetup: false
for every MeshRef id including cell-geometry ids. The parameter is
currently dead in MeshExtractor.PrepareMeshData (dispatch is on the
resolved dat type), but two disagreeing call sites were a latent trap.

Header FormatVersion is no longer set by the bake (PakWriter stamps it
per review finding 2, previous commit).
This commit is contained in:
Erik 2026-07-05 22:14:31 +02:00
parent 84d1956d84
commit 86e0dc4655
6 changed files with 371 additions and 208 deletions
Download patch file Download diff file Expand all files Collapse all files

216
src/AcDream.Bake/Program.cs
View file

@ -1,27 +1,15 @@
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using AcDream.Bake;
using AcDream.Content;
using AcDream.Content.Pak;
using DatReaderWriter;
using DatReaderWriter.DBObjs;
using DatReaderWriter.Options;
using Environment = System.Environment;
// acdream-bake: offline CLI producing a versioned pak file containing every
// ObjectMeshData the client's decode workers would otherwise produce at
// runtime (GfxObj / Setup / EnvCell mesh+texture payloads).
//
// Drives AcDream.Content.MeshExtractor — the SAME extraction code the live
// client runs — so pak-vs-live equivalence is by construction (verified by
// the Task 6 dat-gated equivalence suite), not by a second hand-written
// extraction path.
// Thin arg-parsing shell over BakeRunner (the real pipeline lives there so
// the dat-gated byte-reproducibility test can drive it directly).
//
// Plan: docs/superpowers/plans/2026-07-05-mp1b-pak-and-bake.md, Task 5.
@ -66,160 +54,13 @@ if (!Directory.Exists(datDir)) {
outPath ??= Path.Combine(datDir, "acdream.pak");
Console.WriteLine("acdream-bake");
Console.WriteLine($"dat dir: {datDir}");
Console.WriteLine($"out: {outPath}");
Console.WriteLine($"threads: {threads}");
Console.WriteLine();
var sw = Stopwatch.StartNew();
using var dats = new DatCollection(datDir, DatAccessType.Read);
using var datReaderWriter = new BakeDatCollectionAdapter(dats);
var extractorLogger = new ConsoleErrorLogger(nameof(MeshExtractor));
// A ConcurrentQueue collector for particle-preload meshes MeshExtractor
// side-stages mid-extraction (CollectEmittersFromScript). MeshExtractor is
// documented thread-safe for up to MaxParallelLoads (4) concurrent workers
// sharing one instance and requires this sink argument explicitly — no
// default — so a bake tool can't silently forget it and drop preloads.
// The bake tool treats each side-staged item as its own GfxObjMesh entry,
// deduplicated by id the same way the enumerated id set is.
var sideStaged = new ConcurrentQueue<ObjectMeshData>();
var extractor = new MeshExtractor(datReaderWriter, extractorLogger, data => sideStaged.Enqueue(data));
// ---- enumeration ---------------------------------------------------------
var gfxObjIds = dats.GetAllIdsOfType<GfxObj>().ToList();
var setupIds = dats.GetAllIdsOfType<Setup>().ToList();
var envCellIds = EnumerateEnvCellIds(dats, landblockFilter);
if (idFilter is not null) {
gfxObjIds = gfxObjIds.Where(idFilter.Contains).ToList();
setupIds = setupIds.Where(idFilter.Contains).ToList();
envCellIds = envCellIds.Where(idFilter.Contains).ToList();
}
var work = new List<(PakAssetType Type, uint FileId)>(gfxObjIds.Count + setupIds.Count + envCellIds.Count);
work.AddRange(gfxObjIds.Select(id => (PakAssetType.GfxObjMesh, id)));
work.AddRange(setupIds.Select(id => (PakAssetType.SetupMesh, id)));
work.AddRange(envCellIds.Select(id => (PakAssetType.EnvCellMesh, id)));
Console.WriteLine($"enumerated: {gfxObjIds.Count:N0} GfxObj, {setupIds.Count:N0} Setup, {envCellIds.Count:N0} EnvCell " +
$"({work.Count:N0} total)");
Console.WriteLine();
// ---- parallel extract -----------------------------------------------------
var results = new ConcurrentBag<(PakAssetType Type, uint FileId, ObjectMeshData Data)>();
var failures = new ConcurrentBag<(PakAssetType Type, uint FileId, string Reason)>();
long completed = 0;
var progressLock = new object();
var lastProgressReport = Stopwatch.StartNew();
Parallel.ForEach(
work,
new ParallelOptions { MaxDegreeOfParallelism = threads },
item => {
var (type, fileId) = item;
try {
ulong extractorId = type == PakAssetType.EnvCellMesh ? fileId | 0x1_0000_0000UL : fileId;
bool isSetup = type == PakAssetType.SetupMesh;
var data = extractor.PrepareMeshData(extractorId, isSetup);
if (data is not null) {
// Extractor echoes back its own resolved ObjectId (which may
// differ in the high bits for EnvCell synthetic-geometry
// requests) — always key the pak entry by the ENUMERATED
// fileId so lookups at read time are predictable.
results.Add((type, fileId, data));
}
else {
failures.Add((type, fileId, "extractor returned null (no polygons or unresolvable id)"));
}
}
catch (Exception ex) {
// A malformed dat entry skips that id — never fatal to the bake,
// matching the runtime's own per-id try/catch behavior.
failures.Add((type, fileId, ex.Message));
}
long done = Interlocked.Increment(ref completed);
lock (progressLock) {
if (lastProgressReport.Elapsed.TotalSeconds >= 5) {
ReportProgress(done, work.Count, failures.Count, sw.Elapsed);
lastProgressReport.Restart();
}
}
});
ReportProgress(completed, work.Count, failures.Count, sw.Elapsed);
Console.WriteLine();
// ---- write pak -----------------------------------------------------------
var header = new PakHeader {
FormatVersion = 1,
PortalIteration = (uint)dats.Portal.Iteration!.CurrentIteration,
CellIteration = (uint)dats.Cell.Iteration!.CurrentIteration,
HighResIteration = (uint)dats.HighRes.Iteration!.CurrentIteration,
LanguageIteration = (uint)dats.Local.Iteration!.CurrentIteration,
BakeToolVersion = 1,
};
var seenSideStagedIds = new HashSet<ulong>();
int written = 0, sideStagedWritten = 0, sideStagedDuped = 0;
using (var writer = new PakWriter(outPath, header)) {
foreach (var (type, fileId, data) in results) {
writer.AddBlob(PakKey.Compose(type, fileId), data);
written++;
}
// Side-staged particle-preload GfxObj meshes: dedup by id against BOTH
// each other and the primary GfxObj set already written (a particle's
// preload GfxObj can also be independently enumerated as a top-level
// GfxObj — MeshExtractor.PrepareMeshData(id, isSetup:false) is
// deterministic, so re-extracting would just waste time, but we must
// never write the SAME pak key twice).
var primaryGfxObjIds = new HashSet<uint>(gfxObjIds);
while (sideStaged.TryDequeue(out var staged)) {
uint fileId = (uint)(staged.ObjectId & 0xFFFFFFFFu);
if (primaryGfxObjIds.Contains(fileId) || !seenSideStagedIds.Add(staged.ObjectId)) {
sideStagedDuped++;
continue;
}
writer.AddBlob(PakKey.Compose(PakAssetType.GfxObjMesh, fileId), staged);
sideStagedWritten++;
}
writer.Finish();
}
sw.Stop();
var outSize = new FileInfo(outPath).Length;
Console.WriteLine("=== bake summary ===");
Console.WriteLine($" GfxObj baked: {results.Count(r => r.Type == PakAssetType.GfxObjMesh):N0}");
Console.WriteLine($" Setup baked: {results.Count(r => r.Type == PakAssetType.SetupMesh):N0}");
Console.WriteLine($" EnvCell baked: {results.Count(r => r.Type == PakAssetType.EnvCellMesh):N0}");
Console.WriteLine($" side-staged baked (particle preload GfxObjs): {sideStagedWritten:N0} ({sideStagedDuped:N0} deduped)");
Console.WriteLine($" total blobs: {written + sideStagedWritten:N0}");
Console.WriteLine($" failures: {failures.Count:N0}");
Console.WriteLine($" elapsed: {sw.Elapsed.TotalSeconds:F1} s");
Console.WriteLine($" output size: {outSize / 1024.0 / 1024.0:F1} MB");
Console.WriteLine($" output path: {outPath}");
if (failures.Count > 0) {
Console.WriteLine();
Console.WriteLine($"failures ({failures.Count}):");
foreach (var (type, fileId, reason) in failures.OrderBy(f => f.FileId).Take(200)) {
Console.WriteLine($" {type,-12} 0x{fileId:X8}: {reason}");
}
if (failures.Count > 200) Console.WriteLine($" ... and {failures.Count - 200} more");
}
return 0;
// ---- helpers ---------------------------------------------------------------
return BakeRunner.Run(new BakeOptions {
DatDir = datDir,
OutPath = outPath,
IdFilter = idFilter,
LandblockFilter = landblockFilter,
Threads = threads,
});
static HashSet<uint> ParseHexList(string? raw) {
var result = new HashSet<uint>();
@ -235,42 +76,3 @@ static HashSet<uint> ParseHexList(string? raw) {
}
return result;
}
static void ReportProgress(long done, int total, int failures, TimeSpan elapsed) {
double rate = elapsed.TotalSeconds > 0 ? done / elapsed.TotalSeconds : 0;
double etaSeconds = rate > 0 ? (total - done) / rate : 0;
Console.WriteLine($"[{elapsed:hh\\:mm\\:ss}] baked {done:N0}/{total:N0}, failures={failures:N0}, " +
$"elapsed={elapsed.TotalSeconds:F0}s, ETA={etaSeconds:F0}s");
}
/// <summary>
/// Enumerates EnvCell ids by walking the cell dat's LandBlockInfo entries
/// (0xFFFE low 16 bits) and, for each, the NumCells-derived cell id range —
/// GetAllIdsOfType&lt;T&gt;() does not cover cell-dat range-based types
/// (DatReaderWriter's documented limitation; see the low-16-bit bucketing
/// idiom in src/AcDream.Cli/Program.cs's CountCellByLow16, and the
/// firstCellId/NumCells hydration idiom in GameWindow.BuildPhysicsDatBundle
/// / BuildInteriorEntitiesForStreaming).
/// </summary>
static List<uint> EnumerateEnvCellIds(DatCollection dats, HashSet<uint>? landblockFilter) {
var landblockInfoIds = new List<uint>();
foreach (var file in dats.Cell.Tree) {
if ((file.Id & 0xFFFFu) != 0xFFFEu) continue;
uint landblockId = file.Id & 0xFFFF0000u;
if (landblockFilter is not null && !landblockFilter.Contains(landblockId)) continue;
landblockInfoIds.Add(file.Id);
}
var envCellIds = new List<uint>();
foreach (var lbInfoId in landblockInfoIds) {
if (!dats.Cell.TryGet<LandBlockInfo>(lbInfoId, out var lbInfo) || lbInfo is null) continue;
if (lbInfo.NumCells == 0) continue;
uint landblockId = lbInfoId & 0xFFFF0000u;
uint firstCellId = landblockId | 0x0100u;
for (uint offset = 0; offset < lbInfo.NumCells; offset++) {
envCellIds.Add(firstCellId + offset);
}
}
return envCellIds;
}