acdream/docs/superpowers/plans/2026-07-05-mp1b-pak-and-bake.md

MP1b — Pak Format + acdream-bake + PakReader Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: A versioned pak file containing every ObjectMeshData the client's decode workers would otherwise produce at runtime (GfxObj / Setup / EnvCell mesh+texture payloads), an offline acdream-bake CLI that produces it by driving MP1a's MeshExtractor, a zero-copy mmap PakReader, and an equivalence test proving pak round-trip output is identical to live extraction.

Architecture: The bake tool and the client run the SAME extraction code (AcDream.Content.MeshExtractor, extracted verbatim in MP1a) — the pak stores its serialized output, so equivalence is by construction and verified by test. Nothing in the client consumes the pak yet (that is MP1c); MP1b has no user visual gate — its gates are round-trip tests, the dat-gated equivalence suite, and a timed full bake. Spec: docs/superpowers/specs/2026-07-05-modern-pipeline-design.md §6.2–6.5 including the 2026-07-05 amendments (v1 textures = RGBA8 as produced; PakReader lives in AcDream.Content, not Core).

Scope discipline (v1 pak content): ONLY the ObjectMeshData asset classes — this is what the decode workers produce during the 211 ms / 75.7 MB teleport storms (the MP0 baseline's target). Terrain blittables, flattened physics BSPs, scenery instance lists, and degrade tables are LATER slices (MP1c+/MP2/MP4) — do not add them.

Tech Stack: .NET 10; System.IO.MemoryMappedFiles + MemoryMarshal for the reader; plain BinaryWriter-style serialization (little-endian, versioned); xunit in a new tests/AcDream.Content.Tests.

---

File map

File	Action	Responsibility
tests/AcDream.Content.Tests/AcDream.Content.Tests.csproj	Create	Content-layer test project (rule 6), registered in AcDream.slnx
src/AcDream.Content/Pak/PakKey.cs	Create	64-bit asset key: type:u8 | fileId:u32 | reserved:u24
src/AcDream.Content/Pak/PakFormat.cs	Create	Header/TOC binary layout constants + structs
src/AcDream.Content/Pak/ObjectMeshDataSerializer.cs	Create	ObjectMeshData ⇄ bytes, versioned, deterministic
src/AcDream.Content/Pak/PakWriter.cs	Create	Streams blobs to disk, writes TOC + header
src/AcDream.Content/Pak/PakReader.cs	Create	mmap, O(log n) TOC lookup, span access, typed reads
src/AcDream.Bake/AcDream.Bake.csproj + Program.cs	Create	The CLI: enumerate ids → parallel extract → pak
tests/AcDream.Content.Tests/*.cs	Create	Round-trip + format + dat-gated equivalence tests
AcDream.slnx	Modify	Register both new projects

AcDream.Bake references AcDream.Content (+ transitively Core/DRW). No Silk.NET anywhere in either new project — both csproj files get TreatWarningsAsErrors + LangVersion latest like Content.

Format v1 (normative)

All integers little-endian. File layout: [Header][Blob region][TOC] (TOC last so the writer streams blobs without knowing counts up front).

Header (64 bytes, fixed):

offset size field
0      4    magic 'ACPK' (0x4B504341)
4      4    formatVersion = 1
8      4    portalIteration     (DatCollection.Portal.Iteration)
12     4    cellIteration
16     4    highResIteration
20     4    languageIteration
24     8    tocOffset (u64)
32     4    tocCount  (u32)
36     4    bakeToolVersion = 1
40     24   reserved (zero)

PakKey (u64): ((ulong)type << 56) | ((ulong)fileId << 24) — low 24 bits reserved (variant/zero in v1). type enum (u8): GfxObjMesh = 1, SetupMesh = 2, EnvCellMesh = 3. (fileId is the dat id: 0x01xxxxxx GfxObj, 0x02xxxxxx Setup, cell ids for EnvCells.)

TOC entry (24 bytes each): key u64, offset u64, length u32, crc32 u32 — entries sorted ascending by key (binary search); crc32 over the blob bytes (corruption tripwire; reader verifies lazily on first access of each blob, logs + treats as missing on mismatch).

Blobs: each 64-byte aligned (writer pads). Content = ObjectMeshDataSerializer output (below). Uncompressed in v1 (spec: mmap + page cache; LZ4 is a recorded future option).

Serialized ObjectMeshData v1: field-by-field, versioned by the header's formatVersion. Layout rules the implementer follows: primitives raw LE; arrays as count:i32 + payload; blittable arrays (VertexPositionNormalTexture[], ushort[] indices, byte[] texture data) written via MemoryMarshal.AsBytes bulk copy; strings absent (none in the type); the TextureBatches dictionary is written sorted by key tuple (Width, Height, Format) and each inner list in original order — bakes must be byte-reproducible run-to-run; nullable fields as present:byte + value. Enumerate the actual fields from src/AcDream.Content/ObjectMeshData.cs (the moved records: ObjectMeshData, MeshBatchData, TextureBatchData, StagedEmitter, TextureKey, UploadPixelFormat/Type, plus the DRW Sphere/bounding types stored as their float components) — serialize EVERY field; if a field's type resists obvious serialization, STOP and report rather than skipping it (a skipped field is a silent equivalence failure later).

---

Task 1: Test project scaffold

• Create tests/AcDream.Content.Tests/AcDream.Content.Tests.csproj — copy the shape of tests/AcDream.App.Tests/AcDream.App.Tests.csproj (xunit, net10.0), with a ProjectReference to ..\..\src\AcDream.Content\AcDream.Content.csproj. Register in AcDream.slnx.
• Add a trivial smoke test (PakKeyTests placeholder is fine — it becomes real in Task 2).
• dotnet build + dotnet test tests/AcDream.Content.Tests green.
• Commit: test(pipeline): MP1b - AcDream.Content.Tests scaffold

Task 2: PakKey + header/TOC primitives (TDD)

• Failing tests first (PakKeyTests, PakFormatTests): key compose/decompose round-trip for each asset type incl. max fileId 0xFFFFFFFF; key ordering matches (type, fileId) ordering; header write→read round-trip preserves every field; TOC entry write→read round-trip; TOC binary size = 24 bytes/entry exactly.
• Implement PakKey (static compose/decompose + the PakAssetType enum) and PakFormat (header struct + read/write over Span<byte>/Stream, TOC entry struct + read/write) per the normative layout above.
• Tests green; commit: feat(pipeline): MP1b - pak key + header/TOC primitives

Task 3: ObjectMeshDataSerializer (TDD)

• Failing tests first (ObjectMeshDataSerializerTests): build synthetic ObjectMeshData instances covering — empty object; vertices+indices only; multiple texture batches across multiple (W,H,Format) groups; setup with parts; emitters present; nullable fields both present and absent; non-empty EdgeLines. Round-trip each and deep-compare EVERY field (write a ObjectMeshDataEquality test helper that compares field-by-field with clear failure messages — this helper is also Task 6's comparator). Plus: serializing the same instance twice yields byte-identical output (determinism), and dictionaries inserted in different orders yield byte-identical output (sorted-key rule).
• Implement ObjectMeshDataSerializer.Write(ObjectMeshData, Stream) + Read(ReadOnlySpan<byte>) → ObjectMeshData per the layout rules.
• Tests green; commit: feat(pipeline): MP1b - ObjectMeshData binary serializer (deterministic round-trip)

Task 4: PakWriter + PakReader (TDD)

• Failing tests first (PakRoundTripTests): write a pak with N synthetic blobs (via Task 3 serializer) to a temp file → open with PakReader → header fields match; every key found; blob spans deserialize to deep-equal objects; missing key returns false; corrupted blob (flip one byte) → crc mismatch → treated as missing + logged once; blobs are 64-byte aligned; TOC binary-search lookup verified against a linear scan for all keys.
• Implement PakWriter (streaming: write header placeholder → blobs with padding → TOC → seek back and finalize header) and PakReader (MemoryMappedFile + ReadOnlySpan<byte> accessor, lazily-verified crc set, TryReadObjectMeshData(PakKey, out ObjectMeshData)). No locks; document the immutability argument on the class.
• Tests green; commit: feat(pipeline): MP1b - PakWriter + mmap PakReader

Task 5: acdream-bake CLI

• Create src/AcDream.Bake/ console project (register in slnx). Arguments: --dat-dir <path> (required), --out <file> (default acdream.pak next to the dats), --ids <hexlist> and --landblocks <hexlist> (optional dev filters), --threads <n> (default Environment.ProcessorCount).
• Wiring: DatCollection(datDir, Read) → DatCollectionAdapter → one MeshExtractor per worker OR one shared (extractor is documented 4-worker-safe — shared is fine; sink = a ConcurrentBag collector whose entries are serialized under their own GfxObj keys, deduplicated by id).
• Enumeration: GfxObj ids + Setup ids via Portal.GetAllIdsOfType<...> (mirror how ObjectMeshManager/CLI tools enumerate — check src/AcDream.Cli/Program.cs for the idiom); EnvCell ids by walking landblock LandBlockInfo.NumCells per the streamer's hydration path (find the idiom in GameWindow.BuildPhysicsDatBundle / the EnvCell loader — read, don't invent).
• Parallel extract (Parallel.ForEach, --threads), progress line every 5 s (baked/total, failures, elapsed, ETA), failures logged per-id and counted, never fatal (a malformed dat entry skips that id — same as the runtime's behavior).
• Write pak via PakWriter; stamp the four dat iterations into the header; print a summary (counts per type, failures, output size, elapsed).
• Smoke: dotnet run --project src/AcDream.Bake -- --dat-dir "%USERPROFILE%\Documents\Asheron's Call" --ids 0x01000001,0x02000001 --out scratch-test.pak produces a valid pak that PakReader opens (assert via a quick console read-back or a test). Delete the scratch file.
• Commit: feat(pipeline): MP1b - acdream-bake CLI

Task 6: Dat-gated equivalence suite

• PakEquivalenceTests in Content.Tests, skipping cleanly when dats are absent (copy the skip pattern from tests/AcDream.Core.Tests/Conformance/DatConcurrencyStressTests.cs). For a fixture id set (≥10 GfxObjs incl. known-tricky ones — grep the test tree for ids used by Issue119UpNullGfxObjDumpTests/StipplingSurfaceEquivalenceTests and reuse them; ≥3 Setups; ≥5 EnvCells from the Holtburg landblock 0xA9B4): run MeshExtractor live AND bake the same ids to a temp pak → read back → deep-compare with the Task 3 equality helper. Any field mismatch = failure naming the field.
• Tests green (on this machine, dats present); full dotnet test green.
• Commit: test(pipeline): MP1b - live-vs-pak equivalence suite (dat-gated)

Task 7: Full bake run + gate record (coordinator/user machine)

• dotnet build -c Release green; run the full bake against the real dats (no filters), Release.
• Record in docs/research/2026-07-XX-mp1b-bake-report.md: total assets per type, failures (each listed), wall time, pak size, and the equivalence-suite result. Gate: equivalence green + bake completes + failure list reviewed (a failure on an id the runtime CAN decode = bug; a failure matching runtime behavior = recorded as expected).
• Update roadmap Track MP row (MP1b shipped); commit both docs.

---

Post-plan notes

• No divergence-register rows: nothing consumed at runtime changes in MP1b.
• The pak file itself is user-machine output — never committed; add *.pak to .gitignore in the same commit as Task 5.
• MP1c (the cutover: decode workers try PakReader first, stale-pak refusal UX, hitch re-measure vs the 211 ms baseline) is the next plan; its "before" numbers are already in the MP0 baseline doc.