# MP1b — Pak Format + acdream-bake + PakReader Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** A versioned pak file containing every `ObjectMeshData` the client's decode workers would otherwise produce at runtime (GfxObj / Setup / EnvCell mesh+texture payloads), an offline `acdream-bake` CLI that produces it by driving MP1a's `MeshExtractor`, a zero-copy mmap `PakReader`, and an equivalence test proving pak round-trip output is identical to live extraction. **Architecture:** The bake tool and the client run the SAME extraction code (`AcDream.Content.MeshExtractor`, extracted verbatim in MP1a) — the pak stores its serialized output, so equivalence is by construction and verified by test. Nothing in the client consumes the pak yet (that is MP1c); MP1b has **no user visual gate** — its gates are round-trip tests, the dat-gated equivalence suite, and a timed full bake. Spec: `docs/superpowers/specs/2026-07-05-modern-pipeline-design.md` §6.2–6.5 including the 2026-07-05 amendments (v1 textures = RGBA8 as produced; `PakReader` lives in `AcDream.Content`, not Core). **Scope discipline (v1 pak content):** ONLY the `ObjectMeshData` asset classes — this is what the decode workers produce during the 211 ms / 75.7 MB teleport storms (the MP0 baseline's target). Terrain blittables, flattened physics BSPs, scenery instance lists, and degrade tables are LATER slices (MP1c+/MP2/MP4) — do not add them. **Tech Stack:** .NET 10; `System.IO.MemoryMappedFiles` + `MemoryMarshal` for the reader; plain `BinaryWriter`-style serialization (little-endian, versioned); xunit in a new `tests/AcDream.Content.Tests`. --- ## File map | File | Action | Responsibility | |---|---|---| | `tests/AcDream.Content.Tests/AcDream.Content.Tests.csproj` | Create | Content-layer test project (rule 6), registered in `AcDream.slnx` | | `src/AcDream.Content/Pak/PakKey.cs` | Create | 64-bit asset key: `type:u8 \| fileId:u32 \| reserved:u24` | | `src/AcDream.Content/Pak/PakFormat.cs` | Create | Header/TOC binary layout constants + structs | | `src/AcDream.Content/Pak/ObjectMeshDataSerializer.cs` | Create | `ObjectMeshData` ⇄ bytes, versioned, deterministic | | `src/AcDream.Content/Pak/PakWriter.cs` | Create | Streams blobs to disk, writes TOC + header | | `src/AcDream.Content/Pak/PakReader.cs` | Create | mmap, O(log n) TOC lookup, span access, typed reads | | `src/AcDream.Bake/AcDream.Bake.csproj` + `Program.cs` | Create | The CLI: enumerate ids → parallel extract → pak | | `tests/AcDream.Content.Tests/*.cs` | Create | Round-trip + format + dat-gated equivalence tests | | `AcDream.slnx` | Modify | Register both new projects | `AcDream.Bake` references `AcDream.Content` (+ transitively Core/DRW). No Silk.NET anywhere in either new project — both csproj files get `TreatWarningsAsErrors` + `LangVersion latest` like Content. ## Format v1 (normative) All integers little-endian. File layout: `[Header][Blob region][TOC]` (TOC last so the writer streams blobs without knowing counts up front). **Header (64 bytes, fixed):** ``` offset size field 0 4 magic 'ACPK' (0x4B504341) 4 4 formatVersion = 1 8 4 portalIteration (DatCollection.Portal.Iteration) 12 4 cellIteration 16 4 highResIteration 20 4 languageIteration 24 8 tocOffset (u64) 32 4 tocCount (u32) 36 4 bakeToolVersion = 1 40 24 reserved (zero) ``` **PakKey (u64):** `((ulong)type << 56) | ((ulong)fileId << 24)` — low 24 bits reserved (variant/zero in v1). `type` enum (u8): `GfxObjMesh = 1`, `SetupMesh = 2`, `EnvCellMesh = 3`. (`fileId` is the dat id: `0x01xxxxxx` GfxObj, `0x02xxxxxx` Setup, cell ids for EnvCells.) **TOC entry (24 bytes each):** `key u64, offset u64, length u32, crc32 u32` — entries sorted ascending by key (binary search); `crc32` over the blob bytes (corruption tripwire; reader verifies lazily on first access of each blob, logs + treats as missing on mismatch). **Blobs:** each 64-byte aligned (writer pads). Content = `ObjectMeshDataSerializer` output (below). Uncompressed in v1 (spec: mmap + page cache; LZ4 is a recorded future option). **Serialized ObjectMeshData v1:** field-by-field, versioned by the header's formatVersion. Layout rules the implementer follows: primitives raw LE; arrays as `count:i32 + payload`; blittable arrays (`VertexPositionNormalTexture[]`, `ushort[]` indices, `byte[]` texture data) written via `MemoryMarshal.AsBytes` bulk copy; strings absent (none in the type); **the `TextureBatches` dictionary is written sorted by key tuple `(Width, Height, Format)` and each inner list in original order** — bakes must be byte-reproducible run-to-run; nullable fields as `present:byte + value`. Enumerate the actual fields from `src/AcDream.Content/ObjectMeshData.cs` (the moved records: `ObjectMeshData`, `MeshBatchData`, `TextureBatchData`, `StagedEmitter`, `TextureKey`, `UploadPixelFormat/Type`, plus the DRW `Sphere`/bounding types stored as their float components) — serialize EVERY field; if a field's type resists obvious serialization, STOP and report rather than skipping it (a skipped field is a silent equivalence failure later). --- ### Task 1: Test project scaffold - [ ] Create `tests/AcDream.Content.Tests/AcDream.Content.Tests.csproj` — copy the shape of `tests/AcDream.App.Tests/AcDream.App.Tests.csproj` (xunit, net10.0), with a `ProjectReference` to `..\..\src\AcDream.Content\AcDream.Content.csproj`. Register in `AcDream.slnx`. - [ ] Add a trivial smoke test (`PakKeyTests` placeholder is fine — it becomes real in Task 2). - [ ] `dotnet build` + `dotnet test tests/AcDream.Content.Tests` green. - [ ] Commit: `test(pipeline): MP1b - AcDream.Content.Tests scaffold` ### Task 2: PakKey + header/TOC primitives (TDD) - [ ] **Failing tests first** (`PakKeyTests`, `PakFormatTests`): key compose/decompose round-trip for each asset type incl. max fileId `0xFFFFFFFF`; key ordering matches (type, fileId) ordering; header write→read round-trip preserves every field; TOC entry write→read round-trip; TOC binary size = 24 bytes/entry exactly. - [ ] Implement `PakKey` (static compose/decompose + the `PakAssetType` enum) and `PakFormat` (header struct + read/write over `Span`/`Stream`, TOC entry struct + read/write) per the normative layout above. - [ ] Tests green; commit: `feat(pipeline): MP1b - pak key + header/TOC primitives` ### Task 3: ObjectMeshDataSerializer (TDD) - [ ] **Failing tests first** (`ObjectMeshDataSerializerTests`): build synthetic `ObjectMeshData` instances covering — empty object; vertices+indices only; multiple texture batches across multiple `(W,H,Format)` groups; setup with parts; emitters present; nullable fields both present and absent; non-empty `EdgeLines`. Round-trip each and deep-compare EVERY field (write a `ObjectMeshDataEquality` test helper that compares field-by-field with clear failure messages — this helper is also Task 6's comparator). Plus: serializing the same instance twice yields byte-identical output (determinism), and dictionaries inserted in different orders yield byte-identical output (sorted-key rule). - [ ] Implement `ObjectMeshDataSerializer.Write(ObjectMeshData, Stream)` + `Read(ReadOnlySpan) → ObjectMeshData` per the layout rules. - [ ] Tests green; commit: `feat(pipeline): MP1b - ObjectMeshData binary serializer (deterministic round-trip)` ### Task 4: PakWriter + PakReader (TDD) - [ ] **Failing tests first** (`PakRoundTripTests`): write a pak with N synthetic blobs (via Task 3 serializer) to a temp file → open with `PakReader` → header fields match; every key found; blob spans deserialize to deep-equal objects; missing key returns false; corrupted blob (flip one byte) → crc mismatch → treated as missing + logged once; blobs are 64-byte aligned; TOC binary-search lookup verified against a linear scan for all keys. - [ ] Implement `PakWriter` (streaming: write header placeholder → blobs with padding → TOC → seek back and finalize header) and `PakReader` (`MemoryMappedFile` + `ReadOnlySpan` accessor, lazily-verified crc set, `TryReadObjectMeshData(PakKey, out ObjectMeshData)`). No locks; document the immutability argument on the class. - [ ] Tests green; commit: `feat(pipeline): MP1b - PakWriter + mmap PakReader` ### Task 5: acdream-bake CLI - [ ] Create `src/AcDream.Bake/` console project (register in slnx). Arguments: `--dat-dir ` (required), `--out ` (default `acdream.pak` next to the dats), `--ids ` and `--landblocks ` (optional dev filters), `--threads ` (default `Environment.ProcessorCount`). - [ ] Wiring: `DatCollection(datDir, Read)` → `DatCollectionAdapter` → one `MeshExtractor` per worker OR one shared (extractor is documented 4-worker-safe — shared is fine; sink = a `ConcurrentBag` collector whose entries are serialized under their own GfxObj keys, deduplicated by id). - [ ] Enumeration: GfxObj ids + Setup ids via `Portal.GetAllIdsOfType<...>` (mirror how `ObjectMeshManager`/CLI tools enumerate — check `src/AcDream.Cli/Program.cs` for the idiom); EnvCell ids by walking landblock `LandBlockInfo.NumCells` per the streamer's hydration path (find the idiom in `GameWindow.BuildPhysicsDatBundle` / the EnvCell loader — read, don't invent). - [ ] Parallel extract (`Parallel.ForEach`, `--threads`), progress line every 5 s (`baked/total, failures, elapsed, ETA`), failures logged per-id and counted, never fatal (a malformed dat entry skips that id — same as the runtime's behavior). - [ ] Write pak via `PakWriter`; stamp the four dat iterations into the header; print a summary (counts per type, failures, output size, elapsed). - [ ] Smoke: `dotnet run --project src/AcDream.Bake -- --dat-dir "%USERPROFILE%\Documents\Asheron's Call" --ids 0x01000001,0x02000001 --out scratch-test.pak` produces a valid pak that `PakReader` opens (assert via a quick console read-back or a test). Delete the scratch file. - [ ] Commit: `feat(pipeline): MP1b - acdream-bake CLI` ### Task 6: Dat-gated equivalence suite - [ ] `PakEquivalenceTests` in Content.Tests, skipping cleanly when dats are absent (copy the skip pattern from `tests/AcDream.Core.Tests/Conformance/DatConcurrencyStressTests.cs`). For a fixture id set (≥10 GfxObjs incl. known-tricky ones — grep the test tree for ids used by `Issue119UpNullGfxObjDumpTests`/`StipplingSurfaceEquivalenceTests` and reuse them; ≥3 Setups; ≥5 EnvCells from the Holtburg landblock `0xA9B4`): run `MeshExtractor` live AND bake the same ids to a temp pak → read back → deep-compare with the Task 3 equality helper. Any field mismatch = failure naming the field. - [ ] Tests green (on this machine, dats present); full `dotnet test` green. - [ ] Commit: `test(pipeline): MP1b - live-vs-pak equivalence suite (dat-gated)` ### Task 7: Full bake run + gate record (coordinator/user machine) - [ ] `dotnet build -c Release` green; run the full bake against the real dats (no filters), Release. - [ ] Record in `docs/research/2026-07-XX-mp1b-bake-report.md`: total assets per type, failures (each listed), wall time, pak size, and the equivalence-suite result. Gate: equivalence green + bake completes + failure list reviewed (a failure on an id the runtime CAN decode = bug; a failure matching runtime behavior = recorded as expected). - [ ] Update roadmap Track MP row (MP1b shipped); commit both docs. --- ## Post-plan notes - No divergence-register rows: nothing consumed at runtime changes in MP1b. - The pak file itself is user-machine output — never committed; add `*.pak` to `.gitignore` in the same commit as Task 5. - MP1c (the cutover: decode workers try `PakReader` first, stale-pak refusal UX, hitch re-measure vs the 211 ms baseline) is the next plan; its "before" numbers are already in the MP0 baseline doc.