docs(pipeline): MP1b plan - pak format, acdream-bake, PakReader, equivalence
Normative v1 format (ACPK header with dat-iteration stamps, sorted 24-byte TOC entries with crc32, 64-byte-aligned uncompressed blobs, deterministic ObjectMeshData serialization), the bake CLI that drives MP1a MeshExtractor in parallel, the mmap zero-copy PakReader, and a dat-gated live-vs-pak equivalence suite. Scope-fenced to the ObjectMeshData asset classes (the decode-storm content from the MP0 baseline); terrain/BSP/scenery/degrade blobs are later slices. Also amends the spec: PakReader lives in AcDream.Content (Content->Core direction from MP1a makes a Core home circular). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Erik
parent
4122ac7eaf
commit
b9ceafd4fd
2 changed files with 121 additions and 4 deletions
113
docs/superpowers/plans/2026-07-05-mp1b-pak-and-bake.md
Normal file
113
docs/superpowers/plans/2026-07-05-mp1b-pak-and-bake.md
Normal file
|
|
@ -0,0 +1,113 @@
|
|||
# MP1b — Pak Format + acdream-bake + PakReader Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** A versioned pak file containing every `ObjectMeshData` the client's decode workers would otherwise produce at runtime (GfxObj / Setup / EnvCell mesh+texture payloads), an offline `acdream-bake` CLI that produces it by driving MP1a's `MeshExtractor`, a zero-copy mmap `PakReader`, and an equivalence test proving pak round-trip output is identical to live extraction.
|
||||
|
||||
**Architecture:** The bake tool and the client run the SAME extraction code (`AcDream.Content.MeshExtractor`, extracted verbatim in MP1a) — the pak stores its serialized output, so equivalence is by construction and verified by test. Nothing in the client consumes the pak yet (that is MP1c); MP1b has **no user visual gate** — its gates are round-trip tests, the dat-gated equivalence suite, and a timed full bake. Spec: `docs/superpowers/specs/2026-07-05-modern-pipeline-design.md` §6.2–6.5 including the 2026-07-05 amendments (v1 textures = RGBA8 as produced; `PakReader` lives in `AcDream.Content`, not Core).
|
||||
|
||||
**Scope discipline (v1 pak content):** ONLY the `ObjectMeshData` asset classes — this is what the decode workers produce during the 211 ms / 75.7 MB teleport storms (the MP0 baseline's target). Terrain blittables, flattened physics BSPs, scenery instance lists, and degrade tables are LATER slices (MP1c+/MP2/MP4) — do not add them.
|
||||
|
||||
**Tech Stack:** .NET 10; `System.IO.MemoryMappedFiles` + `MemoryMarshal` for the reader; plain `BinaryWriter`-style serialization (little-endian, versioned); xunit in a new `tests/AcDream.Content.Tests`.
|
||||
|
||||
---
|
||||
|
||||
## File map
|
||||
|
||||
| File | Action | Responsibility |
|
||||
|---|---|---|
|
||||
| `tests/AcDream.Content.Tests/AcDream.Content.Tests.csproj` | Create | Content-layer test project (rule 6), registered in `AcDream.slnx` |
|
||||
| `src/AcDream.Content/Pak/PakKey.cs` | Create | 64-bit asset key: `type:u8 \| fileId:u32 \| reserved:u24` |
|
||||
| `src/AcDream.Content/Pak/PakFormat.cs` | Create | Header/TOC binary layout constants + structs |
|
||||
| `src/AcDream.Content/Pak/ObjectMeshDataSerializer.cs` | Create | `ObjectMeshData` ⇄ bytes, versioned, deterministic |
|
||||
| `src/AcDream.Content/Pak/PakWriter.cs` | Create | Streams blobs to disk, writes TOC + header |
|
||||
| `src/AcDream.Content/Pak/PakReader.cs` | Create | mmap, O(log n) TOC lookup, span access, typed reads |
|
||||
| `src/AcDream.Bake/AcDream.Bake.csproj` + `Program.cs` | Create | The CLI: enumerate ids → parallel extract → pak |
|
||||
| `tests/AcDream.Content.Tests/*.cs` | Create | Round-trip + format + dat-gated equivalence tests |
|
||||
| `AcDream.slnx` | Modify | Register both new projects |
|
||||
|
||||
`AcDream.Bake` references `AcDream.Content` (+ transitively Core/DRW). No Silk.NET anywhere in either new project — both csproj files get `TreatWarningsAsErrors` + `LangVersion latest` like Content.
|
||||
|
||||
## Format v1 (normative)
|
||||
|
||||
All integers little-endian. File layout: `[Header][Blob region][TOC]` (TOC last so the writer streams blobs without knowing counts up front).
|
||||
|
||||
**Header (64 bytes, fixed):**
|
||||
```
|
||||
offset size field
|
||||
0 4 magic 'ACPK' (0x4B504341)
|
||||
4 4 formatVersion = 1
|
||||
8 4 portalIteration (DatCollection.Portal.Iteration)
|
||||
12 4 cellIteration
|
||||
16 4 highResIteration
|
||||
20 4 languageIteration
|
||||
24 8 tocOffset (u64)
|
||||
32 4 tocCount (u32)
|
||||
36 4 bakeToolVersion = 1
|
||||
40 24 reserved (zero)
|
||||
```
|
||||
|
||||
**PakKey (u64):** `((ulong)type << 56) | ((ulong)fileId << 24)` — low 24 bits reserved (variant/zero in v1). `type` enum (u8): `GfxObjMesh = 1`, `SetupMesh = 2`, `EnvCellMesh = 3`. (`fileId` is the dat id: `0x01xxxxxx` GfxObj, `0x02xxxxxx` Setup, cell ids for EnvCells.)
|
||||
|
||||
**TOC entry (24 bytes each):** `key u64, offset u64, length u32, crc32 u32` — entries sorted ascending by key (binary search); `crc32` over the blob bytes (corruption tripwire; reader verifies lazily on first access of each blob, logs + treats as missing on mismatch).
|
||||
|
||||
**Blobs:** each 64-byte aligned (writer pads). Content = `ObjectMeshDataSerializer` output (below). Uncompressed in v1 (spec: mmap + page cache; LZ4 is a recorded future option).
|
||||
|
||||
**Serialized ObjectMeshData v1:** field-by-field, versioned by the header's formatVersion. Layout rules the implementer follows: primitives raw LE; arrays as `count:i32 + payload`; blittable arrays (`VertexPositionNormalTexture[]`, `ushort[]` indices, `byte[]` texture data) written via `MemoryMarshal.AsBytes` bulk copy; strings absent (none in the type); **the `TextureBatches` dictionary is written sorted by key tuple `(Width, Height, Format)` and each inner list in original order** — bakes must be byte-reproducible run-to-run; nullable fields as `present:byte + value`. Enumerate the actual fields from `src/AcDream.Content/ObjectMeshData.cs` (the moved records: `ObjectMeshData`, `MeshBatchData`, `TextureBatchData`, `StagedEmitter`, `TextureKey`, `UploadPixelFormat/Type`, plus the DRW `Sphere`/bounding types stored as their float components) — serialize EVERY field; if a field's type resists obvious serialization, STOP and report rather than skipping it (a skipped field is a silent equivalence failure later).
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Test project scaffold
|
||||
|
||||
- [ ] Create `tests/AcDream.Content.Tests/AcDream.Content.Tests.csproj` — copy the shape of `tests/AcDream.App.Tests/AcDream.App.Tests.csproj` (xunit, net10.0), with a `ProjectReference` to `..\..\src\AcDream.Content\AcDream.Content.csproj`. Register in `AcDream.slnx`.
|
||||
- [ ] Add a trivial smoke test (`PakKeyTests` placeholder is fine — it becomes real in Task 2).
|
||||
- [ ] `dotnet build` + `dotnet test tests/AcDream.Content.Tests` green.
|
||||
- [ ] Commit: `test(pipeline): MP1b - AcDream.Content.Tests scaffold`
|
||||
|
||||
### Task 2: PakKey + header/TOC primitives (TDD)
|
||||
|
||||
- [ ] **Failing tests first** (`PakKeyTests`, `PakFormatTests`): key compose/decompose round-trip for each asset type incl. max fileId `0xFFFFFFFF`; key ordering matches (type, fileId) ordering; header write→read round-trip preserves every field; TOC entry write→read round-trip; TOC binary size = 24 bytes/entry exactly.
|
||||
- [ ] Implement `PakKey` (static compose/decompose + the `PakAssetType` enum) and `PakFormat` (header struct + read/write over `Span<byte>`/`Stream`, TOC entry struct + read/write) per the normative layout above.
|
||||
- [ ] Tests green; commit: `feat(pipeline): MP1b - pak key + header/TOC primitives`
|
||||
|
||||
### Task 3: ObjectMeshDataSerializer (TDD)
|
||||
|
||||
- [ ] **Failing tests first** (`ObjectMeshDataSerializerTests`): build synthetic `ObjectMeshData` instances covering — empty object; vertices+indices only; multiple texture batches across multiple `(W,H,Format)` groups; setup with parts; emitters present; nullable fields both present and absent; non-empty `EdgeLines`. Round-trip each and deep-compare EVERY field (write a `ObjectMeshDataEquality` test helper that compares field-by-field with clear failure messages — this helper is also Task 6's comparator). Plus: serializing the same instance twice yields byte-identical output (determinism), and dictionaries inserted in different orders yield byte-identical output (sorted-key rule).
|
||||
- [ ] Implement `ObjectMeshDataSerializer.Write(ObjectMeshData, Stream)` + `Read(ReadOnlySpan<byte>) → ObjectMeshData` per the layout rules.
|
||||
- [ ] Tests green; commit: `feat(pipeline): MP1b - ObjectMeshData binary serializer (deterministic round-trip)`
|
||||
|
||||
### Task 4: PakWriter + PakReader (TDD)
|
||||
|
||||
- [ ] **Failing tests first** (`PakRoundTripTests`): write a pak with N synthetic blobs (via Task 3 serializer) to a temp file → open with `PakReader` → header fields match; every key found; blob spans deserialize to deep-equal objects; missing key returns false; corrupted blob (flip one byte) → crc mismatch → treated as missing + logged once; blobs are 64-byte aligned; TOC binary-search lookup verified against a linear scan for all keys.
|
||||
- [ ] Implement `PakWriter` (streaming: write header placeholder → blobs with padding → TOC → seek back and finalize header) and `PakReader` (`MemoryMappedFile` + `ReadOnlySpan<byte>` accessor, lazily-verified crc set, `TryReadObjectMeshData(PakKey, out ObjectMeshData)`). No locks; document the immutability argument on the class.
|
||||
- [ ] Tests green; commit: `feat(pipeline): MP1b - PakWriter + mmap PakReader`
|
||||
|
||||
### Task 5: acdream-bake CLI
|
||||
|
||||
- [ ] Create `src/AcDream.Bake/` console project (register in slnx). Arguments: `--dat-dir <path>` (required), `--out <file>` (default `acdream.pak` next to the dats), `--ids <hexlist>` and `--landblocks <hexlist>` (optional dev filters), `--threads <n>` (default `Environment.ProcessorCount`).
|
||||
- [ ] Wiring: `DatCollection(datDir, Read)` → `DatCollectionAdapter` → one `MeshExtractor` per worker OR one shared (extractor is documented 4-worker-safe — shared is fine; sink = a `ConcurrentBag` collector whose entries are serialized under their own GfxObj keys, deduplicated by id).
|
||||
- [ ] Enumeration: GfxObj ids + Setup ids via `Portal.GetAllIdsOfType<...>` (mirror how `ObjectMeshManager`/CLI tools enumerate — check `src/AcDream.Cli/Program.cs` for the idiom); EnvCell ids by walking landblock `LandBlockInfo.NumCells` per the streamer's hydration path (find the idiom in `GameWindow.BuildPhysicsDatBundle` / the EnvCell loader — read, don't invent).
|
||||
- [ ] Parallel extract (`Parallel.ForEach`, `--threads`), progress line every 5 s (`baked/total, failures, elapsed, ETA`), failures logged per-id and counted, never fatal (a malformed dat entry skips that id — same as the runtime's behavior).
|
||||
- [ ] Write pak via `PakWriter`; stamp the four dat iterations into the header; print a summary (counts per type, failures, output size, elapsed).
|
||||
- [ ] Smoke: `dotnet run --project src/AcDream.Bake -- --dat-dir "%USERPROFILE%\Documents\Asheron's Call" --ids 0x01000001,0x02000001 --out scratch-test.pak` produces a valid pak that `PakReader` opens (assert via a quick console read-back or a test). Delete the scratch file.
|
||||
- [ ] Commit: `feat(pipeline): MP1b - acdream-bake CLI`
|
||||
|
||||
### Task 6: Dat-gated equivalence suite
|
||||
|
||||
- [ ] `PakEquivalenceTests` in Content.Tests, skipping cleanly when dats are absent (copy the skip pattern from `tests/AcDream.Core.Tests/Conformance/DatConcurrencyStressTests.cs`). For a fixture id set (≥10 GfxObjs incl. known-tricky ones — grep the test tree for ids used by `Issue119UpNullGfxObjDumpTests`/`StipplingSurfaceEquivalenceTests` and reuse them; ≥3 Setups; ≥5 EnvCells from the Holtburg landblock `0xA9B4`): run `MeshExtractor` live AND bake the same ids to a temp pak → read back → deep-compare with the Task 3 equality helper. Any field mismatch = failure naming the field.
|
||||
- [ ] Tests green (on this machine, dats present); full `dotnet test` green.
|
||||
- [ ] Commit: `test(pipeline): MP1b - live-vs-pak equivalence suite (dat-gated)`
|
||||
|
||||
### Task 7: Full bake run + gate record (coordinator/user machine)
|
||||
|
||||
- [ ] `dotnet build -c Release` green; run the full bake against the real dats (no filters), Release.
|
||||
- [ ] Record in `docs/research/2026-07-XX-mp1b-bake-report.md`: total assets per type, failures (each listed), wall time, pak size, and the equivalence-suite result. Gate: equivalence green + bake completes + failure list reviewed (a failure on an id the runtime CAN decode = bug; a failure matching runtime behavior = recorded as expected).
|
||||
- [ ] Update roadmap Track MP row (MP1b shipped); commit both docs.
|
||||
|
||||
---
|
||||
|
||||
## Post-plan notes
|
||||
|
||||
- No divergence-register rows: nothing consumed at runtime changes in MP1b.
|
||||
- The pak file itself is user-machine output — never committed; add `*.pak` to `.gitignore` in the same commit as Task 5.
|
||||
- MP1c (the cutover: decode workers try `PakReader` first, stale-pak refusal UX, hitch re-measure vs the 211 ms baseline) is the next plan; its "before" numbers are already in the MP0 baseline doc.
|
||||
|
|
@ -174,10 +174,14 @@ Single file `acdream.pak` next to the dats (path via `RuntimeOptions`):
|
|||
|
||||
### 6.3 Runtime consumption
|
||||
|
||||
- `PakReader` in `AcDream.Core` (Core stays GL-free; rule 2): one
|
||||
`MemoryMappedFile`, `GetBlob(key)` → `ReadOnlySpan<byte>` into the view; typed
|
||||
access via `MemoryMarshal.Cast` over blittable structs. **No locks anywhere** —
|
||||
immutable mapped memory is thread-safe to read by construction.
|
||||
- `PakReader` in **`AcDream.Content`** (AMENDED 2026-07-05: the spec originally said
|
||||
Core, but MP1a established the dependency direction Content→Core, and the pak's
|
||||
blob shapes are the `ObjectMeshData` family living in Content — a Core-resident
|
||||
reader would be circular. Content is GL-free, so the layering intent is
|
||||
preserved; Core stays untouched): one `MemoryMappedFile`, `GetBlob(key)` →
|
||||
`ReadOnlySpan<byte>` into the view; typed access via `MemoryMarshal.Cast` over
|
||||
blittable segments. **No locks anywhere** — immutable mapped memory is
|
||||
thread-safe to read by construction.
|
||||
- Streaming workers stop decoding: a landblock load becomes TOC lookups + handing
|
||||
spans to the GL upload queue. The decode worker pool shrinks to the residual
|
||||
runtime-decode set (below). Request-key dedup stays in the streaming layer
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue