acdream/docs/superpowers/specs/2026-05-21-phase-o-dat-path-unification-design.md

Phase O — DatPath Unification — Design Spec

Filed: 2026-05-21 Status: ACTIVE Amended: 2026-05-21 (post O-T1 audit; see docs/research/2026-05-21-phase-o-t1-wb-audit.md) Estimated effort: ~7-8 working days, one ship-window.

Tagline: ONE thing touches the DATs. Today we have two readers in process (acdream's DatCollection + WorldBuilder's DefaultDatReaderWriter) reading the same files independently. Phase O collapses that to one.

Amendment summary (2026-05-21): O-T1 audit revealed three components the original spec listed (LandSurfaceManager, EnvCellRenderManager, PortalRenderManager) plus the open §9.2 question on TerrainRenderManager are NOT in acdream's actual call graph — we already have our own ports or never used them. T6 is eliminated entirely; T5 shrinks to stateless helpers only. T4 grows by 0.5d to include a refactor of ObjectMeshManager to take DatCollection directly (avoiding a permanent adapter indirection). Net effort estimate unchanged. Open question O-Q1 (thread-model) closed: verified safe — no worker-thread access to WB code.

---

1. Problem statement

As of Phase N.4 ship (2026-05-08), WorldBuilder is integrated as our rendering + dat-handling base. Concretely:

• WbMeshAdapter.cs:79 constructs _wbDats = new DefaultDatReaderWriter(datDir) — WB's own dat reader.
• Our DatCollection is constructed independently at startup for the rest of the client (network, physics, animation, clothing, audio, UI, etc.).

Both readers open the same four files (client_portal.dat, client_cell_1.dat, client_highres.dat, client_local_English.dat) with independent file handles and independent in-memory index caches.

Costs:

• ~50-100 MB duplicated index cache memory (per WbMeshAdapter.cs:27 comment).
• Double seek cost when the same dat block is read by both pipelines (mesh path via WB; surface metadata side-table via our DatCollection).
• Cross-check awkwardness — WbMeshAdapter.cs:224-262 has explicit "if WE find the cell but WB doesn't" diagnostic code, born from the divergence.
• Architectural smell — a third-party feedback signal (AC community comment, 2026-05-21) flagged WB's dat-touching as "built for tools, not runtime."

WB is MIT-licensed, so the path to fixing this is to extract its load-bearing code into our repo and route it through our DatCollection. One reader, one cache, one source of dat truth.

---

2. Decisions log

#	Decision	Why
O-D1	Extract WB code verbatim into our repo. No re-port from retail decomp. No "improvements" while extracting. Discipline applies to algorithms (meshing math, texture decode, particle pipeline) — NOT to mechanical changes like parameter type renames.	CLAUDE.md is explicit: "Re-porting from retail decomp when WB already has a tested port is how subtle bugs (scenery edge-vertex, triangle-Z) keep slipping in." Verbatim copy preserves all the corner-case fixes WB already has.
O-D2	Make extracted code consume DatCollection, not IDatReaderWriter.	Single source of dat truth. The whole point of the phase.
O-D3	Drop the WorldBuilder.Shared + Chorizite.OpenGLSDLBackend project references at the end of the phase.	If we still reference WB after extraction, we haven't actually finished the work.
O-D4	Keep WB in references/ for reading/comparison. Don't delete the vendored directory.	We'll still want to grep WB during ports of NEW pieces (e.g., minimap renderer if/when we add it).
O-D5	MIT attribution per WB convention. Add NOTICE.md entry crediting WorldBuilder for the extracted code.	License compliance.
O-D6	One ship-window, not sliced. Either the whole extraction lands and references/WorldBuilder is dropped, or we roll back the entire phase.	Half-extracted state (some WB code in our repo, some still referenced) is worse than either endpoint.
O-D7	Refactor ObjectMeshManager to take DatCollection directly (not via an adapter). Safety check at T4 — fall back to thin DatCollectionAdapter : IDatReaderWriter if _dats.X call-site count inside ObjectMeshManager exceeds 20.	After extraction, ObjectMeshManager is OUR code; our code should use our types. An adapter would be permanent tech debt obscuring data flow. O-D1's "verbatim copy" discipline applies to algorithms, not parameter types.
O-D8	Drop four originally-listed components from the extract list: LandSurfaceManager, EnvCellRenderManager, PortalRenderManager, TerrainRenderManager.	O-T1 audit confirmed these aren't reachable from acdream's code graph. LandSurfaceManager and TerrainRenderManager have our own ports (TerrainBlending.cs, TerrainModernRenderer.cs); EnvCell/Portal are rendered via the mesh pipeline, not via WB's dedicated renderers.
O-D9	Promote 3 internal types in Chorizite to public when extracted: EmbeddedResourceReader, TextureFormatExtensions, BufferUsageExtensions.	We vendor them; we control the namespace. Keeping internal would force same-assembly placement with no benefit.
O-D10	Strip [MemoryPackable] from TerrainEntry when copying into our tree.	We don't serialize the struct. Avoids adding MemoryPack as a NuGet dep for an unused attribute.
O-D11	Namespace AcDream.Core.Rendering.Wb.* for extracted code (vs topic-based namespaces).	Preserves the "this came from WB" audit trail. A later phase can re-organize once the dust settles.
O-D12	Drop ResolveId(uint) and the [indoor-upload] NULL_RESULT diagnostic block in WbMeshAdapter.cs at T7.	Only caller of ResolveId is the diagnostic; the diagnostic depends on the second _wbDats which goes away. The block has served its Phase 2 cell-resolution-divergence investigation purpose.

---

3. Architecture overview

Today (Phase N.4–A.5 state)

┌─────────────────────────┐       ┌──────────────────────────────────┐
│ acdream subsystems      │       │ WorldBuilder (referenced project)│
│ - Network               │       │  ┌────────────────────────────┐  │
│ - Physics               │ ───→  │  │ DefaultDatReaderWriter     │  │
│ - Animation             │       │  │  (opens 4 .dat files)      │  │
│ - Clothing/Audio/UI     │       │  └────────────┬───────────────┘  │
│ - Surface metadata      │       │               │                  │
└──────────┬──────────────┘       │  ┌────────────▼───────────────┐  │
           │                      │  │ ObjectMeshManager          │  │
           │ DatCollection        │  │ TextureHelpers             │  │
           │ (opens same 4 files) │  │ SceneryHelpers             │  │
           │                      │  │ OpenGLGraphicsDevice       │  │
           ▼                      │  └────────────────────────────┘  │
    ┌──────────────┐              └──────────────────────────────────┘
    │ Dat files    │              ▲
    │ (same files, │──────────────┘
    │  two readers)│
    └──────────────┘

Phase O target

┌─────────────────────────────────────────────────────────────────┐
│ acdream subsystems                                              │
│ - Network / Physics / Animation / Clothing / Audio / UI         │
│ - Mesh pipeline   (extracted from WB.ObjectMeshManager,         │
│                    refactored to take DatCollection)            │
│ - Texture decode  (extracted from WB.TextureHelpers)            │
│ - Scenery helpers (extracted from WB.SceneryHelpers)            │
│ - Terrain helpers (extracted from WB.TerrainUtils + TerrainEntry)│
│ - GL infra        (extracted from WB.OpenGLGraphicsDevice etc.) │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           │ DatCollection (the ONLY reader)
                           ▼
                  ┌──────────────┐
                  │ Dat files    │
                  └──────────────┘

What stays acdream-original (unchanged by this phase)

• TerrainModernRenderer.cs (Phase N.5b — uses LandblockMesh.Build with retail's FSplitNESW).
• TerrainBlending.cs (our port of WB's LandSurfaceManager — already lives in acdream).
• Network, physics, animation, movement, UI, audio, chat, streaming controller, plugin API.
• Our DatCollection (becomes the only dat reader).
• The Wb* adapter layer (WbMeshAdapter, WbDrawDispatcher, LandblockSpawnAdapter, EntitySpawnAdapter, etc.) — those stay in AcDream.App/Rendering/Wb/; they bridge our world to the extracted code.

---

4. Component changes (audit-corrected)

4.1 Mesh pipeline (T4 — the load-bearing extraction)

Layer placement: mesh-pipeline files live under src/AcDream.App/Rendering/Wb/ (NOT Core), because ObjectMeshManager and its supports use Silk.NET.OpenGL types directly (ManagedGL*, GLSLShader, etc.) and CLAUDE.md forbids Core depending on GL.

WB source	New acdream home	Adaptation
Chorizite.OpenGLSDLBackend.Lib.ObjectMeshManager	src/AcDream.App/Rendering/Wb/ObjectMeshManager.cs	Refactor: replace IDatReaderWriter field/ctor-param with DatCollection (Core type — App can reference it). Update _dats.X call sites. Safety check at T4: if >20 sites, fall back to thin adapter.
Chorizite.OpenGLSDLBackend.Lib.ObjectRenderBatch + ObjectRenderData	Same directory	Verbatim copy.
Particle batcher + emitter (T4 supports)	Same	Verbatim copy.
Chorizite.OpenGLSDLBackend.Lib.DebugRenderSettings	Same	Verbatim copy (constructor parameter type only).
GlobalMeshBuffer, modern render data structs	Same	Verbatim copy.

4.2 Texture pipeline + GL infrastructure (T3)

Layer split: TextureHelpers is pure (no GL, no dat) → goes to Core. All other GL infra → App.

WB source	New acdream home	Adaptation
Chorizite.OpenGLSDLBackend.Lib.TextureHelpers	src/AcDream.Core/Rendering/Wb/TextureHelpers.cs	Verbatim. Pure functions — no dat / no GL.
Chorizite.OpenGLSDLBackend.OpenGLGraphicsDevice	src/AcDream.App/Rendering/Wb/OpenGLGraphicsDevice.cs	Verbatim. Touches GL.
ManagedGL{Texture,TextureArray,VertexBuffer,IndexBuffer,VertexArray,FrameBuffer,UniformBuffer}	App Wb/ directory	Verbatim. Touches GL.
GLSLShader, GLHelpers, GLStateScope	App Wb/ directory	Verbatim. Touches GL.
EmbeddedResourceReader (internal → public)	App Wb/ directory	Promote internal → public.
TextureFormatExtensions, BufferUsageExtensions (internal → public)	App Wb/ directory	Promote internal → public.

4.3 Stateless helpers (T5)

WB source	New acdream home	Adaptation
Chorizite.OpenGLSDLBackend.Lib.SceneryHelpers	src/AcDream.Core/Rendering/Wb/SceneryHelpers.cs	Verbatim. Pure functions.
WorldBuilder.Shared.Modules.Landscape.Lib.TerrainUtils	src/AcDream.Core/Rendering/Wb/TerrainUtils.cs	Verbatim.
WorldBuilder.Shared.Modules.Landscape.Models.TerrainEntry	Same	Verbatim except strip [MemoryPackable].
WorldBuilder.Shared.Modules.Landscape.Models.CellSplitDirection	Same	Verbatim enum.

4.4 NOT extracted (dropped from spec §4)

Audit confirmed these are not in acdream's reachable closure. Documented here for posterity so the next person looking at the spec knows why they're missing.

Component	Why not extracted
LandSurfaceManager	Already ported as src/AcDream.Core/Terrain/TerrainBlending.cs.
SceneryRenderManager	Acdream uses only the stateless SceneryHelpers. The render pipeline is WbDrawDispatcher.
EnvCellRenderManager	Acdream renders env cells via ObjectMeshManager.PrepareEnvCellMeshData + WbDrawDispatcher.
PortalRenderManager	Same — portal cells go through the same path.
TerrainRenderManager	Already replaced by src/AcDream.App/Rendering/TerrainModernRenderer.cs (Phase N.5b).
FontRenderer, MinimapRenderer, BackendGizmoDrawer, AudioPlaybackEngine	Editor-only or replaced by our own subsystems.
WorldBuilder.Shared/Hubs, Migrations, Repositories, editor Services	Editor-only (SignalR, EF Core).

4.5 What we DROP outright

• _wbDats = new DefaultDatReaderWriter(datDir) in WbMeshAdapter.cs:79 — replaced with the existing _dats: DatCollection field passed straight to ObjectMeshManager.
• The [indoor-upload] NULL_RESULT cross-check block at WbMeshAdapter.cs:224-262 and the _pendingEnvCellRequests tracker. Phase 2 cell-resolution diagnostic; no longer needed.
• _wbDats?.Dispose() in WbMeshAdapter.Dispose().
• The two <ProjectReference> entries in AcDream.App.csproj:38-39 and AcDream.Core.csproj:27-28 to WorldBuilder.Shared + Chorizite.OpenGLSDLBackend.
• The two WorldBuilder.Shared/Services/{IDatReaderWriter,DefaultDatReaderWriter}.cs files — never copied into our repo.
• tests/AcDream.Core.Tests/Terrain/SplitFormulaDivergenceTest.cs — one-time data-collection test that informed the N.5b path-C decision; job done.

---

5. Task breakdown

Task	Description	Effort
O-T1	DONE 2026-05-21. Audit WB call graph — produce closure of every WB type/file we transitively use. Output: docs/research/2026-05-21-phase-o-t1-wb-audit.md.	0.5d ✓
O-T2	Create src/AcDream.Core/Rendering/Wb/ + src/AcDream.App/Rendering/Wb/ (already exists) directory structure. Add NOTICE.md entry with MIT attribution to WB.	0.25d
O-T3	Extract texture / GL infrastructure (~15 files, ~3.1K LOC): TextureHelpers, OpenGLGraphicsDevice, ManagedGL*, GLSLShader, GLHelpers, GLStateScope. Promote 3 internal types to public. Verify: does our closure touch SixLabors.ImageSharp? If yes, strip imports / inline byte handling. If no, document. Build green.	1d
O-T4	Extract mesh pipeline (~8 files, ~3.3K LOC): ObjectMeshManager, ObjectRenderBatch, ObjectRenderData, supports. First 30 min: count _dats.X call sites inside ObjectMeshManager. If ≤20, refactor in place to take DatCollection. If >20, write thin DatCollectionAdapter : IDatReaderWriter and pass that. Document the choice + actual count in the T4 commit. Build green. All existing tests green.	2.5d
O-T5	Extract stateless helpers (5 files, ~782 LOC): SceneryHelpers (Chorizite), TerrainUtils + TerrainEntry + CellSplitDirection (WB.Shared). Strip [MemoryPackable] from TerrainEntry. Update using lines in SceneryGenerator.cs, WbSceneryAdapter.cs, SurfaceDecoder.cs, and tests/AcDream.Core.Tests/Textures/TextureDecodeConformanceTests.cs to point at the new AcDream.Core.Rendering.Wb.* namespace. Build green.	0.5d
O-T6	EnvCell + portal renderers	eliminated by O-D8
O-T7	Drop project references from AcDream.App.csproj and AcDream.Core.csproj. Drop _wbDats field + ctor + dispose + the [indoor-upload] NULL_RESULT block from WbMeshAdapter. Delete SplitFormulaDivergenceTest.cs. Build green + tests green (minus the deleted test).	0.5d
O-T8	Verification gate — visual side-by-side with main against retail in three scenes: Holtburg town (outdoor + scenery), inn interior (EnvCell), and a dungeon (portals). Screenshots captured BEFORE T3 starts, compared after T7. User confirms "looks identical." Measure resident memory at radius=4 + 50 entities visible; confirm ≥50 MB reduction vs main.	1d (incl. user time)
O-T9	Ship — single merge to main with one descriptive commit per task (T2..T7), then a "drop WB references" final commit. Update CLAUDE.md to remove the "WB is referenced as projects" language and replace with "extracted into src/AcDream.Core/Rendering/Wb/." Update docs/architecture/worldbuilder-inventory.md.	0.5d
Total		~6.75d focused work + 1d verification + 0.5d ship ≈ 7.75d

---

6. Acceptance criteria

Build + test:

• dotnet build green across the solution.
• dotnet test green; no regression vs the 1147 + 8 baseline minus SplitFormulaDivergenceTest.cs's test count (delete is deliberate, not a regression).

Reference deletion (the architectural goal):

• Zero references to WorldBuilder.* or Chorizite.OpenGLSDLBackend.* namespaces in AcDream.App.csproj and AcDream.Core.csproj. (PackageReference for Chorizite.DatReaderWriter stays — that's the NuGet DatReaderWriter lib, not WB.)
• Zero using WorldBuilder.* or using Chorizite.OpenGLSDLBackend* in src/AcDream.* (extracted code lives in AcDream.Core.Rendering.Wb.* now).
• DefaultDatReaderWriter referenced in exactly zero places in our source. DatCollection is the only dat reader.
• _wbDats field + ctor + dispose removed from WbMeshAdapter.cs. [indoor-upload] NULL_RESULT block at lines 224-262 removed.
• SplitFormulaDivergenceTest.cs deleted.

Memory + visual (user-facing wins):

• Resident memory at streaming: radius=4 + 50 entities visible: >50 MB reduction vs. pre-Phase-O main.
• Visual side-by-side with main: Holtburg town, an inn interior, a dungeon — all render identically. User confirms via screenshots taken BEFORE T3 and AFTER T7.

New per audit:

• T4 commit message documents the ObjectMeshManager dat-shim path taken (refactor in place if ≤20 sites, or adapter if >20). With the actual count.
• T3 commit message documents the 3 internal-to-public type promotions in Chorizite (EmbeddedResourceReader, TextureFormatExtensions, BufferUsageExtensions).
• T3 commit message states whether SixLabors.ImageSharp was reachable. If yes: documents which paths were stripped. If no: explicit "ImageSharp not reachable" note.

Docs + attribution:

• NOTICE.md includes WB attribution per its MIT license.
• references/WorldBuilder/ directory remains in the repo as a read-reference; not in any csproj.
• CLAUDE.md updated: WB is now a read reference, not a project dependency. The "WB integration cribs" section is rewritten to point at our extracted code.
• docs/architecture/worldbuilder-inventory.md updated to reflect the new ownership.

---

7. Risks + mitigations (audit-updated)

Risk	Likelihood	Severity	Mitigation
Re-introduce bugs WB already fixed (edge-vertex, triangle-Z)	Medium	High	Verbatim copy at the algorithm level. The refactor change at T4 is API-shape only; meshing math / texture decode / particle pipeline stay byte-identical.
ObjectMeshManager refactor reveals >20 _dats.X call sites	Low-Medium	Medium	T4 safety check. First 30 min of T4 is a grep + count. If threshold breached, fall back to thin DatCollectionAdapter : IDatReaderWriter and document. Either path keeps T4 in its 2.5d budget.
DatCollection doesn't implement a method ObjectMeshManager calls	Medium	Medium	T4 audit, fill the gap when found. Add missing methods to DatCollection (we own it) rather than stub them. Bounded — at most a handful of methods.
SixLabors.ImageSharp shows up in the closure at T3	Low	Low	Verify-at-T3 + strip. Grep T3 source on first pass; if found, replace with our existing byte-handling or BCnEncoder calls.
Visual regression we don't catch in side-by-side	Low	Medium	Screenshot Holtburg + inn + dungeon BEFORE starting T3. Compare after T7. Don't trust eyeballs alone.
Loss of [indoor-upload] diagnostic removes useful Phase 2 evidence	Low	Low	The diagnostic's findings are already documented in commit history and the audit. The block was a one-time probe; its job is done.
Hidden transitive WB deps we missed	Low	Low	Audit complete (33 files, ~7.7K LOC, fully bucketed). Build break at T7 would catch any miss. Was Medium pre-audit — reduced to Low.
User upstream-tracks WB for some reason	Low	Low	references/WorldBuilder/ stays in-tree as read-reference. Re-sync diffs are manual ports (same as today).

---

8. Out of scope (explicitly)

• Re-porting from retail decomp anywhere in the extracted code. We copy WB verbatim. If a retail-faithfulness audit is needed later, file a separate phase.
• Performance optimization of extracted code. Even if WB's ObjectMeshManager has a 30% improvement waiting in it, this phase ships it as-is.
• API cleanup of the extracted code beyond the dat-surface change at T4. If the constructor has 8 parameters and it's ugly, ugly it stays. Refactor in a follow-up.
• Refactoring WbMeshAdapter itself. Phase O drops the second reader; the adapter shape stays.
• tools/InspectCoatTex or other tools that use the NuGet Chorizite.DatReaderWriter. Those keep working — they use the NuGet DatReaderWriter, not the WB project reference.
• Re-organizing the extracted namespace. O-D11 picked AcDream.Core.Rendering.Wb.*; topic-based reorganization is a follow-up phase.

---

9. Open questions

All originally-open questions have been resolved by the O-T1 audit or this brainstorm:

1. O-Q1 (thread-model): CLOSED. Verified safe — adapters run render-thread only; WB's own code uses ConcurrentDictionary + locks as defense in depth. See O-T1 audit §6.
2. O-Q2 (TerrainRenderManager): CLOSED. Confirmed not reachable; we use our own TerrainModernRenderer. LandSurfaceManager also not reachable (we have our own port). Both dropped from extract list per O-D8.
3. O-Q3 (namespace): CLOSED. Adopted AcDream.Core.Rendering.Wb.* (option A) per O-D11.
4. O-Q4 (Chorizite.DatReaderWriter NuGet): CLOSED. Stays as NuGet — separate from WB project refs.
5. O-Q5 (NEW — SixLabors.ImageSharp): Deferred to T3. Verify reachability during extraction. Strip if found; document if not.

---

10. Naming + filing

• Phase name: O (letter remains).
• Phase tagline: "ONE thing touches the DATs."
• Roadmap entry: already in docs/plans/2026-04-11-roadmap.md under "currently active". Move to "shipped" at T9.
• Issue tracking: filed as a Phase, not an issue (multi-commit, multi-day, infrastructural).

---

11. After Phase O ships

Things that become unblocked or cheaper:

• Single-cache memory pressure budget. Easier to size streaming radii, MSAA, anisotropic budget.
• Audit a single dat-touching code path instead of two when investigating bugs like the cell-resolution divergence the [indoor-upload] probe was built to investigate.
• Future N.6+ rendering work doesn't have to ask "is this WB's concern or ours?" — it's ours.
• AC community-friendly architecture — the "WB is for tools" criticism is addressed at the structural level.
• A follow-up phase to refactor namespaces from AcDream.Core.Rendering.Wb.* into topic-based (Meshes.*, Textures.*) becomes possible. Estimated 0.5d when scheduled.