acdream/docs/superpowers/specs/2026-05-21-phase-o-dat-path-unification-design.md

Phase O — DatPath Unification — Design Spec

Filed: 2026-05-21 Status: DRAFT (not yet scheduled) Owner: TBD Estimated effort: 1-2 focused weeks of work; one ship-window, no slicing across multiple weeks.

Tagline: ONE thing touches the DATs. Today we have two readers in process (acdream's DatCollection + WorldBuilder's DefaultDatReaderWriter) reading the same files independently. Phase O collapses that to one.

---

1. Problem statement

As of Phase N.4 ship (2026-05-08), WorldBuilder is integrated as our rendering + dat-handling base. Concretely:

• WbMeshAdapter.cs:79 constructs _wbDats = new DefaultDatReaderWriter(datDir) — WB's own dat reader.
• Our DatCollection is constructed independently at startup for the rest of the client (network, physics, animation, clothing, audio, UI, etc.).

Both readers open the same four files (client_portal.dat, client_cell_1.dat, client_highres.dat, client_local_English.dat) with independent file handles and independent in-memory index caches.

Costs:

• ~50-100 MB duplicated index cache memory (per WbMeshAdapter.cs:27 comment).
• Double seek cost when the same dat block is read by both pipelines (mesh path via WB; surface metadata side-table via our DatCollection).
• Cross-check awkwardness — WbMeshAdapter.cs:224-229 has explicit "if WE find the cell but WB doesn't" diagnostic code, born from the divergence.
• Architectural smell — a third-party feedback signal (AC community comment, 2026-05-21) flagged WB's dat-touching as "built for tools, not runtime." The double-reader is the most concrete manifestation.

WB is MIT-licensed, so the path to fixing this is not to "wrap" WB but to extract its load-bearing code into our repo and route it through our DatCollection. This is the long-term clean architecture: one reader, one cache, one source of dat truth.

---

2. Decisions log

#	Decision	Why
O-D1	Extract WB code verbatim into our repo. No re-port from retail decomp. No "improvements" while extracting.	CLAUDE.md is explicit: "Re-porting from retail decomp when WB already has a tested port is how subtle bugs (scenery edge-vertex, triangle-Z) keep slipping in." Verbatim copy preserves all the corner-case fixes WB already has.
O-D2	Make extracted code consume DatCollection, not IDatReaderWriter.	Single source of dat truth. The whole point of the phase.
O-D3	Drop the WorldBuilder.Shared + Chorizite.OpenGLSDLBackend project references at the end of the phase.	If we still reference WB after extraction, we haven't actually finished the work.
O-D4	Keep WB in references/ for reading/comparison. Don't delete the vendored directory.	We'll still want to grep WB during ports of NEW pieces (e.g., minimap renderer if/when we add it).
O-D5	MIT attribution per WB convention. Add NOTICE.md entry crediting WorldBuilder for the extracted code.	License compliance.
O-D6	One ship-window, not sliced. Either the whole extraction lands and references/WorldBuilder is dropped, or we roll back the entire phase.	Half-extracted state (some WB code in our repo, some still referenced) is worse than either endpoint.

---

3. Architecture overview

Today (Phase N.4–A.5 state)

┌─────────────────────────┐       ┌──────────────────────────────────┐
│ acdream subsystems      │       │ WorldBuilder (referenced project)│
│ - Network               │       │  ┌────────────────────────────┐  │
│ - Physics               │ ───→  │  │ DefaultDatReaderWriter     │  │
│ - Animation             │       │  │  (opens 4 .dat files)      │  │
│ - Clothing/Audio/UI     │       │  └────────────┬───────────────┘  │
│ - Surface metadata      │       │               │                  │
└──────────┬──────────────┘       │  ┌────────────▼───────────────┐  │
           │                      │  │ ObjectMeshManager          │  │
           │ DatCollection        │  │ TextureHelpers             │  │
           │ (opens same 4 files) │  │ SceneryHelpers             │  │
           │                      │  │ LandSurfaceManager         │  │
           ▼                      │  │ OpenGLGraphicsDevice       │  │
    ┌──────────────┐              │  └────────────────────────────┘  │
    │ Dat files    │              └──────────────────────────────────┘
    │ (same files, │              ▲
    │  two readers)│──────────────┘
    └──────────────┘

Phase O target

┌─────────────────────────────────────────────────────────────────┐
│ acdream subsystems                                              │
│ - Network / Physics / Animation / Clothing / Audio / UI         │
│ - Mesh pipeline   (extracted from WB.ObjectMeshManager)         │
│ - Texture decode  (extracted from WB.TextureHelpers)            │
│ - Scenery         (extracted from WB.SceneryHelpers etc.)       │
│ - Terrain blend   (extracted from WB.LandSurfaceManager)        │
│ - GL state        (extracted from WB.OpenGLGraphicsDevice)      │
└──────────────────────────┬──────────────────────────────────────┘
                           │
                           │ DatCollection (the ONLY reader)
                           ▼
                  ┌──────────────┐
                  │ Dat files    │
                  └──────────────┘

What stays acdream-original (unchanged by this phase)

• TerrainModernRenderer (Phase N.5b — already ours, uses LandblockMesh.Build with retail's FSplitNESW)
• Network, physics, animation, movement, UI, audio, chat, streaming controller, plugin API
• Our DatCollection itself (becomes the only dat reader)

---

4. Component changes (the actual extract list)

4.1 Mesh pipeline (highest-value extraction)

WB source	New acdream home	Adaptation
Chorizite.OpenGLSDLBackend.Lib.ObjectMeshManager	src/AcDream.Core/Rendering/Wb/ObjectMeshManager.cs	Replace IDatReaderWriter field with DatCollection. Constructor takes our DatCollection. All _dats.Get<T>() calls preserved.
Chorizite.OpenGLSDLBackend.Lib.ObjectRenderBatch + supporting types (TextureKey, etc.)	Same directory	Verbatim copy. No interface changes.
Chorizite.OpenGLSDLBackend.Lib.SceneryHelpers	src/AcDream.Core/Rendering/Wb/SceneryHelpers.cs	Verbatim.
Chorizite.OpenGLSDLBackend.Lib.SceneryRenderManager	Same	Verbatim + DatCollection swap.

4.2 Texture pipeline

WB source	New acdream home	Adaptation
Chorizite.OpenGLSDLBackend.Lib.TextureHelpers	src/AcDream.Core/Rendering/Wb/TextureHelpers.cs	Verbatim copy. Pure functions — no dat dependency.
Chorizite.OpenGLSDLBackend.Lib.LandSurfaceManager	src/AcDream.Core/Rendering/Wb/LandSurfaceManager.cs	Verbatim + DatCollection swap.

4.3 GL infrastructure

WB source	New acdream home	Adaptation
Chorizite.OpenGLSDLBackend.OpenGLGraphicsDevice	src/AcDream.App/Rendering/Wb/OpenGLGraphicsDevice.cs	Verbatim. No dat dependency.
Chorizite.OpenGLSDLBackend.Frustum + VisibilityManager	Same	Verbatim.

4.4 EnvCell decode

WB source	New acdream home	Adaptation
Chorizite.OpenGLSDLBackend.Lib.EnvCellRenderManager (if used at runtime)	src/AcDream.Core/Rendering/Wb/EnvCellRenderManager.cs	Verbatim + DatCollection swap.
Chorizite.OpenGLSDLBackend.Lib.PortalRenderManager	Same	Verbatim.

4.5 What we DROP

• _wbDats = new DefaultDatReaderWriter(datDir) in WbMeshAdapter.cs:79 — replaced with the existing _dats: DatCollection field.
• Cross-check code in WbMeshAdapter.cs:224-229 (if WE find the cell but WB doesn't) — no longer relevant after unification.
• The two project references in AcDream.App.csproj and AcDream.Core.csproj to WorldBuilder.Shared + Chorizite.OpenGLSDLBackend.

---

5. Task breakdown

Task	Description	Estimated effort
O-T1	Audit WB call graph — produce a closure of every WB type/file we transitively use. Generate a "must extract" list with line counts.	0.5d
O-T2	Create src/AcDream.Core/Rendering/Wb/ directory structure. Add NOTICE.md entry with MIT attribution to WB.	0.25d
O-T3	Extract texture / GL infrastructure (TextureHelpers, OpenGLGraphicsDevice, Frustum, VisibilityManager) — verbatim copy, no dat dependency means low risk. Build green.	1d
O-T4	Extract mesh pipeline (ObjectMeshManager, ObjectRenderBatch, TextureKey) — verbatim copy. Replace IDatReaderWriter with DatCollection. Build green. Existing tests green.	2d
O-T5	Extract scenery + terrain pipelines (SceneryHelpers, SceneryRenderManager, LandSurfaceManager) — verbatim + DatCollection swap. Build green. Visual verification on outdoor terrain.	1.5d
O-T6	Extract EnvCell + portal renderers if used at runtime (EnvCellRenderManager, PortalRenderManager) — verbatim + DatCollection swap. Visual verification on indoor scenes.	1d
O-T7	Drop project references from AcDream.App.csproj and AcDream.Core.csproj. Drop _wbDats and the cross-check code from WbMeshAdapter. Build green.	0.5d
O-T8	Verification gate — visual side-by-side with main against retail in three scenes: Holtburg town (outdoor + scenery), inn interior (EnvCell), and a dungeon (portals). User confirms "looks identical."	1d (incl. user time)
O-T9	Ship — single merge to main with one descriptive commit per task (T3..T7), then a "drop WB references" final commit. Update CLAUDE.md to remove the "WB is referenced as projects" language and replace with "extracted into src/AcDream.Core/Rendering/Wb/." Update docs/architecture/worldbuilder-inventory.md.	0.5d
Total		~7-8 working days + visual verification cycles

---

6. Acceptance criteria

• dotnet build green across the solution.
• dotnet test green; no test count regression (currently ~1147).
• Zero references to WorldBuilder.* or Chorizite.OpenGLSDLBackend.* namespaces in AcDream.App.csproj and AcDream.Core.csproj (PackageReference for Chorizite.DatReaderWriter stays — that's the NuGet DatReaderWriter lib, not WB).
• Zero using WorldBuilder.* in src/AcDream.* (the extracted code lives in our namespace now).
• DefaultDatReaderWriter referenced in exactly zero places in our source. DatCollection is the only dat reader.
• Resident memory measured at streaming: radius=4 + 50 entities visible: >50 MB reduction vs. pre-Phase-O main (validates the duplication is actually gone).
• Visual side-by-side with main: Holtburg town, an inn interior, a dungeon — all render identically. User confirms.
• NOTICE.md includes WB attribution per its MIT license.
• references/WorldBuilder/ directory remains in the repo for read-reference, but is not referenced by any csproj.
• CLAUDE.md updated: WB is now a read reference, not a project dependency. The "WB integration cribs" section is rewritten to point at our extracted code.
• docs/architecture/worldbuilder-inventory.md updated to reflect the new ownership.

---

7. Risks + mitigations

Risk	Likelihood	Severity	Mitigation
Re-introduce bugs WB had already fixed (edge-vertex, triangle-Z)	Medium	High	Discipline: verbatim copy first. No "improvements." A separate post-Phase-O ticket can refactor.
WB has implicit assumptions about DefaultDatReaderWriter semantics our DatCollection doesn't match (caching, thread safety)	Medium	High	Task O-T1 must audit the actual call surface used. If DatCollection diverges, add a thin adapter layer rather than monkey-patch WB code.
Hidden transitive WB dependencies (some helper class we didn't notice we use)	Low	Medium	Task O-T1 closure pass should catch this. If it slips, build will fail loudly.
Visual regression we don't catch in side-by-side	Low	Medium	Take screenshots of three reference scenes BEFORE starting; compare AFTER. Don't trust eyeballs alone.
User upstream-tracks WB for some reason	Low	Low	We keep references/WorldBuilder/ in-tree. Re-syncing diffs is a manual port if needed (same as today).
Phase O is scoped too tightly and a piece of WB we use is missed	Medium	Medium	T1's audit is the safety. If we miss one, build fails before T7's reference-drop.

---

8. Out of scope (explicitly)

• Re-porting from retail decomp anywhere in the extracted code. We copy WB verbatim. If a retail-faithfulness audit is needed later, file a separate phase.
• Performance optimization of extracted code. Even if WB's ObjectMeshManager has a 30% improvement waiting in it, this phase ships it as-is.
• API cleanup of the extracted code. If the constructor has 8 parameters and it's ugly, ugly it stays. Refactor in a follow-up.
• Refactoring WbMeshAdapter itself. Phase O drops the second reader; the adapter shape stays.
• tools/InspectCoatTex or other tools that use the NuGet Chorizite.DatReaderWriter. Those keep working — they use the NuGet DatReaderWriter, not the WB project reference.

---

9. Open questions

1. Does WB's ObjectMeshManager thread-model match LandblockStreamer's worker thread?

   • Phase O-T1 must answer. If WB assumes single-threaded mesh-pipeline access and we call into it from the worker, we have a real bug today that gets exposed louder after extraction.
   • Fallback: if WB's code is not thread-safe, route the mesh-prepare call through a dispatcher to the render thread (extra latency, but correct).

2. Should we extract WB's TerrainRenderManager even though we already wrote TerrainModernRenderer?

   • Today: we use OUR terrain renderer (Phase N.5b). WB's TerrainRenderManager is not in our render path.
   • But: LandSurfaceManager (terrain texture blending) IS WB code we use, and it might pull on TerrainRenderManager types.
   • Recommendation: extract LandSurfaceManager only; leave TerrainRenderManager in references/ since we don't use it.

3. What's the right namespace for extracted code?

   • Option A: AcDream.Core.Rendering.Wb.* — preserves "this came from WB" context.
   • Option B: AcDream.Core.Rendering.Meshes.*, AcDream.Core.Rendering.Textures.* — clean namespaces, drops the WB association.
   • Recommendation: Option A for now (keeps the audit trail), with a follow-up phase to fold into Option B once dust settles.

4. Do we extract the Chorizite.DatReaderWriter NuGet package code itself?

   • No. That's a separate, well-maintained library on NuGet. We keep depending on it via PackageReference. The phase is specifically about the WB project references (WorldBuilder.Shared + Chorizite.OpenGLSDLBackend) — the application code, not the protocol library.

---

10. Naming + filing

• Phase name: O (next free letter — A-N + R are in use).
• Phase tagline: "ONE thing touches the DATs."
• Roadmap entry: add to docs/plans/2026-04-11-roadmap.md under "Phases ahead" with cross-reference to this spec.
• Issue tracking: file as a Phase, not an issue (multi-commit, multi-day, infrastructural).

---

11. After Phase O ships

Things that become unblocked or cheaper:

• Single-cache memory pressure budget. Easier to size streaming radii, MSAA, anisotropic budget, etc.
• Audit a single dat-touching code path instead of two when investigating bugs like #37 (the coat-stub palette overlay).
• Future N.6+ rendering work doesn't have to ask "is this WB's concern or ours?" — it's ours.
• AC community-friendly architecture — the "WB is for tools" criticism is addressed at the structural level, not just papered over.