acdream/docs/research/2026-04-26-datreaderwriter-reference.md

DatReaderWriter (DRW) — developer reference

Status as of 2026-04-26. Source for "what's safe to call, what's broken, how to upgrade." Update this doc when DRW version bumps or when we discover a new safety/coverage gap.

1. Current version

• Package: Chorizite.DatReaderWriter
• Version: 2.1.7 (NuGet)
• Pinned in src/AcDream.Core/AcDream.Core.csproj and src/AcDream.Cli/AcDream.Cli.csproj. App + Core both transitively use that one version.
• Some tools/* (WeatherSetupProbe, WeatherEnumerator, SkyObjectInspect, dump-keymap, PesChainAudit) ProjectReference the vendored repo directly instead of NuGet — see §7.

2. Source repo

• Origin: https://github.com/Chorizite/DatReaderWriter
• Vendored at: references/DatReaderWriter/ (HEAD 63e84b6 "Bump version" — no git tag matching the NuGet version; the repo doesn't tag releases, ship version is GitVersion.yml-driven).
• License: MIT, © 2024 ACClientLib (references/DatReaderWriter/LICENSE.txt).
• Targets: net8.0, netstandard2.0, net48. We consume from net10.0 projects; the net8.0 TFM resolves cleanly via netstandard forward-compat.
• Project layout: DatReaderWriter/ (library), …SourceGenerator/ (Roslyn generator emits packing code), …Tests/, …Benchmarks/.

3. Types we use

Top-level entry points:

• DatReaderWriter.DatCollection — the four-database container (Portal, Cell, HighRes, Local).
• DatReaderWriter.Options.DatCollectionOptions (caching mode, paths).
• DatCollection.Get<T>(id) / Portal.Get<T>(id) — generic typed read.
• DatCollection.GetAllIdsOfType<T>() — used by CLI enumerators.

DatReaderWriter.DBObjs.* consumed in src/:

• LandBlock, LandBlockInfo, EnvCell, Region
• GfxObj, Setup, Surface, SurfaceTexture (via SurfaceDecoder)
• Animation, MotionTable
• ParticleEmitter, PhysicsScript
• Environment (note: collides with System.Environment, must be FQN in AcDream.Cli/Program.cs)
• Wave / sound DBObjs (via DatSoundCache, SoundCookbook)

DatReaderWriter.Types.* consumed:

• AnimationFrame, MotionData
• Polygon, VertexArray, Sphere (BSPQuery, collision)
• SoundEntry, AnimationHook, CreateParticleHook
• BSPNode / BSPTree / PhysicsBSPNode / CellBSPNode / DrawingBSPNode (BSP traversal in BSPQuery, PhysicsDataCache)

DatReaderWriter.Enums.* consumed:

• MotionCommand (aliased DRWMotionCommand in MotionCommandResolver)
• Sound (aliased DRWSound)
• StipplingType, plus various surface/material flags

4. Known broken / missing types

• RenderMaterial — not implemented. Per upstream README "Known Issues." We do not consume it.
• LayoutDesc — README marks it "supported but structure will need cleanup." Treat fields as unstable across versions. We do not depend on it directly.
• ParticleEmitterInfo — DRW exposes the ParticleEmitter DBObj only. There is no separate ParticleEmitterInfo type. Our Vfx/VfxModel.cs comment that mentions "ParticleEmitterInfo dat" is conceptual (matches the retail header struct name); the actual lookup in EmitterDescLoader.cs is dats.Portal.Get<ParticleEmitter>(id) with a using DatParticleEmitter = DatReaderWriter.DBObjs.ParticleEmitter; alias because we have our own ParticleEmitter runtime class in AcDream.Core.Vfx.
• Anything not in DBObjs/ — only 5 DBObj files exist (ActionMap, DBProperties, Iteration, LandBlock, MasterProperty) at the top level of that folder; the rest (GfxObj, Setup, etc.) live elsewhere in the source tree. If a type doesn't resolve, search the DRW source — don't assume it's missing.

5. Unsafe patterns

DatCollection is NOT thread-safe. DatBinReader holds a buffer position field per database; concurrent Get<T> calls from two threads corrupt each other and produce silently half-populated payloads (half-empty LandBlock.Height[] in our case). See memory/feedback_phase_a1_hotfix_saga.md rule #2 — Phase A.1 burned ~3 hotfixes mis-diagnosing this as something else.

Rules:

1. All DatCollection.Get<T>() calls run on the render thread.
2. LandblockStreamer runs synchronously (see header comment, lines 17–32 of src/AcDream.App/Streaming/LandblockStreamer.cs). The worker-thread design was reverted; the channel-based outbox is retained so the moment a thread-safe wrapper exists we can swap loaders back to async without touching call sites.
3. Don't claim "DRW is thread-safe per its docs." There are no such docs. The README says nothing about concurrency.
4. The async surface area (*Async methods) is for I/O, not concurrent reads — calling them from two threads is still unsafe.

Caching: DatCollectionOptions caching mode is OnDemand. Don't flip to None without re-benchmarking — terrain mesh build hits the same LandBlock repeatedly.

6. How to upgrade

1. Bump Version="2.1.7" in src/AcDream.Core/AcDream.Core.csproj and src/AcDream.Cli/AcDream.Cli.csproj (and any tools/* using the NuGet package, e.g. TextureDump).
2. Pull references/DatReaderWriter/ to the matching commit so the vendored copy stays in sync (used by tools that ProjectReference it directly + as a search target for "what's in DRW today").
3. dotnet restore && dotnet build from repo root.
4. Run the full test suite (dotnet test). Watch for compile breaks on type renames (DRW has historically renamed DBObj types between minor versions).
5. Smoke-test live: launch against the local ACE server and walk Holtburg → Foundry. Watch for half-loaded landblocks (missing heights / null surfaces) — that's the canonical "thread-safety regression slipped in" signature.

Risks per area:

• DBObj field renames — DRW source-generates packing; field renames break every consumer. Grep before bumping.
• Enum value reordering — MotionCommand, Sound, StipplingType are enum-by-name in our code, so adds are safe but removals will break.
• BSP type changes — BSPNode family is consumed in BSPQuery.cs (collision). Any signature change there is a high-risk diff that needs the conformance sweep re-run.

7. NuGet vs vendored

• Use NuGet (Chorizite.DatReaderWriter package) for everything shipped in src/ and any tools/ that doesn't need DRW internals. This is the default. Currently AcDream.Core, AcDream.Cli, and tools/TextureDump.
• Vendor (ProjectReference to references/DatReaderWriter/…) when:
  • The tool needs a DRW type or method that isn't public in the NuGet build (some helpers are internal).
  • You're prototyping a DRW patch you intend to upstream — edit in place, build, validate, then PR upstream.
  • Diagnostic / probe tools (WeatherSetupProbe, WeatherEnumerator, SkyObjectInspect, dump-keymap, PesChainAudit) where iteration speed against DRW source matters more than ship discipline.
• Never mix within one assembly. A project either NuGet-references DRW or ProjectReferences the vendored copy. Both produces type- identity errors at the boundary.
• Production code stays on NuGet. When a vendored tool proves a DRW change is needed, upstream it, wait for a release, and bump §6.