acdream/docs/research/2026-06-09-dat-reader-thread-safety-investigation.md

Dat-reader thread-safety investigation — read path EXONERATED, teardown bug FIXED, white-walls tripwired

Date: 2026-06-09. Branch: claude/thirsty-goldberg-51bb9b. Trigger: white cottage walls reproduced on a CLEAN (probe-free) launch, refuting the prior session's "heavy probes starve the dat-reader" framing as the whole story; user picked the dat-reader stability fix as the session direction.

Read this before touching any dat-threading code. It REVERSES the long-standing "DatCollection is NOT thread-safe (for reads)" lore for the current package version, and it records exactly what was verified, what was refuted, and what remains open.

---

0. TL;DR

1. Concurrent READS on Chorizite.DatReaderWriter 2.1.7 are SAFE — verified two independent ways: (a) a line-level audit of the actual release/2.1.7 source (cross-checked against the decompiled NuGet DLL); (b) an in-tree hammer test (tests/AcDream.Core.Tests/Conformance/DatConcurrencyStressTests.cs) that replays our four-population access pattern — ~1.1 M concurrent reads, 8 threads, shuffled orders, raw block path AND full typed unpack path — zero anomalies, byte-identical to single-threaded golden fingerprints.
2. The verified crash bug is teardown, not reads: ObjectMeshManager.Dispose never quiesced its Task.Run(ProcessQueueAsync) decode workers and LandblockStreamer.Dispose gave up its join after 2 s — then GameWindow.OnClosing disposed the DatCollection, which unmaps the dats' memory-mapped views (MemoryMappedBlockAllocator.DestroyMappedFile → _viewPtr = null). A worker still inside ReadBlock then dereferences the dead view pointer → uncatchable AccessViolationException with ReadBlock on the stack — the recorded crash signature, firing on close/relaunch during decode storms. FIXED this session (quiesce-before-dispose in ObjectMeshManager.Dispose + entry/loop guards; 15 s loud join in LandblockStreamer.Dispose).
3. The white-walls mechanism is STILL OPEN (issue #105) — but it is now narrowed and instrumented: every silent dat-failure exit on the walls-relevant paths has a tripwire log line ([dat-miss] / [tex-miss] / [tex-skip] / [cell-miss]). They fire ONLY on anomaly (zero cost when healthy) and stay in the build; the next organic occurrence self-attributes in the launch log.
4. Two false trails refuted: both investigation subagents confidently described a ReadBlock instance-field cursor race — the fields don't exist in 2.1.7 (all cursor state is method-local over a stable read-only view). One also cited a vendored references/DatReaderWriter copy that doesn't exist. Verify agent claims against source.

---

1. The threading topology (verified, file:line)

ONE DatCollection (GameWindow.cs:1166, DatAccessType.Read) → four DatDatabases (Portal/Cell/HighRes/Local), each with its own MemoryMappedBlockAllocator. Reader populations on the SAME instances:

Population	Entry	Serialization
Render thread	CellLoader-side hydration, TextureCache.DecodeFromDats, TickAnimations/MotionResolver, WorldPicker	_datLock at 3 sites only (GameWindow.cs:2285,5603,8948,11224); the rest unlocked
Streamer worker	BuildLandblockForStreaming (GameWindow.cs:5142)	the whole load under GameWindow._datLock
Decode pool (≤4)	ObjectMeshManager.ProcessQueueAsync → _dats.* via DatCollectionAdapter	DatDatabaseWrapper._lock (per-db, DatCollectionAdapter.cs:120) — a different lock object than _datLock
Decode pool, raw	DatCollectionAdapter.ResolveId (DatCollectionAdapter.cs:74-96)	none (raw Tree.TryGetFile)

Two disjoint lock domains + unlocked paths ⇒ concurrent same-instance access is constant. Post-investigation verdict: that concurrency is fine for reads (see §2) — the locks are not protecting against anything real on the read path. Do NOT "unify the lock domains" as a fix for a read race that doesn't exist; the locks' remaining value is incidental (they serialize some of our own cache fills).

2. Why the read path is safe (audit summary, v2.1.7 source)

• MemoryMappedBlockAllocator.ReadBlock (Lib/IO/BlockAllocators/MemoryMappedBlockAllocator.cs:137-160): chain cursor + buffer offset are locals; copies are destination-bounded; _viewPtr is written only in the ctor and Expand (write-path only — never runs for DatAccessType.Read).
• BTree: DatBTreeReaderWriter node cache is an internally-locked LRU (:62-126); traversal state is local; nodes are immutable post-unpack.
• DatDatabase.TryGet<T>: rents from the thread-safe ArrayPool, fills via ReadBlock, unpacks through a fresh DatBinReader (per-instance _offset), caches into a ConcurrentDictionary. ObjectFactory = ConcurrentDictionary.GetOrAdd; DBObjAttributeCache = benign last-wins lazy init of immutable dictionaries.
• Hammer test confirms empirically (see §0.1). Known minor wart (not our symptom): _fileCache[fileId] = value is published one statement before value.Id = fileId (DatDatabase.cs:389-393) — a racing reader can see Id == 0. Upstream PR candidate.

Correction to project lore: feedback_phase_a1_hotfix_saga.md's "DatBinReader holds a buffer position field per database; concurrent Get<T> corrupts" does NOT describe 2.1.7 — DatBinReader is created per call. The A.1-era crash predates Phase O's single-reader unification and was likely a different stack (or version). The A.5 spec's "single-threaded by construction" claim was composition-false the day it shipped (N.4's pool decodes already existed) — but per this investigation it also didn't matter for correctness.

3. The teardown fix (this session)

• ObjectMeshManager.Dispose (src/AcDream.App/Rendering/Wb/ObjectMeshManager.cs): sets IsDisposed under the queue lock, cancels + drains _pendingRequests, then waits (≤10 s, 5 ms poll) for _activeWorkers == 0 before returning; logs an error if workers outlive the wait. ProcessQueueAsync re-checks IsDisposed before every dequeue; both Prepare*Async entries and enqueue blocks early-out when disposed.
• LandblockStreamer.Dispose (src/AcDream.App/Streaming/LandblockStreamer.cs): join extended 2 s → 15 s with a loud [streamer] error line on timeout (cancellation is honored between jobs, so the wait is bounded by one landblock load).
• GameWindow.OnClosing order was already correct (streamer → mesh adapter → … → _dats last); with both Disposes now actually quiescing, the order is meaningful.
• Audio was checked and is NOT a dat-reading population (no _dats access under src/AcDream.App/Audio/).

Smoke: 3× close-mid-decode-storm launches, clean exits, no crash signatures (see commit).

4. The white-walls residual (#105) + tripwires

Observed signature (2× user-confirmed): cottage wall surfaces render as background/clear while the cell's statics (paintings, furniture, windows) draw; once broken, broken for the session; strictly intermittent (one clean launch had it, a heavier probe launch didn't); zero error output ever — today's white-wall launch log was 35 lines, no [wb-error].

All silent exits found and tripwired (they print ONLY on anomaly; keep them):

Line	Where	Meaning when it fires
[dat-miss]	DatDatabaseWrapper.TryGet (DatCollectionAdapter.cs)	a read failed for an id whose BTree entry EXISTS — the smoking gun for any real dat-layer fault
[tex-skip]	ObjectMeshManager GfxObj + CellStruct texture chains (5 exits)	a polygon batch (incl. WALL batches) was silently dropped on a dat miss
[tex-miss]	TextureCache.DecodeFromDats (3 exits)	render-thread texture decode fell back to magenta
[cell-miss]	GameWindow interior hydration (EnvCell / Environment null)	a cell's walls were silently never registered — the exact white-walls geometry signature

Discriminator for the next occurrence: magenta walls → TextureCache path; see-through walls + [tex-skip]/[dat-miss] → mesh-build path; see-through + [cell-miss] → hydration; white/missing with NO tripwire output → the failure is GL-side (staged upload / bindless residency under load) — instrument WbMeshAdapter.Tick's upload drain next.

5. Apparatus

• tests/AcDream.Core.Tests/Conformance/DatConcurrencyStressTests.cs — the hammer (8 threads × 25 shuffled passes; raw TryGetFileBytes + typed TryGet with FileCachingStrategy.Never; golden FNV-1a fingerprints). Skips cleanly when dats absent. KEEP as the regression guard for any future dat-reader version bump.
• Package source for reference: git clone --depth 1 --branch release/2.1.7 https://github.com/Chorizite/DatReaderWriter (tag verified == shipped NuGet DLL).
• The tripwire lines (§4) — grep any launch log for dat-miss|tex-miss|tex-skip|cell-miss.