acdream/src/AcDream.Content/DatCollectionAdapter.cs
Erik 7b456b49d6 perf(diag): per-frame history export + checkpoint LOH/cache counters + soak capped mode (2026-07-24 audit review)
An adversarial performance review found our own instruments cannot
measure the project's own performance gates:

- FrameProfiler aggregated CPU/GPU/alloc/stage samples into ~5-second
  windows and reset the ring buffers after each report, so route-wide
  p50/p95/p99 distributions across a whole soak could not be
  reconstructed after the fact. ACDREAM_FRAME_HISTORY=<path> now opts
  into a separate per-frame history (one record per frame, ~72
  bytes/record, accumulated in memory with zero frame-thread I/O) that
  a shutdown-only Dispose() writes as CSV. The aggregated [frame-prof]
  report format and its existing metrics are unchanged.

- The canonical checkpoint JSON tracked cache residency (entry/byte
  counts) but never LOH size/fragmentation, process-wide allocated
  bytes, or cache hit/miss/eviction traffic — a committed audit JSON
  showed 65% LOH fragmentation that no tracked instrument recorded,
  and "does a revisit portal hit or miss the caches" was unanswerable
  from an artifact alone. WorldLifecycleResourceSnapshot now carries
  loh_size_bytes/loh_fragmentation_bytes (GCMemoryInfo.GenerationInfo
  index 3), process_total_allocated_bytes (GC.GetTotalAllocatedBytes),
  and Interlocked hit/miss/eviction counters for the CPU mesh cache,
  decoded-texture cache, and the four bounded DAT-object caches
  (portal/cell/highRes/language, aggregated).

- run-connected-r6-soak.ps1 unconditionally forced
  ACDREAM_UNCAPPED_RENDER=1 with no capped mode, while its sibling
  lifecycle-gate script correctly gated it behind a switch. Added
  -Uncapped (default capped, matching the sibling script's pattern),
  fixed the stationary dwell (12s -> 26s, past the 25s
  LiveEntityLivenessController deadline the adjacent comment already
  cited), and now write an env-disclosure.json into the automation
  artifact directory before every launch listing every ACDREAM_* var
  the script sets plus -Uncapped, since the prior audit could only see
  ACDREAM_DUMP_MOVE_TRUTH and nothing else was ever recorded anywhere.

Cache counters are wired via the existing composition path
(ObjectMeshManager already owns the CPU mesh cache and the mesh
extractor directly; content.Dats is threaded into
WorldLifecycleResourceSnapshotSource the same way every other
composition consumer receives it). The DAT-object cache lives behind
IDatReaderWriter, a third-party interface from the DatReaderWriter
package that cannot be extended; RuntimeDatCollection (the one
production implementation) exposes the aggregate stats directly and a
pattern match reads them, degrading to zero for any test double —
no new static registry was introduced (GpuMemoryTracker remains the
one precedented process-wide static).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 1da2c33c875b41fa383dd79694ee2765f0e21896)
2026-07-24 12:00:30 +02:00

243 lines
10 KiB
C#

using DatReaderWriter;
using DatReaderWriter.Enums;
using DatReaderWriter.Lib.IO;
using System;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.Diagnostics.CodeAnalysis;
namespace AcDream.Content;
/// <summary>
/// THE <see cref="DatCollection"/> → <see cref="IDatReaderWriter"/> adapter,
/// shared by all three consumers: ObjectMeshManager (AcDream.App, via
/// WbMeshAdapter), acdream-bake (AcDream.Bake.BakeRunner), and the pak
/// equivalence suite (AcDream.Content.Tests). MP1b's review found the
/// original App-internal copy had silently forked into three near-identical
/// versions — drift between them is exactly what the live-vs-pak equivalence
/// suite CANNOT detect (both sides would drift together only if they share
/// one implementation), so unification here is load-bearing, not cosmetic.
/// GL-free (DatReaderWriter only), hence Content is the right home
/// (established in MP1a alongside <see cref="IDatReaderWriter"/>).
///
/// <para>History: O-D7 originally introduced this adapter (App-internal)
/// because ObjectMeshManager had 26 <c>_dats.X</c> call sites, letting it
/// stay byte-identical to the WB original while routing all DAT I/O through
/// the single DatCollection.</para>
///
/// <para>Iteration properties return the REAL dat iterations (the App copy
/// hardcoded 0 — a stub nothing read; the bake tool stamps iterations into
/// the pak header from DatCollection directly, and any future consumer of
/// these properties should get truth, so the unification intentionally
/// keeps the real values).</para>
/// </summary>
public sealed class DatCollectionAdapter : IDatReaderWriter {
private readonly DatCollection _dats;
private readonly DatDatabaseWrapper _portal;
private readonly DatDatabaseWrapper _cell;
private readonly DatDatabaseWrapper _highRes;
private readonly DatDatabaseWrapper _language;
private readonly ReadOnlyDictionary<uint, IDatDatabase> _cellRegions;
public DatCollectionAdapter(DatCollection dats) {
ArgumentNullException.ThrowIfNull(dats);
_dats = dats;
_portal = new DatDatabaseWrapper(dats.Portal);
_cell = new DatDatabaseWrapper(dats.Cell);
_highRes = new DatDatabaseWrapper(dats.HighRes);
_language = new DatDatabaseWrapper(dats.Local);
// DatCollection has a single Cell, not multiple cell regions.
// Expose it as region 0 to satisfy callers that iterate CellRegions.
var regions = new Dictionary<uint, IDatDatabase> { [0u] = _cell };
_cellRegions = new ReadOnlyDictionary<uint, IDatDatabase>(regions);
}
/// <summary>Source directory of the underlying DatCollection.</summary>
public string SourceDirectory => _dats.Options.DatDirectory ?? string.Empty;
/// <summary>
/// Aggregate hit/miss/eviction counts across the four bounded per-database
/// object caches (portal/cell/highRes/language). 2026-07-24
/// measurement-tooling review: answers "do revisit portals hit or miss
/// the caches" for the DAT-object layer specifically.
/// </summary>
public CacheStats ObjectCacheStats =>
_portal.ObjectCacheStats + _cell.ObjectCacheStats
+ _highRes.ObjectCacheStats + _language.ObjectCacheStats;
public IDatDatabase Portal => _portal;
public IDatDatabase Cell => _cell;
public ReadOnlyDictionary<uint, IDatDatabase> CellRegions => _cellRegions;
public IDatDatabase HighRes => _highRes;
public IDatDatabase Language => _language;
public IDatDatabase Local => _language;
[return: MaybeNull]
public T Get<T>(uint fileId) where T : IDBObj =>
TryGet<T>(fileId, out var value) ? value : default;
public bool TryGet<T>(uint fileId, [MaybeNullWhen(false)] out T value) where T : IDBObj {
if (typeof(T) == typeof(DatReaderWriter.DBObjs.Iteration)) {
throw new Exception(
"Iteration is not a valid type to get from a dat file collection since it is used in all dat files. Use a specific dat like datCollection.Portal.Get<Iteration>()");
}
switch (_dats.TypeToDatFileType<T>()) {
case DatFileType.Cell:
return _cell.TryGet(fileId, out value);
case DatFileType.Portal:
return _portal.TryGet(fileId, out value)
|| _highRes.TryGet(fileId, out value);
case DatFileType.Local:
return _language.TryGet(fileId, out value);
default:
value = default;
return false;
}
}
public IEnumerable<uint> GetAllIdsOfType<T>() where T : IDBObj =>
_dats.TypeToDatFileType<T>() switch {
DatFileType.Cell => _cell.GetAllIdsOfType<T>(),
DatFileType.Portal => _portal.GetAllIdsOfType<T>()
.Concat(_highRes.GetAllIdsOfType<T>()),
DatFileType.Local => _language.GetAllIdsOfType<T>(),
_ => Array.Empty<uint>(),
};
// RegionFileMap is used by some WB internals but not by any acdream consumer.
public ReadOnlyDictionary<uint, uint> RegionFileMap =>
new ReadOnlyDictionary<uint, uint>(new Dictionary<uint, uint>());
public int PortalIteration => _portal.Iteration;
public int CellIteration => _cell.Iteration;
public int HighResIteration => _highRes.Iteration;
public int LanguageIteration => _language.Iteration;
public bool TryGetFileBytes(uint regionId, uint fileId, ref byte[] bytes, out int bytesRead) {
// Route to cell db (the only region we expose)
return _dats.Cell.TryGetFileBytes(fileId, ref bytes, out bytesRead);
}
/// <summary>
/// Resolves a DAT id to all databases that contain it, along with the DBObjType.
/// Mirrors DefaultDatReaderWriter.ResolveId — checks each underlying DatDatabase
/// via DatDatabase.TypeFromId (which reads the type range tables).
/// </summary>
public IEnumerable<IDatReaderWriter.IdResolution> ResolveId(uint id) {
var results = new List<IDatReaderWriter.IdResolution>();
void CheckDb(DatDatabaseWrapper wrapper) {
var rawDb = wrapper.RawDatabase;
if (rawDb.Tree.TryGetFile(id, out _)) {
var type = rawDb.TypeFromId(id);
if (type != DBObjType.Unknown)
results.Add(new IDatReaderWriter.IdResolution(wrapper, type));
}
}
// Match DefaultDatReaderWriter ordering: HighRes → Portal → Language → Cell
CheckDb(_highRes);
CheckDb(_portal);
CheckDb(_language);
CheckDb(_cell);
return results;
}
public bool TrySave<T>(T obj, int iteration = 0) where T : IDBObj =>
throw new NotSupportedException("DatCollectionAdapter is read-only.");
public bool TrySave<T>(uint regionId, T obj, int iteration = 0) where T : IDBObj =>
throw new NotSupportedException("DatCollectionAdapter is read-only.");
public void Dispose() {
// The underlying DatCollection is owned by the caller — do not dispose it here.
// Individual wrapper objects hold no unmanaged resources.
}
}
/// <summary>
/// Wraps a <see cref="DatDatabase"/> as <see cref="IDatDatabase"/>.
/// Mirrors WorldBuilder.Shared.Services.DefaultDatDatabase (taken into our
/// tree in Phase O; moved here from AcDream.App in the MP1b adapter
/// unification).
/// </summary>
public sealed class DatDatabaseWrapper : IDatDatabase {
private readonly DatDatabase _db;
// One cache per database: a DatCollectionAdapter therefore retains at
// most 4 * 256 decoded entries and 4 * 64 MiB of estimated payload. The
// cache itself documents why estimated bytes are not a hard heap bound.
private readonly BoundedDatObjectCache _cache = new();
private readonly object _databaseLock = new();
public DatDatabaseWrapper(DatDatabase db) {
ArgumentNullException.ThrowIfNull(db);
_db = db;
}
/// <summary>Exposes the raw DatDatabase for ResolveId's Tree.TryGetFile + TypeFromId calls.</summary>
internal DatDatabase RawDatabase => _db;
public DatDatabase Db => _db;
public int Iteration => _db.Iteration?.CurrentIteration ?? 0;
/// <summary>This database's bounded DAT-object cache hit/miss/eviction counts (2026-07-24 measurement-tooling review).</summary>
public CacheStats ObjectCacheStats => _cache.Stats;
public IEnumerable<uint> GetAllIdsOfType<T>() where T : IDBObj =>
_db.GetAllIdsOfType<T>();
public bool TryGet<T>(uint fileId, [MaybeNullWhen(false)] out T value) where T : IDBObj {
if (_cache.TryGet(fileId, out value)) {
return true;
}
lock (_databaseLock) {
// A different reader may have populated the cache while this
// caller waited for the serialized DatDatabase read path.
if (_cache.TryGet(fileId, out value)) {
return true;
}
if (_db.TryGet<T>(fileId, out value)) {
value = _cache.GetOrAdd(fileId, value);
return true;
}
// TEMP diagnostic (dat-race investigation 2026-06-09, strip with fix):
// a miss for an id whose BTree entry EXISTS is always an anomaly —
// either Unpack returned false or the lookup flickered transiently.
// Legit not-found probes (e.g. Portal→HighRes fallback) stay silent.
// Kept through the MP1b unification — the tripwire now covers the
// bake tool and the equivalence suite too, not just the client.
if (_db.Tree.TryGetFile(fileId, out _)) {
Console.WriteLine(
$"[dat-miss] {typeof(T).Name} 0x{fileId:X8} entry EXISTS but TryGet failed " +
$"(thread={Environment.CurrentManagedThreadId})");
}
}
return false;
}
public bool TryGetFileBytes(uint fileId, [MaybeNullWhen(false)] out byte[] value) {
lock (_databaseLock) {
return _db.TryGetFileBytes(fileId, out value);
}
}
public bool TryGetFileBytes(uint fileId, ref byte[] bytes, out int bytesRead) {
lock (_databaseLock) {
return _db.TryGetFileBytes(fileId, ref bytes, out bytesRead);
}
}
public bool TrySave<T>(T obj, int iteration = 0) where T : IDBObj =>
throw new NotSupportedException("DatDatabaseWrapper is read-only.");
public void Dispose() {
// The underlying DatDatabase is owned by DatCollection — do not dispose here.
}
}