Append strict collision/topology payloads to the existing prepared package so later physics cutover can drop parsed DAT graphs without adding a second mapping or changing traversal behavior. The full 2,232,170-key catalog is deterministic across worker counts, exact-byte aliased, corruption-isolated, and cancellation-safe.
Complete Slice D3 by replacing the unbounded animation dictionary with a concurrent byte/count LRU and by putting the three retail alpha scratch owners behind one typed aggregate budget. Preserve immediate growth and draw order while reclaiming one-frame density spikes after sustained under-use. Close stale bounds-cache issue evidence without inventing a cache.
Introduce explicit loaded, missing, and corrupt outcomes over typed pak keys, validate package/DAT identity at open, add zero-I/O TOC probes, preserve cancellation, and route the package path through RuntimeOptions. Production injection follows in the next checkpoint.
An adversarial performance review found our own instruments cannot
measure the project's own performance gates:
- FrameProfiler aggregated CPU/GPU/alloc/stage samples into ~5-second
windows and reset the ring buffers after each report, so route-wide
p50/p95/p99 distributions across a whole soak could not be
reconstructed after the fact. ACDREAM_FRAME_HISTORY=<path> now opts
into a separate per-frame history (one record per frame, ~72
bytes/record, accumulated in memory with zero frame-thread I/O) that
a shutdown-only Dispose() writes as CSV. The aggregated [frame-prof]
report format and its existing metrics are unchanged.
- The canonical checkpoint JSON tracked cache residency (entry/byte
counts) but never LOH size/fragmentation, process-wide allocated
bytes, or cache hit/miss/eviction traffic — a committed audit JSON
showed 65% LOH fragmentation that no tracked instrument recorded,
and "does a revisit portal hit or miss the caches" was unanswerable
from an artifact alone. WorldLifecycleResourceSnapshot now carries
loh_size_bytes/loh_fragmentation_bytes (GCMemoryInfo.GenerationInfo
index 3), process_total_allocated_bytes (GC.GetTotalAllocatedBytes),
and Interlocked hit/miss/eviction counters for the CPU mesh cache,
decoded-texture cache, and the four bounded DAT-object caches
(portal/cell/highRes/language, aggregated).
- run-connected-r6-soak.ps1 unconditionally forced
ACDREAM_UNCAPPED_RENDER=1 with no capped mode, while its sibling
lifecycle-gate script correctly gated it behind a switch. Added
-Uncapped (default capped, matching the sibling script's pattern),
fixed the stationary dwell (12s -> 26s, past the 25s
LiveEntityLivenessController deadline the adjacent comment already
cited), and now write an env-disclosure.json into the automation
artifact directory before every launch listing every ACDREAM_* var
the script sets plus -Uncapped, since the prior audit could only see
ACDREAM_DUMP_MOVE_TRUTH and nothing else was ever recorded anywhere.
Cache counters are wired via the existing composition path
(ObjectMeshManager already owns the CPU mesh cache and the mesh
extractor directly; content.Dats is threaded into
WorldLifecycleResourceSnapshotSource the same way every other
composition consumer receives it). The DAT-object cache lives behind
IDatReaderWriter, a third-party interface from the DatReaderWriter
package that cannot be extended; RuntimeDatCollection (the one
production implementation) exposes the aggregate stats directly and a
pattern match reads them, degrading to zero for any test double —
no new static registry was introduced (GpuMemoryTracker remains the
one precedented process-wide static).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 1da2c33c875b41fa383dd79694ee2765f0e21896)
Three small verified MeshExtractor.cs fixes from an allocation audit:
- CellStruct clip-map clone guard: the CellStruct polygon-surface path
cloned decoded texture data unconditionally before applying clip-map
transparency, even though the comment above it says "if we got this
from the cache, we need to clone it." The GfxObj path already had the
correct pattern (clone only when textureDataIsCached). Wired the same
textureDataIsCached flag through CellStruct's TryGet/GetOrCreate/
RetainOrUse call sites and gated the clone on it, matching GfxObj
exactly — freshly-decoded arrays (the common case) now skip a
redundant clone + BlockCopy.
- Solid-color texture cache: isSolid surfaces (Base1Solid / NoPos-
stippled polys) allocated a fresh 32x32 RGBA array (4 KiB) on every
polygon, even for repeated colors. Added a MeshExtractor-scoped
ConcurrentDictionary<uint, byte[]> keyed by packed ARGB, bounded by
distinct colors observed (a few hundred at most) rather than call
volume. Traced every downstream consumer of the solid-color array
(clip-map clone-guard, translucency clone-guard, GL upload, pak
serialization) — none currently mutate a solid-color array in place,
since isSolid is mutually exclusive with the texture-decode branch
those guards live in — but wired textureDataIsCached = true at both
call sites anyway so the existing clone-before-mutate machinery
protects the shared array if that ever changes.
- Dropped the redundant .ToList() on both _dats.ResolveId(...) calls:
DatCollectionAdapter.ResolveId already returns a materialized List<T>
typed as IEnumerable; the immediate .OrderByDescending().FirstOrDefault()
enumerates once, so the extra copy was pure waste.
Added SolidColorTextureCacheTests (dat-gated, matching suite convention)
exercising the new cache directly via internal visibility.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit e9f9d7539966cc26343e45d653a751c5d1045810)
Replace the incomplete package path with one DatCollection-backed compatibility seam for PhysicsScripts and Animations. Preserve CreateBlockingParticle's inherited payload and following cursor, route every production and audit consumer through the corrected loaders, and apply retail's post-UnPack StartTime ordering.
Add exact stored-order PhysicsScriptTable upper-threshold resolution, high-byte DID and embedded-ID validation, plus live effect profiles with Setup-to-PhysicsDesc precedence across top-level and attached entity lifetimes. Keep blocking execution deferred and narrow TS-11 accordingly.
Pin synthetic malformed/cursor/order fixtures, installed-DAT blocking and recall audits, high-index IDs, IEEE boundaries, profile teardown, and ordinary decoder parity; synchronize architecture, inventory, milestones, roadmap, research, and memory.
Co-Authored-By: Codex <noreply@openai.com>
Establish the executable-backed PhysicsDesc, sequence-gate, PhysicsScript, CreateBlocking, particle-anchor, projectile, and Hidden-state behavior before changing runtime code. Correct stale blocking/threshold claims and synchronize the project instructions with the current UI architecture and matching retail binary.
Add copyright-safe packet and DAT-container fixtures plus a failing installed-DAT conformance audit for projectile shapes, typed tables, recall motion, default scripts, and raw CreateBlocking inventory.
Co-Authored-By: Codex <noreply@openai.com>
Adversarially-verified review findings 7 and 8:
(7) The DatCollection->IDatReaderWriter adapter existed as THREE
near-identical copies (App-internal original, Bake's copy, Content.
Tests' copy) — a structure where adapter drift is exactly what the
live-vs-pak equivalence suite cannot detect (both sides would only
drift together if they shared one implementation). Now ONE public
AcDream.Content.DatCollectionAdapter next to IDatReaderWriter (GL-free
home established in MP1a), carrying App's FULL behavior including the
[dat-miss] TryGet tripwire log (which now also covers the bake tool
and the equivalence suite) and the caching/locking. All three copies
deleted; WbMeshAdapter (App), BakeRunner (Bake), and
PakEquivalenceTests (Content.Tests) resolve the shared class.
Iteration properties return the REAL dat iterations — the App copy's
hardcoded 0 was a stub nothing read; the unification intentionally
keeps truth (noted in the doc comment). Verified post-move: no
Silk.NET anywhere in Content / Bake / Content.Tests / Bake.Tests
resolved dependency graphs.
(8) Two test gaps closed in PakRoundTripTests: (a) direct on-disk TOC
sortedness — blobs added in DESCENDING key order, then the raw file
bytes parsed (not through the reader) and every TOC entry asserted
strictly ascending; (b) corrupt-blob logging — five repeated reads
through both public paths (TryReadObjectMeshData + ContainsKey) with
stderr captured, asserting exactly ONE [pak-corrupt] line for the
victim key.
Full suite: 4120 tests, 0 failures (Content.Tests 56, Bake.Tests 1,
plus the pre-existing 4 skips).
Adversarially-verified review findings 1 and 10:
(1) The bake pipeline no longer accumulates every decoded ObjectMeshData
in one ConcurrentBag before writing (multi-GB OOM risk on the full
bake), and no longer writes blobs in thread-completion order (which
violated the plan's "bakes must be byte-reproducible run-to-run").
New shape in BakeRunner: build the FULL id list, sort by PakKey, chunk
into 512-id batches; Parallel.ForEach WITHIN each batch; sort each
batch's results by key and AddBlob sequentially; release the batch.
Batches are contiguous key ranges, so the blob region lands in global
key order regardless of thread scheduling, and peak memory is one
batch's output. Side-staged particle-preload meshes drain per batch
into a key-deduped map (first instance wins — per-id extraction output
is deterministic, so instance choice cannot affect bytes) and are
written after all batches, sorted by key, skipping keys already
written.
Program.cs is now a thin arg-parsing shell over the public BakeRunner
so the new dat-gated byte-reproducibility test can drive the REAL
pipeline: tests/AcDream.Bake.Tests (new project, rule 6; registered in
slnx; no Silk.NET in its resolved dependency graph — verified) bakes
the same 9-id mixed fixture twice with DIFFERENT thread counts (8 vs
3 — thread scheduling was the nondeterminism source) and asserts the
two pak files are byte-identical. Ran for real against the dats on
this machine: green.
(10) The isSetup argument for EnvCell extraction now matches at both
call sites (BakeRunner and PakEquivalenceTests both pass false) and is
documented at each: the runtime's own request sites
(WbMeshAdapter.IncrementRefCount / EnsureLoaded) pass isSetup: false
for every MeshRef id including cell-geometry ids. The parameter is
currently dead in MeshExtractor.PrepareMeshData (dispatch is on the
resolved dat type), but two disagreeing call sites were a latent trap.
Header FormatVersion is no longer set by the bake (PakWriter stamps it
per review finding 2, previous commit).
Adversarially-verified review findings 2,3,4,5,6,9:
(2) PakFormat.CurrentFormatVersion=1 constant; PakWriter stamps it
unconditionally (caller header templates can no longer produce a
version-0 pak — the default-0 footgun); PakReader refuses any other
version with a message naming found/expected. Tests: versions 0 and 2
both rejected; writer stamps even when the template omits the field.
(3) Reader robustness, all under the documented corrupt-=-missing
CONTRACT (external-file input — surface loudly once, then behave as
absent; never garbage, never throw from a lookup): (a) per-TOC-entry
bounds validation at open (offset/length outside [header, toc) or
past EOF -> logged once, entry missing, siblings unaffected); (b)
TryReadObjectMeshData catches deserialization failures (malformed
structure behind a matching CRC) -> logged once per entry, false; (c)
structurally unopenable files throw at OPEN with a clear message:
unfinalized header (tocOffset < header size — the placeholder-header
crash signature) and TOC-past-EOF truncation. Tests: corrupt TOC
entry, truncated pak, half-written pak, malformed-blob-behind-valid-
CRC (tamper + CRC recompute) — each verifying sibling blobs still read.
(4) Single-pass read: TryReadObjectMeshData now does ONE ReadArray out
of the map; CRC and deserialization run over the same buffer (was: a
separate VerifyCrc traversal + a second ReadArray + a ToArray copy
inside Serializer.Read). New Serializer.Read(byte[]) overload avoids
the defensive copy. Verdict set stays a ConcurrentDictionary (MP1c
calls this from 4 decode workers). True span-over-mmap zero-copy is
deferred to MP1c profiling per the class doc comment.
(5) PakWriter.Dispose restored to try/finally: Finish() does real I/O
and can throw (disk full) — the stream must ALWAYS close so no file
handle leaks mid-unwind. (Reverts the a5926ebc simplification, which
was wrong about this.) Test: dispose after an AddBlob exception leaves
the file deletable.
(6) CRC-32 known-answer vectors: "123456789" -> 0xCBF43926, empty ->
0x00000000, single zero byte -> 0xD202EF8D. The suite was previously
blind to a self-consistent-but-wrong CRC.
(9) Equality comparator floats switched to bit-equality
(SingleToInt32Bits/DoubleToInt64Bits, incl. Vector2/3 + Matrix4x4
components): for byte-identity round-trip purposes `==` was both too
strict (NaN==NaN false — a surviving NaN payload would wrongly FAIL)
and too lax (-0.0==+0.0 true — a sign-bit flip would wrongly PASS).
Content.Tests: 54/54 green (was 43).
PakEquivalenceTests runs MeshExtractor LIVE and bakes the SAME fixture
ids to a temp pak, reads it back via PakReader, and deep-compares every
field via the Task 3 ObjectMeshDataEquality comparator. Since the bake
tool and the live client drive the identical MeshExtractor code (MP1a),
this proves the pak ROUND-TRIP preserves what extraction actually
produces on real content — the serializer's job, not a re-verification
of the extraction algorithm itself (the existing Conformance suite owns
that).
Fixture set (>= the plan's minimums): 10 GfxObjs (3 known-tricky ids
reused from Issue119UpNullGfxObjDumpTests - 0x010002B4, 0x010008A8,
0x010014C3 - plus 7 more from dat order), 3 Setups reused from door/
tower conformance fixtures (0x020019FF, 0x020005D8, 0x020003F2), 5
EnvCells walked from the Holtburg landblock 0xA9B40000's
LandBlockInfo.NumCells range (same idiom as
StipplingSurfaceEquivalenceTests).
Skips cleanly when dats are absent via ContentConformanceDats.
ResolveDatDir(), a deliberate small duplicate of
ConformanceDats.ResolveDatDir()'s exact pattern (env var then
Documents/Asheron's Call fallback) since Content.Tests cannot reference
the AcDream.Core.Tests project. ContentTestDatCollectionAdapter is a
third small copy of the IDatReaderWriter dat-access glue (alongside
AcDream.App's internal original and AcDream.Bake's copy) for the same
layering reason (structure rule 6: tests live in the project matching
the layer under test).
Ran for real against the dats on this machine (not just CI-skip path):
1 test, all fixture ids extracted live with zero failures, baked,
read back, and field-for-field identical. Full solution dotnet test:
4106 tests total (43 Content.Tests + 385 Core.Net.Tests + 425
UI.Abstractions.Tests + 722 App.Tests incl. 2 pre-existing skips + 2531
Core.Tests incl. 2 pre-existing skips) — 0 failures.
TDD: PakRoundTripTests written first, confirmed a compile failure
against the not-yet-existing PakWriter/PakReader types.
PakWriter streams [header placeholder][64-byte-aligned blobs][TOC sorted
by key], then seeks back and finalizes the header once TocOffset/TocCount
are known (plan: "TOC last" so the writer doesn't need blob count
up front). PakReader mmaps the whole file once, loads the TOC into a
sorted array for O(log n) binary-search lookup, and verifies each blob's
CRC-32 LAZILY on first access (cached per-index so a corrupt blob logs
exactly once and is thereafter always treated as missing rather than
handing back garbage bytes). No locks anywhere — the mapped memory is
immutable for the reader's lifetime by construction.
CRC-32 implemented directly (standard IEEE 802.3 table-driven variant)
rather than adding a System.IO.Hashing package reference, keeping
AcDream.Content's dependency surface minimal per its existing
"no GL binaries" intent.
9 tests green: header round-trip, every-key-found, blob deep-equality
via the Task 3 comparator, missing-key returns false, corrupted-blob
(single flipped byte) is detected and treated as missing while leaving
sibling blobs unaffected, 64-byte blob alignment, and TOC binary search
cross-checked against a linear scan for both present and absent keys.
TDD: ObjectMeshDataSerializerTests + the shared ObjectMeshDataEquality
field-by-field comparator (reused by Task 6's equivalence suite) written
first, confirmed a compile failure against the not-yet-existing
ObjectMeshDataSerializer type.
Serializes EVERY field of the ObjectMeshData family per the plan's
normative layout: primitives raw LE, arrays as count:i32+payload,
blittable arrays (VertexPositionNormalTexture[], ushort[], byte[]) via
MemoryMarshal.AsBytes bulk copy, TextureBatches written sorted by the
(Width, Height, Format) key tuple for run-to-run determinism regardless
of dictionary insertion order, nullable fields as present:byte+value.
EnvCellGeometry nests recursively (MeshExtractor can populate one level
today; the serializer supports arbitrary depth rather than assuming it).
Namespace-trap finding: StagedEmitter.Emitter resolves to
DatReaderWriter.DBObjs.ParticleEmitter (the dat DBObj, verified via
reflection against the pinned Chorizite.DatReaderWriter 2.1.7 package
and confirmed live by MeshExtractor's `emitter.HwGfxObjId.DataId` call
site compiling), NOT AcDream.Core.Vfx.ParticleEmitter (the runtime
particle-simulation type with a live Particle[] pool that would NOT be
serializable asset data). All ~31 of its fields are written explicitly
rather than delegating to its own Pack/Unpack, which require a live
DatBinWriter/DatBinReader bound to a DatDatabase — coupling our pak's
determinism to a third-party wire-format helper we don't control the
versioning of.
33 tests green: 9 round-trip fixtures (empty/vertices+indices/multi
texture-batch-groups/setup-parts/emitters/nullable-present/nullable-
absent/edge-lines/nested-EnvCellGeometry), same-instance-twice byte-
identity, and dictionary-insertion-order-independence (two orders ->
identical bytes) plus a key-sort-order assertion on the raw bytes.
TDD: PakKeyTests + PakFormatTests written first (confirmed a compile
failure against the not-yet-existing AcDream.Content.Pak namespace),
then PakKey (64-bit type:u8|fileId:u32|reserved:u24 compose/decompose)
and PakFormat (64-byte PakHeader, 24-byte PakTocEntry) implemented to
the normative layout in the MP1b plan. 21 tests green, including a key-
ordering test proving ascending numeric key order equals ascending
(type, fileId) tuple order (the TOC binary-search precondition) and an
explicit byte-offset test for both structs.
New xunit test project for the Content layer (rule 6: tests live in the
project matching the layer under test). ProjectReference to
AcDream.Content; TreatWarningsAsErrors + LangVersion latest per MP1b's
csproj requirements. Registered in AcDream.slnx. Placeholder PakKeyTests
smoke test becomes the real Task 2 suite next.
