diff --git a/docs/plans/2026-04-11-roadmap.md b/docs/plans/2026-04-11-roadmap.md index 25e48014..17f40b74 100644 --- a/docs/plans/2026-04-11-roadmap.md +++ b/docs/plans/2026-04-11-roadmap.md @@ -1554,16 +1554,18 @@ port in any phase — no separate listing here. ### Track MP — Modern Pipeline (performance side track; user-commissioned 2026-07-05) -> **⏸ PARKED 2026-07-05 (user decision) — WINS BANKED, REMAINDER DEFERRED TO THE FUTURE.** +> **PREPARED-CONTENT WORK RESUMED 2026-07-24; ECS REMAINS PARKED.** > The track's actual trigger — the FPS/ms counter *fluctuating wildly* (read by the > user as inefficiency) — is **RESOLVED**: MP0 (frame profiler), MP1a (Content > extraction), and **MP-Alloc safe batch** shipped and merged to local main -> (`093cdb6d`, suite 4188 green, user-confirmed steady FPS). **ECS (MP3) + the -> rest are DEFERRED**: ECS is the *throughput* lever (raise avg FPS ~230→300), a +> (`093cdb6d`, suite 4188 green, user-confirmed steady FPS). **ECS (MP3) +> remains DEFERRED**: ECS is the *throughput* lever (raise avg FPS ~230→300), a > DIFFERENT goal from steadiness — revisit ONLY if the user wants a higher raw FPS > number and accepts the rewrite risk. Rust rejected. Parked (resume from the table -> below): MP1b pak/bake (needs the 865 GB EnvCell-dedup slice; loading-speed lever), -> MP-Alloc hard sites, MP1c, MP2. Full rationale: `project_mp_track_findings.md`. +> below): MP1b pak/bake is now complete; MP1c remains gated on corrected +> physical-local baselines. The remaining parked items are +> MP-Alloc hard sites and MP1c. MP2 has shipped. Full rationale: +> `project_mp_track_findings.md`. > The 2026-07-18 portal/resource-lifetime repair does not resume MP's ECS, pak, > or throughput redesign. It is corrective ownership/reclamation work for a > reproduced crash-class regression in the mandatory renderer, using the current @@ -1584,8 +1586,9 @@ port in any phase — no separate listing here. > RDP captures have landed. The committed evidence includes capped/uncapped > nine-stop routes, dense Arwic, portal windows, observer-effect measurement, > binary provenance, and contention stacks. The physical-local AMD-display gate -> remains open and blocks Slice C. Slice B EnvCell dedup is active under -> [`2026-07-24-modern-runtime-slice-b-envcell-dedup.md`](2026-07-24-modern-runtime-slice-b-envcell-dedup.md). +> remains open and blocks Slice C. Slice B EnvCell dedup is complete; its +> [full-bake report](../research/2026-07-24-slice-b-full-bake-report.md) +> records the 42.6× result and corrected geometry-identity collision. **Spec:** `docs/superpowers/specs/2026-07-05-modern-pipeline-design.md` (the umbrella design — read it first). **Goal:** smoothness first (no frame over @@ -1607,10 +1610,10 @@ hitch and follows. |---|---|---| | MP0 | Honest frame profiler (`[frame-prof]`) + baseline capture | ✅ SHIPPED 2026-07-05 — gate PASSED (`docs/research/2026-07-05-mp0-baseline.md`); headline: town spikes = GC churn (1.5–3 MB/frame), teleport hitch 211 ms | | MP1a | `AcDream.Content` extraction (GL-free MeshExtractor + boundary records out of App) | ✅ SHIPPED 2026-07-05 — user-gated (renders identical, zero tripwires, perf-neutral); 8 commits `651d041e`..`b0758d77` | -| MP1b | Pak format + `acdream-bake` CLI + mmap zero-copy `PakReader` + equivalence test | 🔵 **ACTIVE 2026-07-24** — implementing content-identity EnvCell dedup, shared TOC aliases, atomic output, and a new full-bake gate; detailed plan: [`2026-07-24-modern-runtime-slice-b-envcell-dedup.md`](2026-07-24-modern-runtime-slice-b-envcell-dedup.md) | +| MP1b | Pak format + `acdream-bake` CLI + mmap `PakReader` + equivalence/full-scale gate | ✅ **SHIPPED 2026-07-24** — 729,888 EnvCell keys → 17,117 unique geometries + 712,771 aliases (42.6×), 751,141 total keys / 38,370 physical blobs, zero failures, 28,192.4 MiB, 81.4 s validated atomic publish; full-DAT gate also corrected a real collision in WB's legacy geometry hash. [Report](../research/2026-07-24-slice-b-full-bake-report.md) | | **MP-Alloc (safe batch)** | Reuse per-frame buffers: anim pose, particle draw-list, interior partition, animatedIds/drawableCells sets | ✅ SHIPPED + USER-GATED 2026-07-05 — dense-town frame-time spikes 20–87ms → 6–10ms, alloc spikes (30–75MB single-frame) eliminated, gen2 GC 5–11/window → ~0; user confirms FPS steady. 4 commits `b8c05e2b`..`91afea24` | | MP-Alloc (hard sites) | EnvCell settled-camera rebuild gate + physics `Transition` pooling — the residual steady ~1.6MB/frame | ⚪ OPTIONAL follow-up — would lower the median too, but the wild-swing complaint is already resolved; each needs its own careful gate (batch correctness / physics faithfulness) | -| MP1c | Streaming cutover to pak + hitch gate (vs the 211 ms baseline) | ⚪ after MP-Alloc | +| MP1c | Streaming cutover to pak + hitch gate | ⏸ blocked on the physical-local corrected Slice A baseline; Slice B package foundation is complete | | MP2 | Retail particle distance/cell-view degradation: DAT-authored range, exact infinite/finite off-view update branches, emitter-first render scan, user-requested 2× default range | ✅ SHIPPED 2026-07-17 — Aerlinthe hotspot `0x32000223` resolves to retail 64m and defaults to 128m; landscape/entity streaming distance unchanged | | MP3 | Arch ECS render world + delta submission (the 300-FPS lever) | ⚪ — note: does NOT fix the steady-state GC churn (that's MP-Alloc); MP3 is the draw-submission throughput lever | | MP4 | Zero-alloc frame loop + flat physics data (residual, post-MP-Alloc) | ⚪ hard-queued behind M1.5 #137 | diff --git a/docs/plans/2026-07-24-modern-runtime-architecture.md b/docs/plans/2026-07-24-modern-runtime-architecture.md index acddb516..ffda8e8e 100644 --- a/docs/plans/2026-07-24-modern-runtime-architecture.md +++ b/docs/plans/2026-07-24-modern-runtime-architecture.md @@ -2,8 +2,8 @@ **Date:** 2026-07-24 -**Status:** Slice A tooling and diagnostic RDP baselines landed; Slice B active. -The physical-local Slice A gate still blocks Slice C. +**Status:** Slice A tooling and diagnostic RDP baselines landed; Slice B +complete. The physical-local Slice A gate still blocks Slice C. **Scope:** Reconcile and sequence the existing Modern Pipeline (`MP`) and Linux/headless (`LH`) tracks using the 2026-07-24 connected performance audit. @@ -578,7 +578,7 @@ effect, portal windows, and contention stacks are committed in [`../research/2026-07-24-slice-a-rdp-baselines.md`](../research/2026-07-24-slice-a-rdp-baselines.md) and its machine-readable companion. They prove the tooling is usable but are diagnostic-only. The physical-local AMD-display repetitions remain required, -so Slice C is still blocked. Slice B is baseline-independent and is now active. +so Slice C is still blocked. Slice B was baseline-independent and is complete. ### Slice B — Finish MP1b EnvCell dedup and full bake @@ -599,6 +599,13 @@ so Slice C is still blocked. Slice B is baseline-independent and is now active. sample equivalence tests pass; duplicate EnvCells share blob offsets; no runtime code consumes the pak yet. +**Completed 2026-07-24:** 729,888 EnvCell keys collapsed to 17,117 unique +geometries plus 712,771 aliases (42.6×), with 751,141 total keys, 38,370 +physical blobs, zero failures, a 28,192.4 MiB artifact, and an 81.4-second +validated atomic publication. The gate also exposed and corrected a real +collision in WorldBuilder's legacy 31× geometry hash. Evidence: +[`../research/2026-07-24-slice-b-full-bake-report.md`](../research/2026-07-24-slice-b-full-bake-report.md). + ### Slice C — Runtime prepared-asset source and MP1c cutover **Purpose:** Remove mesh/texture decode from portal frames. @@ -1116,12 +1123,11 @@ an improvement" carries the same evidence burden as a regression claim. ## 14. Current execution point -Slice A's tooling and diagnostic RDP evidence are complete. Its physical-local -exit gate remains open. Execute **Slice B** now: it is baseline-independent, -completes an implementation already present in the tree, and directly attacks -the benchmark's largest process-wide allocation source. Do not start Slice C -until the capped/uncapped nine-stop and pinned dense-Arwic routes have been -repeated on the physical local AMD display path and committed. +Slice A's tooling and diagnostic RDP evidence are complete, and Slice B's +prepared-content bake is complete. The next implementation is Slice C, but its +physical-local prerequisite remains open. Do not start Slice C until the +capped/uncapped nine-stop and pinned dense-Arwic routes have been repeated on +the physical local AMD display path and committed. The intended order is therefore: diff --git a/docs/plans/2026-07-24-modern-runtime-slice-b-envcell-dedup.md b/docs/plans/2026-07-24-modern-runtime-slice-b-envcell-dedup.md index 04b2f1f3..b454967d 100644 --- a/docs/plans/2026-07-24-modern-runtime-slice-b-envcell-dedup.md +++ b/docs/plans/2026-07-24-modern-runtime-slice-b-envcell-dedup.md @@ -1,7 +1,7 @@ # Modern Runtime Slice B — EnvCell geometry deduplication and full bake **Date:** 2026-07-24 -**Status:** Planned; implementation follows the Slice A diagnostic capture +**Status:** Complete 2026-07-24; full installed-DAT gate passed **Parent:** [`2026-07-24-modern-runtime-architecture.md`](2026-07-24-modern-runtime-architecture.md), Slice B **Prior implementation:** [`../superpowers/plans/2026-07-05-mp1b-pak-and-bake.md`](../superpowers/plans/2026-07-05-mp1b-pak-and-bake.md) @@ -9,7 +9,7 @@ Complete the existing MP1b bake without changing runtime behavior. Every EnvCell file ID remains an independently addressable pak key, while cells with -the same WorldBuilder geometry identity share one serialized blob and one +the same collision-resistant geometry identity share one serialized blob and one physical byte range. The client does not consume the pak in this slice. The full gate must replace the failed 865 GB / 52 minute per-cell bake with a @@ -136,6 +136,21 @@ as a live unique-geometry extraction. equivalence and determinism gates are green, duplicate EnvCells demonstrably share physical offsets, and no runtime code consumes the pak. +## Completion record + +All checkpoints B1-B5 are complete. The full installed-DAT bake produced +751,141 addressable keys backed by 38,370 physical blobs with zero failures. +The 729,888 EnvCell keys resolve to 17,117 unique geometries and 712,771 aliases +(42.6×). The validated package is 28,192.4 MiB and publishes atomically in +81.4 seconds on the reference machine. A cross-landblock eight-key alias fixture +is byte-identical with one versus eight workers. Full evidence: +[`../research/2026-07-24-slice-b-full-bake-report.md`](../research/2026-07-24-slice-b-full-bake-report.md). + +No runtime code reads the pak in this slice. Slice C remains blocked by the +physical-local Slice A baseline gate. Release build and the complete solution +suite pass (8,059 passed, five skipped, zero failed); generated pak artifacts +were deleted after validation. + ## Verification Run: diff --git a/docs/research/2026-07-24-slice-b-full-bake-report.json b/docs/research/2026-07-24-slice-b-full-bake-report.json new file mode 100644 index 00000000..6dc43b40 --- /dev/null +++ b/docs/research/2026-07-24-slice-b-full-bake-report.json @@ -0,0 +1,76 @@ +{ + "schemaVersion": 1, + "capturedUtcDate": "2026-07-24", + "binaryCommit": "90bf6bbf455654dd1cfe1749b19bfd36b769939c", + "pakFormatVersion": 1, + "bakeToolVersion": 2, + "datIterations": { + "portal": 2072, + "cell": 982, + "highRes": 497, + "language": 994 + }, + "threads": 8, + "enumerated": { + "gfxObj": 15318, + "setup": 5935, + "envCell": 729888 + }, + "output": { + "gfxObjKeys": 15318, + "setupKeys": 5935, + "envCellKeys": 729888, + "uniqueEnvCellGeometries": 17117, + "envCellAliases": 712771, + "envCellDedupRatio": 42.64, + "sideStagedNewKeys": 0, + "sideStagedDuplicateKeys": 13284, + "physicalBlobs": 38370, + "totalKeys": 751141, + "failures": 0, + "bytes": 29561864760, + "mibibytes": 28192.4 + }, + "timing": { + "extractAndWriteSeconds": 81.2, + "validatedAndPublishedSeconds": 81.4 + }, + "memory": { + "peakWorkingSetMibibytes": 5073.7, + "peakPrivateMibibytes": 4507.3, + "note": "Offline bake-process peak while constructing and serializing 28.2 GiB of immutable payloads; this is not client runtime residency." + }, + "validation": { + "headerAndDatIterations": "pass", + "tocCountOrderAndEveryRange": "pass", + "atomicOverwrite": "pass", + "temporaryFilesRemaining": 0, + "crossLandblockAliasSample": { + "sharedOffset": 16170873792, + "aliasCountAtReceipt": 25, + "sampleFileIds": [ + "0x00A90230", + "0x00B901A8", + "0x00E50602", + "0x00E50610", + "0x00E5061F", + "0x00F90784", + "0x00F9078E", + "0x00FA073E" + ] + }, + "filteredThreadCountDeterminism": { + "threads": [1, 8], + "envCellKeys": 8, + "physicalBlobs": 1, + "sha256": "C8187761ED30CE9412C9081F52067159AECA5B02EB316E73DA29DC88D4339642" + }, + "releaseBuild": "pass-with-17-pre-existing-test-warnings", + "completeReleaseSuite": { + "passed": 8059, + "skipped": 5, + "failed": 0 + } + }, + "artifactPolicy": "The generated pak is an untracked machine artifact and is deleted after final verification. This compact path-free report is the committed evidence." +} diff --git a/docs/research/2026-07-24-slice-b-full-bake-report.md b/docs/research/2026-07-24-slice-b-full-bake-report.md new file mode 100644 index 00000000..1c41c7bb --- /dev/null +++ b/docs/research/2026-07-24-slice-b-full-bake-report.md @@ -0,0 +1,76 @@ +# Slice B full-bake report + +**Date:** 2026-07-24 +**Measured binary:** `90bf6bbf455654dd1cfe1749b19bfd36b769939c` +**Result:** pass, zero extraction failures + +Slice B replaces the failed 865 GB per-cell bake with a deterministic, +content-identity package. The client does not consume this package yet; this is +the offline prepared-content foundation for Slice C. + +## Full installed-DAT result + +| Measure | Result | +|---|---:| +| GfxObj keys | 15,318 | +| Setup keys | 5,935 | +| EnvCell keys | 729,888 | +| Unique EnvCell geometries | 17,117 | +| EnvCell aliases | 712,771 | +| EnvCell deduplication | 42.6× | +| Physical blobs | 38,370 | +| Total TOC keys | 751,141 | +| Failures | 0 | +| Extract + write | 81.2 seconds | +| Validate + atomic publish total | 81.4 seconds | +| Artifact size | 29,561,864,760 bytes (28,192.4 MiB) | +| Peak working set | 5,073.7 MiB | +| Peak private bytes | 4,507.3 MiB | + +The memory figures belong to the short-lived offline bake process while it +constructs and serializes 28.2 GiB of immutable payloads. They are not expected +client residency: the later runtime source memory-maps the package and +deserializes only demanded assets. Sixteen-result extraction batches and +immediate rejection of already-written side-staged meshes keep live bake graphs +bounded; 13,284 redundant particle-preload results were discarded without +retention. + +The header carries format version 1, bake-tool version 2, and source DAT +iterations portal 2072, cell 982, high-res 497, language 994. Validation opened +the completed temporary package, checked those values, strict TOC ordering, +all 751,141 physical ranges, and then published with a same-volume atomic +overwrite move. No temporary artifact remained. + +## Identity correction discovered by the gate + +The first guarded full catalog stopped on a real collision in WorldBuilder’s +31× polynomial: + +- Cell `0x00030175`: environment `0x277`, structure 0, surfaces + `0x013B, 0x0034`. +- Cell `0x01BC0105`: environment `0x276`, structure 0, surfaces + `0x04FC, 0x0034`. +- Both legacy identities: `0x00000002020E8C13`. + +Those cells have different vertices and polygons. Silently aliasing them would +render the wrong room. Runtime and bake now share a namespaced FNV-1a identity +over the complete typed tuple, while the catalog still compares full tuples and +fails loudly on any future collision. The legacy calculation remains as a +conformance test proving this exact installed-DAT case. + +## Alias and determinism gates + +A sampled physical receipt at offset `16170873792` is shared by 25 EnvCell keys +across multiple landblocks. Eight of those keys were baked independently with +one and eight workers. Each output contained eight keys backed by one physical +blob, and both files were byte-identical: + +`C8187761ED30CE9412C9081F52067159AECA5B02EB316E73DA29DC88D4339642` + +The generated full pak remains an untracked machine artifact and is deleted +after final verification. The machine-readable, path-free evidence is +[`2026-07-24-slice-b-full-bake-report.json`](2026-07-24-slice-b-full-bake-report.json). + +Release build passed with 17 pre-existing test-source warnings. The complete +Release suite passed 8,059 tests with five skips and zero failures. The generated +full and filtered pak artifacts were then deleted as required.