acdream/docs/research/2026-07-24-runtime-performance-audit.md

# Runtime CPU/GPU/memory performance audit

**Capture date:** 2026-07-24

**Benchmark commit:** `2c3da8e153123240a289417856d0f937887cdace`

**Status:** Complete, report-only audit. No runtime code, settings, or game
behavior were changed during the benchmark.

**Compact data:**

- [`2026-07-24-runtime-performance-audit-summary.json`](2026-07-24-runtime-performance-audit-summary.json)
- [`2026-07-24-runtime-performance-audit-checkpoints.csv`](2026-07-24-runtime-performance-audit-checkpoints.csv)

**Architecture response:**
[`../plans/2026-07-24-modern-runtime-architecture.md`](../plans/2026-07-24-modern-runtime-architecture.md)

---

## 1. Executive result

Steady capped play was inexpensive. Portal transitions were not.

The client maintained approximately 32 FPS on a 32 Hz Remote Desktop display.
Normal update work was approximately 0.4–1.3 ms median, and external GPU 3D use
was 7.6% median. The dominant performance failure was the burst of runtime
content preparation, publication, and retirement performed around portal
transitions:

- Up to approximately 204 MiB allocated in one frame.
- Approximately 2.22 GiB/s maximum managed allocation rate.
- 38 Gen-2 collections over the six-minute route.
- 256 ms maximum recorded GC pause.
- 276 ms maximum frame.

The final live managed heap was approximately 194 MiB even though the process
had previously reached 1.66 GiB working set and 2.27 GiB private bytes. Entity,
particle, effect, and tracked GPU counts returned to repeatable plateaus when
locations were revisited. The audit therefore did **not** reproduce an
unbounded logical-object or GPU-resource leak. It reproduced excessive
allocation and retained high-water memory.

Allocation stacks identify runtime mesh/texture preparation as the largest
portal source. Update-thread landblock publication and retirement then amplify
the burst with complete dictionary/set/sort work and unmetered per-entity
operations.

---

## 2. Test conditions

### Hardware and host

| Item | Value |
|---|---|
| CPU | AMD Ryzen 7 9800X3D, 8 cores / 16 logical processors, up to 4.7 GHz |
| GPU | AMD Radeon RX 9070 XT |
| GPU driver | `32.0.31021.5001` |
| Active display path | Microsoft Remote Display Adapter |
| Active resolution | 2056 × 1290 at 32 Hz |
| Physical display | 2560 × 1440 at 240 Hz |
| OS | Windows NT 10.0.26200.0 |
| Runtime | .NET 10 / SDK 10.0.300 |

### Client configuration

- Release build.
- Normal graphical client.
- Retail retained UI.
- Developer UI disabled.
- Normal capped presentation.
- `ACDREAM_FRAME_PROF=1` enabled for observational frame metrics.
- Local ACE server.
- Graceful client shutdown.

The RDP display and normal presentation cap prevent this capture from
establishing uncapped maximum FPS or maximum GPU throughput. Capped and uncapped
local-display results must remain separate in future comparisons.

### Route

The existing connected R6 route produced nine canonical checkpoints:

1. Caul baseline.
2. Sawato baseline.
3. Rynthid world edge.
4. Aerlinthe.
5. Sawato return.
6. Holtburg.
7. Caul return.
8. Sawato plateau.
9. Caul plateau.

The route includes portal transitions, location revisits, camera turns, input
exercises, stable dwell periods, checkpoint capture, and graceful disconnect.

---

## 3. Measurement methods

The primary process ran for approximately 359 seconds. Measurements combined:

- The permanent `[frame-prof]` ring profiler.
- `System.Runtime` counters through `dotnet-counters`.
- Windows per-process CPU, memory, handle, thread, I/O, and GPU-engine samples.
- A 30-second CPU-sampling EventPipe trace at Aerlinthe.
- A final GC heap dump.
- A short allocation-sampling trace covering startup and Caul.
- A short exception-stack trace covering a connected route segment.
- Canonical world lifecycle/resource checkpoint JSON.

The allocation trace reports sampled allocation intervals. A type attached to
an allocation tick is not proof that a single object of that type had the full
sampled byte count. Stack attribution is the reliable part of that capture.

The CPU trace was collected while counters were also attached. It is suitable
for ranking active code, not for absolute standalone CPU percentages.

---

## 4. Headline measurements

| Measurement | Result |
|---|---:|
| Stable capped rate | approximately 32 FPS / 31.2 ms wall frame |
| Steady update work | approximately 0.4–1.3 ms median |
| Normal update p95 | approximately 0.5–2.0 ms |
| CPU normalized p50 / p95 / max | 0.05% / 1.30% / 2.46% |
| CPU as one-core p50 / p95 / max | 0.8% / 20.8% / 39.4% |
| GPU 3D p50 / p95 / max | 7.6% / 15.9% / 16.8% |
| Working set range | 776.0–1,661.9 MiB |
| Private-byte range | 1,286.4–2,269.1 MiB |
| Dedicated GPU range | 632.5–818.0 MiB |
| Handle range | 680–704 |
| Thread range | 26–34 |
| Managed allocation average | approximately 40.79 MB/s |
| Managed allocation maximum | 2,386,818,304 B/s |
| Total sampled managed allocation | 14,643,958,472 B |
| Maximum single-frame allocation | approximately 203.6 MiB |
| Maximum frame | 276.4 ms |
| Worst five-second p95 / p99 | 52.6 ms / 147 ms |
| Maximum update p95 | 37.5 ms |
| Maximum upload p95 | 3.5 ms |
| Total GC pause time | approximately 2.07 s |
| Maximum one-second GC pause | 256 ms |
| Collections Gen0 / Gen1 / Gen2 | 99 / 105 / 38 |
| Lock contentions | 4,614 total; 952/s maximum |
| Thread-pool queue | 0 throughout capture |

Steady per-frame allocation varied by scene:

- Sawato: approximately 27–28 KiB/frame.
- Caul: approximately 42–44 KiB/frame.
- Holtburg: approximately 53.5 KiB/frame outside the portal burst.

Portal windows produced maximum frames of approximately 203.6 MiB, 141.6 MiB,
141.5 MiB, 58.5 MiB, and 35.1 MiB.

---

## 5. Checkpoint and leak analysis

The complete sanitized checkpoint table is in the companion CSV. Representative
revisit comparisons:

| Location | Visit | Landblocks | Entities | Emitters | Particles | Tracked GPU MiB |
|---|---|---:|---:|---:|---:|---:|
| Caul | baseline | 625 | 21,031 | 1,712 | 4,562 | 275.3 |
| Caul | return | 625 | 21,031 | 1,714 | 4,530 | 439.2 |
| Caul | plateau | 625 | 21,025 | 1,720 | 4,502 | 430.7 |
| Sawato | baseline | 625 | 6,659 | 586 | 3,021 | 347.6 |
| Sawato | return | 625 | 6,659 | 563 | 2,666 | 345.6 |
| Sawato | plateau | 625 | 6,667 | 581 | 2,808 | 455.2 |

Entity, emitter, and particle populations converged. GPU residence warmed and
then plateaued rather than increasing without bound. The different Sawato GPU
plateau reflects the process-wide cache working set accumulated through the
complete route, not an increasing Sawato entity population.

All three Caul materializations recorded `viewport-before-ready` invariant
failures. This is separate evidence that destination work and reveal are not
currently one atomic generation-ready transaction.

### Final managed heap

- Live heap: 203,771,023 B, approximately 194.3 MiB.
- Live objects: 1,314,128.
- Last observed committed managed memory: approximately 830.6 MiB.

Large retained categories included:

| Category | Approximate live size |
|---|---:|
| `System.Byte[]` | 27.5 MiB |
| `DeferredAlphaInstance[]` | 5.0 MiB |
| `Matrix4x4[]` | 4.6 MiB |
| `Int32[]` | 4.2 MiB |
| `CachedBatch[]` | 4.2 MiB |
| `Single[]` | 4.0 MiB |
| `WorldEntity` objects | 3.9 MiB |
| `InstanceLightSet[]` | 3.5 MiB |
| Fixed `ParticleInstance[]` | 3.5 MiB |

There were approximately 20,000 records in several static presentation
registries, including pose, script activation, static owner state, and effect
profile tables. Individually they are modest, but together they add portal
create/teardown work and several MiB of retained state.

The gap between 194 MiB live heap and much larger committed/private memory is
primarily GC segment high-water, fragmentation, native/driver allocations,
mapped/cached data, and GPU resource residence.

---

## 6. CPU attribution

Most CPU samples represented waits and presentation pacing:

| Stack/category | Inclusive share of all samples |
|---|---:|
| `ManualResetEventSlim.Wait` | 28.79% |
| `WaitHandle.WaitOneNoCheck` | 21.68% |
| `LowLevelLifoSemaphore.WaitForSignal` | 20.48% |
| `Socket.ReceiveFrom` | 7.23% |
| `FramePacingController.CompleteFrame` | 6.44% |

Named active App work was comparatively small in this capped scene:

| App path | Inclusive share of all samples |
|---|---:|
| `RenderFrameOrchestrator` | 0.60% |
| `WorldSceneRenderer` | 0.48% |
| `RetailPViewRenderer.DrawInside` | 0.44% |
| `DrawLandscapeThroughOutsideView` | 0.33% |
| `DrawLandscapeSlice` | 0.23% |
| `DatDatabaseWrapper.TryGet` | 0.21% |
| `WbDrawDispatcher.Draw` | 0.19% |
| `UpdateFrameOrchestrator` | 0.15% |
| `MeshExtractor.PrepareMeshData` | 0.13% |

The largest named exclusive App sites were `InteriorEntityPartition.Partition`,
`WbDrawDispatcher.Draw`, terrain clip AABB testing, PView outside work, and
deferred-alpha preparation. These are small under the 32 Hz cap but scale with
uncapped frame rate.

There was no evidence of a spinning thread-pool worker or a growing thread-pool
queue.

---

## 7. GPU analysis and timing limitation

External GPU 3D utilization was low in this capture, so the client was not
GPU-saturated. Tracked GPU resources warmed from approximately 275 MiB to
approximately 431 MiB, with OS-reported dedicated memory reaching 818 MiB.
Approximately 599 tracked textures and 43–48 atlas arrays were resident near the
end of the route.

The difference between tracked and OS-reported GPU memory can include driver
allocation, texture-array slack, alignment, metadata, framebuffer resources,
and resources not covered by acdream's tracker.

The internal `gpu_ms` value is not valid active-render time under a cap.
`GpuFrameTimer.FrameBoundary` starts at one frame boundary and completes at the
next, so the GPU query includes idle/pacing gaps. Its 23–30 ms result must not be
interpreted as GPU saturation. Correct pass/submission timestamp measurement is
Slice A of the modern runtime plan.

---

## 8. Allocation and exception attribution

### Runtime mesh and texture preparation

The largest sampled allocation stacks were:

```text
AcDream.Content.MeshExtractor.AddSurfaceToBatch
  -> PrepareGfxObjMeshData
  -> PrepareMeshData
  -> AcDream.App.Rendering.Wb.ObjectMeshManager.ProcessQueue
```

The path decodes compressed and palette textures into RGBA arrays, clones cached
pixels for surface-local alpha changes, constructs vertex/index collections,
and produces upload batches. Rebuilding those products during portals creates
large short-lived graphs even when the same world content has been seen before.

Relevant implementation:
[`../src/AcDream.Content/MeshExtractor.cs`](../../src/AcDream.Content/MeshExtractor.cs).

### Invalid Setup probes

The focused exception trace captured 1,587 exceptions with no lost events:

- 1,335 `ArgumentOutOfRangeException`.
- 252 `SocketException`.

The DAT exceptions followed:

```text
QualifiedDataId<T>.Unpack
  -> Setup.Unpack
  -> DatDatabase.TryGet
  -> RuntimeDatCollection.Get<Setup>
  -> LivePresentationComposition.ResolveActivation
  -> EntityScriptActivator.OnCreate
  -> GpuWorldState.ActivateLandblockPresentation
  -> StreamingController.ApplyResult
```

`ResolveActivation` attempts `Get<Setup>` for every static source ID and catches
all exceptions. Many IDs are GfxObj IDs, not Setup IDs. The mesh path already
contains the required DID-prefix validation.

Relevant implementation:
[`../src/AcDream.App/Composition/LivePresentationComposition.cs`](../../src/AcDream.App/Composition/LivePresentationComposition.cs).

### Network receive timeout

`NetClient` configures a receive timeout and catches `SocketError.TimedOut` as a
normal heartbeat every approximately 250 ms. Every send also materializes
`datagram.ToArray()`.

Relevant implementation:
[`../src/AcDream.Core.Net/NetClient.cs`](../../src/AcDream.Core.Net/NetClient.cs).

---

## 9. Ranked findings

### P0 — Portal content preparation and publication storm

**Sources**

- `MeshExtractor` runtime decode and texture cloning.
- `StreamingController.DrainAndApply`.
- `LandblockRetirementCoordinator.RunForEachEntity`.
- Landblock render, physics, static presentation, effect, light, and plugin
  publication.

**Impact**

- 204 MiB maximum single-frame allocation.
- 276 ms maximum frame.
- Gen-2 GC and committed-heap high-water.
- Destination readiness failures.

**Recommendation**

- Finish the deduplicated baked pak and consume prepared data at runtime.
- Keep visibility/collision detachment immediate.
- Budget expensive publication/teardown by time, bytes, and entity operations.
- Prepare the destination across portal frames and reveal one ready generation.

**Plan mapping:** Slices B, C, and E.

### P1 — Exception-driven DAT type resolution

**Source**

`LivePresentationComposition.ResolveActivation` parses arbitrary source IDs as
Setup and catches failure.

**Impact**

Thousands of constructed/thrown exceptions during streaming, avoidable DAT
parsing, noisy diagnostics, and main-thread activation work.

**Recommendation**

Use explicit DID type metadata, non-allocating typed lookup, negative caching,
and prepared Setup presentation metadata. Exceptions must represent corruption
or failure, not ordinary type selection.

**Plan mapping:** Slice C.

### P1 — Complete collection rebuilds during static publication

**Source**

`LandblockStaticPresentationPublisher` creates entity/snapshot dictionaries,
copies previous keys to a set, and sorts IDs/snapshots per publication.

**Impact**

Portal allocation and update-thread CPU scale with complete landblock contents.

**Recommendation**

Carry canonical ordered prepared arrays, use cursor-based publication receipts,
and publish versioned deltas rather than reconstructing complete collections.

**Plan mapping:** Slices C, E, and F.

### P1 — DAT resolution/cache churn

**Sources**

- `DatCollectionAdapter.ResolveId`.
- `MeshExtractor.ResolveId().ToList().OrderByDescending().FirstOrDefault()`.
- Small bounded typed-object cache plus serialized database access.

**Impact**

Repeated parsing on revisits, allocation from resolution candidates/LINQ, 4,614
lock contentions, and repeated invalid type probes.

**Recommendation**

Use typed non-allocating resolution, prepared compact metadata, negative/type
caches, and remove world render/streaming DAT reads after the pak cutover.

**Plan mapping:** Slices B and C.

### P2 — Whole-world render partition every frame

**Source**

`InteriorEntityPartition.Partition` scans every entity in every loaded
landblock each render frame. Caul had approximately 21,000 entities.

**Impact**

CPU cost grows with the complete resident world rather than visible cells and
changes. Reused lists reduce allocation but not traversal.

**Recommendation**

Maintain outdoor-static, per-cell, dynamic, translucent, selectable, and light
indices incrementally in a render-only scene database. Existing retail PView
remains the visible-cell authority.

**Plan mapping:** Slices F and G.

### P2 — Unconditional diagnostic and retained-UI work

**Sources**

- `WorldSceneDiagnosticsController` performs every-landblock AABB tests without
  an active debug consumer.
- `GpuWorldState.LandblockBounds` uses an iterator.
- Retained UI reapplies anchors and walks the overlay tree every frame.
- Item cooldown UI scans complete item lists.
- Dialog handling snapshots open-order arrays.

**Impact**

Moderate capped cost, but it scales directly with uncapped FPS and adds small
steady allocations.

**Recommendation**

Reuse renderer facts, skip inactive diagnostics, add UI dirty-layout/event
registries, and replace frame iterators/snapshots with borrowed stable views.

**Plan mapping:** Slice H.

### P2 — High but bounded GPU/cache residency

**Sources**

- Object mesh and texture caches.
- Standalone and composite texture arrays.
- Fixed 65,536-instance particle CPU/GPU buffer.
- Deferred-alpha scratch arrays that grow to scene high-water.

**Impact**

633–818 MiB dedicated GPU memory during the route, plus driver/native
difference from tracked resources.

**Recommendation**

Introduce typed asset leases and unified accounting, measure atlas occupancy and
fragmentation, use generation-safe cost-aware eviction, and shrink or segment
large scratch buffers only with hysteresis.

Do not reduce view distance or particle range as the primary correction.

**Plan mapping:** Slice D.

### P3 — Smaller continuous allocations and background overhead

**Sources**

- Per-frame `RetailPViewFrameInput`.
- Yield iterators and periodic liveness sets.
- Full candidate light sorting.
- UDP send copies.
- Socket timeout exceptions.

**Impact**

Not responsible for the portal storm, but measurable at high uncapped FPS or
across many headless sessions.

**Recommendation**

Use reusable frame views, stable spans, spatial light candidates, span-based
socket sends, and cancellable blocking/async receive without exception-driven
timeouts.

**Plan mapping:** Slices H and J.

---

## 10. Recommended execution order

The audit supports this order:

1. Correct GPU/pass measurement and checkpoint accounting.
2. Complete EnvCell dedup and the full pak bake.
3. Cut world mesh/texture/Setup presentation preparation over to prepared data.
4. Add explicit CPU/staging/GPU residency ownership and accounting.
5. Replace portal budget bypasses with staged cost budgets and one reveal
   generation.
6. Add the incremental render scene and delta submission.
7. Remove remaining per-frame UI/diagnostic/network allocation.
8. Extract the presentation-independent runtime for Linux/headless use.

The complete slice definitions and acceptance gates are in the linked modern
runtime architecture plan.

---

## 11. Reproduction

### Canonical route

```powershell
dotnet build AcDream.slnx -c Release

$env:ACDREAM_TEST_USER = '<account>'
$env:ACDREAM_TEST_PASS = '<password>'
$env:ACDREAM_FRAME_PROF = '1'

.\tools\run-connected-r6-soak.ps1 -Repository (Get-Location).Path -SkipBuild
```

Do not place credentials in committed logs or reports.

### Runtime counters

After identifying the client PID:

```powershell
dotnet-counters collect `
  --process-id <pid> `
  --counters System.Runtime `
  --format csv `
  --output dotnet-counters.csv
```

### CPU trace

```powershell
dotnet-trace collect `
  --process-id <pid> `
  --profile cpu-sampling `
  --duration 00:00:30 `
  --output cpu.nettrace
```

### Final live heap

```powershell
dotnet-gcdump collect --process-id <pid> --output final.gcdump
```

For comparable results:

- Use the same commit, Release configuration, route, and camera/input script.
- Record RDP/capped and local/uncapped modes separately.
- Gracefully close the client.
- Capture checkpoint JSON before interpreting process memory.
- Compare live heap, GC committed memory, tracked GPU bytes, and OS GPU memory
  separately.

---

## 12. Artifact retention policy

The local raw capture contained 96 files totaling approximately 157.9 MiB.
Heavy and environment-specific artifacts were deliberately excluded from Git:

- `.nettrace` and generated `.etlx`.
- `.gcdump`.
- Full `dotnet-counters.csv`.
- Complete process/GPU sample CSV.
- Full client stdout/stderr.
- Generated trace-reader project binaries.
- Process IDs, timestamps, absolute local paths, and session-specific logs.

Those files are valuable for local forensic work but poor Git artifacts: they
are binary or noisy, inflate permanent history, are difficult for code-review
tools to inspect, and can expose local/session metadata.

This report plus the compact JSON and checkpoint CSV preserve the benchmark's
method, provenance, measurements, interpretation, and decision-driving evidence
in a form another engineer or model can review directly.