docs(architecture): consolidate modern runtime roadmap
Unify the unfinished Modern Pipeline work with Linux/headless goals around measured content, residency, streaming, render-scene, and runtime boundaries. Record twelve gated slices and preserve retail-shaped simulation plus the mandatory N.5 renderer. Co-authored-by: Codex <codex@openai.com>
This commit is contained in:
Erik
parent
2c3da8e153
commit
944ee55584
2 changed files with 808 additions and 0 deletions
|
|
@ -1568,6 +1568,17 @@ port in any phase — no separate listing here.
|
||||||
> or throughput redesign. It is corrective ownership/reclamation work for a
|
> or throughput redesign. It is corrective ownership/reclamation work for a
|
||||||
> reproduced crash-class regression in the mandatory renderer, using the current
|
> reproduced crash-class regression in the mandatory renderer, using the current
|
||||||
> architecture and preserving pixels/ranges.
|
> architecture and preserving pixels/ranges.
|
||||||
|
>
|
||||||
|
> **2026-07-24 planning reconciliation:** a new connected nine-stop CPU/GPU/
|
||||||
|
> memory audit reproduced a larger portal allocation/GC storm (approximately
|
||||||
|
> 204 MiB in one frame, 276 ms maximum frame, 38 Gen-2 collections) while
|
||||||
|
> confirming stable entity/effect/GPU plateaus and low steady GPU utilization.
|
||||||
|
> The user requested a detailed modern architecture plan. The resulting
|
||||||
|
> [`2026-07-24-modern-runtime-architecture.md`](2026-07-24-modern-runtime-architecture.md)
|
||||||
|
> unifies the unfinished MP pak/render-world work with future Track LH, preserves
|
||||||
|
> retail-shaped simulation and the N.5 renderer, and reorders execution around
|
||||||
|
> prepared content and cost-budgeted streaming before render-ECS throughput.
|
||||||
|
> This records the plan only; implementation remains not started.
|
||||||
|
|
||||||
**Spec:** `docs/superpowers/specs/2026-07-05-modern-pipeline-design.md` (the
|
**Spec:** `docs/superpowers/specs/2026-07-05-modern-pipeline-design.md` (the
|
||||||
umbrella design — read it first). **Goal:** smoothness first (no frame over
|
umbrella design — read it first). **Goal:** smoothness first (no frame over
|
||||||
|
|
|
||||||
797
docs/plans/2026-07-24-modern-runtime-architecture.md
Normal file
797
docs/plans/2026-07-24-modern-runtime-architecture.md
Normal file
|
|
@ -0,0 +1,797 @@
|
||||||
|
# Modern Runtime Architecture — detailed execution plan
|
||||||
|
|
||||||
|
**Date:** 2026-07-24
|
||||||
|
|
||||||
|
**Status:** Planning complete; implementation not started
|
||||||
|
|
||||||
|
**Scope:** Reconcile and sequence the existing Modern Pipeline (`MP`) and
|
||||||
|
Linux/headless (`LH`) tracks using the 2026-07-24 connected performance audit.
|
||||||
|
|
||||||
|
**Primary objective:** Minimize CPU, managed/native memory, GPU memory,
|
||||||
|
allocation, and frame-time variance without reducing view distance, particles,
|
||||||
|
world detail, or retail behavior.
|
||||||
|
|
||||||
|
This plan refines rather than replaces
|
||||||
|
[`docs/superpowers/specs/2026-07-05-modern-pipeline-design.md`](../superpowers/specs/2026-07-05-modern-pipeline-design.md).
|
||||||
|
It preserves the shipped N.5 bindless/MDI renderer, the completed `GameWindow`
|
||||||
|
decomposition, canonical `LiveEntityRuntime` ownership, the retail PView/cell
|
||||||
|
visibility ports, and all behavior already accepted by the user.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Why the architecture should change
|
||||||
|
|
||||||
|
The 2026-07-24 six-minute, nine-stop connected route established:
|
||||||
|
|
||||||
|
| Observation | Measurement | Architectural implication |
|
||||||
|
|---|---:|---|
|
||||||
|
| Steady capped update work | approximately 0.4–1.3 ms median | The gameplay/update architecture is not the steady-state crisis |
|
||||||
|
| External GPU use | 7.6% median, 15.9% p95 | The tested route was not GPU-bound |
|
||||||
|
| Portal allocation | up to approximately 204 MiB in one frame | Runtime content preparation and publication must leave the frame transaction |
|
||||||
|
| Managed allocation rate | approximately 40.8 MB/s average, 2.22 GiB/s maximum | Portal work is creating short-lived graphs and pixel/mesh arrays faster than GC can absorb |
|
||||||
|
| GC | 38 Gen-2 collections; 256 ms longest pause | Frame spikes are dominated by allocation/publication, not upload time alone |
|
||||||
|
| Longest frame | 276 ms | Landblock-count budgets are not real cost budgets |
|
||||||
|
| Final live GC heap | approximately 194 MiB | The 1–2 GiB process high-water is not an unbounded live-object leak |
|
||||||
|
| Dedicated GPU memory | 633–818 MiB | Texture arrays/caches are bounded but expensive and insufficiently observable |
|
||||||
|
| Revisited locations | entity/emitter/particle counts returned to the same plateaus | Logical lifetime is broadly convergent; optimize rather than replace it |
|
||||||
|
| Exception traffic | 4,520 invalid DAT probes plus 1,168 receive timeouts | Exceptions are being used as normal control flow |
|
||||||
|
|
||||||
|
Allocation attribution names the runtime mesh/texture path
|
||||||
|
(`MeshExtractor.PrepareMeshData` → `PrepareGfxObjMeshData` →
|
||||||
|
`AddSurfaceToBatch`) as the largest portal source. The update-thread
|
||||||
|
publication/retirement graph then amplifies it with complete dictionaries,
|
||||||
|
sorts, effect profiles, and unmetered per-entity teardown.
|
||||||
|
|
||||||
|
The conclusion is:
|
||||||
|
|
||||||
|
> Keep retail-shaped gameplay and the modern GPU backend. Replace the runtime
|
||||||
|
> content, scene-projection, residency, and work-scheduling architecture that
|
||||||
|
> feeds the renderer.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Fixed architectural decisions
|
||||||
|
|
||||||
|
These decisions are binding unless a later measured gate disproves one.
|
||||||
|
|
||||||
|
1. **No whole-client rewrite.** Existing retail ports, network behavior,
|
||||||
|
physics, animation, UI, plugins, and the N.5 renderer remain in place.
|
||||||
|
2. **No simulation ECS.** Retail-shaped object-oriented code remains the oracle-
|
||||||
|
traceable form for physics, motion, sequencers, combat, inventory, and magic.
|
||||||
|
3. **A data-oriented render world only.** Arch remains the selected storage
|
||||||
|
implementation, hidden behind acdream-owned interfaces so it can be replaced
|
||||||
|
without changing simulation or rendering contracts.
|
||||||
|
4. **One authoritative live-object owner.** `LiveEntityRuntime` is evolved and
|
||||||
|
later extracted; no second GUID map, gameplay world, or mirrored entity state
|
||||||
|
may be introduced during migration.
|
||||||
|
5. **Headless is not a hidden window.** A headless session constructs no
|
||||||
|
Silk.NET window, GL context, render assets, particles, audio, or gameplay UI.
|
||||||
|
6. **One content interpretation.** Bake and runtime validation use the exact
|
||||||
|
existing `AcDream.Content` interpretation code. A baked artifact must be
|
||||||
|
byte/field equivalent before runtime uses it.
|
||||||
|
7. **No visual-quality optimization by default.** View horizon, retail PView,
|
||||||
|
particles, effects, and texture pixels stay unchanged. Retail distance LOD is
|
||||||
|
a faithful mechanism and remains separately configurable.
|
||||||
|
8. **Work is budgeted by cost, not object count.** Streaming stages use elapsed
|
||||||
|
microseconds, prepared bytes, uploaded bytes, and entity operations. “Four
|
||||||
|
landblocks” is not a meaningful budget.
|
||||||
|
9. **GL remains render-thread owned.** Workers may perform I/O, parsing,
|
||||||
|
decompression, mesh preparation, and collision preparation, but never mutate
|
||||||
|
GL or live scene state.
|
||||||
|
10. **Frame products are borrowed immutable views, not object graphs.** Snapshot
|
||||||
|
boundaries use double-buffered arrays/spans and generation stamps; they do
|
||||||
|
not allocate a deep immutable tree each frame.
|
||||||
|
11. **Parallelism follows data cleanup.** Do not add a job system to compensate
|
||||||
|
for repeated decode, sorting, allocation, or whole-world scans.
|
||||||
|
12. **Every cutover deletes the replaced production path.** Temporary shadow
|
||||||
|
execution is allowed for comparison, but no permanent dual renderer,
|
||||||
|
duplicate world, or silent decode fallback remains after its gate.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Target dependency architecture
|
||||||
|
|
||||||
|
```text
|
||||||
|
AcDream.GraphicalHost (current AcDream.App)
|
||||||
|
├── window/input/OpenGL/OpenAL/retail UI
|
||||||
|
├── render scene projection + renderer
|
||||||
|
└── owns one AcDream.Runtime.GameRuntime
|
||||||
|
|
||||||
|
AcDream.Headless
|
||||||
|
├── deterministic host clock
|
||||||
|
├── bot/plugin command surface
|
||||||
|
└── owns one or more AcDream.Runtime.GameRuntime instances
|
||||||
|
|
||||||
|
AcDream.Runtime [new, extracted incrementally]
|
||||||
|
├── session lifecycle and inbound ordering
|
||||||
|
├── authoritative live world and properties
|
||||||
|
├── movement/physics/interaction/combat/magic/inventory/chat
|
||||||
|
├── instance-scoped clocks, queues, commands, and events
|
||||||
|
└── no Silk.NET, GL, OpenAL, retained UI, or OS window dependency
|
||||||
|
|
||||||
|
AcDream.Content [already exists]
|
||||||
|
├── DAT interpretation
|
||||||
|
├── deterministic bake pipeline
|
||||||
|
├── mapped pak/content manifest
|
||||||
|
├── immutable prepared render/collision metadata
|
||||||
|
└── shared read-only content store for multiple sessions
|
||||||
|
|
||||||
|
AcDream.Core / AcDream.Core.Net [existing retail-shaped logic]
|
||||||
|
AcDream.Plugin.Abstractions [existing BCL-only contracts]
|
||||||
|
```
|
||||||
|
|
||||||
|
`AcDream.Runtime` is a target boundary, not the first implementation step. App
|
||||||
|
types move only after their presentation dependencies have been removed and
|
||||||
|
parity tests prove that the graphical host is still driving the same owner.
|
||||||
|
|
||||||
|
### Dependency rules
|
||||||
|
|
||||||
|
- `Core` never depends on Runtime, App, Content, or a backend.
|
||||||
|
- `Core.Net` depends on Core only.
|
||||||
|
- `Content` depends on Core, never App or a graphics backend.
|
||||||
|
- `Runtime` may depend on Core, Core.Net, Content, and Plugin.Abstractions.
|
||||||
|
- The graphical and headless hosts depend on Runtime; Runtime never depends on
|
||||||
|
either host.
|
||||||
|
- UI and rendering consume read-only runtime views plus explicit commands/events.
|
||||||
|
- Shared multi-session caches contain immutable content only. Credentials,
|
||||||
|
GUIDs, clocks, packets, plugins, object state, and automation remain
|
||||||
|
instance-scoped.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Runtime ownership model
|
||||||
|
|
||||||
|
| Concern | Canonical owner | Consumers |
|
||||||
|
|---|---|---|
|
||||||
|
| Server GUID/incarnation/timestamps/properties | `GameRuntime` evolved from `LiveEntityRuntime` | physics, UI, render projection, plugins, bots |
|
||||||
|
| Static prepared content | `ContentStore` | streaming, collision, render projection |
|
||||||
|
| Desired spatial window | `StreamingRegion` | streaming scheduler |
|
||||||
|
| Landblock generation and stage receipts | `StreamingWorkScheduler` | reveal, physics publication, render publication |
|
||||||
|
| Render entity/component data | `IRenderScene` / Arch implementation | visibility, snapshot builder |
|
||||||
|
| CPU/GPU asset residence | `ResidencyManager` | render scene, upload owner |
|
||||||
|
| GL names, fences, physical retirement | renderer resource owners | renderer only |
|
||||||
|
| Retail PView/visible-cell result | existing PView owner | render-scene query/snapshot builder |
|
||||||
|
| UI state | focused UI controllers/ViewModels | retained UI and optional dev UI |
|
||||||
|
|
||||||
|
Identity types must remain explicit:
|
||||||
|
|
||||||
|
```text
|
||||||
|
ServerGuid authoritative network identity
|
||||||
|
ObjectIncarnation ServerGuid + generation
|
||||||
|
RuntimeEntityId runtime-local logical identity
|
||||||
|
RenderEntityId presentation projection identity
|
||||||
|
AssetHandle<T> immutable content identity + generation
|
||||||
|
GpuHandle<T> render-thread physical resource identity
|
||||||
|
```
|
||||||
|
|
||||||
|
No layer may infer one identity from another through unchecked integer casts.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Content and asset architecture
|
||||||
|
|
||||||
|
### 5.1 Prepared assets
|
||||||
|
|
||||||
|
The runtime world hot path should consume prepared records, not DBObj graphs:
|
||||||
|
|
||||||
|
- Packed mesh vertices, indices, batch/material records, bounds, and part tables.
|
||||||
|
- Exact decoded texture pixels used by the current renderer.
|
||||||
|
- EnvCell geometry and portal records with content-deduplicated aliases.
|
||||||
|
- Terrain and deterministic scenery placement.
|
||||||
|
- Flattened collision/BSP arrays.
|
||||||
|
- Compact Setup presentation metadata:
|
||||||
|
default script, script table, animation, part availability, sound table,
|
||||||
|
collision references, and effect-relevant flags.
|
||||||
|
- Retail degrade tables.
|
||||||
|
|
||||||
|
Dynamic character palette/appearance composition, low-volume UI assets, audio,
|
||||||
|
and motion/animation tables may remain runtime DAT consumers until separately
|
||||||
|
measured and migrated.
|
||||||
|
|
||||||
|
### 5.2 Content-addressed pak manifest
|
||||||
|
|
||||||
|
Finish the existing pak implementation rather than replacing it:
|
||||||
|
|
||||||
|
```text
|
||||||
|
Source asset key -> manifest entry -> content hash/shared blob
|
||||||
|
```
|
||||||
|
|
||||||
|
- EnvCell file IDs may alias the same blob offset.
|
||||||
|
- Blob contents are deterministic and 64-byte aligned.
|
||||||
|
- DAT iteration stamps, serializer version, and bake-tool version invalidate
|
||||||
|
stale content.
|
||||||
|
- The writer builds a temporary artifact, validates it, then atomically replaces
|
||||||
|
the prior artifact.
|
||||||
|
- The reader memory-maps immutable data and exposes typed borrowed views.
|
||||||
|
- Corrupt/stale entries fail loudly and name the required rebake.
|
||||||
|
- Production streaming does not silently fall back to runtime decode after the
|
||||||
|
cutover. Developer equivalence tools retain both sources explicitly.
|
||||||
|
|
||||||
|
### 5.3 Asset handles and residence
|
||||||
|
|
||||||
|
Callers receive typed handles rather than retaining decoded arrays:
|
||||||
|
|
||||||
|
```csharp
|
||||||
|
readonly record struct AssetHandle<T>(uint Index, ushort Generation);
|
||||||
|
readonly record struct AssetLease<T>(AssetHandle<T> Handle, OwnerToken Owner);
|
||||||
|
```
|
||||||
|
|
||||||
|
The eventual `ResidencyManager` tracks:
|
||||||
|
|
||||||
|
- CPU prepared bytes.
|
||||||
|
- Decoded/pinned/staging bytes.
|
||||||
|
- GPU buffer and texture bytes.
|
||||||
|
- Current owners and reference count.
|
||||||
|
- Last used generation/frame.
|
||||||
|
- Rebuild/reload cost.
|
||||||
|
- Priority: destination-critical, visible, near, far, speculative.
|
||||||
|
- State: absent, requested, prepared, upload-pending, resident, retiring.
|
||||||
|
|
||||||
|
Eviction is generation-safe and owner-scoped. A stale completion cannot revive a
|
||||||
|
retired world generation or release the replacement generation's resource.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Streaming and reveal architecture
|
||||||
|
|
||||||
|
### 6.1 Staged pipeline
|
||||||
|
|
||||||
|
```text
|
||||||
|
Request
|
||||||
|
-> I/O/map lookup
|
||||||
|
-> parse/borrow prepared records
|
||||||
|
-> build landblock publication
|
||||||
|
-> physics/collision publish
|
||||||
|
-> render asset request
|
||||||
|
-> GL upload
|
||||||
|
-> render-scene publish
|
||||||
|
-> reveal-ready
|
||||||
|
```
|
||||||
|
|
||||||
|
Every item carries:
|
||||||
|
|
||||||
|
- World/session generation.
|
||||||
|
- Landblock/cell identity.
|
||||||
|
- Stage and priority.
|
||||||
|
- Estimated and actual byte cost.
|
||||||
|
- Entity-operation count.
|
||||||
|
- Cancellation token.
|
||||||
|
- Retryable stage receipt.
|
||||||
|
|
||||||
|
Bounded queues provide back-pressure. Background stages may run concurrently;
|
||||||
|
authoritative world, render-scene publication, and GL upload remain ordered on
|
||||||
|
their owning thread.
|
||||||
|
|
||||||
|
### 6.2 Cost budget
|
||||||
|
|
||||||
|
Each frame receives independently configurable budgets:
|
||||||
|
|
||||||
|
- Update-thread publication time.
|
||||||
|
- Entity create/retire operations.
|
||||||
|
- CPU bytes adopted.
|
||||||
|
- GPU bytes uploaded.
|
||||||
|
- GL resource retire operations.
|
||||||
|
|
||||||
|
Immediate work:
|
||||||
|
|
||||||
|
- Remove old landblocks from visibility, collision, picking, radar, and target
|
||||||
|
eligibility.
|
||||||
|
- Mark the old generation unavailable to new consumers.
|
||||||
|
|
||||||
|
Budgeted work:
|
||||||
|
|
||||||
|
- Walk old presentation owners.
|
||||||
|
- Release scripts/effects/lights/plugin snapshots.
|
||||||
|
- Retire CPU/GPU resources after fences permit.
|
||||||
|
- Publish far-ring content.
|
||||||
|
|
||||||
|
This preserves correct visible lifetime while preventing a 600-landblock
|
||||||
|
retirement from becoming one frame transaction.
|
||||||
|
|
||||||
|
### 6.3 Portal generation and reveal
|
||||||
|
|
||||||
|
A destination generation progresses through:
|
||||||
|
|
||||||
|
```text
|
||||||
|
Requested -> Prepared -> CollisionReady -> NearSceneReady -> Revealed
|
||||||
|
|
|
||||||
|
+-> FarSceneConverging
|
||||||
|
```
|
||||||
|
|
||||||
|
Reveal is an atomic edge. The world cannot become visible until the destination
|
||||||
|
near ring, collision root, camera identity, and required scene publication all
|
||||||
|
belong to the same generation. Far content continues under normal budgets after
|
||||||
|
reveal.
|
||||||
|
|
||||||
|
This replaces “priority work bypasses the budget” with a prepared-behind-portal
|
||||||
|
contract and eliminates `viewport-before-ready`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Data-oriented render scene
|
||||||
|
|
||||||
|
### 7.1 Scope
|
||||||
|
|
||||||
|
Arch stores render projections only:
|
||||||
|
|
||||||
|
- Static landblock objects and scenery.
|
||||||
|
- EnvCell objects.
|
||||||
|
- Live entities and equipped children.
|
||||||
|
- Lights and effect anchors where a packed projection is beneficial.
|
||||||
|
|
||||||
|
It never owns gameplay properties, network sequencing, physics authority,
|
||||||
|
inventory, combat, or interaction state.
|
||||||
|
|
||||||
|
### 7.2 Components
|
||||||
|
|
||||||
|
Initial components are blittable or stable handles:
|
||||||
|
|
||||||
|
```text
|
||||||
|
RenderTransform
|
||||||
|
PreviousRenderTransform
|
||||||
|
MeshAsset
|
||||||
|
MaterialVariant
|
||||||
|
SpatialResidency (landblock, cell)
|
||||||
|
WorldBounds
|
||||||
|
RenderFlags
|
||||||
|
DegradeState
|
||||||
|
SortKey
|
||||||
|
OwnerIncarnation
|
||||||
|
DirtyMask
|
||||||
|
```
|
||||||
|
|
||||||
|
Separate archetypes cover static, dynamic, equipped-child, translucent, and
|
||||||
|
light-bearing projections. Do not force every entity to carry every component.
|
||||||
|
|
||||||
|
### 7.3 Incremental indices
|
||||||
|
|
||||||
|
Creation, rebucketing, mutation, and removal maintain:
|
||||||
|
|
||||||
|
- Outdoor-static set.
|
||||||
|
- Per-cell static sets.
|
||||||
|
- Dynamic set.
|
||||||
|
- Translucent set.
|
||||||
|
- Selectable/pickable spatial index.
|
||||||
|
- Light candidates.
|
||||||
|
- Dirty transform/material/mesh ranges.
|
||||||
|
|
||||||
|
The frame loop no longer partitions every loaded entity. Existing retail PView
|
||||||
|
continues to determine visible cells; the render scene enumerates only those
|
||||||
|
cell buckets plus the dynamic set.
|
||||||
|
|
||||||
|
### 7.4 Simulation-to-render seam
|
||||||
|
|
||||||
|
`LiveEntityRuntime` and static publication emit ordered projection deltas:
|
||||||
|
|
||||||
|
```text
|
||||||
|
Register
|
||||||
|
UpdateTransform
|
||||||
|
UpdateAppearance
|
||||||
|
UpdateFlags
|
||||||
|
Rebucket
|
||||||
|
Unregister
|
||||||
|
```
|
||||||
|
|
||||||
|
Deltas contain exact incarnation identity. The render world rejects stale
|
||||||
|
generation updates. It never calls back into simulation dictionaries while
|
||||||
|
drawing.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Render-frame product and GPU submission
|
||||||
|
|
||||||
|
The snapshot is a double-buffered, generation-stamped borrowed view:
|
||||||
|
|
||||||
|
```text
|
||||||
|
RenderFrameView
|
||||||
|
Visible opaque instance ranges
|
||||||
|
Visible alpha instance ranges
|
||||||
|
Dynamic transforms
|
||||||
|
Light set
|
||||||
|
Effect draw records
|
||||||
|
Selection records
|
||||||
|
Existing PView/clip products
|
||||||
|
```
|
||||||
|
|
||||||
|
The update/render order initially remains on the accepted host thread. The
|
||||||
|
snapshot seam prevents mutation during draw and permits later thread separation
|
||||||
|
without requiring it now.
|
||||||
|
|
||||||
|
GPU submission evolves in measured steps:
|
||||||
|
|
||||||
|
1. Persistent global mesh/instance buffers remain.
|
||||||
|
2. Dirty ranges update only changed instance records.
|
||||||
|
3. MDI command buffers are reused and rewritten in place.
|
||||||
|
4. Static command templates are cached by scene generation/cell visibility.
|
||||||
|
5. Accurate timestamp queries bracket actual render passes.
|
||||||
|
6. GPU culling, command compaction, GPU particles, or GPU light selection are
|
||||||
|
introduced only when the corrected profile names them as the next bottleneck.
|
||||||
|
|
||||||
|
The existing portal/PView pass graph, clipping, translucency ordering, and
|
||||||
|
bindless material behavior remain authoritative.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Presentation-independent runtime and headless host
|
||||||
|
|
||||||
|
### 9.1 Extraction rule
|
||||||
|
|
||||||
|
Do not create a new parallel `GameRuntime` and synchronize it with
|
||||||
|
`LiveEntityRuntime`. Instead:
|
||||||
|
|
||||||
|
1. Define narrow runtime read/command/event contracts around the current owner.
|
||||||
|
2. Remove presentation-specific fields from that owner into App projections.
|
||||||
|
3. Move the now presentation-independent owner and collaborators into
|
||||||
|
`AcDream.Runtime`.
|
||||||
|
4. Keep the graphical host using the same instance throughout the move.
|
||||||
|
|
||||||
|
### 9.2 `GameRuntime`
|
||||||
|
|
||||||
|
One instance owns:
|
||||||
|
|
||||||
|
- Connection/authentication/character/session lifetime.
|
||||||
|
- Packet receive ordering and retail update phases.
|
||||||
|
- Authoritative objects, properties, containers, inventory, spell state,
|
||||||
|
enchantments, vitals, targets, and combat state.
|
||||||
|
- Movement, collision state required for gameplay, and interaction commands.
|
||||||
|
- Instance clock, random sources where applicable, queues, plugins/behaviors,
|
||||||
|
diagnostics identity, and teardown.
|
||||||
|
|
||||||
|
It exposes:
|
||||||
|
|
||||||
|
- Immutable or borrowed read views.
|
||||||
|
- Typed commands.
|
||||||
|
- Ordered events/deltas.
|
||||||
|
- Deterministic `Tick`.
|
||||||
|
- Retryable, complete shutdown.
|
||||||
|
|
||||||
|
### 9.3 Hosts
|
||||||
|
|
||||||
|
`GraphicalGameHost` provides input, camera, rendering, UI, audio, frame pacing,
|
||||||
|
and a single runtime instance.
|
||||||
|
|
||||||
|
`HeadlessGameHost` provides a monotonic scheduler, navigation collision/content,
|
||||||
|
bot actions, diagnostics, and one or more runtime instances. It performs no
|
||||||
|
render-content bake lookup unless navigation/collision requires that content.
|
||||||
|
|
||||||
|
For 30 clients:
|
||||||
|
|
||||||
|
- Immutable content and flattened collision assets are shared.
|
||||||
|
- Every mutable session structure is instance-scoped.
|
||||||
|
- One process may host many sessions, but one-process-per-session remains a
|
||||||
|
supported diagnostic/isolation mode.
|
||||||
|
- Scheduling uses a deterministic round-robin/time-wheel rather than 30 busy
|
||||||
|
loops.
|
||||||
|
- Server-safe outbound rate limits are per session.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Detailed implementation sequence
|
||||||
|
|
||||||
|
Each slice is independently buildable, testable, bisectable, and behavior-
|
||||||
|
preserving. Each gets a focused implementation plan when it starts.
|
||||||
|
|
||||||
|
### Slice A — Measurement and contract correction
|
||||||
|
|
||||||
|
**Purpose:** Make every later gate trustworthy.
|
||||||
|
|
||||||
|
- Correct `GpuFrameTimer` so queries cover actual GL submission/pass intervals,
|
||||||
|
not frame pacing.
|
||||||
|
- Extend canonical checkpoint JSON with CPU/GPU resident bytes, staging bytes,
|
||||||
|
per-stage queue depth, stage work time, exception counts, GC pause/heap
|
||||||
|
fields, and render-scene generation.
|
||||||
|
- Add a tracked analysis script that summarizes the existing nine-stop route.
|
||||||
|
- Record capped-RDP and uncapped-local results separately.
|
||||||
|
- Add a deterministic allocation/exception attribution recipe without enabling
|
||||||
|
developer UI or changing gameplay.
|
||||||
|
|
||||||
|
**Gate:** repeated idle measurements have low observer effect; CPU stage sums
|
||||||
|
reconcile with active frame time; external GPU engine direction agrees with GL
|
||||||
|
timestamps.
|
||||||
|
|
||||||
|
### Slice B — Finish MP1b EnvCell dedup and full bake
|
||||||
|
|
||||||
|
**Purpose:** Make the existing pak physically usable.
|
||||||
|
|
||||||
|
- Add shared-blob/alias support to `PakWriter` while retaining unique source
|
||||||
|
keys.
|
||||||
|
- Compute the existing runtime EnvCell geometry identity before extraction.
|
||||||
|
- Extract each unique geometry once; map all source file IDs to the shared blob.
|
||||||
|
- Make side-staged particle preloads and ordinary assets use the same
|
||||||
|
deterministic content table.
|
||||||
|
- Add alias, collision, determinism, stale-version, corruption, cancellation,
|
||||||
|
and atomic-replace tests.
|
||||||
|
- Run the complete bake and publish counts, unique ratios, size, time, failures,
|
||||||
|
and peak memory.
|
||||||
|
|
||||||
|
**Gate:** full bake finishes in practical time/space; all fixture and random
|
||||||
|
sample equivalence tests pass; duplicate EnvCells share blob offsets; no runtime
|
||||||
|
code consumes the pak yet.
|
||||||
|
|
||||||
|
### Slice C — Runtime prepared-asset source and MP1c cutover
|
||||||
|
|
||||||
|
**Purpose:** Remove mesh/texture decode from portal frames.
|
||||||
|
|
||||||
|
- Introduce `IPreparedAssetSource` in Content and inject it into App streaming.
|
||||||
|
- Teach `ObjectMeshManager`/landblock builders to adopt pak-backed prepared
|
||||||
|
payloads without reconstructing complete intermediate object graphs.
|
||||||
|
- Add compact Setup presentation metadata to the bake or a keyed prepared cache
|
||||||
|
so static activation does not parse arbitrary IDs as Setup.
|
||||||
|
- Replace `ResolveId().ToList().OrderBy...` with typed non-allocating lookup.
|
||||||
|
- Add explicit negative/type metadata; exceptions are not type tests.
|
||||||
|
- Retain live extraction only in bake/equivalence tooling and required dynamic
|
||||||
|
appearance paths.
|
||||||
|
- Remove production streaming fallback at the gate.
|
||||||
|
|
||||||
|
**Gate:** byte/field equivalence; no invalid Setup exception storm; portal
|
||||||
|
single-frame allocation and p99 materially improve from the 2026-07-24 baseline;
|
||||||
|
connected nine-stop route and screenshots pass.
|
||||||
|
|
||||||
|
### Slice D — Typed asset handles and unified residency
|
||||||
|
|
||||||
|
**Purpose:** Bound and explain CPU/GPU memory.
|
||||||
|
|
||||||
|
- Add `AssetHandle<T>`, owner tokens, leases, generations, and accounting.
|
||||||
|
- Place object mesh, standalone texture, composite texture-array, staging, and
|
||||||
|
prepared-content residence behind one policy owner while retaining specialized
|
||||||
|
physical caches.
|
||||||
|
- Track logical, CPU, staging, GPU-requested, GPU-resident, and retiring bytes
|
||||||
|
separately.
|
||||||
|
- Add configurable budgets through `RuntimeOptions`/settings with current visual
|
||||||
|
behavior as the default.
|
||||||
|
- Implement generation-safe LRU/cost-aware eviction and fence-delayed physical
|
||||||
|
release.
|
||||||
|
- Expose exact budget/occupancy/fragmentation facts to diagnostics.
|
||||||
|
|
||||||
|
**Gate:** same-location third-visit residence plateaus; no stale-generation
|
||||||
|
release; no missing textures; no resource growth after repeated portal loops.
|
||||||
|
|
||||||
|
### Slice E — Cost-budgeted streaming and retirement
|
||||||
|
|
||||||
|
**Purpose:** Remove update-thread portal transactions.
|
||||||
|
|
||||||
|
- Introduce explicit stage queues and `StreamingWorkBudget`.
|
||||||
|
- Split immediate logical/spatial detach from budgeted owner/resource teardown.
|
||||||
|
- Make per-entity retirement cursor/time bounded.
|
||||||
|
- Make publication cursor/time/byte bounded.
|
||||||
|
- Reserve destination-critical work across portal frames rather than bypassing
|
||||||
|
all budgets on one frame.
|
||||||
|
- Preserve FIFO within priority/generation and exact retry receipts.
|
||||||
|
- Connect reveal to the single destination-generation readiness state.
|
||||||
|
|
||||||
|
**Gate:** zero `viewport-before-ready`; traversal p99 at or below 16.67 ms target
|
||||||
|
and maximum below 33.3 ms target on the reference local run; no stranded old
|
||||||
|
generation, staged upload, collision, effect, or GPU owner.
|
||||||
|
|
||||||
|
### Slice F — Incremental render scene foundation
|
||||||
|
|
||||||
|
**Purpose:** Stop rebuilding/partitioning the world each frame.
|
||||||
|
|
||||||
|
- Add acdream-owned `IRenderScene`, identifiers, components, and delta journal.
|
||||||
|
- Implement it with Arch in App only.
|
||||||
|
- Mirror static publication and live projection into a non-drawing shadow world.
|
||||||
|
- Add deterministic scene digest and compare it with current owners at
|
||||||
|
checkpoints.
|
||||||
|
- Maintain cell/outdoor/dynamic/translucent/light indices incrementally.
|
||||||
|
- Add stale-incarnation, duplicate-create, rebucket, delete/recreate, hidden,
|
||||||
|
parent-child, and session-reset tests.
|
||||||
|
|
||||||
|
**Gate:** shadow scene matches canonical world/resource checkpoints through the
|
||||||
|
nine-stop route with bounded memory and zero authoritative ownership.
|
||||||
|
|
||||||
|
### Slice G — Render snapshot and delta submission cutover
|
||||||
|
|
||||||
|
**Purpose:** Make render cost proportional to visible/changed data.
|
||||||
|
|
||||||
|
- Build the double-buffered `RenderFrameView`.
|
||||||
|
- Feed it the existing PView visible-cell/clip product.
|
||||||
|
- Replace full `InteriorEntityPartition` scans with render-scene bucket queries.
|
||||||
|
- Upload only dirty persistent instance ranges.
|
||||||
|
- Reuse MDI command/sort buffers.
|
||||||
|
- Build instance-set comparison and fixed-camera screenshot comparison.
|
||||||
|
- Run old and new submissions in compare mode without drawing twice.
|
||||||
|
- At the gate, delete the replaced production enumeration/submission path.
|
||||||
|
|
||||||
|
**Gate:** instance sets and screenshots match; dense-town uncapped target is
|
||||||
|
300 FPS or the corrected profile identifies a new dominant stage; steady frame
|
||||||
|
allocation is near zero.
|
||||||
|
|
||||||
|
### Slice H — Event-driven UI, diagnostics, lights, and frame cleanup
|
||||||
|
|
||||||
|
**Purpose:** Remove remaining work that scales with uncapped FPS.
|
||||||
|
|
||||||
|
- Skip diagnostics with no consumer and reuse renderer visibility facts.
|
||||||
|
- Dirty-layout retained UI: apply anchors only after geometry changes.
|
||||||
|
- Maintain an overlay-participant registry instead of a second full-tree walk.
|
||||||
|
- Track active cooldown items/effects/dialogs through events.
|
||||||
|
- Replace per-frame frame-input objects, LINQ arrays, iterator objects, and
|
||||||
|
liveness collections with reusable storage or borrowed views.
|
||||||
|
- Make light candidates spatial/cell-driven and select top-k without a complete
|
||||||
|
per-frame sort.
|
||||||
|
- Replace socket-timeout exceptions and datagram `ToArray` copies with normal
|
||||||
|
cancellable/poll/span-based I/O.
|
||||||
|
|
||||||
|
**Gate:** no steady-state Gen-2 collections; steady allocation target
|
||||||
|
≤ 4 KiB/frame initially and zero for the core world/render loop; UI and network
|
||||||
|
behavior tests unchanged.
|
||||||
|
|
||||||
|
### Slice I — Flat collision assets and residual zero-allocation work
|
||||||
|
|
||||||
|
**Purpose:** Complete MP4 without changing retail math.
|
||||||
|
|
||||||
|
- Bake flattened index-based BSP/collision records.
|
||||||
|
- Port traversal data access line-for-line while preserving ordering and
|
||||||
|
arithmetic.
|
||||||
|
- Remove parsed DBObj graphs from streaming collision publication.
|
||||||
|
- Address remaining measured allocation sites, including any physics transition
|
||||||
|
pooling only after identity/lifetime tests prove it safe.
|
||||||
|
|
||||||
|
**Gate:** trajectory and retail conformance suites remain bit-equivalent;
|
||||||
|
navigation/collision fixtures match; portal and steady allocation targets pass.
|
||||||
|
|
||||||
|
### Slice J — Presentation-independent `AcDream.Runtime`
|
||||||
|
|
||||||
|
**Purpose:** Establish the graphical/headless shared client kernel.
|
||||||
|
|
||||||
|
- Add the Runtime project and dependency guards.
|
||||||
|
- Define `IGameRuntimeView`, commands, ordered events, clock, and lifecycle.
|
||||||
|
- Adapt the current graphical host to those interfaces first.
|
||||||
|
- Remove App presentation dependencies from canonical gameplay owners.
|
||||||
|
- Move owners by coherent lifetime groups; never mirror state.
|
||||||
|
- Add host-parity and no-backend construction tests after each move.
|
||||||
|
- End with `GameWindow`/App composing one `GameRuntime`.
|
||||||
|
|
||||||
|
**Gate:** the connected graphical route is unchanged; a no-window integration
|
||||||
|
test connects, enters world, moves, receives inventory/chat/world updates,
|
||||||
|
portals, logs out, reconnects, and tears down without loading App/Silk/OpenAL.
|
||||||
|
|
||||||
|
### Slice K — Linux headless and multi-session host
|
||||||
|
|
||||||
|
**Purpose:** Deliver efficient automated bots.
|
||||||
|
|
||||||
|
- Add Linux CI for Core, Core.Net, Content, Runtime, and Headless.
|
||||||
|
- Implement portable path/config/credential handling.
|
||||||
|
- Add `AcDream.Headless` CLI and deterministic scheduler.
|
||||||
|
- Add bot command/event APIs for movement, selection, use, combat, spells,
|
||||||
|
looting, chat, and commands.
|
||||||
|
- Share immutable content/collision stores across sessions.
|
||||||
|
- Audit and eliminate mutable statics and process-wide session state.
|
||||||
|
- Add clean cancellation, reconnect, plugin isolation, and credential-safe logs.
|
||||||
|
- Stress 1, 5, 10, and 30 sessions.
|
||||||
|
|
||||||
|
**Gate:** 30 local-server sessions in one Linux process, no GPU/display/audio
|
||||||
|
dependency, bounded incremental memory per session, no busy-loop CPU, clean
|
||||||
|
teardown/reconnect, and parity with graphical runtime command/event behavior.
|
||||||
|
|
||||||
|
### Slice L — Linux graphical host and evidence-gated GPU work
|
||||||
|
|
||||||
|
**Purpose:** Finish platform portability and only then pursue remaining GPU
|
||||||
|
opportunities.
|
||||||
|
|
||||||
|
- Validate Linux OpenGL extension/driver matrix and package native dependencies.
|
||||||
|
- Port paths, frame pacing, input, audio, and packaging without renderer
|
||||||
|
fallback.
|
||||||
|
- Reprofile locally with accurate pass timing.
|
||||||
|
- Add GPU culling, command compaction, particle simulation, or light selection
|
||||||
|
only for stages proven dominant.
|
||||||
|
|
||||||
|
**Gate:** graphical Linux connected route passes the same lifecycle, screenshot,
|
||||||
|
resource, and performance checks; every GPU migration has a CPU/GPU before/after
|
||||||
|
and visual equivalence result.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. Performance gates
|
||||||
|
|
||||||
|
Reference measurements use a Release build, the same account/route, stable
|
||||||
|
camera scripts, and both capped and uncapped modes. RDP results are kept
|
||||||
|
separate from local-display results.
|
||||||
|
|
||||||
|
| Metric | Target |
|
||||||
|
|---|---:|
|
||||||
|
| Traversal frame p99 | ≤ 16.67 ms on reference local hardware |
|
||||||
|
| Traversal maximum | ≤ 33.3 ms after warm process startup |
|
||||||
|
| Dense-town uncapped frame p50 | ≤ 3.33 ms or a newly attributed blocker |
|
||||||
|
| Steady update p95 | ≤ 2 ms |
|
||||||
|
| Core world/render allocation | 0 B/frame after warmup |
|
||||||
|
| Whole-client steady allocation | initial gate ≤ 4 KiB/frame |
|
||||||
|
| Portal single-frame allocation | ≤ 4 MiB, then tighten from evidence |
|
||||||
|
| Gen-2 collections during canonical route | 0 after startup/bake warmup |
|
||||||
|
| Same-location third-visit resource growth | ≤ 5% and explained |
|
||||||
|
| World-visible-before-ready events | 0 |
|
||||||
|
| Staged resources at stable checkpoints | 0 |
|
||||||
|
| Exception-as-control-flow | 0 known sites |
|
||||||
|
| Headless renderer/audio/window allocations | 0 |
|
||||||
|
|
||||||
|
Memory budgets are measured by category rather than only process working set:
|
||||||
|
|
||||||
|
- Live managed heap.
|
||||||
|
- GC committed/fragmented.
|
||||||
|
- Native prepared/staging buffers.
|
||||||
|
- Mapped content pages.
|
||||||
|
- Tracked GPU buffers/textures.
|
||||||
|
- Driver-reported dedicated/shared GPU memory.
|
||||||
|
|
||||||
|
Absolute process/VRAM targets are fixed after Slice D produces trustworthy
|
||||||
|
category accounting; until then, the binding rule is plateau plus no regression
|
||||||
|
in pixels/range.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 12. Test and review matrix
|
||||||
|
|
||||||
|
Every slice must pass the tests relevant to its boundary:
|
||||||
|
|
||||||
|
1. **Pure unit tests:** handles, generations, budgets, queues, manifest,
|
||||||
|
deduplication, stale/corrupt data, snapshot buffers.
|
||||||
|
2. **DAT equivalence:** live extractor versus prepared asset for representative
|
||||||
|
outdoor, dungeon, town, portal, animated, translucent, and particle assets.
|
||||||
|
3. **Scene equivalence:** canonical entity/instance digest at fixed checkpoints.
|
||||||
|
4. **Render equivalence:** instance-set diff plus fixed-camera screenshots.
|
||||||
|
5. **Retail conformance:** named-retail citations for any AC-specific selection,
|
||||||
|
degradation, visibility, collision, or timing behavior touched.
|
||||||
|
6. **Lifecycle:** fresh login, same-location revisit, world edge, dungeon,
|
||||||
|
repeated recalls, rapid generation replacement, logout/reconnect, graceful
|
||||||
|
close, cancellation, and failure injection.
|
||||||
|
7. **Resource:** exact create/retire counts, heap diff, GPU accounting, no stale
|
||||||
|
owner or queued work at stable checkpoints.
|
||||||
|
8. **Performance:** capped and uncapped CPU/GPU/allocation/GC percentiles.
|
||||||
|
9. **Headless parity:** same runtime packet/order/command results with
|
||||||
|
presentation attached or absent.
|
||||||
|
10. **Platform:** Windows graphical/headless plus Linux headless; Linux graphical
|
||||||
|
begins in Slice L.
|
||||||
|
|
||||||
|
For a replacement cutover:
|
||||||
|
|
||||||
|
- Shadow/compare mode may exist only during development.
|
||||||
|
- Confirmed differences are fixed at their source.
|
||||||
|
- The gate commit removes the old production path.
|
||||||
|
- No test threshold is loosened to accept a regression.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 13. Risks and controls
|
||||||
|
|
||||||
|
| Risk | Control |
|
||||||
|
|---|---|
|
||||||
|
| Pak repeats the 865 GB failure | content identity and aliasing land before another full bake; bake reports unique/shared ratios continuously |
|
||||||
|
| Prepared data changes pixels/geometry | byte/field equivalence against the current extractor; no compression in the equivalence phase |
|
||||||
|
| Streaming budget reveals an incomplete world | one destination-generation readiness barrier; reveal is an atomic state edge |
|
||||||
|
| Deferred retirement leaks or revives stale resources | incarnation/generation owner tokens and stable-checkpoint zero-outstanding assertions |
|
||||||
|
| Render ECS becomes a second game world | render-only component contract; no network properties or authoritative GUID map; digest checks against canonical owner |
|
||||||
|
| Snapshot adds a frame of latency | same-thread borrowed snapshot first; frame-number assertions; concurrency remains deferred |
|
||||||
|
| Arch dependency constrains the design | acdream-owned `IRenderScene`; Arch only in App; no Arch types cross the boundary |
|
||||||
|
| Headless extraction duplicates state | move the existing canonical owner in lifetime groups; adapters first, never synchronize two stores |
|
||||||
|
| Shared 30-client cache gains mutable state | immutable content only; tests create conflicting sessions/characters concurrently |
|
||||||
|
| Linux work forces renderer fallback | graphical host retains mandatory modern extensions and fails clearly on unsupported drivers |
|
||||||
|
| “Optimization” changes retail behavior | unchanged behavior is a hard gate; retail ports remain OO and traceable; visual/connected gates stay mandatory |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 14. Recommended starting point
|
||||||
|
|
||||||
|
Begin with **Slice A**, then **Slice B**.
|
||||||
|
|
||||||
|
Slice A is small and makes all later GPU and frame gates honest. Slice B completes
|
||||||
|
an implementation already present in the tree and directly attacks the
|
||||||
|
benchmark's largest allocation source. The first meaningful runtime cutover is
|
||||||
|
Slice C; do not begin the render-scene migration before the portal content storm
|
||||||
|
has been removed and measured.
|
||||||
|
|
||||||
|
The intended order is therefore:
|
||||||
|
|
||||||
|
```text
|
||||||
|
honest metrics
|
||||||
|
-> prepared content
|
||||||
|
-> typed residence
|
||||||
|
-> cost-budgeted streaming
|
||||||
|
-> incremental render scene
|
||||||
|
-> delta GPU submission
|
||||||
|
-> residual frame cleanup
|
||||||
|
-> presentation-independent runtime
|
||||||
|
-> Linux/headless/multi-session
|
||||||
|
-> evidence-gated GPU jobs
|
||||||
|
```
|
||||||
|
|
||||||
|
This order improves the current graphical client first while laying clean
|
||||||
|
boundaries for Linux and approximately 30 automated headless clients later.
|
||||||
Loading…
Add table
Add a link
Reference in a new issue