diff --git a/docs/ISSUES.md b/docs/ISSUES.md index 72f6f3a7..24ecb5dc 100644 --- a/docs/ISSUES.md +++ b/docs/ISSUES.md @@ -99,11 +99,23 @@ Copy this block when adding a new issue: ## #260 — Portal-network wedge: outbound actions die + native/GPU memory climbs -**Status:** OPEN — investigated 2026-07-29 (two-agent report-only pass over the -captured evidence); fix authorized by the user. **Verdict: two separate roots.** +**Status:** ROOT-CAUSED 2026-07-29 — **fix in flight as Campaign N** +([`docs/plans/2026-07-29-network-transport-campaign.md`](plans/2026-07-29-network-transport-campaign.md)). +The wedge is missing packet-loss recovery in BOTH directions: (a) no outbound +retransmission — ACE's `RequestRetransmit` lists are parsed and consumed +nowhere, no sent-packet cache exists, so one lost C2S datagram permanently +stalls ACE's ordered processing of everything we send (actions void, position +updates void → ACE stops streaming new areas → the #256 invisible portals); +(b) the inbound ISAAC keystream is burned in ARRIVAL order +(`PacketCodec.TryDecodeBorrowed` consumes before comparing), so one lost S2C +datagram permanently desyncs the inbound cipher and silences inbound. Local +ACE is loopback (zero loss) — every historical gate was structurally blind. +The memory half is CLOSED as benign: WS +2.5 GB vs private +0.7 GB over a +session = mapped-pak page residency (27.85 GB mmap) + GPU caches filling to +designed ceilings and plateauing; no leak. **Severity:** HIGH (renders the client unplayable after sustained portal use) **Filed:** 2026-07-29 (instrumented Coldeve session, Vulkan backend) -**Component:** WorldSession outbound boundary + Vulkan resource lifecycle +**Component:** Core.Net reliable transport (both directions) **Supersedes framing of:** #256 (invisible portal-network objects) and #257 (~1.5 GB working set) — both are facets of this. diff --git a/docs/plans/2026-07-29-network-transport-campaign.md b/docs/plans/2026-07-29-network-transport-campaign.md new file mode 100644 index 00000000..3d602578 --- /dev/null +++ b/docs/plans/2026-07-29-network-transport-campaign.md @@ -0,0 +1,260 @@ +# Campaign N — Retail-Faithful Network Transport + +**Status: ACTIVE (approved 2026-07-29). Slices N0–N6.** + +acdream cannot survive a single lost UDP packet in either direction. This +campaign ports retail's reliable-transport mechanism (the Sept 2013 named +decomp is the oracle; ACE and holtburger are cross-checks) so live-server +sessions survive real packet loss. Root-cause record: `docs/ISSUES.md` #260. + +Read this before touching `src/AcDream.Core.Net/`. + +--- + +## 1. Why (the two fatal bugs) + +1. **Outbound (the #260 wedge):** no sent-packet cache, no resend. ACE NAKs a + client-sequence gap with `RequestRetransmit`; we parse the list and consume + it nowhere. One lost C2S datagram → ACE buffers everything after the gap + forever → all actions void, position updates void → ACE stops streaming new + areas (the #256 "invisible portals") while the session stays alive on + cleartext acks. Reproduced twice on Coldeve 2026-07-29 with the + `ACDREAM_PROBE_NET` probe (`artifacts/coldeve-probe-20260729/`, local). +2. **Inbound (found at design time):** `PacketCodec.TryDecodeBorrowed` + consumes the inbound ISAAC word BEFORE comparing. One lost S2C packet + shifts the keystream permanently; every later encrypted packet fails + checksum and burns another word. Inbound silent forever. + +Loopback ACE never drops packets — every historical gate was structurally +blind to both bugs. Slice N5's loss gate removes that blindness permanently. + +## 2. The retail mechanism (port target) + +Owners: inbound = `ReceiverData` (per-connection, in `SharedNet::receivers_`); +outbound = `RecipientData` + `ClientFlowQueue` + `SentPacketStore` (under +`PacketController`). Both pumped once per frame from `Client::UseTime +@ 0x00411C40` (receive: `SharedNet::UseTime @ 0x00542450`; send: +`PacketController::UseTime @ 0x005410D0`). + +### 2.1 Outbound resend + +- Every reliable packet (fragments aboard) is cached AFTER a successful send + (`FlowQueue::TransmitNewPackets @ 0x00547A60` → `SentPacketStore:: + AddSentPacket @ 0x0054AB00`), with disposable optional headers stripped + first (`NetPacket::RemoveDisposableOptionalHeaders @ 0x00549510`). Under + our standalone-control design (§4) reliable packets never carry optional + headers, so the strip is a provable no-op — assert `optionalLength == 0` + in `SentPacketStore.Add`. +- Server `RequestRetransmit` (0x1000) → merge-insert ids wrap-safe sorted + (`FlowQueue::EnqueueAcks @ 0x005488E0`); `ids[0]` doubles as an implicit + cumulative ack (`RecipientData::ProcessNaks @ 0x00547010`). Ids no longer + cached → retail answers `RejectRetransmit`; we drop silently (row TS-57). +- Resend (`FlowQueue::TransmitAcks @ 0x005485B0` / `DequeueAck @ + 0x005472F0`): re-emit with a REBUILT 20-byte header — flags + `Retransmission|EncryptedChecksum` (=3; |`BlobFragments` =7 with + fragments), `Time` = the CURRENT interval id, `Sequence`/`DataSize` + verbatim, checksum = fresh header hash + stored sealed checksum. Body + bytes untouched; **the original ISAAC key is reused** (`CryptoSystem:: + EncryptData @ 0x0065FF40` takes the key as an optional in/out — null draws, + non-null reuses). Never a new keystream word. +- `AckSequence` (0x4000) inbound → wrap-safe max watermark → prune strictly + older (`SentPacketStore::Flush @ 0x0054ACD0`, pops from FIFO head while + `seqNum < watermark`). No timer-based resend exists — resend only on + explicit NAK. +- Sequence allocation: `highestIDSent_` starts 1; `++`; wrap 0xFFFFFFFF → 1 + (never 0). Unsequenced packets reuse the current watermark, cleartext, no + ISAAC word. +- Pump order per frame (`FlowQueue::Empty @ 0x00548A20`): NAKs → retransmits + → new packets. + +### 2.2 Inbound sequencing + ISAAC discipline + +**No transport reorder buffer** — packets process on arrival +(`SharedNet::ProcessPacket @ 0x00544790` → `ProcessNewSeqNum @ 0x00544690`): + +1. `SeqIDSanityCheck @ 0x00543A20`: drop if seq is newer than + `highestIDReceived_ + 0x7FFF` (wrap-safe). +2. Encrypted + not newer than the watermark = duplicate/late arrival: remove + from the NAK set — hit → decrypt with the **parked pre-drawn key**; + miss → drop silently at zero keystream cost. +3. Newer than the watermark → gap walk (`ProcessNewestSeqNum @ 0x00541930`): + for each missing id (skip 0) `AddNakked(id, null)` — **pre-draws one + inbound ISAAC word per missing id, in sequence order, BEFORE the arriving + packet's own key**, parking it beside the id (`ReceiverData::AddNakked @ + 0x00549240`, idempotent). Cleartext packets walk to `seq+1` (they borrow + an already-delivered sequence). Then watermark = seq. +4. Encrypted packets then decrypt with the parked key or the next drawn word. +5. Parse failure on a sequenced packet → re-park its consumed key + (`AddNakked(seq, &key)`), so the retransmission decodes. +6. Inbound `RejectRetransmit` (`SharedNet::HandleEmptyAck @ 0x005448F0`) → + remove ids from the NAK set, silent abandonment. + +This keeps the inbound keystream aligned to SEQUENCE order, not arrival order +— the invariant the whole inbound port hangs on. + +### 2.3 Ack/NAK sweep + +Per-frame per-connection sweep (`ClientNet::ProcessConnection @ 0x00545450`), +mutually exclusive on ONE shared timestamp (`ReceiverData::timeStamp_`): + +- NAK set non-empty → `RequestRetransmit`: ≤114 ids ascending, cleartext, + 0.6 s gate (`SharedNet::EnqueueNaks @ 0x00543BD0`, `ReceiverData::GetNaks + @ 0x005490C0`), and NO ack this sweep. +- Else → one cumulative `AckSequence` carrying `highestIDReceived_`, + unsequenced + cleartext, 2.0 s gate (`SharedNet::EnqueuePak @ 0x00543B10` + — the ONLY 0x4000 construction site in the binary; retail never acks + per-packet). + +### 2.4 Keepalive/time/flow (mostly deferred — §5) + +TimeSync (8-byte double) + EchoRequest (4-byte float) every ~3 s +(`ClientFlowQueue::IncrementLocalInterval @ 0x00547F10`); inbound TimeSync +adopts the server clock; Flow report sent on remote-interval advance; +inbound Flow ignored, no throttle (`WireRoomLeft` is a folded `return 1` — +verified); dead link at 140 s without inbound data (self-stall guarded) with +referral auto-reconnect; 0.5 s interval clock feeds `ProtoHeader::interval_` +(our `PacketHeader.Time`). + +### 2.5 Constants + +464 max payload after the 20-byte header · 448 fragment payload · 114 NAK +list cap · +0x7FFF sanity window · 0.5 s interval tick · 2.0 s ack gate · +0.6 s NAK gate · ~3 s TimeSync/Echo · 140 s dead link · 10 s disconnect +drain · `0xBADD70DD` checksum placeholder · sequence init 1, wrap +0xFFFFFFFF→1 · 0.333 s ConnectResponse handshake resend. + +## 3. ACE constraints (Coldeve runs ACE; references in the MAIN repo checkout) + +| Constraint | Source | Consequence | +|---|---|---| +| NAKs client gaps at `expected+2`, 1 s limit, arrival-driven ONLY | NetworkSession.cs:351-363 | after one C2S loss two more sends must arrive before ACE NAKs; a quiet client is never NAKed | +| Crypto search window 256 keys; re-key orphans permanently | CryptoSystem.cs:30-49 | NEVER re-key a resend; NEVER resend an already-accepted packet | +| Watermark advances on ANY packet with flags ≠ exactly `AckSequence` | NetworkSession.cs:474-476 | standalone unsequenced control packets other than exact-`AckSequence`/exact-`RequestRetransmit` can skip a REAL packet forever (the self-induced wedge) | +| Honours only CLEARTEXT NAKs; encrypted NAKs silently ignored | NetworkSession.cs:283-284 | no NAK piggybacking; NAKs never refresh ACE's 60 s timeout | +| Ack-only dedup exemption requires flags == `AckSequence` exactly | NetworkSession.cs:342-343 | equality check, not HasFlag | +| ACE never proactively resends S2C; S2C cache prunes at 120 s | NetworkSession.cs:675-708, :251-262 | the client MUST NAK or inbound stalls; old NAKs get RejectRetransmit | +| C2S fragment sequence strictly contiguous or dispatch stalls silently | NetworkSession.cs:532-543 | packet-level retransmission heals it automatically | +| ACE sends TimeSync/20 s, cumulative ack/2 s, EchoResponse on request; NO disconnect/error packets ever | NetworkSession.cs:207-216, grep | every transport death is silence | +| Rejects AckSequence/TimeSync/EchoRequest/Flow during AuthLoginRequest | Session.cs:101-102 | no sweep before negotiation completes | +| Inbound `Header.Time` ignored | grep | our interval field is cosmetic against ACE — safe | + +## 4. Design + +`src/AcDream.Core.Net/Transport/` — retail's split adapted to one login + one +world connection on one frame thread. `TransportClock` (injectable monotonic +source + 0.5 s interval counter), `SequenceMath` (`IsNewer(a,b) => +unchecked((int)(a-b)) > 0`), `SentPacketStore` (FIFO of rented wire buffers), +`OutboundFlowQueue` (outbound ISAAC, HighestIdSent, fragment seq, pending +resends, NAK/ack consumption), `InboundSequenceTracker` (inbound ISAAC, +HighestIdReceived, NAK set `SortedDictionary` seq→parked key), +`AckNakScheduler` (ONE shared timestamp; per sweep NAK xor ack), +`ReliableTransport` (composition + `Sweep()`), `TransportStats` +(unconditional counters; printing probe-gated). + +Cache entry: `{ Sequence, rented Buffer (header+body), BodyLength, +SealedChecksum (= payloadHash ^ isaacKey — retail `NetPacket::checksum_`), +IsaacKey, HasFragments }`. Resend rewrites ONLY the header per §2.1; +`PacketCodec.FinalizeInPlace` gains an overload returning +`(isaacKeyUsed, sealedChecksum)`; the old signature forwards. + +Inbound: codec splits pure-parse (`TryParseBorrowed` — no keystream) from +`VerifyChecksum(header, headerHash, payloadHash, uint? isaacKey)`; the +tracker owns every key decision (retail's own factoring — `EncryptData`'s +optional key parameter). `TryDecodeBorrowed(datagram, IsaacRandom?)` is +deleted; owned `TryDecode` stays (test-only). + +`WorldSession` keeps its entire ~60-method public surface; the two +`SendGameMessage` overloads delegate to the transport; `GameActionCapture` +is untouched. `Sweep()` runs at the end of `Tick()` (after the budget break) +AND inside the blocking handshake pump loops (the EnterWorld flood arrives +before `Tick` ever runs), gated on `_transportNegotiated`. Cache is +unbounded like retail (ACE acks every 2 s; steady state is tens of entries); +`cache=N` in `[net-tick]` is the watchdog, not a silent cap. + +## 5. Scope deferrals (divergence-register rows, filed in-slice) + +- **TS-57** no outbound RejectRetransmit — ACE no-ops it; the standalone + unsequenced form trips the watermark hole (§3 row 3). +- **AP-125** standalone control packets only, no `CoalesceData` piggyback — + piggybacked NAKs become encrypted, which ACE ignores. +- **TS-58** no TimeSync/EchoRequest keepalive — standalone-unsafe against + the watermark hole; our 2 s ack already refreshes ACE's 60 s timeout. +- **TS-59** no Flow report — ACE parses and has no handler. +- **AD-49** blob-layer ephemeral ordering stamps not ported — provably a + no-op against ACE (fragment `Id` is constant 0x80000000 and the stamp + table keys on ACE's per-message-unique fragment sequence, so + `FragIsObsoleteEmphemeral @ 0x0054A450` can never fire). +- **TS-60** no 140 s dead-link/referral auto-reconnect — LinkStatus exposes + the input; reconnect is Runtime's, its own campaign. +- **TS-61** UDP send-failure burns the sequence+key (retail retries from the + queue head) — effectively unreachable. +- **AP-126** one monotonic clock for all gates (retail's cur/local split + immaterial to the gates we port). +- Multi-fragment outbound (>448 B) stays unimplemented (nothing sends >448; + existing behavior, keep its row current). + +## 6. Slices + +| # | Slice | Size | Parallel? | Review | Gate | +|---|---|---|---|---|---| +| N0 | ACE-behaviour test double + virtual clock + lossy link | M | yes (test-only) | Opus | unit | +| N1 | Outbound cache + resend on NAK (the #260 fix) | L | no | **Fable** | unit + local ACE lifecycle | +| N2 | Inbound sequence-aligned ISAAC + NAK set | L | no | **Fable** | unit + lifecycle + nine-stop | +| N3 | AckNakScheduler + 2.0 s cumulative ack | M | no | Opus | unit + lifecycle + nine-stop | +| N4 | Client NAK emission + RejectRetransmit consumption | M | no | Opus | unit + local ACE + loss gate | +| N5 | Observability + LossyTransportDecorator + connected loss gate | M | partly | Opus | unit + loss gate | +| N6 | Optional: ConnectResponse 0.333 s retransmit; assembler TTL | S | yes | Opus | unit + local ACE | + +N1–N5 all touch `WorldSession.cs`: strictly sequential, ONE agent at a time. +Implementers Sonnet; a redo escalates to Opus; every diff reviewed by the +Review-column model before commit. Key per-slice test specs are in the +approved plan (`~/.claude/plans/mac-os-is-not-robust-sphinx.md`) and travel +verbatim in each implementer prompt. + +## 7. Landmines (verbatim in every implementer prompt) + +1. Resends are NOT byte-identical — flags gain `Retransmission`, `Time` + advances, the header hash MUST be recomputed; checksum = new header hash + + stored sealed checksum. A verbatim resend fails ACE's CRC silently. +2. Never draw a new ISAAC word on a resend — one re-key permanently orphans + a key in ACE's 256-entry window. +3. Never resend unrequested — a duplicate of an accepted packet burns up to + the whole window. +4. The inbound gap-walk draws parked keys BEFORE the arriving packet's own + key, in sequence order — reversed, the stream is off by the gap size + forever. +5. Ack flags are an EQUALITY check (`Flags == AckSequence`), never HasFlag. +6. NAKs are cleartext (`Flags == RequestRetransmit` exactly) or ACE ignores + them; NAKs do not refresh ACE's timeout. +7. The 0.6 s NAK gate and 2.0 s ack gate share ONE timestamp — a NAK delays + the next ack and vice versa; never both in one sweep. +8. The sweep must also run inside the blocking handshake pump loops — the + EnterWorld CreateObject flood precedes the first `Tick()`. +9. ACE/holtburger reference sources live in the MAIN repo checkout + (`C:\Users\erikn\source\repos\acdream\references\`), not in worktrees. + +## 8. Verification ladder + +1. Per slice: `dotnet build` + full `dotnet test -c Release` + slice gate + + one commit (retail symbol+address citations) + model review + revert SHA + recorded here. +2. N1–N5: `tools/run-connected-world-lifecycle-gate.ps1` vs local ACE; + N2/N3 add the canonical nine-stop route. +3. From N4: `tools/run-connected-loss-gate.ps1` at `ACDREAM_NET_DROP_PCT=2` + vs local ACE — passes only with NON-ZERO resend/NAK counters. +4. Final: one user Coldeve session past the 15-minute wedge horizon with + portal churn, after N5. `[net-tick] nak-in/s` becomes the first direct + measure of real C2S loss the project has had. +5. Rollback: one commit per slice, `git revert`; no runtime kill switch. + +## 9. Slice ledger + +| Slice | Status | Commit | Notes | +|---|---|---|---| +| N0 | pending | — | | +| N1 | pending | — | | +| N2 | pending | — | | +| N3 | pending | — | | +| N4 | pending | — | | +| N5 | pending | — | | +| N6 | pending | — | |