feat(net): N5 - loss observability, lossy decorator, the connected loss gate

Campaign N Slice N5 (docs/plans/2026-07-29-network-transport-campaign.md
section 8 rung 3): the permanent removal of the loopback blindness that let
#260 ship. Local ACE never drops a datagram, so every historical connected
gate was structurally incapable of exercising the N1-N4 recovery machinery;
from this slice on, tools/run-connected-loss-gate.ps1 runs the standard
lifecycle route through deterministic seeded loss and passes only on proven
non-zero recovery.

Observability:
- [net-tick] gains resend/s nak-out/s nak-in/s rej-in/s dup-drop/s parked/s
  reclaim/s cache= nakset= - TransportStats window deltas mirroring the
  acks/s cumulative-delta pattern, plus the two instantaneous depths (the
  unbounded-like-retail sent-packet cache watchdog and the inbound NAK set).
  TransportStats gains RejectsReceived (inbound RejectRetransmit packets).
  Counters increment unconditionally; every string is behind
  NetDiagnostics.ProbeNet (Code Structure Rule 5).
- WorldSession.Dispose emits one cumulative [net-final] totals line so the
  loss gate asserts exact counters instead of reconstructing them from
  rounded per-second rates.
- LinkStatusSnapshot.PacketLossPercentage is deliberately NOT wired: filed
  #261 - retail's CLinkStatusAverages formula
  (LinkStatusHolder::GetPacketLossPercentage @ 0x00411370) must be located
  first; inventing a ratio is forbidden.

N4-review F3 fold-in:
- Fresh reliable sends stamp Header.Iteration = the session iteration
  through the same shared retail header build already cited for Time (N3)
  and the N4 control packets: FlowQueue::TransmitNewPackets @ 0x00547A60,
  the stack build at 0x00547A84/0x00547AA8. The control-header rule now
  holds across all three send shapes (fresh reliable, ack, NAK). ACE reads
  neither Time nor Iteration inbound (campaign section 3) - wire-safe, and
  resends keep the stamp verbatim per the N1 rebuild rule.

Loss injection (Transport/LossyTransportDecorator):
- IWorldSessionTransport wrapper with deterministic seeded per-direction
  loss. Config via NetDiagnostics typed env properties read once:
  ACDREAM_NET_DROP_PCT (0 = off = default), ACDREAM_NET_DROP_SEED (default
  1), ACDREAM_NET_DROP_DIR (out|in|both, default both).
- Arming gate: NOTHING drops in either direction until the decorator has
  FORWARDED the first ENCRYPTED outbound datagram - parse-free check on
  length > 20 with EncryptedChecksum set in the LE flags word at bytes
  4..8. The cleartext handshake always survives and the arming datagram is
  never a casualty; handshake-loss testing belongs to N6's ConnectResponse
  0.333 s retransmit.
- Structurally absent at 0%: WrapIfConfigured returns the raw transport -
  WorldSession's default factory is the only production seam and a normal
  run never constructs the decorator.

Root-cause fix the gate immediately exposed:
- The logoff-confirmation wait in Dispose processed inbound datagrams but
  never pumped the transport, so a lost S2C logoff confirmation was
  gap-detected but its healing NAK never went out. Retail's pump
  (Client::UseTime @ 0x00411C40 -> PacketController::UseTime @ 0x005410D0)
  runs until LogOffServer; the wait now sweeps per processed datagram,
  making the logoff wait the third covered blocking pump (after Tick and
  the handshake loops). A lost C2S logoff REQUEST remains unrecoverable by
  ACE design (arrival-driven NAK; a quiet client is never NAKed - campaign
  section 3 row 1), recorded in the gate header.

Gates:
- tools/run-connected-loss-gate.ps1 (-DropPct 2 -Seed 1): PASS vs local
  ACE - the first automated observation of packet loss in project history.
  Decorator ledger: dropped out=3 in=10 of forwarded out=183 in=496.
  [net-final] resends=2 nak-in=2 nak-out=6 rej-in=0 acks-out=114
  acks-in=119 dup-drop=0 sanity-drop=0 cksum-fail=0 parked=9 reclaimed=0
  uncached-nak=0 cache=1 nakset=0. Every injected loss healed: both
  ACE-driven C2S resend recovery (nak-in=2 -> resends=2) and client-driven
  S2C NAK recovery (parked=9 -> nak-out=6) fired on a real connected
  route, all six checkpoints validated, graceful logout confirmed, ACE
  recorded the transport Disconnect.
- tools/run-connected-world-lifecycle-gate.ps1 (decorator absent): PASS -
  zero behavior change on the no-loss baseline; the gate now defensively
  clears the drop env vars.
- Core.Net Release: 747/747 (737 + 10 N5: decorator determinism/direction/
  arming/structural-absence/env parsing, the 5% seeded WorldSession lossy
  lifecycle with zero message loss both ways + ACE Headroom 256, the
  [net-tick] field pins, the Iteration stamps).
- Full solution Release: 9,763 passed / 5 skipped / 0 failed.

Test-fixture note: FakeAceTransport gains AutoAdvanceOnBlockingReceive so
virtual time can move during the blocking Connect()/EnterWorld() pumps -
with the clock frozen there, a dropped handshake-window datagram could
never be NAK-healed (a fixture artifact, not a transport property).

Campaign section 9 ledger row added (SHA recorded at N6 kickoff).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Erik 2026-07-29 16:26:06 +02:00
parent 396838bb40
commit 4e290f00d8
14 changed files with 1629 additions and 27 deletions

View file

@ -256,5 +256,5 @@ verbatim in each implementer prompt.
| N2 | complete | `46d209d0`**Fable review PASS** (tracker/codec-split/admission verified pre-commit against retail rules; gates lifecycle+nine-stop both PASS) | Inbound sequence-aligned ISAAC + NAK set (`Transport/InboundSequenceTracker`: watermark, sanity window, duplicate/parked-key path, sequence-ordered gap-walk pre-draw, verify-failure re-park, RejectRetransmit abandonment — `ProcessNewSeqNum @ 0x00544690`, `ProcessNewestSeqNum @ 0x00541930`, `SeqIDSanityCheck @ 0x00543A20`, `AddNakked @ 0x00549240`, `HandleEmptyAck @ 0x005448F0`); `PacketCodec` split into keystream-free `TryParseBorrowed` + `VerifyChecksum` (`TryDecodeBorrowed` deleted); `RejectRetransmit` ids exposed on both optional-header decoders; `ReliableTransport` now owns both keystreams; stats gained `InboundDupsDropped`/`InboundSanityDrops`/`ChecksumFailures`/`KeysParked`. Watermark init 1 is the AD-50 ACE adaptation (retail zero-init vs ACE's re-prime dance — first encrypted S2C is sequence 2; holtburger api.rs:30 agrees); pinned by the clean-lifecycle conformance test (zero NAKs, min encrypted S2C sequence == 2). 716 Core.Net tests green. **N3/N4 handoff note:** the interim per-packet reflex ack acks the ARRIVING sequence even while a gap is parked, so ACE prunes the lost id from its S2C cache (`AcknowledgeSequence` strictly-below) before N4 can NAK it — message-level recovery of a real loss needs N3's retail NAK-xor-ack sweep (§2.3's mutual exclusivity is load-bearing). Also noted for N4: ACE's `RejectRetransmit` consumes a fresh CLEARTEXT sequence via FlushPackets (no keystream word), so the client's gap walk parks a word for an id that never had one server-side — a real retail-vs-ACE incompatibility to resolve in N4's design (retail never assigns new sequences to cleartext). |
| N3 | complete | `0265cc42` — Opus review PASS (advisories folded into N4: transitional-state wording, SharedInit citation, pump-order wording, control-packet Time/Iteration rule, stale budget-break comment) | AckNakScheduler + 2.0 s cumulative ack (`Transport/AckNakScheduler`): ONE shared timestamp (`ReceiverData::timeStamp_` @ +0x10) arbitrating NAK-xor-ack per sweep (`ClientNet::ProcessConnection @ 0x00545450`); the ack is one cleartext exact-flags `AckSequence` carrying the tracker's `highestIDReceived_` behind the >= 2.0 s gate (`SharedNet::EnqueuePak @ 0x00543B10` — the binary's only 0x4000 construction site), armed at connection birth (`ReceiverData::Init @ 0x00548EF0`). The Phase 4.9 per-packet reflex ack and `WorldSession.SendAck` are DELETED; the `[net-tick]` acks/s probe now reads `Stats.AcksSent`. Sweep order per `FlowQueue::Empty @ 0x00548A20`: interval clock, NAK/ack arbitration, pending resends, prune. **N3 transitional state (closed by N4):** a non-empty NAK set suppressed the ack and emitted NOTHING — the exposure was real even on loopback, just low-probability: one receive-buffer drop parks an id, every later sweep takes the silent NAK branch, acks stop (witness: `[net-tick] acks/s=0`), and ACE disconnects the quiet session at its 60 s timeout. N4 completed the branch. **N1 advisory retired (fold-in):** fresh reliable sends now stamp `Header.Time` = the current interval id (`FlowQueue::TransmitNewPackets @ 0x00547A60`, header build at 0x00547A84); ACE ignores inbound `Header.Time`, so the wire is unaffected. New `WorldSession.TransportClockSource` seam drives the gate on virtual time. 723 Core.Net tests green (keepalive property proven: a quiet session's 2 s acks refresh ACE's 60 s deadline across a 120 s virtual horizon; storm collapse: a 50-packet flood → ONE ack; the model accepts the reused-sequence ack without advancing its watermark). Connected lifecycle + canonical nine-stop gates PASS. |
| N4 | complete | `852a59e3`**Opus review PASS** (reclaim invariant attacked from five angles, held; NAK fidelity verified to the x87 masks; AP-125 filed + F1 false-arithmetic wording + F5 ordinal sentinel fixed in the acceptance commit; F3 Iteration-on-fresh-sends folds into N5) | Client NAK emission + RejectRetransmit consumption. `AckNakScheduler` completes the NAK branch: one cleartext exact-flags `RequestRetransmit` per sweep behind the STRICT 0.6 s gate on the ONE shared timestamp (`SharedNet::EnqueueNaks @ 0x00543BD0` — the 0x41-mask x87 test at 0x00543C03 proceeds only on strictly-greater, contrast the ack's >=), body u32 count + ids ascending capped at 114 (`ReceiverData::GetNaks @ 0x005490C0`, cap 0x72), borrowed sequence, never an ack in a NAK sweep. Control-header rule decided for BOTH emissions: `Time` = interval id, `Iteration` = session iteration, per the shared retail header build (`FlowQueue::TransmitNewPackets @ 0x00547A60` @ 0x00547A84); ACE reads neither. THE design piece: the AD-51 reclaimed-word pool in `InboundSequenceTracker` closes the ACE cleartext-reject keystream hazard the N2 ledger row recorded — ACE's `RejectRetransmit` consumes a fresh cleartext sequence with NO keystream word, so the gap walk mis-parks a word for it and the whole inbound stream runs one word ahead. On a VALIDATED cleartext reject (`WorldSession` calls `OnCleartextRejectSequence` post-checksum), the tracker removes the mis-park, bubble-shifts every later-drawn parked word down one position (per-word draw ordinals; ascending id ⇔ ascending draw order), and pools the excess for the next fresh draws, consumed lowest-draw-order-first — exact for any number of interleaved rejects in any arrival order (a plain FIFO is NOT: reject-after-higher-arrival crosses the parked chain, and dual out-of-order rejects pool out of draw order — both pinned by tests). Reject BODY ids keep N2's discard (word-bearing server-side, consumed-in-place). N3 advisories all folded: honest transitional wording (above), `ReceiverData::SharedInit @ 0x00548EF0` (from `Init @ 0x00548FA0`) citation, `FlowQueue::Empty` pump-order comment (TransmitNaks → TransmitAcks → TransmitNewPackets, interval increment LAST @ 0x00548A9D; our clock-first order is cosmetic vs ACE), the Time/Iteration rule, and the stale `WorldSession` budget-break comment. Gate arithmetic hardened: gate ticks now round (0.6 has no exact double; truncation opened the strict gate AT the boundary). 737 Core.Net tests green, including: strict-gate boundary, shared-timestamp both directions, NAK-suppresses-ack, full wire-shape + 114-cap pins, model-served retransmission round trip, five tracker reclaim proofs, the 130-s virtual prune → fresh-sequence reject system test (victim abandoned, later traffic decodes, pool drains to zero), 10 s long-loss (NAKs on the gate cadence, zero acks, heal inside the window), and the capstone soak: 2% seeded bidirectional loss × 10,000 messages → zero message loss both ways, ACE crypto headroom 256 at convergence with a ≥250 no-erosion floor mid-flight, NAK set / reclaim pool / pending resends / ACE out-of-order buffer all zero, cache at the single watermark entry (retail Flush prunes STRICTLY below the ack). Soak notes: ACE never NAKs a quiet client (§3 row 1), so convergence keeps a C2S trickle flowing — a real idle-client tail loss heals only on the next action, an ACE constraint outside N4's scope. |
| N5 | pending | — | |
| N5 | complete | SHA recorded at N6 kickoff | Loss observability + the LossyTransportDecorator + the connected loss gate — the permanent removal of the loopback blindness (§1). `[net-tick]` gains `resend/s nak-out/s nak-in/s rej-in/s dup-drop/s parked/s reclaim/s cache= nakset=` (window deltas mirroring acks/s; `TransportStats` gains `RejectsReceived`; string work probe-gated, counters unconditional) and `WorldSession.Dispose` emits one cumulative `[net-final]` totals line so the gate asserts exact counters, not rounded rates. N4-review F3 folded: fresh reliable sends stamp `Iteration` = the session iteration through the same shared retail header build already cited for `Time` and the N4 control packets (`FlowQueue::TransmitNewPackets @ 0x00547A60`, the build at 0x00547A84/0x00547AA8) — the control-header rule now holds across all three send shapes; ACE reads neither field inbound. `Transport/LossyTransportDecorator`: deterministic seeded per-direction loss (`ACDREAM_NET_DROP_PCT`/`_SEED`/`_DIR` via `NetDiagnostics` typed properties, Rule 5), armed only after the first ENCRYPTED outbound datagram is forwarded (parse-free flags-word check — the cleartext handshake always survives; handshake loss belongs to N6), structurally absent at 0% (`WrapIfConfigured` returns the raw transport; the default factory is the only production seam). The logoff-confirmation wait now runs the transport sweep — retail's pump (`Client::UseTime @ 0x00411C40`) never stops before `LogOffServer`, and the loss gate exposed that a dropped S2C confirmation was gap-detected but never NAKed during `Dispose`. `tools/run-connected-loss-gate.ps1` (default 2%/seed 1) runs the standard lifecycle route through the decorator vs local ACE and FAILS unless `[net-final]` shows resends>0 OR nak-out>0 OR nak-in>0 AND the decorator's own dropped ledger is non-zero — a loss gate that never dropped proves nothing, asserted explicitly. The lifecycle gate defensively clears the drop vars (decorator-absent baseline). **First loss-observing gate evidence (2026-07-29, 2%/seed 1, local ACE):** decorator dropped out=3 in=10 of forwarded out=183 in=496; `[net-final] resends=2 nak-in=2 nak-out=6 rej-in=0 acks-out=114 acks-in=119 dup-drop=0 sanity-drop=0 cksum-fail=0 parked=9 reclaimed=0 uncached-nak=0 cache=1 nakset=0` — both recovery directions fired on a real connected route (ACE NAK → cached resend; client gap-walk park → NAK → ACE retransmit), all six checkpoints validated, graceful logout confirmed, ACE recorded the transport Disconnect, RESULT=PASS. The gate immediately paid for itself: it exposed that the Dispose logoff-confirmation wait processed inbound but never swept the transport, so a lost S2C confirmation could be gap-detected yet never NAKed — fixed by running the sweep in that third blocking pump (retail's `Client::UseTime @ 0x00411C40` pump runs until `LogOffServer`). Known tail caveat recorded in the gate header: a drop landing on the single-shot logoff request or transport Disconnect (~pct each) is unrecoverable by ACE's arrival-driven NAK design (§3 row 1) — rerun with another seed, never widen teardown tolerances. #261 filed for `LinkStatusSnapshot.PacketLossPercentage` (retail `CLinkStatusAverages` formula required; inventing a ratio forbidden). 747 Core.Net tests green (decorator determinism/direction/arming/structural-absence, the 5% seeded WorldSession lossy lifecycle with zero message loss + Headroom 256, `[net-tick]` field pins, Iteration-stamp pins). |
| N6 | pending | — | |