feat(net): N5 - loss observability, lossy decorator, the connected loss gate
Campaign N Slice N5 (docs/plans/2026-07-29-network-transport-campaign.md section 8 rung 3): the permanent removal of the loopback blindness that let #260 ship. Local ACE never drops a datagram, so every historical connected gate was structurally incapable of exercising the N1-N4 recovery machinery; from this slice on, tools/run-connected-loss-gate.ps1 runs the standard lifecycle route through deterministic seeded loss and passes only on proven non-zero recovery. Observability: - [net-tick] gains resend/s nak-out/s nak-in/s rej-in/s dup-drop/s parked/s reclaim/s cache= nakset= - TransportStats window deltas mirroring the acks/s cumulative-delta pattern, plus the two instantaneous depths (the unbounded-like-retail sent-packet cache watchdog and the inbound NAK set). TransportStats gains RejectsReceived (inbound RejectRetransmit packets). Counters increment unconditionally; every string is behind NetDiagnostics.ProbeNet (Code Structure Rule 5). - WorldSession.Dispose emits one cumulative [net-final] totals line so the loss gate asserts exact counters instead of reconstructing them from rounded per-second rates. - LinkStatusSnapshot.PacketLossPercentage is deliberately NOT wired: filed #261 - retail's CLinkStatusAverages formula (LinkStatusHolder::GetPacketLossPercentage @ 0x00411370) must be located first; inventing a ratio is forbidden. N4-review F3 fold-in: - Fresh reliable sends stamp Header.Iteration = the session iteration through the same shared retail header build already cited for Time (N3) and the N4 control packets: FlowQueue::TransmitNewPackets @ 0x00547A60, the stack build at 0x00547A84/0x00547AA8. The control-header rule now holds across all three send shapes (fresh reliable, ack, NAK). ACE reads neither Time nor Iteration inbound (campaign section 3) - wire-safe, and resends keep the stamp verbatim per the N1 rebuild rule. Loss injection (Transport/LossyTransportDecorator): - IWorldSessionTransport wrapper with deterministic seeded per-direction loss. Config via NetDiagnostics typed env properties read once: ACDREAM_NET_DROP_PCT (0 = off = default), ACDREAM_NET_DROP_SEED (default 1), ACDREAM_NET_DROP_DIR (out|in|both, default both). - Arming gate: NOTHING drops in either direction until the decorator has FORWARDED the first ENCRYPTED outbound datagram - parse-free check on length > 20 with EncryptedChecksum set in the LE flags word at bytes 4..8. The cleartext handshake always survives and the arming datagram is never a casualty; handshake-loss testing belongs to N6's ConnectResponse 0.333 s retransmit. - Structurally absent at 0%: WrapIfConfigured returns the raw transport - WorldSession's default factory is the only production seam and a normal run never constructs the decorator. Root-cause fix the gate immediately exposed: - The logoff-confirmation wait in Dispose processed inbound datagrams but never pumped the transport, so a lost S2C logoff confirmation was gap-detected but its healing NAK never went out. Retail's pump (Client::UseTime @ 0x00411C40 -> PacketController::UseTime @ 0x005410D0) runs until LogOffServer; the wait now sweeps per processed datagram, making the logoff wait the third covered blocking pump (after Tick and the handshake loops). A lost C2S logoff REQUEST remains unrecoverable by ACE design (arrival-driven NAK; a quiet client is never NAKed - campaign section 3 row 1), recorded in the gate header. Gates: - tools/run-connected-loss-gate.ps1 (-DropPct 2 -Seed 1): PASS vs local ACE - the first automated observation of packet loss in project history. Decorator ledger: dropped out=3 in=10 of forwarded out=183 in=496. [net-final] resends=2 nak-in=2 nak-out=6 rej-in=0 acks-out=114 acks-in=119 dup-drop=0 sanity-drop=0 cksum-fail=0 parked=9 reclaimed=0 uncached-nak=0 cache=1 nakset=0. Every injected loss healed: both ACE-driven C2S resend recovery (nak-in=2 -> resends=2) and client-driven S2C NAK recovery (parked=9 -> nak-out=6) fired on a real connected route, all six checkpoints validated, graceful logout confirmed, ACE recorded the transport Disconnect. - tools/run-connected-world-lifecycle-gate.ps1 (decorator absent): PASS - zero behavior change on the no-loss baseline; the gate now defensively clears the drop env vars. - Core.Net Release: 747/747 (737 + 10 N5: decorator determinism/direction/ arming/structural-absence/env parsing, the 5% seeded WorldSession lossy lifecycle with zero message loss both ways + ACE Headroom 256, the [net-tick] field pins, the Iteration stamps). - Full solution Release: 9,763 passed / 5 skipped / 0 failed. Test-fixture note: FakeAceTransport gains AutoAdvanceOnBlockingReceive so virtual time can move during the blocking Connect()/EnterWorld() pumps - with the clock frozen there, a dropped handshake-window datagram could never be NAK-healed (a fixture artifact, not a transport property). Campaign section 9 ledger row added (SHA recorded at N6 kickoff). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Erik
parent
396838bb40
commit
4e290f00d8
14 changed files with 1629 additions and 27 deletions
|
|
@ -67,10 +67,13 @@ internal sealed class NetClientWorldSessionTransport(IPEndPoint remote)
|
|||
/// </code>
|
||||
///
|
||||
/// <para>
|
||||
/// <b>Still deferred:</b> inbound sequence-aligned ISAAC + client NAK
|
||||
/// emission (Campaign N slices N2/N4) and unsolicited-disconnect recovery.
|
||||
/// The outbound sent-packet cache + resend on server NAK (N1), ACKs, world
|
||||
/// updates, chat, and retail-ordered graceful logout are live.
|
||||
/// <b>Still deferred:</b> unsolicited-disconnect recovery and the optional
|
||||
/// N6 handshake hardening (ConnectResponse 0.333 s retransmit). The full
|
||||
/// Campaign N reliable transport is live in both directions: outbound
|
||||
/// sent-packet cache + resend on server NAK (N1), inbound sequence-aligned
|
||||
/// ISAAC + NAK set (N2), the retail 2.0 s cumulative-ack sweep (N3), client
|
||||
/// NAK emission + RejectRetransmit reclaim (N4), and the N5 loss
|
||||
/// observability + deterministic loss injection seam.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class WorldSession : IDisposable
|
||||
|
|
@ -761,7 +764,13 @@ public sealed class WorldSession : IDisposable
|
|||
public WorldSession(IPEndPoint serverLogin)
|
||||
: this(
|
||||
serverLogin,
|
||||
static endpoint => new NetClientWorldSessionTransport(endpoint))
|
||||
// N5: ACDREAM_NET_DROP_PCT > 0 wraps the socket transport in the
|
||||
// deterministic LossyTransportDecorator (the connected loss
|
||||
// gate's injection point). At the default 0 the decorator is
|
||||
// structurally absent — WrapIfConfigured returns the raw
|
||||
// transport and never constructs the wrapper.
|
||||
static endpoint => LossyTransportDecorator.WrapIfConfigured(
|
||||
new NetClientWorldSessionTransport(endpoint)))
|
||||
{
|
||||
}
|
||||
|
||||
|
|
@ -1124,6 +1133,16 @@ public sealed class WorldSession : IDisposable
|
|||
private int _probeBudgetBreaks;
|
||||
private int _probeSendWindow;
|
||||
private long _probeAckSeenTotal;
|
||||
// N5 loss-observability window baselines — the cumulative TransportStats
|
||||
// value each counter had when the probe last printed, mirroring
|
||||
// _probeAckSeenTotal. Only touched when NetDiagnostics.ProbeNet is set.
|
||||
private long _probeResendSeenTotal;
|
||||
private long _probeNakOutSeenTotal;
|
||||
private long _probeNakInSeenTotal;
|
||||
private long _probeRejInSeenTotal;
|
||||
private long _probeDupDropSeenTotal;
|
||||
private long _probeParkedSeenTotal;
|
||||
private long _probeReclaimSeenTotal;
|
||||
// Probe-owned queue depth: the SingleReader channel's Reader.Count
|
||||
// throws NotSupportedException, so the net thread increments on
|
||||
// enqueue and the frame thread decrements on dequeue instead.
|
||||
|
|
@ -1158,23 +1177,100 @@ public sealed class WorldSession : IDisposable
|
|||
double windowSeconds = (double)windowTicks / Stopwatch.Frequency;
|
||||
double maxGapMs = _probeMaxGapTicks * 1000.0 / Stopwatch.Frequency;
|
||||
int sends = Interlocked.Exchange(ref _probeSendWindow, 0);
|
||||
long ackTotal = _transport?.Stats.AcksSent ?? 0;
|
||||
long acks = ackTotal - _probeAckSeenTotal;
|
||||
_probeAckSeenTotal = ackTotal;
|
||||
Console.WriteLine(
|
||||
$"[net-tick] in/s={_probeProcessedWindow / windowSeconds:F0}"
|
||||
+ $" q={Volatile.Read(ref _probeInboundDepth)}"
|
||||
+ $" budget-breaks={_probeBudgetBreaks}"
|
||||
+ $" maxgap={maxGapMs:F0}ms"
|
||||
+ $" out/s={sends / windowSeconds:F0}"
|
||||
+ $" acks/s={acks / windowSeconds:F0}"
|
||||
+ $" st={CurrentState}");
|
||||
ReliableTransport? transport = _transport;
|
||||
TransportStats? stats = transport?.Stats;
|
||||
long acks = WindowDelta(stats?.AcksSent ?? 0, ref _probeAckSeenTotal);
|
||||
// N5: reliable-transport window deltas (same cumulative-delta shape
|
||||
// as acks/s) + the two instantaneous depths. The counters increment
|
||||
// unconditionally in TransportStats; only this string work is
|
||||
// probe-gated.
|
||||
long resends = WindowDelta(
|
||||
stats?.ResendsSent ?? 0, ref _probeResendSeenTotal);
|
||||
long naksOut = WindowDelta(
|
||||
stats?.NaksSent ?? 0, ref _probeNakOutSeenTotal);
|
||||
long naksIn = WindowDelta(
|
||||
stats?.NakRequestsReceived ?? 0, ref _probeNakInSeenTotal);
|
||||
long rejsIn = WindowDelta(
|
||||
stats?.RejectsReceived ?? 0, ref _probeRejInSeenTotal);
|
||||
long dupDrops = WindowDelta(
|
||||
stats?.InboundDupsDropped ?? 0, ref _probeDupDropSeenTotal);
|
||||
long parked = WindowDelta(
|
||||
stats?.KeysParked ?? 0, ref _probeParkedSeenTotal);
|
||||
long reclaimed = WindowDelta(
|
||||
stats?.RejectWordsReclaimed ?? 0, ref _probeReclaimSeenTotal);
|
||||
Console.WriteLine(FormatNetTickLine(
|
||||
windowSeconds,
|
||||
_probeProcessedWindow,
|
||||
Volatile.Read(ref _probeInboundDepth),
|
||||
_probeBudgetBreaks,
|
||||
maxGapMs,
|
||||
sends,
|
||||
acks,
|
||||
resends,
|
||||
naksOut,
|
||||
naksIn,
|
||||
rejsIn,
|
||||
dupDrops,
|
||||
parked,
|
||||
reclaimed,
|
||||
stats?.CacheDepth ?? 0,
|
||||
transport?.Inbound.NakCount ?? 0,
|
||||
CurrentState));
|
||||
_probeWindowStartTs = tickStartTs;
|
||||
_probeMaxGapTicks = 0;
|
||||
_probeProcessedWindow = 0;
|
||||
_probeBudgetBreaks = 0;
|
||||
}
|
||||
|
||||
private static long WindowDelta(long cumulative, ref long seenTotal)
|
||||
{
|
||||
long delta = cumulative - seenTotal;
|
||||
seenTotal = cumulative;
|
||||
return delta;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The <c>[net-tick]</c> line shape, extracted so the N5 field extension
|
||||
/// is string-assertable without a wall-clock window. <c>cache</c> and
|
||||
/// <c>nakset</c> are instantaneous depths (the sent-packet cache is the
|
||||
/// unbounded-like-retail watchdog value — campaign §4); every other
|
||||
/// transport field is a per-second rate over the probe window.
|
||||
/// </summary>
|
||||
internal static string FormatNetTickLine(
|
||||
double windowSeconds,
|
||||
int processed,
|
||||
int queueDepth,
|
||||
int budgetBreaks,
|
||||
double maxGapMs,
|
||||
int sends,
|
||||
long acks,
|
||||
long resends,
|
||||
long naksOut,
|
||||
long naksIn,
|
||||
long rejsIn,
|
||||
long dupDrops,
|
||||
long parked,
|
||||
long reclaimed,
|
||||
int cacheDepth,
|
||||
int nakSetDepth,
|
||||
State state) =>
|
||||
$"[net-tick] in/s={processed / windowSeconds:F0}"
|
||||
+ $" q={queueDepth}"
|
||||
+ $" budget-breaks={budgetBreaks}"
|
||||
+ $" maxgap={maxGapMs:F0}ms"
|
||||
+ $" out/s={sends / windowSeconds:F0}"
|
||||
+ $" acks/s={acks / windowSeconds:F0}"
|
||||
+ $" resend/s={resends / windowSeconds:F0}"
|
||||
+ $" nak-out/s={naksOut / windowSeconds:F0}"
|
||||
+ $" nak-in/s={naksIn / windowSeconds:F0}"
|
||||
+ $" rej-in/s={rejsIn / windowSeconds:F0}"
|
||||
+ $" dup-drop/s={dupDrops / windowSeconds:F0}"
|
||||
+ $" parked/s={parked / windowSeconds:F0}"
|
||||
+ $" reclaim/s={reclaimed / windowSeconds:F0}"
|
||||
+ $" cache={cacheDepth}"
|
||||
+ $" nakset={nakSetDepth}"
|
||||
+ $" st={state}";
|
||||
|
||||
/// <summary>
|
||||
/// Pure, testable decision for the per-frame inbound bound: stop draining only when
|
||||
/// in-world AND the elapsed Stopwatch ticks have reached the budget. Extracted so the
|
||||
|
|
@ -1446,6 +1542,7 @@ public sealed class WorldSession : IDisposable
|
|||
// consumed-in-place.
|
||||
if ((serverHeader.Flags & PacketHeaderFlags.RejectRetransmit) != 0)
|
||||
{
|
||||
transport.Stats.RejectsReceived++;
|
||||
if (packet.Optional.RejectRetransmitCount > 0)
|
||||
{
|
||||
transport.Inbound.OnRejectRetransmit(
|
||||
|
|
@ -2569,6 +2666,29 @@ public sealed class WorldSession : IDisposable
|
|||
}
|
||||
_netCancel.Dispose();
|
||||
|
||||
// N5: one cumulative TransportStats summary so the connected loss
|
||||
// gate asserts exact totals instead of reconstructing them from the
|
||||
// rounded per-second [net-tick] rates.
|
||||
if (NetDiagnostics.ProbeNet && _transport is { } finalTransport)
|
||||
{
|
||||
TransportStats finalStats = finalTransport.Stats;
|
||||
Console.WriteLine(
|
||||
$"[net-final] resends={finalStats.ResendsSent}"
|
||||
+ $" nak-in={finalStats.NakRequestsReceived}"
|
||||
+ $" nak-out={finalStats.NaksSent}"
|
||||
+ $" rej-in={finalStats.RejectsReceived}"
|
||||
+ $" acks-out={finalStats.AcksSent}"
|
||||
+ $" acks-in={finalStats.AcksConsumed}"
|
||||
+ $" dup-drop={finalStats.InboundDupsDropped}"
|
||||
+ $" sanity-drop={finalStats.InboundSanityDrops}"
|
||||
+ $" cksum-fail={finalStats.ChecksumFailures}"
|
||||
+ $" parked={finalStats.KeysParked}"
|
||||
+ $" reclaimed={finalStats.RejectWordsReclaimed}"
|
||||
+ $" uncached-nak={finalStats.UncachedNakIds}"
|
||||
+ $" cache={finalStats.CacheDepth}"
|
||||
+ $" nakset={finalTransport.Inbound.NakCount}");
|
||||
}
|
||||
|
||||
// N1: return every rented sent-packet cache buffer before the
|
||||
// socket goes away.
|
||||
_transport?.Dispose();
|
||||
|
|
@ -2674,6 +2794,23 @@ public sealed class WorldSession : IDisposable
|
|||
ProcessDatagram(
|
||||
datagram.Memory,
|
||||
dispatchWorldEvents: false);
|
||||
// N5: keep the transport pumped while waiting for the
|
||||
// logoff confirmation — retail's frame pump
|
||||
// (Client::UseTime @ 0x00411C40 →
|
||||
// PacketController::UseTime @ 0x005410D0) keeps running
|
||||
// until LogOffServer, so the logoff wait is the third
|
||||
// blocking pump the sweep must cover (after Tick and the
|
||||
// handshake loops). The connected loss gate exposed the
|
||||
// gap: without a sweep here, a lost S2C confirmation can
|
||||
// be gap-detected (ACE's next sequenced packet arrives and
|
||||
// parks a key) but the NAK that would heal it never goes
|
||||
// out, and the graceful logout dies at the 35 s timeout.
|
||||
// ACE's 2 s ack cadence guarantees arrivals to hang this
|
||||
// callback on. (A lost C2S logoff REQUEST remains
|
||||
// unrecoverable against ACE — its NAK is arrival-driven
|
||||
// and a quiet client is never NAKed, campaign §3 row 1 —
|
||||
// the same idle-tail constraint the N4 soak recorded.)
|
||||
SweepTransport();
|
||||
return Volatile.Read(ref _characterLogOffConfirmed) != 0;
|
||||
},
|
||||
ReturnInboundDatagram);
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue