feat(net): N5 - loss observability, lossy decorator, the connected loss gate
Campaign N Slice N5 (docs/plans/2026-07-29-network-transport-campaign.md section 8 rung 3): the permanent removal of the loopback blindness that let #260 ship. Local ACE never drops a datagram, so every historical connected gate was structurally incapable of exercising the N1-N4 recovery machinery; from this slice on, tools/run-connected-loss-gate.ps1 runs the standard lifecycle route through deterministic seeded loss and passes only on proven non-zero recovery. Observability: - [net-tick] gains resend/s nak-out/s nak-in/s rej-in/s dup-drop/s parked/s reclaim/s cache= nakset= - TransportStats window deltas mirroring the acks/s cumulative-delta pattern, plus the two instantaneous depths (the unbounded-like-retail sent-packet cache watchdog and the inbound NAK set). TransportStats gains RejectsReceived (inbound RejectRetransmit packets). Counters increment unconditionally; every string is behind NetDiagnostics.ProbeNet (Code Structure Rule 5). - WorldSession.Dispose emits one cumulative [net-final] totals line so the loss gate asserts exact counters instead of reconstructing them from rounded per-second rates. - LinkStatusSnapshot.PacketLossPercentage is deliberately NOT wired: filed #261 - retail's CLinkStatusAverages formula (LinkStatusHolder::GetPacketLossPercentage @ 0x00411370) must be located first; inventing a ratio is forbidden. N4-review F3 fold-in: - Fresh reliable sends stamp Header.Iteration = the session iteration through the same shared retail header build already cited for Time (N3) and the N4 control packets: FlowQueue::TransmitNewPackets @ 0x00547A60, the stack build at 0x00547A84/0x00547AA8. The control-header rule now holds across all three send shapes (fresh reliable, ack, NAK). ACE reads neither Time nor Iteration inbound (campaign section 3) - wire-safe, and resends keep the stamp verbatim per the N1 rebuild rule. Loss injection (Transport/LossyTransportDecorator): - IWorldSessionTransport wrapper with deterministic seeded per-direction loss. Config via NetDiagnostics typed env properties read once: ACDREAM_NET_DROP_PCT (0 = off = default), ACDREAM_NET_DROP_SEED (default 1), ACDREAM_NET_DROP_DIR (out|in|both, default both). - Arming gate: NOTHING drops in either direction until the decorator has FORWARDED the first ENCRYPTED outbound datagram - parse-free check on length > 20 with EncryptedChecksum set in the LE flags word at bytes 4..8. The cleartext handshake always survives and the arming datagram is never a casualty; handshake-loss testing belongs to N6's ConnectResponse 0.333 s retransmit. - Structurally absent at 0%: WrapIfConfigured returns the raw transport - WorldSession's default factory is the only production seam and a normal run never constructs the decorator. Root-cause fix the gate immediately exposed: - The logoff-confirmation wait in Dispose processed inbound datagrams but never pumped the transport, so a lost S2C logoff confirmation was gap-detected but its healing NAK never went out. Retail's pump (Client::UseTime @ 0x00411C40 -> PacketController::UseTime @ 0x005410D0) runs until LogOffServer; the wait now sweeps per processed datagram, making the logoff wait the third covered blocking pump (after Tick and the handshake loops). A lost C2S logoff REQUEST remains unrecoverable by ACE design (arrival-driven NAK; a quiet client is never NAKed - campaign section 3 row 1), recorded in the gate header. Gates: - tools/run-connected-loss-gate.ps1 (-DropPct 2 -Seed 1): PASS vs local ACE - the first automated observation of packet loss in project history. Decorator ledger: dropped out=3 in=10 of forwarded out=183 in=496. [net-final] resends=2 nak-in=2 nak-out=6 rej-in=0 acks-out=114 acks-in=119 dup-drop=0 sanity-drop=0 cksum-fail=0 parked=9 reclaimed=0 uncached-nak=0 cache=1 nakset=0. Every injected loss healed: both ACE-driven C2S resend recovery (nak-in=2 -> resends=2) and client-driven S2C NAK recovery (parked=9 -> nak-out=6) fired on a real connected route, all six checkpoints validated, graceful logout confirmed, ACE recorded the transport Disconnect. - tools/run-connected-world-lifecycle-gate.ps1 (decorator absent): PASS - zero behavior change on the no-loss baseline; the gate now defensively clears the drop env vars. - Core.Net Release: 747/747 (737 + 10 N5: decorator determinism/direction/ arming/structural-absence/env parsing, the 5% seeded WorldSession lossy lifecycle with zero message loss both ways + ACE Headroom 256, the [net-tick] field pins, the Iteration stamps). - Full solution Release: 9,763 passed / 5 skipped / 0 failed. Test-fixture note: FakeAceTransport gains AutoAdvanceOnBlockingReceive so virtual time can move during the blocking Connect()/EnterWorld() pumps - with the clock frozen there, a dropped handshake-window datagram could never be NAK-healed (a fixture artifact, not a transport property). Campaign section 9 ledger row added (SHA recorded at N6 kickoff). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Erik
parent
396838bb40
commit
4e290f00d8
14 changed files with 1629 additions and 27 deletions
234
src/AcDream.Core.Net/Transport/LossyTransportDecorator.cs
Normal file
234
src/AcDream.Core.Net/Transport/LossyTransportDecorator.cs
Normal file
|
|
@ -0,0 +1,234 @@
|
|||
using System.Buffers.Binary;
|
||||
using System.Diagnostics;
|
||||
using System.Net;
|
||||
using AcDream.Core.Net.Packets;
|
||||
|
||||
namespace AcDream.Core.Net.Transport;
|
||||
|
||||
/// <summary>
|
||||
/// Campaign N Slice N5: deterministic seeded packet-loss injection around a
|
||||
/// real <see cref="IWorldSessionTransport"/> — the permanent removal of the
|
||||
/// loopback blindness that let #260 ship (local ACE never drops a datagram,
|
||||
/// so every historical connected gate was structurally incapable of
|
||||
/// exercising the N1–N4 recovery machinery). The connected loss gate
|
||||
/// (<c>tools/run-connected-loss-gate.ps1</c>) runs the standard lifecycle
|
||||
/// route through this decorator and passes only when the transport counters
|
||||
/// prove real loss was injected AND healed.
|
||||
///
|
||||
/// <para>
|
||||
/// <b>Configuration</b> comes from the <see cref="NetDiagnostics"/> typed
|
||||
/// env-var owner (Code Structure Rule 5 — read once at startup):
|
||||
/// <c>ACDREAM_NET_DROP_PCT</c> (0 = off), <c>ACDREAM_NET_DROP_SEED</c>, and
|
||||
/// <c>ACDREAM_NET_DROP_DIR</c>. <see cref="WrapIfConfigured"/> is the ONLY
|
||||
/// production entry point, and it returns the inner transport untouched when
|
||||
/// the percentage is zero — a normal run never constructs the decorator
|
||||
/// (structural absence, not an inert wrapper).
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// <b>The arming gate:</b> nothing is dropped in either direction until the
|
||||
/// decorator has FORWARDED the first ENCRYPTED outbound datagram — detected
|
||||
/// parse-free as length > 20 with <see
|
||||
/// cref="PacketHeaderFlags.EncryptedChecksum"/> set in the little-endian
|
||||
/// flags word at bytes 4..8. That first encrypted send only exists after
|
||||
/// ISAAC negotiation, so the cleartext handshake (LoginRequest →
|
||||
/// ConnectRequest → ConnectResponse) always completes intact and the arming
|
||||
/// datagram itself is never dropped. Handshake-loss testing is deliberately
|
||||
/// out of scope here — it belongs to N6's ConnectResponse 0.333 s
|
||||
/// retransmit (campaign doc §6).
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// <b>Determinism:</b> one <see cref="Random"/> per direction (outbound
|
||||
/// seeded with <c>seed</c>, inbound with <c>~seed</c> so the two streams
|
||||
/// differ), consumed only for droppable datagrams (post-arming, direction
|
||||
/// enabled), so a given seed yields an identical per-direction drop pattern
|
||||
/// for a given datagram sequence. Threading matches the transport contract:
|
||||
/// sends happen on the session's frame thread, receives on the single
|
||||
/// receive owner, so each PRNG is single-consumer; only the arming latch
|
||||
/// crosses the two.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
internal sealed class LossyTransportDecorator : IWorldSessionTransport
|
||||
{
|
||||
private readonly IWorldSessionTransport _inner;
|
||||
private readonly int _dropPercent;
|
||||
private readonly Random _outboundRandom;
|
||||
private readonly Random _inboundRandom;
|
||||
private readonly bool _dropOutbound;
|
||||
private readonly bool _dropInbound;
|
||||
private volatile bool _armed;
|
||||
|
||||
private int _outboundDropped;
|
||||
private int _inboundDropped;
|
||||
private int _outboundForwarded;
|
||||
private int _inboundForwarded;
|
||||
|
||||
/// <summary>Outbound datagrams eaten so far (diagnostic evidence for the
|
||||
/// loss gate's "the decorator actually dropped" assertion).</summary>
|
||||
public int OutboundDropped => Volatile.Read(ref _outboundDropped);
|
||||
|
||||
/// <summary>Inbound datagrams eaten so far.</summary>
|
||||
public int InboundDropped => Volatile.Read(ref _inboundDropped);
|
||||
|
||||
/// <summary>True once the first encrypted outbound datagram has been
|
||||
/// forwarded (the arming gate above).</summary>
|
||||
public bool IsArmed => _armed;
|
||||
|
||||
public LossyTransportDecorator(
|
||||
IWorldSessionTransport inner,
|
||||
int dropPercent,
|
||||
int seed,
|
||||
NetDropDirection direction)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(inner);
|
||||
ArgumentOutOfRangeException.ThrowIfLessThan(dropPercent, 1);
|
||||
ArgumentOutOfRangeException.ThrowIfGreaterThan(dropPercent, 100);
|
||||
|
||||
_inner = inner;
|
||||
_dropPercent = dropPercent;
|
||||
_outboundRandom = new Random(seed);
|
||||
_inboundRandom = new Random(~seed);
|
||||
_dropOutbound = direction is NetDropDirection.Out or NetDropDirection.Both;
|
||||
_dropInbound = direction is NetDropDirection.In or NetDropDirection.Both;
|
||||
Console.WriteLine(
|
||||
$"[net-loss] active pct={dropPercent} seed={seed} dir={direction}");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The production seam: wrap <paramref name="inner"/> only when
|
||||
/// <see cref="NetDiagnostics.NetDropPercent"/> is non-zero. At zero the
|
||||
/// decorator is never constructed — the default transport path is
|
||||
/// byte-for-byte the pre-N5 one.
|
||||
/// </summary>
|
||||
public static IWorldSessionTransport WrapIfConfigured(
|
||||
IWorldSessionTransport inner) =>
|
||||
NetDiagnostics.NetDropPercent > 0
|
||||
? new LossyTransportDecorator(
|
||||
inner,
|
||||
NetDiagnostics.NetDropPercent,
|
||||
NetDiagnostics.NetDropSeed,
|
||||
NetDiagnostics.NetDropDir)
|
||||
: inner;
|
||||
|
||||
// ---- outbound ----
|
||||
|
||||
public void Send(ReadOnlySpan<byte> datagram)
|
||||
{
|
||||
if (DropOutbound())
|
||||
return;
|
||||
_inner.Send(datagram);
|
||||
AfterOutboundForwarded(datagram);
|
||||
}
|
||||
|
||||
public void Send(IPEndPoint remote, ReadOnlySpan<byte> datagram)
|
||||
{
|
||||
if (DropOutbound())
|
||||
return;
|
||||
_inner.Send(remote, datagram);
|
||||
AfterOutboundForwarded(datagram);
|
||||
}
|
||||
|
||||
private bool DropOutbound()
|
||||
{
|
||||
if (!_armed || !_dropOutbound)
|
||||
return false;
|
||||
if (_outboundRandom.Next(100) >= _dropPercent)
|
||||
return false;
|
||||
Interlocked.Increment(ref _outboundDropped);
|
||||
return true;
|
||||
}
|
||||
|
||||
private void AfterOutboundForwarded(ReadOnlySpan<byte> datagram)
|
||||
{
|
||||
Interlocked.Increment(ref _outboundForwarded);
|
||||
if (_armed)
|
||||
return;
|
||||
// The arming gate: bytes 4..8 are the little-endian
|
||||
// PacketHeaderFlags word of the fixed 20-byte header. The first
|
||||
// forwarded datagram carrying EncryptedChecksum proves negotiation
|
||||
// completed; from the NEXT datagram on, drops are live.
|
||||
if (datagram.Length > PacketHeader.Size
|
||||
&& (BinaryPrimitives.ReadUInt32LittleEndian(datagram.Slice(4))
|
||||
& (uint)PacketHeaderFlags.EncryptedChecksum) != 0)
|
||||
{
|
||||
_armed = true;
|
||||
Console.WriteLine("[net-loss] armed");
|
||||
}
|
||||
}
|
||||
|
||||
// ---- inbound ----
|
||||
|
||||
public int Receive(
|
||||
Span<byte> destination,
|
||||
TimeSpan timeout,
|
||||
out IPEndPoint? from)
|
||||
{
|
||||
// The first inner call gets the caller's timeout verbatim, so the
|
||||
// no-drop path is behavior-identical to the undecorated transport.
|
||||
// Only a drop re-enters the loop with the remaining time.
|
||||
long deadline = Stopwatch.GetTimestamp()
|
||||
+ (long)(timeout.TotalSeconds * Stopwatch.Frequency);
|
||||
TimeSpan next = timeout;
|
||||
while (true)
|
||||
{
|
||||
int length = _inner.Receive(destination, next, out from);
|
||||
if (length < 0)
|
||||
return length; // inner timeout contract: -1
|
||||
if (!DropInbound())
|
||||
{
|
||||
Interlocked.Increment(ref _inboundForwarded);
|
||||
return length;
|
||||
}
|
||||
|
||||
double remainingMs =
|
||||
(deadline - Stopwatch.GetTimestamp())
|
||||
* 1000.0 / Stopwatch.Frequency;
|
||||
if (remainingMs < 1.0)
|
||||
{
|
||||
// Expired while eating datagrams. NetClient treats a 0 ms
|
||||
// socket timeout as INFINITE, so never pass ≤ 0 back down.
|
||||
from = null;
|
||||
return -1;
|
||||
}
|
||||
|
||||
next = TimeSpan.FromMilliseconds(remainingMs);
|
||||
}
|
||||
}
|
||||
|
||||
public async ValueTask<NetReceiveResult> ReceiveAsync(
|
||||
Memory<byte> destination,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
while (true)
|
||||
{
|
||||
NetReceiveResult result = await _inner
|
||||
.ReceiveAsync(destination, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
if (DropInbound())
|
||||
continue;
|
||||
Interlocked.Increment(ref _inboundForwarded);
|
||||
return result;
|
||||
}
|
||||
}
|
||||
|
||||
private bool DropInbound()
|
||||
{
|
||||
if (!_armed || !_dropInbound)
|
||||
return false;
|
||||
if (_inboundRandom.Next(100) >= _dropPercent)
|
||||
return false;
|
||||
Interlocked.Increment(ref _inboundDropped);
|
||||
return true;
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
Console.WriteLine(
|
||||
$"[net-loss] dropped out={OutboundDropped} in={InboundDropped}"
|
||||
+ $" forwarded out={Volatile.Read(ref _outboundForwarded)}"
|
||||
+ $" in={Volatile.Read(ref _inboundForwarded)}"
|
||||
+ $" armed={_armed}");
|
||||
_inner.Dispose();
|
||||
}
|
||||
}
|
||||
|
|
@ -58,6 +58,7 @@ internal sealed class OutboundFlowQueue : IDisposable
|
|||
private readonly SentPacketStore _store;
|
||||
private readonly ArrayPool<byte> _pool;
|
||||
private readonly ushort _sessionClientId;
|
||||
private readonly ushort _sessionIteration;
|
||||
|
||||
/// <summary>Wrap-safe sorted pending NAKed ids awaiting the next sweep
|
||||
/// (<c>FlowQueue::EnqueueAcks @ 0x005488E0</c> merge-insert).</summary>
|
||||
|
|
@ -83,6 +84,7 @@ internal sealed class OutboundFlowQueue : IDisposable
|
|||
public OutboundFlowQueue(
|
||||
IsaacRandom outboundIsaac,
|
||||
ushort sessionClientId,
|
||||
ushort sessionIteration,
|
||||
TransportClock clock,
|
||||
TransportStats stats,
|
||||
DatagramSendDelegate send,
|
||||
|
|
@ -97,6 +99,7 @@ internal sealed class OutboundFlowQueue : IDisposable
|
|||
|
||||
_outboundIsaac = outboundIsaac;
|
||||
_sessionClientId = sessionClientId;
|
||||
_sessionIteration = sessionIteration;
|
||||
_clock = clock;
|
||||
_stats = stats;
|
||||
_send = send;
|
||||
|
|
@ -118,12 +121,16 @@ internal sealed class OutboundFlowQueue : IDisposable
|
|||
/// it, THEN cache it (retail commits to the sent-packet store only after
|
||||
/// a successful send — <c>FlowQueue::TransmitNewPackets @ 0x00547C85</c>).
|
||||
/// Flags <c>BlobFragments|EncryptedChecksum</c>, session client id, one
|
||||
/// ISAAC word, and — the N3 fold-in of the N1 review advisory —
|
||||
/// <c>Time</c> = the current interval id: retail stamps
|
||||
/// <c>CurLocalInterval_.intervalID_</c> on every fresh packet
|
||||
/// (<c>FlowQueue::TransmitNewPackets @ 0x00547A60</c>, the header build
|
||||
/// at 0x00547A84). ACE never reads inbound <c>Header.Time</c>
|
||||
/// (campaign §3), so wire compatibility is unaffected.
|
||||
/// ISAAC word, and the retail control-header rule completed across all
|
||||
/// three send shapes (fresh reliable here; ack + NAK in
|
||||
/// <see cref="AckNakScheduler"/>): <c>Time</c> = the current interval id
|
||||
/// (the N3 fold-in of the N1 review advisory) and <c>Iteration</c> = the
|
||||
/// session iteration (the N5 fold-in of the N4 review advisory F3).
|
||||
/// Retail stamps both through one shared header build
|
||||
/// (<c>FlowQueue::TransmitNewPackets @ 0x00547A60</c>, the stack build
|
||||
/// at 0x00547A84/0x00547AA8 — <c>CurLocalInterval_.intervalID_</c> plus
|
||||
/// the receiver iteration). ACE reads neither field inbound (campaign
|
||||
/// §3), so wire compatibility is unaffected.
|
||||
/// </summary>
|
||||
public void SendGameMessage(
|
||||
ReadOnlySpan<byte> gameMessageBody,
|
||||
|
|
@ -147,6 +154,7 @@ internal sealed class OutboundFlowQueue : IDisposable
|
|||
| PacketHeaderFlags.EncryptedChecksum,
|
||||
Id = _sessionClientId,
|
||||
Time = _clock.IntervalId,
|
||||
Iteration = _sessionIteration,
|
||||
};
|
||||
int datagramLength = PacketCodec.FinalizeInPlace(
|
||||
header,
|
||||
|
|
|
|||
|
|
@ -54,6 +54,7 @@ internal sealed class ReliableTransport : IDisposable
|
|||
Outbound = new OutboundFlowQueue(
|
||||
outboundIsaac,
|
||||
sessionClientId,
|
||||
sessionIteration,
|
||||
Clock,
|
||||
Stats,
|
||||
send,
|
||||
|
|
|
|||
|
|
@ -62,6 +62,11 @@ internal sealed class TransportStats
|
|||
/// adaptation; always zero against a retail server).</summary>
|
||||
public long RejectWordsReclaimed;
|
||||
|
||||
/// <summary>N5: inbound packets carrying <c>RejectRetransmit</c> — the
|
||||
/// server abandoned ids we NAKed (its cache pruned them). The
|
||||
/// <c>rej-in/s</c> field of <c>[net-tick]</c>.</summary>
|
||||
public long RejectsReceived;
|
||||
|
||||
/// <summary>Live sent-packet cache depth — the N5 watchdog value
|
||||
/// (<c>cache=N</c> in <c>[net-tick]</c>; the cache is unbounded like
|
||||
/// retail's, so depth is the health signal, not a cap).</summary>
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue