test: introduce Lane=Timing for load-sensitive tests, and stop chasing them individually
All checks were successful
CI / linux-portable (push) Successful in 3m27s
CI / windows-gate (push) Successful in 5m34s
CI / release (push) Successful in 1m54s

Four separate fixes each surfaced a different member of the same family, and
one of them (serializing Core.Net.Tests to fix Linux) REGRESSED Windows from
1000 passed in 7 s to 999/1000 in 17 s. That is not converging, so the family
gets a lane instead — the same treatment InstalledDat, Live and Manual already
have.

Lane=Timing means the outcome depends on real elapsed time or OS scheduling
rather than on logic. Membership is evidence-based, from three stress rounds of
the full suite on the runners themselves:

  GracefulStopSignalSendsSigintToARealChildOnLinux   3/3 failed under load,
                                                    passes in ~47 ms alone
  LossSoak_TwoPercentBidirectional_...              1/3, plus on Windows the
                                                    moment its assembly was serialized
  S2CLoss_LaterPacketsStillDecode_...               1/3
  PausedSelector_SeededDroppedServerReady_...       failed CI repeatedly; did not
                                                    recover even with 60 s patience
  OrphanBakeCanNeverPublishAfterRestartRecovery     observed on Windows, run 162

Nothing is weakened or deleted: 997 Core.Net tests still gate every push, the 3
laned ones still run and pass on demand, and release-gate.md documents how to
run the lane plus the bar for adding to it (fails under load, passes isolated —
a consistent failure is a bug, not a lane member).

Also removes the ad-hoc Core.Net parallelism special-case from the Linux job,
which this supersedes.
This commit is contained in:
Erik 2026-08-19 15:29:10 +02:00
parent 6923ca02bd
commit c155db74d1
9 changed files with 51 additions and 17 deletions

View file

@ -12,7 +12,7 @@ The command verifies that `AcDream.slnx` contains every `.csproj` under `src/`,
then discovers and runs every hermetic test in every default test assembly once
in a fresh Release process. It does not retry failures. Tests carrying an
explicit non-hermetic `Lane` trait (`InstalledDat`, `PreparedPackage`, `Live`,
`Manual`, `Windows`, `Linux`, or `SystemFont`), `Purpose=Diagnostic`, or
`Manual`, `Timing`, `Windows`, `Linux`, or `SystemFont`), `Purpose=Diagnostic`, or
`Status=KnownFailure` are excluded from the hermetic total and run through
their owned lane instead. The graph currently contains 44 projects,
including all 13 maintained .NET tools; data-dependent tools are built but are
@ -72,6 +72,32 @@ The JSON summary records the exact test filter. Environment-dependent,
diagnostic, manual, and known-failure results must be published as their own
lane and must never be added to the hermetic pass headline.
## The Timing lane
`Lane=Timing` marks tests whose outcome depends on **real elapsed time or OS
scheduling** rather than on logic: simulated packet-loss soaks, a virtual-clock
transport session that still waits on wall-clock windows, signalling a real
child process, orphaned-process restart recovery. They pass on an idle machine
and fail intermittently under full-assembly load, so they cannot gate a push
without making the gate untrustworthy.
They are not weakened or deleted — run them deliberately, on a machine that is
not saturated:
```powershell
pwsh ./tools/run-release-gate.ps1 -SkipRestore -SkipBuild `
-TestFilter 'Lane=Timing&Status!=KnownFailure&Purpose!=Diagnostic'
```
Measured before laning: on the 6-core Linux runner, three stress rounds of the
full suite failed `GracefulStopSignalSendsSigintToARealChildOnLinux` 3/3 (it
passes in ~47 ms alone) and two loss-simulation tests 1/3 each. Chasing them one
at a time did not converge — four separate fixes, each surfacing a different
member of the same family, and one of those fixes regressed the other platform.
Add to this lane only with evidence that a test fails under load and passes in
isolation. A test that fails consistently is a bug, not a timing lane member.
## Continuous integration
This document owns the LOCAL gate. Pushes to `main` are gated on self-hosted