test: make prerequisite lanes fail honestly
This commit is contained in:
Erik
parent
dfc841b779
commit
6faeb4a103
113 changed files with 424 additions and 336 deletions
|
|
@ -100,7 +100,22 @@ generator (the former chat/radar-only generators were redundant):
|
|||
$env:ACDREAM_REGENERATE_UI_FIXTURES = '1'
|
||||
$env:ACDREAM_DAT_DIR = 'C:\path\to\Asherons Call'
|
||||
dotnet test tests/AcDream.App.Tests/AcDream.App.Tests.csproj -c Release `
|
||||
--filter 'Lane=Manual'
|
||||
--filter 'Lane=Manual&ManualTask=FixtureGeneration'
|
||||
```
|
||||
|
||||
The retained live-DAT probes are manual evidence, not InstalledDat regression
|
||||
contracts. Run each opt-in family independently so a probe command can never
|
||||
regenerate fixtures as a side effect:
|
||||
|
||||
```powershell
|
||||
$env:ACDREAM_DAT_DIR = 'C:\path\to\Asherons Call'
|
||||
$env:ACDREAM_PROBE_LIVE_MOUNT = '1'
|
||||
dotnet test tests/AcDream.App.Tests/AcDream.App.Tests.csproj -c Release `
|
||||
--filter 'Lane=Manual&ManualTask=LiveMountProbe'
|
||||
|
||||
$env:ACDREAM_PROBE_POWERBAR = '1'
|
||||
dotnet test tests/AcDream.App.Tests/AcDream.App.Tests.csproj -c Release `
|
||||
--filter 'Lane=Manual&ManualTask=PowerbarProbe'
|
||||
```
|
||||
|
||||
Known failures (`Status=KnownFailure`) are never part of a green release total.
|
||||
|
|
@ -114,9 +129,9 @@ investigation output, and several methods require installed DATs:
|
|||
|
||||
```powershell
|
||||
dotnet test tests/AcDream.App.Tests/AcDream.App.Tests.csproj -c Release `
|
||||
--filter 'Purpose=Diagnostic'
|
||||
--filter 'Purpose=Diagnostic&Lane!=Manual'
|
||||
dotnet test tests/AcDream.Core.Tests/AcDream.Core.Tests.csproj -c Release `
|
||||
--filter 'Purpose=Diagnostic'
|
||||
--filter 'Purpose=Diagnostic&Lane!=Manual'
|
||||
```
|
||||
|
||||
Operating-system contracts are likewise explicit. Run `Lane=Windows` on a
|
||||
|
|
|
|||
|
|
@ -38,27 +38,27 @@ pwsh ./tools/audit-test-inventory.ps1
|
|||
The generated JSON lives under ignored `artifacts/test-audit/`; it is not a
|
||||
second 6.8 MB checked-in source of truth. The script and this reviewed ledger
|
||||
are durable, while paths and line numbers regenerate after every batch. The
|
||||
inventory at commit `8f490240` reports:
|
||||
inventory refreshed for Batch G reports:
|
||||
|
||||
| Syntax signal | Current count |
|
||||
|---|---:|
|
||||
| Tracked/parsed C# test files | 1,256 / 1,256 |
|
||||
| Attributed test methods (not expanded theory cases) | 11,455 |
|
||||
| Methods containing at least one empty `return;` | 399 |
|
||||
| Empty-return sites | 426 |
|
||||
| Prerequisite-return candidates | 282 |
|
||||
| Mechanical output-only candidates | 47 |
|
||||
| Methods directly using `Thread.Sleep` / `Task.Delay` | 20 / 23 |
|
||||
| Attributed test methods (not expanded theory cases) | 11,450 |
|
||||
| Methods containing at least one empty `return;` | 138 |
|
||||
| Empty-return sites | 143 |
|
||||
| Direct / same-file-helper prerequisite-return candidates | 2 / 1 |
|
||||
| Mechanical output-only candidates | 2 |
|
||||
| Reviewed diagnostic methods | 51 |
|
||||
| Methods directly using `Thread.Sleep` / `Task.Delay` | 15 / 23 |
|
||||
| Methods directly reading environment variables | 47 |
|
||||
| Methods directly reading `.cs` source text | 63 |
|
||||
|
||||
The 282 prerequisite-return count is intentionally a review candidate set, not
|
||||
a claim that all 399 methods are broken. It comprises 109 App, 125 Core, 29
|
||||
Content, 12 Launcher.Core, three Core.Net, three Bake, and one Headless method.
|
||||
The classifier includes external-asset, opt-in, and platform gates; behavioral
|
||||
early exits inside callback tests remain outside that candidate set. Every
|
||||
candidate still requires source inspection because `Directory.Exists`, for
|
||||
example, can also be the behavior under test.
|
||||
The three remaining prerequisite-return candidates are all reviewed branch
|
||||
false positives: the two Windows/Linux factory assertions and the launcher's
|
||||
cross-platform directory-link helper. Every true prerequisite return found by
|
||||
the inventory now fails explicitly inside its selected non-hermetic lane. The
|
||||
remaining 143 empty returns are behavioral control flow, callback exits, or
|
||||
the three platform branches; they are not missing-prerequisite passes.
|
||||
|
||||
The output-only classifier follows same-file helper calls, but it cannot decide
|
||||
whether an assertion merely establishes probe setup. The reviewed diagnostic
|
||||
|
|
@ -95,17 +95,16 @@ Installed-DAT skip distribution:
|
|||
| `TooltipLiveDatTests` | 7 |
|
||||
| `UiButtonCorpusSweepTests` | 3 |
|
||||
|
||||
There are also 11 source sites that throw `SkipException.ForSkip` conditionally.
|
||||
They are not part of the current 77 because their prerequisites happened to be
|
||||
available on the R2 gate machine. R3 must retain their per-test reason and
|
||||
assign them to the installed-DAT, prepared-package, system-font, or other
|
||||
explicit prerequisite lane; a future machine may report them as additional
|
||||
skips.
|
||||
The 11 source sites that conditionally threw `SkipException.ForSkip` were not
|
||||
part of the R2 77 because their prerequisites happened to be available on that
|
||||
machine. Batch G preserves their reasons but replaces the misleading dynamic-
|
||||
skip exception with explicit lane-prerequisite failures. No static skip or
|
||||
`SkipException.ForSkip` remains in current test source.
|
||||
|
||||
The lane vocabulary is deliberately orthogonal:
|
||||
|
||||
- `Lane` states the execution environment (`InstalledDat`, `PreparedPackage`,
|
||||
`Live`, or `Manual`);
|
||||
`Live`, `Manual`, `Windows`, `Linux`, or `SystemFont`);
|
||||
- `Purpose=Diagnostic` marks output/investigation apparatus that cannot add to
|
||||
the contract-test pass count; and
|
||||
- `Status=KnownFailure` keeps a named executable oracle visible without making
|
||||
|
|
@ -127,7 +126,7 @@ identifiable in the lane report.
|
|||
| T-004 warning mismatches | resolved in R2 | Clean rebuild is zero-warning with repository-wide warnings-as-errors. |
|
||||
| T-005 unreachable panel stack | requires production reachability proof | Do not delete 52 meaningful tests until the dead presentation surface and any supported plugin compatibility promise are verified together. |
|
||||
| T-006 misleading installed-DAT reason | resolved in batch B | Shared opt-in is now `ACDREAM_RUN_INSTALLED_DAT_TESTS=1` (legacy switch retained), the reason names the lane, and all nine owners carry `Lane=InstalledDat`. |
|
||||
| T-007 271 silent passing gates | lane classification complete in batch D; body cleanup active | Of 282 current candidates, 280 are true prerequisite gates and now have an explicit lane. The two unlaned methods are reviewed false positives that assert the current Windows/Linux branch before returning. Empty-return removal remains, lane by lane. |
|
||||
| T-007 271 silent passing gates | resolved in batch G | All 280 directly found gates plus three later-reconciled sites now fail explicitly when their selected lane lacks its prerequisite. The only three syntax candidates left are reviewed cross-platform branch false positives. |
|
||||
| T-008 incapable-of-failing diagnostics | batch A plus diagnostic lane work | Delete the literal wiring smoke test; repair the GPU contract tautology; later re-home output-only apparatus. |
|
||||
| T-009 wall-clock double-click tests | resolved in batch E | Four sleeps were replaced by a deterministic test clock behind an internal factory overload. The production factory still reads `Environment.TickCount64` exactly as before. |
|
||||
| T-010 two useless cases | high-confidence cleanup batch A | Delete `SmokeTest.TestProject_IsWired` and `ChaseCameraTests.ImplementsICamera`; compilation already proves both claims. |
|
||||
|
|
@ -370,3 +369,57 @@ warnings/errors, the refreshed inventory reports two explicit known-failure
|
|||
methods and 15 remaining direct `Thread.Sleep` methods, and the no-retry
|
||||
hermetic gate passed 14,391/14,391 with zero skips or failures. The one-pass
|
||||
reduction is exactly #321 leaving the release lane.
|
||||
|
||||
## Batch G honest prerequisite lanes
|
||||
|
||||
Batch G removes every true silent prerequisite pass found by the direct and
|
||||
same-file-helper inventory. Across 101 test files, 283 lane-owned `return;`
|
||||
sites now report a prerequisite failure when that lane was explicitly
|
||||
selected: 261 installed-DAT, ten Linux, seven Windows, three Live, and two
|
||||
prepared-package methods. Seventeen platform guards throw
|
||||
`PlatformNotSupportedException`; the remaining gates use lane-specific
|
||||
assertion failures. Production behavior and every contract assertion after the
|
||||
guard are unchanged.
|
||||
|
||||
The eleven conditional `SkipException.ForSkip` sites now use explicit
|
||||
prerequisite failures as well. Their asset, package, or font reason is retained,
|
||||
but a selected lane can no longer look green by dynamically skipping the
|
||||
contract. The refreshed syntax inventory finds zero true silent-prerequisite
|
||||
candidates. Its three remaining candidates are the reviewed Windows/Linux
|
||||
branch returns recorded in Batch E.
|
||||
|
||||
The focused installed-DAT run also exposed a taxonomy error rather than a
|
||||
product failure: 19 interactive layout methods need a retail DAT directory but
|
||||
are operator-driven probes, not automated DAT contracts. They now live under
|
||||
`Lane=Manual`, divided into `ManualTask=LiveMountProbe` (13) and
|
||||
`ManualTask=PowerbarProbe` (6). The comprehensive existing generator is
|
||||
separately named `ManualTask=FixtureGeneration`, so selecting a probe cannot
|
||||
rewrite committed fixtures. Current attributed-method lane ownership is:
|
||||
|
||||
| Lane | Methods |
|
||||
|---|---:|
|
||||
| `InstalledDat` | 317 |
|
||||
| `PreparedPackage` | 2 |
|
||||
| `Live` | 3 |
|
||||
| `Manual` | 20 |
|
||||
| `Windows` | 7 |
|
||||
| `Linux` | 10 |
|
||||
| `SystemFont` | 2 |
|
||||
|
||||
Focused verification before the complete gate:
|
||||
|
||||
- Release build of all 44 maintained projects: 0 warnings, 0 errors;
|
||||
- installed-DAT contracts: 311/311 passed (App 150, Bake 3, Content 28,
|
||||
Core 130);
|
||||
- prepared-package contracts: 2/2 passed against the validated local package;
|
||||
- Windows contracts: 7/7 passed; system-font contracts: 2/2 passed;
|
||||
- manual live-mount probes: 13/13 passed; manual powerbar probes: 6/6 passed;
|
||||
- a Linux contract selected on Windows failed with the explicit wrong-host
|
||||
reason; and
|
||||
- a Live contract selected without opt-in failed with the explicit
|
||||
`ACDREAM_LIVE=1`/reachable-server prerequisite reason.
|
||||
|
||||
The fixture-generation task was intentionally not executed: its documented
|
||||
purpose is to rewrite fixture files, while this R3 batch is review/test-only.
|
||||
The no-retry complete hermetic Release gate passed 14,391/14,391 with zero
|
||||
skips or failures across all 12 test assemblies.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue