test: separate non-hermetic release lanes
This commit is contained in:
Erik
parent
52015f5052
commit
8f490240d4
17 changed files with 131 additions and 114 deletions
|
|
@ -9,8 +9,12 @@ pwsh ./tools/run-release-gate.ps1
|
|||
|
||||
The command verifies that `AcDream.slnx` contains every `.csproj` under `src/`,
|
||||
`tests/`, and `tools/`, performs a locked restore, builds that complete graph,
|
||||
then discovers and runs every default test assembly once in a fresh Release
|
||||
process. It does not retry failures. The graph currently contains 44 projects,
|
||||
then discovers and runs every hermetic test in every default test assembly once
|
||||
in a fresh Release process. It does not retry failures. Tests carrying an
|
||||
explicit non-hermetic `Lane` trait (`InstalledDat`, `PreparedPackage`, `Live`,
|
||||
or `Manual`), `Purpose=Diagnostic`, or `Status=KnownFailure` are excluded from
|
||||
the hermetic total and run through their owned lane instead. The graph currently
|
||||
contains 44 projects,
|
||||
including all 13 maintained .NET tools; data-dependent tools are built but are
|
||||
not executed as tests.
|
||||
|
||||
|
|
@ -64,9 +68,36 @@ are preserved for later use. Until then, the repository command above is the
|
|||
authoritative gate. Focused portability or Vulkan jobs are not substitutes for
|
||||
the complete gate.
|
||||
|
||||
Environment-dependent tests retain their current skip behavior and are counted
|
||||
explicitly in the JSON/TRX report. Classifying or replacing those tests belongs
|
||||
to R3, not to this gate checkpoint.
|
||||
The JSON summary records the exact test filter. Environment-dependent,
|
||||
diagnostic, manual, and known-failure results must be published as their own
|
||||
lane and must never be added to the hermetic pass headline.
|
||||
|
||||
## Non-hermetic test lanes
|
||||
|
||||
Installed-DAT tests require an explicit opt-in and a retail DAT directory:
|
||||
|
||||
```powershell
|
||||
$env:ACDREAM_RUN_INSTALLED_DAT_TESTS = '1'
|
||||
$env:ACDREAM_DAT_DIR = 'C:\path\to\Asherons Call'
|
||||
dotnet test tests/AcDream.App.Tests/AcDream.App.Tests.csproj -c Release `
|
||||
--filter 'Lane=InstalledDat&Status!=KnownFailure&Purpose!=Diagnostic'
|
||||
```
|
||||
|
||||
Regenerate all committed UI fixtures through the one comprehensive manual
|
||||
generator (the former chat/radar-only generators were redundant):
|
||||
|
||||
```powershell
|
||||
$env:ACDREAM_REGENERATE_UI_FIXTURES = '1'
|
||||
$env:ACDREAM_DAT_DIR = 'C:\path\to\Asherons Call'
|
||||
dotnet test tests/AcDream.App.Tests/AcDream.App.Tests.csproj -c Release `
|
||||
--filter 'Lane=Manual'
|
||||
```
|
||||
|
||||
Known failures (`Status=KnownFailure`) are never part of a green release total.
|
||||
Run them explicitly with their prerequisite lane configured; a failure is
|
||||
expected until the linked defect is fixed. Diagnostic apparatus
|
||||
(`Purpose=Diagnostic`) likewise reports separately and does not inflate the
|
||||
contract-test pass count.
|
||||
|
||||
## Updating dependencies
|
||||
|
||||
|
|
|
|||
|
|
@ -32,8 +32,8 @@ The latest clean-gate TRX files account for every reported skip:
|
|||
|
||||
| Classification | Count | Current source | R3 decision |
|
||||
|---|---:|---|---|
|
||||
| Installed retail DAT lane | 73 | `InstalledDatFactAttribute` across nine files | Retain the contracts, replace the campaign-specific shared reason with prerequisite- and suite-specific identity, and report this lane separately. |
|
||||
| Manual fixture generation | 2 | `ChatLayoutFixtureGenerator.GenerateChatFixture`; `RadarLayoutFixtureGenerator.GenerateRadarFixture` | Remove from default discovery and expose documented explicit generator commands. |
|
||||
| Installed retail DAT lane | 73 | `InstalledDatFactAttribute` across nine files | Retain the contracts, replace the campaign-specific shared reason with prerequisite- and suite-specific identity, and report this lane separately. Batch B assigns all nine classes to `Lane=InstalledDat`. |
|
||||
| Manual fixture generation | 2 | `ChatLayoutFixtureGenerator.GenerateChatFixture`; `RadarLayoutFixtureGenerator.GenerateRadarFixture` | Delete both redundant generators: the existing comprehensive `RetailLayoutFixtureGenerator` already owns their chat and radar outputs. Assign that generator to the explicit Manual lane. |
|
||||
| Empty future scaffold | 1 | `PvsConformanceTests.Pvs_CottageInterior_MatchesRetailCellDrawList` | Delete the test. Preserve its retail oracle and capture recipe here and in the existing P0 plan. |
|
||||
| Known product failure | 1 | `TowerAscentReplayTests.TowerAscent_StaircaseStaysConeVisible_EveryStep` | Do not pretend this is unavailable coverage. Keep the oracle and move it to an explicitly reported known-failure/non-blocking lane until #119 is fixed. |
|
||||
|
||||
|
|
@ -58,6 +58,21 @@ assign them to the installed-DAT, prepared-package, system-font, or other
|
|||
explicit prerequisite lane; a future machine may report them as additional
|
||||
skips.
|
||||
|
||||
The lane vocabulary is deliberately orthogonal:
|
||||
|
||||
- `Lane` states the execution environment (`InstalledDat`, `PreparedPackage`,
|
||||
`Live`, or `Manual`);
|
||||
- `Purpose=Diagnostic` marks output/investigation apparatus that cannot add to
|
||||
the contract-test pass count; and
|
||||
- `Status=KnownFailure` keeps a named executable oracle visible without making
|
||||
a green release claim.
|
||||
|
||||
The default gate excludes all three dimensions. An explicitly selected lane
|
||||
must fail clearly when its prerequisite is missing; it may not pass through an
|
||||
empty return. The 73 custom installed-DAT facts retain discovery-time skips
|
||||
when their opt-in is absent so each unavailable contract and reason remains
|
||||
identifiable in the lane report.
|
||||
|
||||
## Finding disposition
|
||||
|
||||
| Finding | Current disposition | Evidence / next action |
|
||||
|
|
@ -67,7 +82,7 @@ skips.
|
|||
| T-003 duplicate theory row | high-confidence cleanup batch A | Remove literal `0x41000012`, which is identical to `MotionCommand.Crouch`; remove the temporary analyzer suppression. |
|
||||
| T-004 warning mismatches | resolved in R2 | Clean rebuild is zero-warning with repository-wide warnings-as-errors. |
|
||||
| T-005 unreachable panel stack | requires production reachability proof | Do not delete 52 meaningful tests until the dead presentation surface and any supported plugin compatibility promise are verified together. |
|
||||
| T-006 misleading installed-DAT reason | open | Replace the LA8-specific shared attribute reason with truthful suite/prerequisite identity. |
|
||||
| T-006 misleading installed-DAT reason | resolved in batch B | Shared opt-in is now `ACDREAM_RUN_INSTALLED_DAT_TESTS=1` (legacy switch retained), the reason names the lane, and all nine owners carry `Lane=InstalledDat`. |
|
||||
| T-007 271 silent passing gates | open | Re-run the syntax inventory against current source, then convert each to a hermetic fixture, explicit prerequisite skip, or non-default lane. |
|
||||
| T-008 incapable-of-failing diagnostics | batch A plus diagnostic lane work | Delete the literal wiring smoke test; repair the GPU contract tautology; later re-home output-only apparatus. |
|
||||
| T-009 wall-clock double-click tests | open | Introduce a behavior-preserving injectable monotonic clock and deterministic fake time. |
|
||||
|
|
@ -75,8 +90,8 @@ skips.
|
|||
| T-011 51 output-only methods | open | Validate the historical 51-method list against current source; add stable oracles or move them outside default discovery. |
|
||||
| T-012 source-text freezes | requires semantic replacement map | Retain whole-tree dependency rules; remove exact-text freezes only when an equivalent semantic/behavioral guard is identified. |
|
||||
| T-013 controller self-comparison | high-confidence cleanup batch A | Capture the first controller next to the first body and compare every retry with that reference. |
|
||||
| T-014 seven load-sensitive tests | open | Reproduce and classify each mechanism; no retries or blanket tolerances. |
|
||||
| T-015 four non-prerequisite skips | batch A plus lane work | Delete the PVS scaffold now; move two generators and the known failure in later focused batches. |
|
||||
| T-014 seven load-sensitive tests | active; first mechanism reproduced | `PortalProjectionTests.ClipToRegion_FrameOwnedStore_ReusesExactResultArray` reported 6,904 B only while three App test hosts ran concurrently, then passed in an isolated run. Preserve both observations and remove ambient process-allocation sensitivity rather than loosening the zero-allocation oracle. |
|
||||
| T-015 four non-prerequisite skips | resolved in batches A/B | PVS scaffold deleted with rationale preserved; redundant chat/radar generators deleted in favor of the comprehensive Manual lane; tower oracle is `Status=KnownFailure`. |
|
||||
| T-016 historical test taxonomy | open | Rename/re-home only after each test's durable owner and oracle are established. |
|
||||
| T-017 Avalonia ownership | resolved in R2 | All seven editor variants now execute in one owned application session; 11 focused passes plus the complete gate. |
|
||||
| T-018 stderr observer race | resolved in R2 | Live reader shares writes/deletes; 25 focused repetitions plus the complete gate. |
|
||||
|
|
@ -130,3 +145,37 @@ The count delta is intentional and classified: Core removed the two useless
|
|||
compile-only cases and the skipped PVS scaffold; App gained one case by
|
||||
splitting the old mixed/tautological depth-stencil test into two independently
|
||||
named contracts. Core emitted no duplicate-case discovery warning.
|
||||
|
||||
## Preserved rationale for removals in batch B
|
||||
|
||||
The chat- and radar-only fixture generators were not the sole way to recreate
|
||||
their committed JSON. `RetailLayoutFixtureGenerator` already includes retail
|
||||
layout IDs `0x2100006F` and `0x21000074` and writes the same
|
||||
`chat_2100006f.json` and `radar_21000074.json` destinations as part of its
|
||||
complete 20-layout run. Deleting the two permanently skipped duplicates removes
|
||||
two false test entries without losing the regeneration recipe or retail notes;
|
||||
the comprehensive generator and `docs/release-gate.md` now own that workflow.
|
||||
|
||||
The tower staircase oracle is not unavailable or useless. Its former static
|
||||
skip documents issue #119 and the exact failing steps. Batch B keeps the
|
||||
executable assertion under `Status=KnownFailure`, separates two output-only
|
||||
methods with `Purpose=Diagnostic`, and makes missing installed DATs a clear lane
|
||||
prerequisite failure instead of a passing early return.
|
||||
|
||||
## Batch B verification
|
||||
|
||||
The lane-classification batch was verified without changing product code:
|
||||
|
||||
- App Release build: 0 warnings, 0 errors;
|
||||
- hermetic App lane, isolated: 5,505 passed / 0 skipped / 0 failed;
|
||||
- Installed-DAT contract lane without opt-in: 3 passed / 73 individually named
|
||||
skips / 0 failed (the local tower DAT prerequisite was available);
|
||||
- known-failure discovery: exactly one tower staircase oracle; and
|
||||
- Manual discovery: exactly one comprehensive fixture generator.
|
||||
|
||||
One concurrent three-host verification attempt reproduced T-014 rather than
|
||||
being hidden: the hermetic host failed the portal-clipping zero-allocation test
|
||||
at 6,904 bytes while the Installed-DAT and discovery hosts ran beside it. The
|
||||
same unmodified hermetic command then passed 5,505/5,505 in isolation. This is
|
||||
evidence for removing cross-host allocation sensitivity; it is not grounds to
|
||||
weaken the assertion or claim a clean concurrent result.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue