test: separate non-hermetic release lanes

This commit is contained in:
Erik 2026-08-18 10:49:22 +02:00
parent 52015f5052
commit 8f490240d4
17 changed files with 131 additions and 114 deletions

View file

@ -9,8 +9,12 @@ pwsh ./tools/run-release-gate.ps1
The command verifies that `AcDream.slnx` contains every `.csproj` under `src/`,
`tests/`, and `tools/`, performs a locked restore, builds that complete graph,
then discovers and runs every default test assembly once in a fresh Release
process. It does not retry failures. The graph currently contains 44 projects,
then discovers and runs every hermetic test in every default test assembly once
in a fresh Release process. It does not retry failures. Tests carrying an
explicit non-hermetic `Lane` trait (`InstalledDat`, `PreparedPackage`, `Live`,
or `Manual`), `Purpose=Diagnostic`, or `Status=KnownFailure` are excluded from
the hermetic total and run through their owned lane instead. The graph currently
contains 44 projects,
including all 13 maintained .NET tools; data-dependent tools are built but are
not executed as tests.
@ -64,9 +68,36 @@ are preserved for later use. Until then, the repository command above is the
authoritative gate. Focused portability or Vulkan jobs are not substitutes for
the complete gate.
Environment-dependent tests retain their current skip behavior and are counted
explicitly in the JSON/TRX report. Classifying or replacing those tests belongs
to R3, not to this gate checkpoint.
The JSON summary records the exact test filter. Environment-dependent,
diagnostic, manual, and known-failure results must be published as their own
lane and must never be added to the hermetic pass headline.
## Non-hermetic test lanes
Installed-DAT tests require an explicit opt-in and a retail DAT directory:
```powershell
$env:ACDREAM_RUN_INSTALLED_DAT_TESTS = '1'
$env:ACDREAM_DAT_DIR = 'C:\path\to\Asherons Call'
dotnet test tests/AcDream.App.Tests/AcDream.App.Tests.csproj -c Release `
--filter 'Lane=InstalledDat&Status!=KnownFailure&Purpose!=Diagnostic'
```
Regenerate all committed UI fixtures through the one comprehensive manual
generator (the former chat/radar-only generators were redundant):
```powershell
$env:ACDREAM_REGENERATE_UI_FIXTURES = '1'
$env:ACDREAM_DAT_DIR = 'C:\path\to\Asherons Call'
dotnet test tests/AcDream.App.Tests/AcDream.App.Tests.csproj -c Release `
--filter 'Lane=Manual'
```
Known failures (`Status=KnownFailure`) are never part of a green release total.
Run them explicitly with their prerequisite lane configured; a failure is
expected until the linked defect is fixed. Diagnostic apparatus
(`Purpose=Diagnostic`) likewise reports separately and does not inflate the
contract-test pass count.
## Updating dependencies

View file

@ -32,8 +32,8 @@ The latest clean-gate TRX files account for every reported skip:
| Classification | Count | Current source | R3 decision |
|---|---:|---|---|
| Installed retail DAT lane | 73 | `InstalledDatFactAttribute` across nine files | Retain the contracts, replace the campaign-specific shared reason with prerequisite- and suite-specific identity, and report this lane separately. |
| Manual fixture generation | 2 | `ChatLayoutFixtureGenerator.GenerateChatFixture`; `RadarLayoutFixtureGenerator.GenerateRadarFixture` | Remove from default discovery and expose documented explicit generator commands. |
| Installed retail DAT lane | 73 | `InstalledDatFactAttribute` across nine files | Retain the contracts, replace the campaign-specific shared reason with prerequisite- and suite-specific identity, and report this lane separately. Batch B assigns all nine classes to `Lane=InstalledDat`. |
| Manual fixture generation | 2 | `ChatLayoutFixtureGenerator.GenerateChatFixture`; `RadarLayoutFixtureGenerator.GenerateRadarFixture` | Delete both redundant generators: the existing comprehensive `RetailLayoutFixtureGenerator` already owns their chat and radar outputs. Assign that generator to the explicit Manual lane. |
| Empty future scaffold | 1 | `PvsConformanceTests.Pvs_CottageInterior_MatchesRetailCellDrawList` | Delete the test. Preserve its retail oracle and capture recipe here and in the existing P0 plan. |
| Known product failure | 1 | `TowerAscentReplayTests.TowerAscent_StaircaseStaysConeVisible_EveryStep` | Do not pretend this is unavailable coverage. Keep the oracle and move it to an explicitly reported known-failure/non-blocking lane until #119 is fixed. |
@ -58,6 +58,21 @@ assign them to the installed-DAT, prepared-package, system-font, or other
explicit prerequisite lane; a future machine may report them as additional
skips.
The lane vocabulary is deliberately orthogonal:
- `Lane` states the execution environment (`InstalledDat`, `PreparedPackage`,
`Live`, or `Manual`);
- `Purpose=Diagnostic` marks output/investigation apparatus that cannot add to
the contract-test pass count; and
- `Status=KnownFailure` keeps a named executable oracle visible without making
a green release claim.
The default gate excludes all three dimensions. An explicitly selected lane
must fail clearly when its prerequisite is missing; it may not pass through an
empty return. The 73 custom installed-DAT facts retain discovery-time skips
when their opt-in is absent so each unavailable contract and reason remains
identifiable in the lane report.
## Finding disposition
| Finding | Current disposition | Evidence / next action |
@ -67,7 +82,7 @@ skips.
| T-003 duplicate theory row | high-confidence cleanup batch A | Remove literal `0x41000012`, which is identical to `MotionCommand.Crouch`; remove the temporary analyzer suppression. |
| T-004 warning mismatches | resolved in R2 | Clean rebuild is zero-warning with repository-wide warnings-as-errors. |
| T-005 unreachable panel stack | requires production reachability proof | Do not delete 52 meaningful tests until the dead presentation surface and any supported plugin compatibility promise are verified together. |
| T-006 misleading installed-DAT reason | open | Replace the LA8-specific shared attribute reason with truthful suite/prerequisite identity. |
| T-006 misleading installed-DAT reason | resolved in batch B | Shared opt-in is now `ACDREAM_RUN_INSTALLED_DAT_TESTS=1` (legacy switch retained), the reason names the lane, and all nine owners carry `Lane=InstalledDat`. |
| T-007 271 silent passing gates | open | Re-run the syntax inventory against current source, then convert each to a hermetic fixture, explicit prerequisite skip, or non-default lane. |
| T-008 incapable-of-failing diagnostics | batch A plus diagnostic lane work | Delete the literal wiring smoke test; repair the GPU contract tautology; later re-home output-only apparatus. |
| T-009 wall-clock double-click tests | open | Introduce a behavior-preserving injectable monotonic clock and deterministic fake time. |
@ -75,8 +90,8 @@ skips.
| T-011 51 output-only methods | open | Validate the historical 51-method list against current source; add stable oracles or move them outside default discovery. |
| T-012 source-text freezes | requires semantic replacement map | Retain whole-tree dependency rules; remove exact-text freezes only when an equivalent semantic/behavioral guard is identified. |
| T-013 controller self-comparison | high-confidence cleanup batch A | Capture the first controller next to the first body and compare every retry with that reference. |
| T-014 seven load-sensitive tests | open | Reproduce and classify each mechanism; no retries or blanket tolerances. |
| T-015 four non-prerequisite skips | batch A plus lane work | Delete the PVS scaffold now; move two generators and the known failure in later focused batches. |
| T-014 seven load-sensitive tests | active; first mechanism reproduced | `PortalProjectionTests.ClipToRegion_FrameOwnedStore_ReusesExactResultArray` reported 6,904 B only while three App test hosts ran concurrently, then passed in an isolated run. Preserve both observations and remove ambient process-allocation sensitivity rather than loosening the zero-allocation oracle. |
| T-015 four non-prerequisite skips | resolved in batches A/B | PVS scaffold deleted with rationale preserved; redundant chat/radar generators deleted in favor of the comprehensive Manual lane; tower oracle is `Status=KnownFailure`. |
| T-016 historical test taxonomy | open | Rename/re-home only after each test's durable owner and oracle are established. |
| T-017 Avalonia ownership | resolved in R2 | All seven editor variants now execute in one owned application session; 11 focused passes plus the complete gate. |
| T-018 stderr observer race | resolved in R2 | Live reader shares writes/deletes; 25 focused repetitions plus the complete gate. |
@ -130,3 +145,37 @@ The count delta is intentional and classified: Core removed the two useless
compile-only cases and the skipped PVS scaffold; App gained one case by
splitting the old mixed/tautological depth-stencil test into two independently
named contracts. Core emitted no duplicate-case discovery warning.
## Preserved rationale for removals in batch B
The chat- and radar-only fixture generators were not the sole way to recreate
their committed JSON. `RetailLayoutFixtureGenerator` already includes retail
layout IDs `0x2100006F` and `0x21000074` and writes the same
`chat_2100006f.json` and `radar_21000074.json` destinations as part of its
complete 20-layout run. Deleting the two permanently skipped duplicates removes
two false test entries without losing the regeneration recipe or retail notes;
the comprehensive generator and `docs/release-gate.md` now own that workflow.
The tower staircase oracle is not unavailable or useless. Its former static
skip documents issue #119 and the exact failing steps. Batch B keeps the
executable assertion under `Status=KnownFailure`, separates two output-only
methods with `Purpose=Diagnostic`, and makes missing installed DATs a clear lane
prerequisite failure instead of a passing early return.
## Batch B verification
The lane-classification batch was verified without changing product code:
- App Release build: 0 warnings, 0 errors;
- hermetic App lane, isolated: 5,505 passed / 0 skipped / 0 failed;
- Installed-DAT contract lane without opt-in: 3 passed / 73 individually named
skips / 0 failed (the local tower DAT prerequisite was available);
- known-failure discovery: exactly one tower staircase oracle; and
- Manual discovery: exactly one comprehensive fixture generator.
One concurrent three-host verification attempt reproduced T-014 rather than
being hidden: the hermetic host failed the portal-clipping zero-allocation test
at 6,904 bytes while the Installed-DAT and discovery hosts ran beside it. The
same unmodified hermetic command then passed 5,505/5,505 in isolation. This is
evidence for removing cross-host allocation sensitivity; it is not grounds to
weaken the assertion or claim a clean concurrent result.

View file

@ -36,6 +36,7 @@ namespace AcDream.App.Tests.Rendering;
/// those runs as Skipped so the counts show "ran" separately from "no-op'd".
/// </para>
/// </summary>
[Trait("Lane", "InstalledDat")]
public sealed class ChargenPreviewControllerTests
{
private readonly ITestOutputHelper _out;

View file

@ -29,6 +29,7 @@ namespace AcDream.App.Tests.Rendering;
/// the vanish; root/visibility instability across adjacent steps pins the
/// flap.
/// </summary>
[Trait("Lane", "InstalledDat")]
public class TowerAscentReplayTests
{
private readonly ITestOutputHelper _out;
@ -96,10 +97,9 @@ public class TowerAscentReplayTests
}
}
private List<AscentStep>? RunAscent()
private List<AscentStep> RunAscent()
{
var datDir = CornerFloodReplayTests.ResolveDatDir();
if (datDir is null) { _out.WriteLine("SKIP: dats unavailable"); return null; }
var datDir = RequireDatDir();
using var dats = new DatCollection(datDir, DatAccessType.Read);
var cells = Issue120ReciprocalPingPongTests.LoadAllInteriorCells(dats, Landblock);
@ -203,12 +203,11 @@ public class TowerAscentReplayTests
/// through this band (curr_cell keep-curr above open-top cells), or can
/// ConstructView(CBldPortal) seed with an in-plane eye?
/// </summary>
[Fact(Skip = "#119-residual: pins the roof-lip flood gap (steps 195-201) — un-skip with the fix; see the doc comment")]
[Fact]
[Trait("Status", "KnownFailure")]
public void TowerAscent_StaircaseStaysConeVisible_EveryStep()
{
var steps = RunAscent();
if (steps is null) return;
var failures = steps.FindAll(s => !s.StairsConeVisible);
if (failures.Count > 0)
{
@ -229,8 +228,6 @@ public class TowerAscentReplayTests
public void TowerAscent_RootDoesNotPingPong()
{
var steps = RunAscent();
if (steps is null) return;
var flips = new List<int>();
for (int i = 2; i < steps.Count; i++)
{
@ -265,8 +262,7 @@ public class TowerAscentReplayTests
public void CapturedTopOfStairs_MainCellStaysInFlood(
float ex, float ey, float ez, float fx, float fy, float fz)
{
var datDir = CornerFloodReplayTests.ResolveDatDir();
if (datDir is null) { _out.WriteLine("SKIP: dats unavailable"); return; }
var datDir = RequireDatDir();
using var dats = new DatCollection(datDir, DatAccessType.Read);
var cells = Issue120ReciprocalPingPongTests.LoadAllInteriorCells(dats, Landblock);
@ -336,10 +332,10 @@ public class TowerAscentReplayTests
/// footprint containment).
/// </summary>
[Fact]
[Trait("Purpose", "Diagnostic")]
public void Diagnostic_TopOfStairs_GateByGate()
{
var datDir = CornerFloodReplayTests.ResolveDatDir();
if (datDir is null) { _out.WriteLine("SKIP: dats unavailable"); return; }
var datDir = RequireDatDir();
using var dats = new DatCollection(datDir, DatAccessType.Read);
var cells = Issue120ReciprocalPingPongTests.LoadAllInteriorCells(dats, Landblock);
@ -419,10 +415,15 @@ public class TowerAscentReplayTests
/// <summary>Full per-step table for the investigation record.</summary>
[Fact]
[Trait("Purpose", "Diagnostic")]
public void Diagnostic_TowerAscent_PerStepTable()
{
var steps = RunAscent();
if (steps is null) return;
DumpSteps(steps);
}
private static string RequireDatDir()
=> CornerFloodReplayTests.ResolveDatDir()
?? throw new InvalidOperationException(
"Installed client_portal.dat is required for the tower-ascent lane.");
}

View file

@ -17,6 +17,7 @@ namespace AcDream.App.Tests.UI.Layout;
/// authored master-shell and page ids the campaign plan and CC4's own
/// decomp research cite, pinning them against the real installed layout.
/// </summary>
[Trait("Lane", "InstalledDat")]
public sealed class CharacterCreationLiveDatTests
{
private static string DatDirectory =>

View file

@ -16,6 +16,7 @@ namespace AcDream.App.Tests.UI.Layout;
/// <c>ACDREAM_PROBE_LIVE_MOUNT=1</c>; <c>ACDREAM_DAT_DIR</c> can override the
/// ordinary Documents/Asheron's Call location. Reads the DATs read-only.
/// </summary>
[Trait("Lane", "InstalledDat")]
public sealed class CharacterManagementLiveDatTests
{
[InstalledDatFact]
@ -368,9 +369,14 @@ internal sealed class InstalledDatFactAttribute : FactAttribute
{
public InstalledDatFactAttribute()
{
if (Environment.GetEnvironmentVariable("ACDREAM_PROBE_LIVE_MOUNT") != "1")
bool requested = Environment.GetEnvironmentVariable(
"ACDREAM_RUN_INSTALLED_DAT_TESTS") == "1"
|| Environment.GetEnvironmentVariable("ACDREAM_PROBE_LIVE_MOUNT") == "1";
if (!requested)
{
Skip = "Set ACDREAM_PROBE_LIVE_MOUNT=1 to run the installed-DAT LA8 gate.";
Skip = "Installed-DAT lane not requested; set "
+ "ACDREAM_RUN_INSTALLED_DAT_TESTS=1. The legacy "
+ "ACDREAM_PROBE_LIVE_MOUNT=1 switch is also accepted.";
return;
}

View file

@ -235,7 +235,7 @@ public class ChatLayoutConformanceTests
/// §2.5/§2.6): the chat transcript's base style (<c>0x10000372</c> in layout
/// <c>0x2100003F</c>) authors NO property 0x21 (Outline) anywhere in its
/// inheritance chain — regenerated straight from the real installed DAT
/// (<c>ChatLayoutFixtureGenerator</c>), proving the property-0x21/0x22
/// (<c>RetailLayoutFixtureGenerator</c>), proving the property-0x21/0x22
/// importer (<see cref="ElementReader.ApplyCanonicalLegacyProjection"/>)
/// resolves this correctly end to end, not merely by a missing-field JSON
/// default.

View file

@ -1,50 +0,0 @@
using System;
using System.IO;
using System.Runtime.CompilerServices;
using System.Text.Json;
using AcDream.App.UI.Layout;
using DatReaderWriter;
using DatReaderWriter.Options;
namespace AcDream.App.Tests.UI.Layout;
/// <summary>
/// One-off generator for the committed chat golden fixture. Skipped by default —
/// run manually with the real dats present (set ACDREAM_DAT_DIR) to regenerate
/// chat_2100006f.json, then commit it. Mirrors how vitals_2100006C.json was made.
///
/// <para>
/// Campaign CH slice CH6a swapped the imported main-chat LayoutDesc from the
/// wrong <c>0x21000006</c> (a different, unrelated chat layout whose root
/// <c>0x1000000E</c> and 800px resize bar <c>0x1000000F</c> appear nowhere in
/// the EoR gameplay UI) to retail's actual main chat window,
/// <c>0x2100006F</c> (window element <c>0x10000601</c>, layout root
/// <c>0x10000600</c>, authored 410x100 — see
/// docs/research/2026-08-09-chat-retail-window-shell.md §2.1 and §5). The old
/// <c>chat_21000006.json</c> fixture is retired with it.
/// </para>
/// </summary>
public class ChatLayoutFixtureGenerator
{
[Fact(Skip = "manual: regenerates the committed chat fixture; needs the real dats (ACDREAM_DAT_DIR)")]
public void GenerateChatFixture()
{
var datDir = Environment.GetEnvironmentVariable("ACDREAM_DAT_DIR")
?? Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.UserProfile),
"Documents", "Asheron's Call");
using var dats = new DatCollection(datDir, DatAccessType.Read);
var info = LayoutImporter.ImportInfos(dats, 0x2100006Fu);
Assert.NotNull(info);
var json = JsonSerializer.Serialize(info, new JsonSerializerOptions
{
IncludeFields = true,
WriteIndented = true,
});
File.WriteAllText(FixturePath(), json);
}
// Resolve the SOURCE fixtures dir (not bin/) from this file's compile-time path.
private static string FixturePath([CallerFilePath] string thisFile = "")
=> Path.Combine(Path.GetDirectoryName(thisFile)!, "fixtures", "chat_2100006f.json");
}

View file

@ -17,6 +17,7 @@ namespace AcDream.App.Tests.UI.Layout;
/// proves the escape class genuinely exists in shipping data and prints
/// which tables carry it.
/// </summary>
[Trait("Lane", "InstalledDat")]
public sealed class DatStringEscapeSweepTests
{
[InstalledDatFact]

View file

@ -24,6 +24,7 @@ namespace AcDream.App.Tests.UI.Layout;
/// elements (not a brittle exact global total) so the gate survives a
/// future DAT revision without going red on an unrelated content change.
/// </summary>
[Trait("Lane", "InstalledDat")]
public sealed class LayoutImporterMediaBearingChildSweepTests
{
private static string DatDirectory =>

View file

@ -34,6 +34,7 @@ namespace AcDream.App.Tests.UI.Layout;
/// Documents/Asheron's Call location.
/// </para>
/// </summary>
[Trait("Lane", "InstalledDat")]
public sealed class MapHousePanelLiveDatMountTests
{
private static string DatDirectory =>

View file

@ -30,6 +30,7 @@ namespace AcDream.App.Tests.UI.Layout;
/// A=FF R=00 G=FF B=00) via per-state <c>P0x3B</c>.
/// </para>
/// </summary>
[Trait("Lane", "InstalledDat")]
public sealed class MapNoteLiveDatTests
{
private static string DatDirectory =>

View file

@ -1,36 +0,0 @@
using System;
using System.IO;
using System.Runtime.CompilerServices;
using System.Text.Json;
using AcDream.App.UI.Layout;
using DatReaderWriter;
using DatReaderWriter.Options;
namespace AcDream.App.Tests.UI.Layout;
/// <summary>One-off generator for the committed retail radar LayoutDesc fixture.</summary>
public sealed class RadarLayoutFixtureGenerator
{
[Fact(Skip = "manual: regenerates the committed radar fixture; needs the real dats (ACDREAM_DAT_DIR)")]
public void GenerateRadarFixture()
{
var datDir = Environment.GetEnvironmentVariable("ACDREAM_DAT_DIR")
?? Path.Combine(
Environment.GetFolderPath(Environment.SpecialFolder.UserProfile),
"Documents",
"Asheron's Call");
using var dats = new DatCollection(datDir, DatAccessType.Read);
var info = LayoutImporter.ImportInfos(dats, RadarController.LayoutId);
Assert.NotNull(info);
var json = JsonSerializer.Serialize(info, new JsonSerializerOptions
{
IncludeFields = true,
WriteIndented = true,
});
File.WriteAllText(FixturePath(), json);
}
private static string FixturePath([CallerFilePath] string thisFile = "")
=> Path.Combine(Path.GetDirectoryName(thisFile)!, "fixtures", "radar_21000074.json");
}

View file

@ -9,9 +9,10 @@ namespace AcDream.App.Tests.UI.Layout;
/// <summary>
/// Regenerates every committed retail layout fixture from production portal.dat.
/// The test is inert unless <c>ACDREAM_REGENERATE_UI_FIXTURES=1</c> is set, keeping
/// The manual lane requires <c>ACDREAM_REGENERATE_UI_FIXTURES=1</c>, keeping
/// normal test runs deterministic and dat-independent.
/// </summary>
[Trait("Lane", "Manual")]
public sealed class RetailLayoutFixtureGenerator
{
private static readonly (uint Id, string FileName)[] Layouts =
@ -46,7 +47,10 @@ public sealed class RetailLayoutFixtureGenerator
"1",
StringComparison.Ordinal))
{
return;
throw new InvalidOperationException(
"Manual fixture generation was not requested; set "
+ "ACDREAM_REGENERATE_UI_FIXTURES=1 and ACDREAM_DAT_DIR, "
+ "then run the Manual lane explicitly.");
}
var datDir = Environment.GetEnvironmentVariable("ACDREAM_DAT_DIR")

View file

@ -19,6 +19,7 @@ namespace AcDream.App.Tests.UI.Layout;
/// sweep landmark, mirroring <see cref="CharacterManagementLiveDatTests"/>'s
/// pattern.
/// </summary>
[Trait("Lane", "InstalledDat")]
public sealed class TooltipLiveDatTests
{
private static string DatDirectory =>

View file

@ -21,6 +21,7 @@ namespace AcDream.App.Tests.UI.Layout;
/// enumeration for the commit message, pins landmark counts rather than a
/// brittle exact global total.
/// </summary>
[Trait("Lane", "InstalledDat")]
public sealed class UiButtonCorpusSweepTests
{
private static string DatDirectory =>

View file

@ -23,6 +23,7 @@ param(
[int]$BuildTimeoutSeconds = 900,
[int]$TestTimeoutSeconds = 600,
[int]$HangTimeoutSeconds = 180,
[string]$TestFilter = 'Lane!=InstalledDat&Lane!=PreparedPackage&Lane!=Live&Lane!=Manual&Purpose!=Diagnostic&Status!=KnownFailure',
[switch]$SkipRestore,
[switch]$SkipBuild
)
@ -373,6 +374,8 @@ try {
'--no-restore'
'--no-build'
'--nologo'
'--filter'
$TestFilter
'--results-directory'
$projectResultDirectory
'--logger'
@ -434,6 +437,7 @@ try {
DurationSeconds = [Math]::Round(($gateEndedUtc - $gateStartedUtc).TotalSeconds, 3)
Solution = 'AcDream.slnx'
TestProjectCount = $testProjects.Count
TestFilter = $TestFilter
Timeouts = [ordered]@{
RestoreSeconds = $RestoreTimeoutSeconds
BuildSeconds = $BuildTimeoutSeconds