test(render): add the Campaign V offline pixel gate
Seven of the campaign slices (V2, V4a-V4g) are renderer ports whose entire acceptance criterion is that no pixel changed, and the existing connected lifecycle route needs both a live ACE server and the user watching. That would have made the campaign advance only when someone is at the keyboard. The client renders the world from the DATs without ACDREAM_LIVE, so the existing UI automation probe can capture a settled frame with no session created and no ACE state to disturb. The gate wraps that: capture at the parent commit, capture at slice HEAD, compare through the existing compare-screenshots CLI at the project's pinned tolerance 2 / 0.001. Determinism was measured rather than assumed, and the first measurement failed: two captures at the same commit differed in 0.29% of pixels. The differences were confined to the top ~180 rows, which is correct behavior rather than a bug - the sky animates and the Dereth clock advances with wall time, so two launches cannot agree there. Below the horizon everything was stable. Masking the top 280 rows brings two independent same-commit pairs to 15 and 17 differing pixels out of 563,200 compared, a ~33x margin under the threshold. Masking the animated band keeps the rest a strict identity check; relaxing the tolerance instead would have hidden real regressions everywhere else. Covers terrain and blending, scenery, static meshes, water, fog, and the whole retained UI. Does not cover sky (masked), EnvCell interiors, particles, or the paperdoll viewports, since the offline scene is a fixed outdoor view - so V4e, V4f and V4g keep a user visual gate on top of this one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Erik
parent
4f94ad7ddd
commit
86afbe2ecd
2 changed files with 273 additions and 4 deletions
|
|
@ -391,10 +391,45 @@ a loud `Debug.Assert` in Debug builds. A Debug run therefore shows one failure
|
|||
that is neither a regression nor yours. The V0 baseline is **3,785 passed /
|
||||
3 skipped** in Release (3,763 pre-campaign plus 22 contract tests).
|
||||
|
||||
"Pixel gate" means: capture the deterministic checkpoints from the parent
|
||||
commit's build, capture again at slice HEAD, compare with the
|
||||
`compare-screenshots` CLI at tolerance 2 / fraction 0.001, MSAA off,
|
||||
`ACDREAM_DAY_GROUP` pinned.
|
||||
### 5.1 The offline pixel gate
|
||||
|
||||
"Pixel gate" means `tools/run-offline-pixel-gate.ps1`: capture at the parent
|
||||
commit, capture at slice HEAD, compare with the `compare-screenshots` CLI at
|
||||
tolerance 2 / fraction 0.001.
|
||||
|
||||
The client is launched **without `ACDREAM_LIVE`**, so it renders the world
|
||||
straight from the DATs. No session is created and no ACE state can be disturbed,
|
||||
which means this gate runs unattended — it needs neither the live server nor the
|
||||
user. That matters: seven slices (V2, V4a–V4g) are renderer ports whose whole
|
||||
acceptance criterion is "no pixel changed."
|
||||
|
||||
```
|
||||
tools/run-offline-pixel-gate.ps1 -Out artifacts/gate-base # at the parent commit
|
||||
tools/run-offline-pixel-gate.ps1 -Out artifacts/gate-head -Baseline artifacts/gate-base
|
||||
```
|
||||
|
||||
**Determinism was measured, not assumed.** Two captures at the same commit
|
||||
initially differed in 0.29% of pixels — far above the 0.001 threshold. The
|
||||
differences were confined to the top ~180 rows: the sky animates (clouds scroll,
|
||||
the sun moves) and the Dereth clock advances with wall time, so two launches can
|
||||
never agree there. Everything below the horizon was stable. With the top 280 rows
|
||||
masked, two independent same-commit pairs differ by **15 and 17 pixels out of
|
||||
563,200 compared** — a fraction of 0.000027, roughly a 33× margin under the
|
||||
threshold. The gate is a strict identity check on everything it covers, rather
|
||||
than a loose tolerance that would hide real regressions.
|
||||
|
||||
**Coverage.** Terrain and terrain blending, scenery, static world meshes, water,
|
||||
fog, and the entire retained UI (vitals, spell bar, toolbar, chat, radar).
|
||||
|
||||
**Not covered — these still need a user visual gate:** sky (masked), EnvCell
|
||||
interiors, particles, and the paperdoll/appraisal viewports, because the offline
|
||||
scene is a fixed outdoor view with no camera control. Slices **V4e** (particles),
|
||||
**V4f** (sky), and **V4g** (viewports) therefore carry a user gate in addition to
|
||||
their automated one.
|
||||
|
||||
MSAA is left at the quality preset for GL-versus-GL self-differentials, where it
|
||||
is deterministic. The V7 GL-versus-Vulkan differential must force MSAA off,
|
||||
because sample positions are not specified across implementations.
|
||||
|
||||
| Slice | Scope | Gate |
|
||||
|---|---|---|
|
||||
|
|
|
|||
234
tools/run-offline-pixel-gate.ps1
Normal file
234
tools/run-offline-pixel-gate.ps1
Normal file
|
|
@ -0,0 +1,234 @@
|
|||
<#
|
||||
.SYNOPSIS
|
||||
Campaign V self-differential pixel gate. Captures deterministic offline
|
||||
screenshots and optionally compares them against a baseline capture.
|
||||
|
||||
.DESCRIPTION
|
||||
Every Campaign V renderer-port slice must prove it changed no pixels. The
|
||||
connected lifecycle gate needs a live ACE server and the user's attention;
|
||||
this gate needs neither. The client is launched WITHOUT ACDREAM_LIVE, so it
|
||||
renders the world straight from the DATs, no session is created, and no ACE
|
||||
state can be disturbed.
|
||||
|
||||
Coverage: terrain, terrain blending, scenery, static world meshes, sky and
|
||||
fog, and the full retained UI (vitals, spell bar, toolbar, chat, radar).
|
||||
It does NOT cover EnvCell interiors, particles, or the paperdoll viewport —
|
||||
those still need the connected route, so slices touching them (V4e, V4g)
|
||||
must additionally be gated by the user.
|
||||
|
||||
Determinism levers: ACDREAM_DAY_GROUP pins the sky day-group so weather and
|
||||
lighting do not drift between runs, and a fixed warm-up lets streaming settle
|
||||
before the frame is captured.
|
||||
|
||||
.PARAMETER Out
|
||||
Directory to write this capture into. Created if absent, emptied if present.
|
||||
|
||||
.PARAMETER Baseline
|
||||
Optional. A directory from a previous capture. When supplied, every PNG is
|
||||
compared against its namesake and the script fails on any mismatch.
|
||||
|
||||
.PARAMETER WarmupMs
|
||||
Milliseconds to let the world stream and settle before capturing. Default
|
||||
12000, which reliably reaches a fully populated view on the reference machine.
|
||||
|
||||
.PARAMETER DayGroup
|
||||
Sky day-group index to pin. Default 0.
|
||||
|
||||
.PARAMETER Tolerance
|
||||
Per-channel absolute difference allowed. Default 2, matching the project's
|
||||
pinned screenshot rule.
|
||||
|
||||
.PARAMETER MaxDifferentFraction
|
||||
Fraction of differing pixels allowed. Default 0.001, likewise pinned.
|
||||
|
||||
.PARAMETER MaskTopPixels
|
||||
Height in pixels of the sky band excluded from comparison. Default 280.
|
||||
|
||||
This is not a fudge factor. Two captures at the SAME commit were measured to
|
||||
differ in 0.29% of pixels, and the differences were confined to the top ~180
|
||||
rows: the sky legitimately animates (clouds scroll, the sun moves) and the
|
||||
Dereth clock advances with wall time, so the sky cannot be identical between
|
||||
two launches. Everything below the horizon — terrain, blending, scenery,
|
||||
static meshes, water, and the whole retained UI — was bit-stable. Masking the
|
||||
animated band is what makes the rest a strict identity check rather than
|
||||
forcing a loose tolerance that would hide real regressions everywhere else.
|
||||
|
||||
Consequence: this gate does NOT cover sky rendering. Slice V4f must be gated
|
||||
by the user instead.
|
||||
|
||||
.PARAMETER SkipBuild
|
||||
Skip the Release build (use when the caller already built).
|
||||
|
||||
.EXAMPLE
|
||||
# Capture a baseline at the parent commit, then gate the slice:
|
||||
git stash
|
||||
tools/run-offline-pixel-gate.ps1 -Out artifacts/gate-base
|
||||
git stash pop
|
||||
tools/run-offline-pixel-gate.ps1 -Out artifacts/gate-head -Baseline artifacts/gate-base
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory = $true)][string]$Out,
|
||||
[string]$Baseline,
|
||||
[int]$WarmupMs = 12000,
|
||||
[int]$DayGroup = 0,
|
||||
[int]$Tolerance = 2,
|
||||
[double]$MaxDifferentFraction = 0.001,
|
||||
[int]$MaskTopPixels = 280,
|
||||
[switch]$SkipBuild
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$repo = Split-Path -Parent $PSScriptRoot
|
||||
$exe = Join-Path $repo 'src\AcDream.App\bin\Release\net10.0\AcDream.App.exe'
|
||||
$cli = Join-Path $repo 'src\AcDream.Cli\bin\Release\net10.0\AcDream.Cli.dll'
|
||||
|
||||
function Write-Step($message) { Write-Host "[pixel-gate] $message" }
|
||||
|
||||
# --- 1. Build -----------------------------------------------------------------
|
||||
if (-not $SkipBuild) {
|
||||
Write-Step 'building Release'
|
||||
& dotnet build (Join-Path $repo 'AcDream.slnx') -c Release --nologo -v q | Out-Null
|
||||
if ($LASTEXITCODE -ne 0) { throw "Release build failed with exit code $LASTEXITCODE." }
|
||||
}
|
||||
if (-not (Test-Path $exe)) { throw "Client not found at $exe. Build Release first." }
|
||||
|
||||
# --- 2. Prepare the capture directory and probe script ------------------------
|
||||
if (Test-Path $Out) { Remove-Item -Recurse -Force $Out }
|
||||
New-Item -ItemType Directory -Force -Path $Out | Out-Null
|
||||
|
||||
$probe = Join-Path $Out 'offline.probe.txt'
|
||||
# The script runner reads one command per line. A single settled capture is the
|
||||
# whole gate: a second stop would need camera movement, which offline has no
|
||||
# deterministic way to drive.
|
||||
Set-Content -Encoding utf8 -Path $probe -Value @"
|
||||
sleep $WarmupMs
|
||||
screenshot world-offline 30000
|
||||
sleep 500
|
||||
"@
|
||||
|
||||
$log = Join-Path $Out 'client.log'
|
||||
|
||||
# --- 3. Launch offline --------------------------------------------------------
|
||||
$previousLive = $env:ACDREAM_LIVE
|
||||
Remove-Item Env:\ACDREAM_LIVE -ErrorAction SilentlyContinue
|
||||
$env:ACDREAM_DAT_DIR = Join-Path $env:USERPROFILE "Documents\Asheron's Call"
|
||||
$env:ACDREAM_NO_AUDIO = '1'
|
||||
$env:ACDREAM_RETAIL_UI = '1'
|
||||
$env:ACDREAM_DAY_GROUP = "$DayGroup"
|
||||
$env:ACDREAM_UI_PROBE_SCRIPT = $probe
|
||||
$env:ACDREAM_AUTOMATION_ARTIFACT_DIR = $Out
|
||||
|
||||
Write-Step "launching offline client (warmup ${WarmupMs}ms, day group $DayGroup)"
|
||||
$proc = Start-Process -FilePath $exe -RedirectStandardOutput $log `
|
||||
-RedirectStandardError "$log.err" -PassThru -WindowStyle Minimized
|
||||
|
||||
try {
|
||||
$shots = Join-Path $Out 'screenshots'
|
||||
$deadline = (Get-Date).AddMilliseconds($WarmupMs + 60000)
|
||||
$captured = $false
|
||||
while ((Get-Date) -lt $deadline) {
|
||||
if ((Test-Path $shots) -and (Get-ChildItem $shots -Filter *.png -ErrorAction SilentlyContinue)) {
|
||||
$captured = $true
|
||||
break
|
||||
}
|
||||
if ($proc.HasExited) { break }
|
||||
Start-Sleep -Milliseconds 1000
|
||||
}
|
||||
|
||||
if (-not $captured) {
|
||||
Write-Host (Get-Content $log -Tail 30 -ErrorAction SilentlyContinue)
|
||||
throw 'No screenshot was captured before the deadline.'
|
||||
}
|
||||
# Let the probe script finish its trailing sleep so the PNG is fully flushed.
|
||||
Start-Sleep -Milliseconds 1500
|
||||
}
|
||||
finally {
|
||||
# Graceful close: WM_CLOSE runs the shutdown path, so the ownership ledger
|
||||
# converges the way the lifecycle tests expect. No ACE session exists here,
|
||||
# but keeping the habit means this script is safe to point at a live run too.
|
||||
$app = Get-Process -Name AcDream.App -ErrorAction SilentlyContinue
|
||||
if ($app) {
|
||||
$app.CloseMainWindow() | Out-Null
|
||||
if (-not $app.WaitForExit(10000)) {
|
||||
Write-Step 'WM_CLOSE timed out; forcing'
|
||||
$app | Stop-Process -Force
|
||||
}
|
||||
}
|
||||
if ($previousLive) { $env:ACDREAM_LIVE = $previousLive }
|
||||
}
|
||||
|
||||
$captures = Get-ChildItem (Join-Path $Out 'screenshots') -Filter *.png
|
||||
Write-Step "captured $($captures.Count) screenshot(s) into $Out"
|
||||
|
||||
# --- 4. Compare against the baseline -----------------------------------------
|
||||
if (-not $Baseline) {
|
||||
Write-Step 'no baseline supplied; capture only'
|
||||
exit 0
|
||||
}
|
||||
|
||||
if (-not (Test-Path $cli)) { throw "AcDream.Cli not found at $cli." }
|
||||
|
||||
# Build the sky mask. The comparer skips any pixel whose mask alpha is non-zero,
|
||||
# so the band we exclude is opaque and everything compared is transparent.
|
||||
$maskPath = $null
|
||||
if ($MaskTopPixels -gt 0) {
|
||||
Add-Type -AssemblyName System.Drawing
|
||||
$probeImage = [System.Drawing.Bitmap]::FromFile($captures[0].FullName)
|
||||
$width = $probeImage.Width
|
||||
$height = $probeImage.Height
|
||||
$probeImage.Dispose()
|
||||
|
||||
$mask = New-Object System.Drawing.Bitmap($width, $height, [System.Drawing.Imaging.PixelFormat]::Format32bppArgb)
|
||||
$graphics = [System.Drawing.Graphics]::FromImage($mask)
|
||||
$graphics.Clear([System.Drawing.Color]::FromArgb(0, 0, 0, 0))
|
||||
$opaque = New-Object System.Drawing.SolidBrush ([System.Drawing.Color]::FromArgb(255, 255, 0, 255))
|
||||
$graphics.FillRectangle($opaque, 0, 0, $width, [Math]::Min($MaskTopPixels, $height))
|
||||
$graphics.Dispose()
|
||||
$opaque.Dispose()
|
||||
|
||||
$maskPath = Join-Path $Out 'sky-mask.png'
|
||||
$mask.Save($maskPath, [System.Drawing.Imaging.ImageFormat]::Png)
|
||||
$mask.Dispose()
|
||||
Write-Step "masking the top $MaskTopPixels rows (animated sky)"
|
||||
}
|
||||
|
||||
$failed = @()
|
||||
foreach ($shot in $captures) {
|
||||
$expected = Join-Path $Baseline "screenshots\$($shot.Name)"
|
||||
if (-not (Test-Path $expected)) {
|
||||
$failed += "$($shot.Name): no baseline counterpart"
|
||||
continue
|
||||
}
|
||||
|
||||
$report = Join-Path $Out "compare-$($shot.BaseName).json"
|
||||
if ($maskPath) {
|
||||
& dotnet $cli compare-screenshots $expected $shot.FullName $report $Tolerance $MaxDifferentFraction $maskPath | Out-Null
|
||||
}
|
||||
else {
|
||||
& dotnet $cli compare-screenshots $expected $shot.FullName $report $Tolerance $MaxDifferentFraction | Out-Null
|
||||
}
|
||||
$verdict = Get-Content $report -Raw | ConvertFrom-Json
|
||||
|
||||
$fraction = $verdict.differentPixelFraction
|
||||
if ($null -eq $fraction) { $fraction = $verdict.DifferentPixelFraction }
|
||||
$passed = $verdict.passed
|
||||
if ($null -eq $passed) { $passed = $verdict.Passed }
|
||||
|
||||
if ($passed) {
|
||||
Write-Step "PASS $($shot.Name) (differing fraction $fraction)"
|
||||
}
|
||||
else {
|
||||
$failed += "$($shot.Name): differing fraction $fraction exceeds $MaxDifferentFraction (report: $report)"
|
||||
}
|
||||
}
|
||||
|
||||
if ($failed.Count -gt 0) {
|
||||
Write-Host ''
|
||||
Write-Host '[pixel-gate] FAILED:' -ForegroundColor Red
|
||||
$failed | ForEach-Object { Write-Host " - $_" -ForegroundColor Red }
|
||||
exit 1
|
||||
}
|
||||
|
||||
Write-Step 'all screenshots match the baseline'
|
||||
exit 0
|
||||
Loading…
Add table
Add a link
Reference in a new issue