Exact-pixel and production-perf comparison of6c79d35c(+ only the A2 normal files, so no terrain mask is needed) againstc51b07efwith no pack: connected as +Acdream, visible window, one isolated config clone per variant, pinned clocks. Open field: the only differences are idle pose, mana digits and a passing flyer. Holtburg: same-binary-twice defines the dynamic mask (9.8%); in the static 90% both self-diffs have ZERO pixels with |d|>=8 while base+normals vs HEAD-off has 841/729 - all streaks inside the animated lifestone. Buildings, ground, trees, sky and UI are clean. Perf (uncapped Release, no automation observer, ACDREAM_FRAME_PROF=1): Holtburg CPU p50 4.7 -> 4.1 ms, Arwic 6.0 -> 5.2 ms, GPU unchanged, alloc/frame 574 KB -> 21 KB. No regression; F5b's '27.8 ms retail CPU' was the observer. Three false alarms recorded so nobody repeats them: the isolated gate settings lack fieldOfView (90 vs the real 86.33 -> a 0.952 zoom); the real %APPDATA% settings still selected acdream.atmospheric/low (pack ON); a minimized GLFW window is throttled and never settles. Tools: tools/vm0/capture-visible.ps1 (pre-campaign gate + -Exe/-Live/ -ConfigDir/-CharacterName/-PreCaptureCommand), tools/vm0/perf-run.sh, and -BuildingDetailTextures on run-offline-pixel-gate.ps1. Baseline patches under docs/research/evidence/vm0/. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
352 lines
15 KiB
PowerShell
352 lines
15 KiB
PowerShell
# Campaign VM slice VM0 capture helper. The pre-campaign pixel gate (6c79d35c)
|
|
# with three changes: an explicit -Exe so one script drives every binary under
|
|
# test, a VISIBLE normal window (the real product condition - a minimized GLFW
|
|
# surface is throttled by Windows and HEAD's hidden borderless mode changes the
|
|
# framing, see docs/research/2026-08-22-vm0-default-path-invariance.md), and an
|
|
# optional -ConfigDir for binaries that honour ACDREAM_CONFIG_DIR.
|
|
<#
|
|
.SYNOPSIS
|
|
Campaign V self-differential pixel gate. Captures deterministic offline
|
|
screenshots and optionally compares them against a baseline capture.
|
|
|
|
.DESCRIPTION
|
|
Every Campaign V renderer-port slice must prove it changed no pixels. The
|
|
connected lifecycle gate needs a live ACE server and the user's attention;
|
|
this gate needs neither. The client is launched WITHOUT ACDREAM_LIVE, so it
|
|
renders the world straight from the DATs, no session is created, and no ACE
|
|
state can be disturbed.
|
|
|
|
Coverage: terrain, terrain blending, scenery, static world meshes, sky and
|
|
fog, and the full retained UI (vitals, spell bar, toolbar, chat, radar).
|
|
It does NOT cover EnvCell interiors, particles, or the paperdoll viewport —
|
|
those still need the connected route, so slices touching them (V4e, V4g)
|
|
must additionally be gated by the user.
|
|
|
|
Determinism levers, all three forced here rather than left to the ambient
|
|
environment, because a capture that inherits any of them is not comparable
|
|
against one that did not:
|
|
|
|
* ACDREAM_DAY_GROUP -- the sky day group, so weather and the keyframe
|
|
do not drift between runs.
|
|
* ACDREAM_WORLD_TIME -- the Dereth day fraction. This moves the SUN, so
|
|
it moves every lit surface, not just the sky. Slice V7 measured 22.3% of
|
|
the frame differing between two captures 45 s apart at one stop on one
|
|
backend with this unpinned.
|
|
* ACDREAM_SKY_PHASE_SECONDS -- the cloud sheet's UV scroll, which runs on
|
|
wall time and is independent of the world clock by design.
|
|
|
|
ACDREAM_MSAA_SAMPLES is forced to 0 because multisample resolve positions
|
|
are explicitly unspecified across implementations, and at 4x they swamp
|
|
the threshold ninety-fold with two drivers' sample patterns. See plan
|
|
section 5.5.16.
|
|
|
|
A fixed warm-up lets streaming settle before the frame is captured.
|
|
|
|
.PARAMETER Out
|
|
Directory to write this capture into. Created if absent, emptied if present.
|
|
|
|
.PARAMETER Baseline
|
|
Optional. A directory from a previous capture. When supplied, every PNG is
|
|
compared against its namesake and the script fails on any mismatch.
|
|
|
|
.PARAMETER WarmupMs
|
|
Milliseconds to let the world stream and settle before capturing. Default
|
|
12000, which reliably reaches a fully populated view on the reference machine.
|
|
|
|
.PARAMETER DayGroup
|
|
Sky day-group index to pin. Default 0.
|
|
|
|
.PARAMETER Tolerance
|
|
Per-channel absolute difference allowed. Default 2, matching the project's
|
|
pinned screenshot rule.
|
|
|
|
.PARAMETER MaxDifferentFraction
|
|
Fraction of differing pixels allowed. Default 0.001, likewise pinned.
|
|
|
|
.PARAMETER WorldDayFraction
|
|
Dereth day fraction pinned on the launch, in [0, 1). Default 0.5 -- noon, the
|
|
brightest and flattest lighting the day has.
|
|
|
|
.PARAMETER SkyPhaseSeconds
|
|
Elapsed-seconds value the sky's UV scroll is pinned to. Default 0, the cloud
|
|
sheet's authored origin.
|
|
|
|
.PARAMETER MsaaSamples
|
|
MSAA sample count forced on the launch. Default 0 -- off, which is what plan
|
|
section 2's pixel-acceptance contract specifies. Pass -1 to leave the quality
|
|
preset alone.
|
|
|
|
.PARAMETER MaskTopPixels
|
|
Height in pixels of the sky and treeline band excluded from comparison.
|
|
Default 280.
|
|
|
|
This is not a fudge factor, and slice V10 re-measured it rather than assuming
|
|
the V7 clock pins had retired it. Two captures at the SAME commit once
|
|
differed in 0.29% of pixels, essentially all of it in the top rows, because
|
|
the sky legitimately animates: clouds scroll on wall time and the Dereth
|
|
clock moves the sun, and therefore every lit surface. Pinning both clocks --
|
|
which this script now does -- shrinks that but does NOT remove it. Measured
|
|
at V10, two launches of the same binary with all four levers pinned:
|
|
|
|
whole frame, GL vs GL 1,011 px of 921,600 (1.10e-03)
|
|
whole frame, VK vs VK 482 px of 921,600 (5.23e-04)
|
|
top 280 masked, GL vs GL 10 px of 563,200 (1.78e-05)
|
|
top 280 masked, VK vs VK 8 px of 563,200 (1.42e-05)
|
|
|
|
So the band still moves between launches on both backends -- enough that an
|
|
unmasked self-differential on GL fails the 0.001 threshold on noise alone --
|
|
and everything below it is bit-stable to about ten pixels. Masking the band
|
|
is what makes the rest a strict identity check instead of forcing a loose
|
|
tolerance that would hide real regressions everywhere else.
|
|
|
|
Consequence, unchanged: this gate does NOT cover sky rendering.
|
|
|
|
Do NOT reach for a LARGER value to make a failing gate pass: plan section 7.1
|
|
rule 2. Its legitimate use is diagnostic -- reporting a masked/unmasked split
|
|
so a known, registered band (for instance AD-46's treeline) can be quantified
|
|
separately from the rest of the frame.
|
|
|
|
.PARAMETER SkipBuild
|
|
Skip the Release build (use when the caller already built).
|
|
|
|
.EXAMPLE
|
|
# Capture a baseline at the parent commit, then gate the slice:
|
|
git stash
|
|
tools/run-offline-pixel-gate.ps1 -Out artifacts/gate-base
|
|
git stash pop
|
|
tools/run-offline-pixel-gate.ps1 -Out artifacts/gate-head -Baseline artifacts/gate-base
|
|
#>
|
|
[CmdletBinding()]
|
|
param(
|
|
[Parameter(Mandatory = $true)][string]$Out,
|
|
[string]$Baseline,
|
|
[int]$WarmupMs = 12000,
|
|
[int]$DayGroup = 0,
|
|
[double]$WorldDayFraction = 0.5,
|
|
[double]$SkyPhaseSeconds = 0,
|
|
[int]$MsaaSamples = 0,
|
|
[int]$Tolerance = 2,
|
|
[double]$MaxDifferentFraction = 0.001,
|
|
[int]$MaskTopPixels = 280,
|
|
[switch]$SkipBuild,
|
|
[Parameter(Mandatory = $true)][string]$Exe,
|
|
[string]$ConfigDir,
|
|
# Connect to the local ACE (ACDREAM_TEST_HOST/PORT/USER/PASS must already be in the
|
|
# environment - never written into this script). Uses the retail chase camera.
|
|
[switch]$Live,
|
|
# Character NAME to log in as. ACE orders the roster by last login, so an index is
|
|
# not stable across consecutive runs; a name is.
|
|
[string]$CharacterName = '+Acdream',
|
|
# Optional slash command run after entering the world (e.g. '/telepoi Holtburg'),
|
|
# followed by a materialization wait, before the warmup and capture.
|
|
[string]$PreCaptureCommand
|
|
)
|
|
|
|
$ErrorActionPreference = 'Stop'
|
|
$repo = Split-Path -Parent $PSScriptRoot
|
|
$exe = $Exe
|
|
$cli = Join-Path $repo 'src\AcDream.Cli\bin\Release\net10.0\AcDream.Cli.dll'
|
|
|
|
function Write-Step($message) { Write-Host "[pixel-gate] $message" }
|
|
|
|
# --- 1. Build -----------------------------------------------------------------
|
|
if (-not $SkipBuild) {
|
|
Write-Step 'building Release'
|
|
& dotnet build (Join-Path $repo 'AcDream.slnx') -c Release --nologo -v q | Out-Null
|
|
if ($LASTEXITCODE -ne 0) { throw "Release build failed with exit code $LASTEXITCODE." }
|
|
}
|
|
if (-not (Test-Path $exe)) { throw "Client not found at $exe. Build Release first." }
|
|
|
|
# --- 2. Prepare the capture directory and probe script ------------------------
|
|
if (Test-Path $Out) { Remove-Item -Recurse -Force $Out }
|
|
New-Item -ItemType Directory -Force -Path $Out | Out-Null
|
|
|
|
$probe = Join-Path $Out 'offline.probe.txt'
|
|
# The script runner reads one command per line. A single settled capture is the
|
|
# whole gate: a second stop would need camera movement, which offline has no
|
|
# deterministic way to drive.
|
|
if ($Live) {
|
|
$tele = if ($PreCaptureCommand) { "command $PreCaptureCommand`nwait materialized 1 60000`n" } else { '' }
|
|
Set-Content -Encoding utf8 -Path $probe -Value @"
|
|
wait world-ready 90000
|
|
wait world-visible 30000
|
|
${tele}sleep $WarmupMs
|
|
screenshot world-offline 30000
|
|
sleep 500
|
|
"@
|
|
} else {
|
|
Set-Content -Encoding utf8 -Path $probe -Value @"
|
|
sleep $WarmupMs
|
|
screenshot world-offline 30000
|
|
sleep 500
|
|
"@
|
|
}
|
|
|
|
$log = Join-Path $Out 'client.log'
|
|
|
|
# --- 3. Launch offline --------------------------------------------------------
|
|
$previousLive = $env:ACDREAM_LIVE
|
|
if ($Live) { $env:ACDREAM_LIVE = '1' } else { Remove-Item Env:\ACDREAM_LIVE -ErrorAction SilentlyContinue }
|
|
$env:ACDREAM_DAT_DIR = Join-Path $env:USERPROFILE "Documents\Asheron's Call"
|
|
$env:ACDREAM_NO_AUDIO = '1'
|
|
if ($ConfigDir) { $env:ACDREAM_CONFIG_DIR = $ConfigDir; $env:ACDREAM_DATA_DIR = (Join-Path $Out 'isolated-data'); $env:ACDREAM_CACHE_DIR = (Join-Path $Out 'isolated-cache') }
|
|
else { Remove-Item Env:\ACDREAM_CONFIG_DIR,Env:\ACDREAM_DATA_DIR,Env:\ACDREAM_CACHE_DIR -ErrorAction SilentlyContinue }
|
|
$env:ACDREAM_RETAIL_UI = '1'
|
|
$env:ACDREAM_DAY_GROUP = "$DayGroup"
|
|
$env:ACDREAM_UI_PROBE_SCRIPT = $probe
|
|
$env:ACDREAM_AUTOMATION_ARTIFACT_DIR = $Out
|
|
|
|
# The determinism pins, forced rather than inherited. See .DESCRIPTION.
|
|
$invariant = [System.Globalization.CultureInfo]::InvariantCulture
|
|
$env:ACDREAM_WORLD_TIME = $WorldDayFraction.ToString($invariant)
|
|
$env:ACDREAM_SKY_PHASE_SECONDS = $SkyPhaseSeconds.ToString($invariant)
|
|
if ($MsaaSamples -ge 0) { $env:ACDREAM_MSAA_SAMPLES = "$MsaaSamples" }
|
|
else { Remove-Item Env:\ACDREAM_MSAA_SAMPLES -ErrorAction SilentlyContinue }
|
|
|
|
Write-Step "launching offline client (warmup ${WarmupMs}ms, day group $DayGroup, day fraction $WorldDayFraction, sky phase $SkyPhaseSeconds, MSAA $MsaaSamples)"
|
|
$launchArgs = @()
|
|
if ($Live) {
|
|
$sessionConfigPath = Join-Path $Out 'graphical-session.json'
|
|
[ordered]@{
|
|
version = 1
|
|
sessions = @([ordered]@{
|
|
id = 'vm0-capture'
|
|
endpoint = [ordered]@{ host = $env:ACDREAM_TEST_HOST; port = [int]$env:ACDREAM_TEST_PORT }
|
|
account = $env:ACDREAM_TEST_USER
|
|
character = [ordered]@{ name = $CharacterName }
|
|
credential = [ordered]@{ provider = 'Environment'; reference = 'ACDREAM_TEST_PASS' }
|
|
})
|
|
} | ConvertTo-Json -Depth 8 | Set-Content -LiteralPath $sessionConfigPath -Encoding utf8
|
|
$launchArgs = @('--session-config', ('"' + $sessionConfigPath + '"'))
|
|
}
|
|
$proc = Start-Process -FilePath $exe -ArgumentList $launchArgs -RedirectStandardOutput $log `
|
|
-RedirectStandardError "$log.err" -PassThru -WindowStyle Normal
|
|
|
|
try {
|
|
$shots = Join-Path $Out 'screenshots'
|
|
$deadline = (Get-Date).AddMilliseconds($WarmupMs + 60000 + $(if ($Live) { 120000 } else { 0 }))
|
|
$captured = $false
|
|
while ((Get-Date) -lt $deadline) {
|
|
if ((Test-Path $shots) -and (Get-ChildItem $shots -Filter *.png -ErrorAction SilentlyContinue)) {
|
|
$captured = $true
|
|
break
|
|
}
|
|
if ($proc.HasExited) { break }
|
|
Start-Sleep -Milliseconds 1000
|
|
}
|
|
|
|
if (-not $captured) {
|
|
Write-Host (Get-Content $log -Tail 30 -ErrorAction SilentlyContinue)
|
|
throw 'No screenshot was captured before the deadline.'
|
|
}
|
|
# Let the probe script finish its trailing sleep so the PNG is fully flushed.
|
|
Start-Sleep -Milliseconds 1500
|
|
}
|
|
finally {
|
|
# Graceful close: WM_CLOSE runs the shutdown path, so the ownership ledger
|
|
# converges the way the lifecycle tests expect. No ACE session exists here,
|
|
# but keeping the habit means this script is safe to point at a live run too.
|
|
$app = Get-Process -Name AcDream.App -ErrorAction SilentlyContinue
|
|
if ($app) {
|
|
$app.CloseMainWindow() | Out-Null
|
|
if (-not $app.WaitForExit(10000)) {
|
|
Write-Step 'WM_CLOSE timed out; forcing'
|
|
$app | Stop-Process -Force
|
|
}
|
|
}
|
|
Remove-Item Env:\ACDREAM_MSAA_SAMPLES -ErrorAction SilentlyContinue
|
|
Remove-Item Env:\ACDREAM_WORLD_TIME -ErrorAction SilentlyContinue
|
|
Remove-Item Env:\ACDREAM_SKY_PHASE_SECONDS -ErrorAction SilentlyContinue
|
|
if ($previousLive) { $env:ACDREAM_LIVE = $previousLive }
|
|
}
|
|
|
|
$captures = Get-ChildItem (Join-Path $Out 'screenshots') -Filter *.png
|
|
Write-Step "captured $($captures.Count) screenshot(s) into $Out"
|
|
|
|
# The offline window is minimised but still focusable, so a stray scroll or key
|
|
# press from whoever is at the keyboard can move the camera mid-capture. That
|
|
# produces two screenshots of the same scene from different camera positions —
|
|
# a huge, entirely spurious pixel difference. This actually happened during
|
|
# slice V4b. Detect it and fail loudly rather than let a comparison be
|
|
# interpreted as a rendering regression.
|
|
$cameraInput = Select-String -Path $log -Pattern 'ScrollUp|ScrollDown|ZoomIn|ZoomOut|CameraZoom' `
|
|
-CaseSensitive -ErrorAction SilentlyContinue
|
|
if ($cameraInput) {
|
|
Write-Host ''
|
|
Write-Host '[pixel-gate] ABORTED: camera-affecting input reached the capture window.' -ForegroundColor Red
|
|
Write-Host ' The capture is not comparable. Re-run without touching the machine.' -ForegroundColor Red
|
|
$cameraInput | Select-Object -First 5 | ForEach-Object { Write-Host " $($_.Line)" -ForegroundColor Red }
|
|
exit 2
|
|
}
|
|
|
|
# --- 4. Compare against the baseline -----------------------------------------
|
|
if (-not $Baseline) {
|
|
Write-Step 'no baseline supplied; capture only'
|
|
exit 0
|
|
}
|
|
|
|
if (-not (Test-Path $cli)) { throw "AcDream.Cli not found at $cli." }
|
|
|
|
# Build the sky mask. The comparer skips any pixel whose mask alpha is non-zero,
|
|
# so the band we exclude is opaque and everything compared is transparent.
|
|
$maskPath = $null
|
|
if ($MaskTopPixels -gt 0) {
|
|
Add-Type -AssemblyName System.Drawing
|
|
$probeImage = [System.Drawing.Bitmap]::FromFile($captures[0].FullName)
|
|
$width = $probeImage.Width
|
|
$height = $probeImage.Height
|
|
$probeImage.Dispose()
|
|
|
|
$mask = New-Object System.Drawing.Bitmap($width, $height, [System.Drawing.Imaging.PixelFormat]::Format32bppArgb)
|
|
$graphics = [System.Drawing.Graphics]::FromImage($mask)
|
|
$graphics.Clear([System.Drawing.Color]::FromArgb(0, 0, 0, 0))
|
|
$opaque = New-Object System.Drawing.SolidBrush ([System.Drawing.Color]::FromArgb(255, 255, 0, 255))
|
|
$graphics.FillRectangle($opaque, 0, 0, $width, [Math]::Min($MaskTopPixels, $height))
|
|
$graphics.Dispose()
|
|
$opaque.Dispose()
|
|
|
|
$maskPath = Join-Path $Out 'sky-mask.png'
|
|
$mask.Save($maskPath, [System.Drawing.Imaging.ImageFormat]::Png)
|
|
$mask.Dispose()
|
|
Write-Step "masking the top $MaskTopPixels rows (animated sky and treeline)"
|
|
}
|
|
|
|
$failed = @()
|
|
foreach ($shot in $captures) {
|
|
$expected = Join-Path $Baseline "screenshots\$($shot.Name)"
|
|
if (-not (Test-Path $expected)) {
|
|
$failed += "$($shot.Name): no baseline counterpart"
|
|
continue
|
|
}
|
|
|
|
$report = Join-Path $Out "compare-$($shot.BaseName).json"
|
|
if ($maskPath) {
|
|
& dotnet $cli compare-screenshots $expected $shot.FullName $report $Tolerance $MaxDifferentFraction $maskPath | Out-Null
|
|
}
|
|
else {
|
|
& dotnet $cli compare-screenshots $expected $shot.FullName $report $Tolerance $MaxDifferentFraction | Out-Null
|
|
}
|
|
$verdict = Get-Content $report -Raw | ConvertFrom-Json
|
|
|
|
$fraction = $verdict.differentPixelFraction
|
|
if ($null -eq $fraction) { $fraction = $verdict.DifferentPixelFraction }
|
|
$passed = $verdict.passed
|
|
if ($null -eq $passed) { $passed = $verdict.Passed }
|
|
|
|
if ($passed) {
|
|
Write-Step "PASS $($shot.Name) (differing fraction $fraction)"
|
|
}
|
|
else {
|
|
$failed += "$($shot.Name): differing fraction $fraction exceeds $MaxDifferentFraction (report: $report)"
|
|
}
|
|
}
|
|
|
|
if ($failed.Count -gt 0) {
|
|
Write-Host ''
|
|
Write-Host '[pixel-gate] FAILED:' -ForegroundColor Red
|
|
$failed | ForEach-Object { Write-Host " - $_" -ForegroundColor Red }
|
|
exit 1
|
|
}
|
|
|
|
Write-Step 'all screenshots match the baseline'
|
|
exit 0
|