test(render): add blank-world attribution apparatus and a post-world GL sample
Investigating the V4c connected blank-world failure needed two things the tree
did not have: a way to tell whether a blank run is caused by the binary under
test, and a way to see GL state at the end of the world phase rather than only
at the frame clear. Both are apparatus only - no production behaviour changes,
and the new probe emits nothing unless ACDREAM_PROBE_GLSTATE=1.
run-blank-world-ab-probe.ps1 interleaves two client builds over the repeat
gate's exact connected route and reports the blank rate per arm. This exists
because the blank rate is not stable across blocks: the same V4c binary
measured 3/10 in one block and 7/10 in another an hour later, so a block of A
followed by a block of B confounds the change with whatever else moved on the
machine in between. Strict alternation shares that drift between both arms.
Run against V4c and its parent it reported 4/5 versus 0/5 (Fisher exact
p~0.024), which is what established the defect follows the binary.
run-blank-world-surface-probe.ps1 grabs the composited window off the desktop
with CopyFromScreen at the same moment the client writes its own screenshot.
No instrument inside the GL context can separate "the renderer drew nothing"
from "the read did not return what the renderer drew", because both live on
the same side of the readback; an independent witness can. It is what showed
the two disagree - see below.
EmitPostWorldGlStateIfChanged is a second sample of the existing [gl-state]
snapshot, taken at the end of the normal-world phase. The existing tripwire
samples just after the clear phase's RestoreFrameDefaults, so it can only
observe state that survives from one frame into the next, and the draw
framebuffer is restored by no frame-global path. A binding established during
the world phase and put back before the next clear was therefore invisible to
it. Sampling at both ends brackets the phase.
What the apparatus established, recorded here rather than in the campaign doc
because no fix landed and the doc's re-land conditions are unchanged:
* The world draw path is not what is missing from the frame. On a blank run
the desktop grab shows the atmosphere clear over the whole viewport and the
complete retained UI - chat, radar, toolbar, vitals - in their normal
places, with every 3-D surface absent. Terrain and sky are still raw GL and
V4c does not touch them, so whatever V4c disturbs is shared, not per-
renderer.
* The CPU issues the same work either way. With ACDREAM_PROBE_FLAP=1 the
render signature is identical between blank and rendered runs: same
RetailPViewInside branch, same resolved root, terrain drawn, 3,331 outdoor
statics and 6 live dynamics dispatched.
* Both GL-state samples read fbo=0, full 1280x720 viewport, scissor off and
err=0x0, byte-identical between blank and rendered runs.
* The client's own capture disagrees with the screen. glReadPixels returns
uniformly RGBA(0,0,0,0) on a frame the desktop grab shows as fog plus UI.
The default framebuffer is 4x multisampled (SampleBuffers=1, Samples=4 in
the capability report) and glReadPixels against a multisampled read
framebuffer is undefined per the GL spec, so the gate's blank-versus-
rendered verdict rests on undefined behaviour in both directions.
Baseline App tests 3,864 passed / 3 skipped, unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Erik
parent
c090cd693d
commit
fed636b9c0
4 changed files with 352 additions and 0 deletions
132
tools/run-blank-world-ab-probe.ps1
Normal file
132
tools/run-blank-world-ab-probe.ps1
Normal file
|
|
@ -0,0 +1,132 @@
|
|||
<#
|
||||
.SYNOPSIS
|
||||
Campaign V blank-world A/B attribution probe: interleaves two client builds
|
||||
over the same connected route and reports the blank rate per build.
|
||||
|
||||
.DESCRIPTION
|
||||
tools/run-repeat-connected-gate.ps1 answers "does THIS build blank?" It cannot
|
||||
answer "does this build blank BECAUSE of the change?", because the blank rate
|
||||
drifts with machine state - desktop composition, foreground window, driver
|
||||
residency - between one ten-run block and the next. A block of A followed by a
|
||||
block of B therefore confounds the change with whatever else moved in those
|
||||
twenty minutes, and that is precisely how a build gets blamed for an ambient
|
||||
defect.
|
||||
|
||||
This script runs A and B strictly alternately in one block, so any drift is
|
||||
shared by both arms, and prints a per-arm verdict table. Identical rates
|
||||
exonerate the change under test; a rate that follows the binary implicates it.
|
||||
|
||||
Each cycle is the repeat gate's cycle exactly: launch connected -> teleloc to
|
||||
the pinned cell -> wait world-visible -> settle -> screenshot -> graceful close
|
||||
-> cooldown. The verdict is the same screenshot-size test, so the two scripts'
|
||||
numbers are directly comparable.
|
||||
|
||||
.PARAMETER Pairs
|
||||
A/B pairs to run. Default 5 (ten launches).
|
||||
|
||||
.PARAMETER ExeA / ExeB
|
||||
The two client executables. Give each a stable staging directory - not the
|
||||
build output, which the next `dotnet build` overwrites.
|
||||
|
||||
.PARAMETER LabelA / LabelB
|
||||
Names for the arms in the verdict table.
|
||||
|
||||
.PARAMETER Teleloc
|
||||
The pinned location, as the full /teleloc argument string. Same default as the
|
||||
repeat gate; the failure rate is location-sensitive.
|
||||
|
||||
.PARAMETER MinRenderedBytes
|
||||
Screenshot size below which a run is called blank. Default 500000.
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[int]$Pairs = 5,
|
||||
[Parameter(Mandatory = $true)][string]$ExeA,
|
||||
[Parameter(Mandatory = $true)][string]$ExeB,
|
||||
[string]$LabelA = 'A',
|
||||
[string]$LabelB = 'B',
|
||||
[string]$Teleloc = "0x2E430012 57.895 42.116 16.802 1 0 0 0",
|
||||
[int]$MinRenderedBytes = 500000
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
foreach ($exe in @($ExeA, $ExeB)) {
|
||||
if (-not (Test-Path $exe)) { throw "Client not found at $exe." }
|
||||
}
|
||||
|
||||
# Never race a session someone is already playing: same account, same server.
|
||||
if (Get-Process -Name AcDream.App -ErrorAction SilentlyContinue) {
|
||||
throw 'AcDream.App is already running. This probe uses the shared test account and must not steal its session.'
|
||||
}
|
||||
|
||||
$root = Join-Path $env:TEMP "claude\blank-world-ab-$([DateTime]::Now.ToString('HHmmss'))"
|
||||
New-Item -ItemType Directory -Force -Path $root | Out-Null
|
||||
$results = @()
|
||||
|
||||
for ($pair = 1; $pair -le $Pairs; $pair++) {
|
||||
foreach ($arm in @(
|
||||
@{ Label = $LabelA; Exe = $ExeA },
|
||||
@{ Label = $LabelB; Exe = $ExeB })) {
|
||||
|
||||
$dir = Join-Path $root ("pair-{0}-{1}" -f $pair, $arm.Label)
|
||||
New-Item -ItemType Directory -Force -Path $dir | Out-Null
|
||||
Set-Content -Encoding utf8 (Join-Path $dir 'probe.txt') -Value @"
|
||||
wait world-ready 90000
|
||||
command /teleloc $Teleloc
|
||||
wait materialized 1 90000
|
||||
wait world-visible 30000
|
||||
sleep 15000
|
||||
screenshot ab-run 30000
|
||||
sleep 1000
|
||||
"@
|
||||
|
||||
$env:ACDREAM_DAT_DIR = Join-Path $env:USERPROFILE "Documents\Asheron's Call"
|
||||
$env:ACDREAM_LIVE = '1'
|
||||
$env:ACDREAM_TEST_HOST = '127.0.0.1'
|
||||
$env:ACDREAM_TEST_PORT = '9000'
|
||||
$env:ACDREAM_TEST_USER = 'testaccount'
|
||||
$env:ACDREAM_TEST_PASS = 'testpassword'
|
||||
$env:ACDREAM_RETAIL_UI = '1'
|
||||
$env:ACDREAM_UI_PROBE_SCRIPT = Join-Path $dir 'probe.txt'
|
||||
$env:ACDREAM_AUTOMATION_ARTIFACT_DIR = $dir
|
||||
|
||||
$log = Join-Path $dir 'client.log'
|
||||
$proc = Start-Process -FilePath $arm.Exe -RedirectStandardOutput $log `
|
||||
-RedirectStandardError "$log.err" -PassThru -WindowStyle Minimized
|
||||
|
||||
$shot = Join-Path $dir 'screenshots\ab-run.png'
|
||||
$deadline = (Get-Date).AddSeconds(180)
|
||||
while ((Get-Date) -lt $deadline) {
|
||||
if (Test-Path $shot) { Start-Sleep -Seconds 2; break }
|
||||
if ($proc.HasExited) { break }
|
||||
Start-Sleep -Seconds 2
|
||||
}
|
||||
|
||||
$app = Get-Process -Name AcDream.App -ErrorAction SilentlyContinue
|
||||
if ($app) {
|
||||
$app.CloseMainWindow() | Out-Null
|
||||
if (-not $app.WaitForExit(12000)) { $app | Stop-Process -Force }
|
||||
}
|
||||
|
||||
$size = if (Test-Path $shot) { (Get-Item $shot).Length } else { 0 }
|
||||
$verdict = if ($size -ge $MinRenderedBytes) { 'RENDERED' }
|
||||
elseif ($size -gt 0) { 'BLANK' }
|
||||
else { 'NO-CAPTURE' }
|
||||
$results += [pscustomobject]@{
|
||||
Pair = $pair; Arm = $arm.Label; Verdict = $verdict; Bytes = $size
|
||||
}
|
||||
Write-Host ("[ab-probe] pair {0}/{1} arm {2}: {3} ({4} bytes)" -f `
|
||||
$pair, $Pairs, $arm.Label, $verdict, $size)
|
||||
|
||||
Start-Sleep -Seconds 10 # let ACE clear the graceful logout before the next login
|
||||
}
|
||||
}
|
||||
|
||||
Write-Host ''
|
||||
$results | Format-Table -AutoSize | Out-String | Write-Host
|
||||
foreach ($label in @($LabelA, $LabelB)) {
|
||||
$arm = @($results | Where-Object Arm -eq $label)
|
||||
$bad = @($arm | Where-Object Verdict -ne 'RENDERED')
|
||||
Write-Host ("[ab-probe] {0}: {1}/{2} blank" -f $label, $bad.Count, $arm.Count)
|
||||
}
|
||||
Write-Host ("[ab-probe] artifacts: {0}" -f $root)
|
||||
Loading…
Add table
Add a link
Reference in a new issue