test(render): Campaign V slice V11 step 0 — the #256/#257 discriminator, both arms
Issues #256 (server-spawned signs and portals go invisible after repeated portal runs, while staying interactive) and #257 (working set grows to ~1.5 GB over the same session) were observed together in one long live Vulkan session, and both filings demanded the same thing before V11 deletes the OpenGL backend: run the churn on GL too. Vulkan-only growth or drift would mean the new arm's resource lifecycle is broken, and deleting its only reference implementation while that was true would be wrong even with the cutover signed. So the discriminator is built and run first, and it can stop the slice. tools/run-portal-churn-soak.ps1 generates a route of N cycles over three portal-bearing stops taken from the two existing connected routes, runs it once per backend from one binary, and measures three things the existing instruments do not measure together: * working set and private bytes, sampled from the OS every two seconds and joined to each checkpoint by timestamp -- the client's own snapshot has no view of its own working set, which is exactly #257's quantity; * the published-versus-live pair already in the checkpoint JSON, because "alive in the object table, gone from the presentation" is #256's whole symptom and a drift between those halves at the SAME stop across cycles is what would show it; * a within-arm capture comparison -- cycle 1 against cycles 10, 20 and 30 at a pinned viewpoint -- plus a difference map and a row histogram, because a number cannot tell an absent object from a walking NPC and the map can. Every teleloc carries the identity quaternion so the heading repeats, and the four determinism levers the differential gate forces are forced here for the same reason: an unpinned sun would swamp the signal. Result at 90 transits per arm, 91 checkpoints, zero errors, graceful exits: neither arm reproduces either symptom. Working set means agree to 1 MiB (GL 1864, VK 1863) and warm-half drift is NEGATIVE on both (-48.0, -27.0). GPU accounting is exactly constant per arm. worldEntities holds 10,382 at all thirty cycles on both. The difference maps show every building, the portal, the statue and the treeline still drawn at cycle 30. That refutes the one outcome that would have blocked V11, and it does not identify the pre-existing bug -- so both issues stay OPEN with the negative recorded, and the follow-up named: walked portal transits rather than /teleloc, which do not take the same path into the transit state machine. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Erik
parent
122fe8a7e2
commit
db4426d5ef
2 changed files with 701 additions and 0 deletions
611
tools/run-portal-churn-soak.ps1
Normal file
611
tools/run-portal-churn-soak.ps1
Normal file
|
|
@ -0,0 +1,611 @@
|
|||
<#
|
||||
.SYNOPSIS
|
||||
Campaign V slice V11 step 0: the #256/#257 discriminator, run on both backends.
|
||||
|
||||
.DESCRIPTION
|
||||
Issues #256 (server-spawned signs and portals go invisible after repeated
|
||||
portal runs, while staying interactive) and #257 (working set grows to
|
||||
~1.5 GB over the same session) were both observed on the Vulkan backend in
|
||||
one long live session. Both filings demand the SAME discriminator, and both
|
||||
demand it BEFORE V11 deletes the OpenGL backend:
|
||||
|
||||
run the same repeated-churn route on GL and on Vulkan.
|
||||
|
||||
growth / drift on BOTH arms -> a pre-existing publication or resource
|
||||
lifetime bug that the campaign merely
|
||||
witnessed;
|
||||
growth / drift on VULKAN ONLY -> the new arm's resource lifecycle, and
|
||||
deleting its predecessor would destroy
|
||||
the only reference implementation that
|
||||
can answer the question.
|
||||
|
||||
After V11 the GL arm does not exist and this question costs far more to
|
||||
answer, which is why this script runs first and why its verdict can stop the
|
||||
slice.
|
||||
|
||||
WHAT IT MEASURES, per stop, per cycle:
|
||||
|
||||
* working set and private bytes, sampled from the OS every -SampleSeconds
|
||||
and joined to each checkpoint by timestamp. This is #257's quantity: the
|
||||
client's own snapshot has no view of its own working set.
|
||||
* the published-versus-live entity pair from the checkpoint JSON --
|
||||
liveEntities / materializedLiveEntities against worldEntities and
|
||||
meshRenderData. #256's symptom is "alive in the object table, gone from
|
||||
the presentation", so a drift between those two halves at the SAME stop
|
||||
across cycles is the instrument.
|
||||
* the GPU accounting the checkpoint already carries: trackedGpuBytes,
|
||||
residencyGpuBytes, their difference, tracked buffer and texture counts,
|
||||
owned composite textures and their owners. GpuMemoryTracker's categories
|
||||
reach the checkpoint as those counts.
|
||||
* managed bytes, committed bytes, LOH size, and the streaming/retirement
|
||||
backlogs, so a growth finding can be attributed to a heap rather than
|
||||
left as "the process is bigger".
|
||||
|
||||
HOW THE ROUTE IS BUILT. -Cycles repetitions of three stops taken from the
|
||||
two existing connected routes: Holtburg town (the dense outdoor stop the
|
||||
differential route calls its smoke stop, and the one that has a portal in
|
||||
frame), the Facility Hub interior EnvCell, and the Aerlinthe island. Every
|
||||
teleloc carries the identity quaternion so the heading at a stop is the same
|
||||
on every visit, which is what makes cycle 1 and cycle N comparable at all.
|
||||
Each stop writes a checkpoint; Holtburg additionally captures a screenshot on
|
||||
the cycles named by -ShotCycles.
|
||||
|
||||
THE PIXEL HALF. Within one arm, the Holtburg capture from cycle 1 and the
|
||||
capture from the last shot cycle are compared against each other. Same
|
||||
process, same position, same heading, same pinned clock: an object that has
|
||||
stopped being drawn shows up as a contiguous mid-frame difference. This is
|
||||
the closest automated instrument to what the user actually saw. The four
|
||||
determinism levers the differential gate forces are forced here for the same
|
||||
reason -- an unpinned sun or cloud sheet would swamp the signal.
|
||||
|
||||
NOT A PIXEL GATE. This script never compares the two backends to each other;
|
||||
that is run-backend-differential-gate.ps1's job and its thresholds are the
|
||||
campaign's. Here each arm is compared only with itself, and the verdict is
|
||||
about growth and drift.
|
||||
|
||||
.PARAMETER Out
|
||||
Directory for both arms' logs, checkpoints, captures, samples and the verdict.
|
||||
|
||||
.PARAMETER Cycles
|
||||
Repetitions of the three-stop loop. Default 30 -- #257's acceptance criterion
|
||||
is "a repeated-portal soak (>= 30 transits) holds working set flat after
|
||||
warmup", and three stops per cycle makes 90 transits at the default.
|
||||
|
||||
.PARAMETER Backends
|
||||
Which arms to run, in order. Default gl then vulkan, so the arm under
|
||||
suspicion runs second and cannot be blamed for warming the machine.
|
||||
|
||||
.PARAMETER ShotCycles
|
||||
Cycles on which Holtburg is captured. Default 1, 10, 20 and the last cycle.
|
||||
|
||||
.PARAMETER SettleMilliseconds
|
||||
Stationary window between arrival and the checkpoint. Default 6000.
|
||||
|
||||
.EXAMPLE
|
||||
tools/run-portal-churn-soak.ps1 -Out artifacts/v11-churn
|
||||
tools/run-portal-churn-soak.ps1 -Out artifacts/v11-churn-short -Cycles 4 -SkipBuild
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[Parameter(Mandatory = $true)][string]$Out,
|
||||
[int]$Cycles = 30,
|
||||
[string[]]$Backends = @('gl', 'vulkan'),
|
||||
[int[]]$ShotCycles,
|
||||
[int]$SettleMilliseconds = 6000,
|
||||
[int]$ShotSettleMilliseconds = 12000,
|
||||
[int]$SampleSeconds = 2,
|
||||
[int]$DayGroup = 0,
|
||||
[double]$WorldDayFraction = 0.5,
|
||||
[double]$SkyPhaseSeconds = 0,
|
||||
[int]$MaskTopPixels = 280,
|
||||
[int]$Tolerance = 2,
|
||||
[double]$MaxDifferentFraction = 0.001,
|
||||
[int]$MinRenderedBytes = 500000,
|
||||
# A soak whose subject is unbounded growth must not be allowed to take the
|
||||
# machine down with it. Crossing this stops the arm early and keeps every
|
||||
# sample and checkpoint taken so far -- which is the finding, not a failure.
|
||||
[int]$AbortWorkingSetMiB = 10000,
|
||||
[int]$CooldownSeconds = 20,
|
||||
[int]$RouteTimeoutSeconds = 2400,
|
||||
[switch]$SkipBuild
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
Set-StrictMode -Version Latest
|
||||
$repo = Split-Path -Parent $PSScriptRoot
|
||||
$exe = Join-Path $repo 'src\AcDream.App\bin\Release\net10.0\AcDream.App.exe'
|
||||
$cli = Join-Path $repo 'src\AcDream.Cli\bin\Release\net10.0\AcDream.Cli.dll'
|
||||
|
||||
function Write-Step($message) { Write-Host "[churn] $message" }
|
||||
|
||||
if (-not $ShotCycles) {
|
||||
$ShotCycles = @(1, 10, 20, $Cycles) | Where-Object { $_ -le $Cycles } | Sort-Object -Unique
|
||||
}
|
||||
|
||||
# --- Preconditions ------------------------------------------------------------
|
||||
if (Get-Process -Name AcDream.App -ErrorAction SilentlyContinue) {
|
||||
throw 'AcDream.App is already running. This soak uses the shared test account and must not steal its session.'
|
||||
}
|
||||
if (@(Get-NetUDPEndpoint -LocalPort 9000 -ErrorAction SilentlyContinue).Count -eq 0) {
|
||||
throw 'local ACE is not listening on UDP port 9000.'
|
||||
}
|
||||
if (-not $SkipBuild) {
|
||||
Write-Step 'building Release'
|
||||
& dotnet build (Join-Path $repo 'AcDream.slnx') -c Release --nologo -v q | Out-Null
|
||||
if ($LASTEXITCODE -ne 0) { throw "Release build failed with exit code $LASTEXITCODE." }
|
||||
}
|
||||
if (-not (Test-Path $exe)) { throw "Client not found at $exe." }
|
||||
if (-not (Test-Path $cli)) { throw "AcDream.Cli not found at $cli." }
|
||||
|
||||
if (Test-Path $Out) { Remove-Item -Recurse -Force $Out }
|
||||
New-Item -ItemType Directory -Force -Path $Out | Out-Null
|
||||
|
||||
# --- The route ----------------------------------------------------------------
|
||||
# Three stops per cycle, all three taken from routes that already exist. The
|
||||
# identity quaternion on every teleloc is what pins the heading across visits.
|
||||
$stops = @(
|
||||
[pscustomobject]@{ Key = 'holtburg'; Teleloc = '0xA9B40019 84.0 7.1 94.005 1 0 0 0' }
|
||||
[pscustomobject]@{ Key = 'hub'; Teleloc = '0x8A020164 70.35 -40.66 -5.9 1 0 0 0' }
|
||||
[pscustomobject]@{ Key = 'aerlinthe'; Teleloc = '0x09040008 11.4 188.6 87.705 1 0 0 0' }
|
||||
)
|
||||
|
||||
$lines = [System.Collections.Generic.List[string]]::new()
|
||||
$lines.Add('# GENERATED by tools/run-portal-churn-soak.ps1 -- the V11 step-0 discriminator.')
|
||||
$lines.Add("# $Cycles cycles x $($stops.Count) stops. Do not hand-edit; regenerate.")
|
||||
$lines.Add('')
|
||||
$lines.Add('wait world-ready 90000')
|
||||
$lines.Add('wait world-visible 30000')
|
||||
$lines.Add('sleep 15000')
|
||||
$lines.Add('checkpoint c00_login')
|
||||
$lines.Add('')
|
||||
|
||||
$transit = 0
|
||||
foreach ($cycle in 1..$Cycles) {
|
||||
$tag = '{0:d2}' -f $cycle
|
||||
$shot = $ShotCycles -contains $cycle
|
||||
foreach ($stop in $stops) {
|
||||
$transit++
|
||||
$settle = if ($shot -and $stop.Key -eq 'holtburg') { $ShotSettleMilliseconds } else { $SettleMilliseconds }
|
||||
$lines.Add("command /teleloc $($stop.Teleloc)")
|
||||
$lines.Add("wait materialized $transit 90000")
|
||||
$lines.Add('wait world-visible 30000')
|
||||
$lines.Add("sleep $settle")
|
||||
$lines.Add("checkpoint c${tag}_$($stop.Key)")
|
||||
if ($shot -and $stop.Key -eq 'holtburg') {
|
||||
$lines.Add("screenshot holtburg_c$tag 15000")
|
||||
}
|
||||
}
|
||||
$lines.Add('')
|
||||
}
|
||||
$lines.Add('sleep 5000')
|
||||
|
||||
$routePath = Join-Path $Out 'route.txt'
|
||||
Set-Content -Encoding utf8 -LiteralPath $routePath -Value $lines
|
||||
Write-Step "route: $Cycles cycles, $transit transits, $($lines.Count) lines -> $routePath"
|
||||
|
||||
Add-Type -AssemblyName System.Drawing
|
||||
Add-Type @'
|
||||
using System;
|
||||
using System.Runtime.InteropServices;
|
||||
public static class ChurnSoakSurface {
|
||||
[DllImport("user32.dll")] public static extern bool SetForegroundWindow(IntPtr hWnd);
|
||||
[DllImport("user32.dll")] public static extern bool ShowWindow(IntPtr hWnd, int nCmdShow);
|
||||
[DllImport("user32.dll")] public static extern bool GetClientRect(IntPtr hWnd, out RECT r);
|
||||
[DllImport("user32.dll")] public static extern bool ClientToScreen(IntPtr hWnd, ref POINT p);
|
||||
[StructLayout(LayoutKind.Sequential)] public struct RECT { public int L, T, R, B; }
|
||||
[StructLayout(LayoutKind.Sequential)] public struct POINT { public int X, Y; }
|
||||
}
|
||||
'@
|
||||
|
||||
# A number cannot tell an absent object from a walking NPC. The map and the row
|
||||
# histogram can: a vanished sign is a contiguous blob in one band, a creature is
|
||||
# a thin moving silhouette, and phase noise is scattered.
|
||||
Add-Type -ReferencedAssemblies System.Drawing @'
|
||||
using System;
|
||||
using System.Drawing;
|
||||
using System.Drawing.Imaging;
|
||||
using System.Runtime.InteropServices;
|
||||
public static class ChurnDiffMap {
|
||||
public static int[] Write(string expected, string actual, string outPath, int tolerance, int maskTop) {
|
||||
int[] bands = new int[18];
|
||||
using (Bitmap a = new Bitmap(expected))
|
||||
using (Bitmap b = new Bitmap(actual))
|
||||
using (Bitmap map = new Bitmap(a.Width, a.Height, PixelFormat.Format32bppArgb)) {
|
||||
Rectangle rect = new Rectangle(0, 0, a.Width, a.Height);
|
||||
BitmapData da = a.LockBits(rect, ImageLockMode.ReadOnly, PixelFormat.Format32bppArgb);
|
||||
BitmapData db = b.LockBits(rect, ImageLockMode.ReadOnly, PixelFormat.Format32bppArgb);
|
||||
BitmapData dm = map.LockBits(rect, ImageLockMode.WriteOnly, PixelFormat.Format32bppArgb);
|
||||
int count = a.Width * a.Height;
|
||||
int[] pa = new int[count], pb = new int[count], pm = new int[count];
|
||||
Marshal.Copy(da.Scan0, pa, 0, count);
|
||||
Marshal.Copy(db.Scan0, pb, 0, count);
|
||||
for (int y = 0; y < a.Height; y++) {
|
||||
for (int x = 0; x < a.Width; x++) {
|
||||
int i = y * a.Width + x;
|
||||
int ca = pa[i], cb = pb[i];
|
||||
int d = Math.Max(Math.Abs(((ca >> 16) & 255) - ((cb >> 16) & 255)),
|
||||
Math.Max(Math.Abs(((ca >> 8) & 255) - ((cb >> 8) & 255)),
|
||||
Math.Abs((ca & 255) - (cb & 255))));
|
||||
if (y >= maskTop && d > tolerance) {
|
||||
pm[i] = unchecked((int)0xFFFF00FF);
|
||||
bands[Math.Min(17, y / 40)]++;
|
||||
} else {
|
||||
int grey = (int)((((cb >> 16) & 255) * 0.3 + ((cb >> 8) & 255) * 0.59 + (cb & 255) * 0.11) / 4);
|
||||
pm[i] = unchecked((int)0xFF000000) | (grey << 16) | (grey << 8) | grey;
|
||||
}
|
||||
}
|
||||
}
|
||||
Marshal.Copy(pm, 0, dm.Scan0, count);
|
||||
a.UnlockBits(da); b.UnlockBits(db); map.UnlockBits(dm);
|
||||
map.Save(outPath, ImageFormat.Png);
|
||||
}
|
||||
return bands;
|
||||
}
|
||||
}
|
||||
'@
|
||||
|
||||
function Invoke-Arm([string]$Backend) {
|
||||
$dir = Join-Path $Out $Backend
|
||||
New-Item -ItemType Directory -Force -Path $dir | Out-Null
|
||||
$log = Join-Path $dir 'client.log'
|
||||
$grab = Join-Path $dir 'desktop-grab.png'
|
||||
$timeline = Join-Path $dir 'world-lifecycle.checkpoints.jsonl'
|
||||
|
||||
$env:ACDREAM_DAT_DIR = Join-Path $env:USERPROFILE "Documents\Asheron's Call"
|
||||
$env:ACDREAM_LIVE = '1'
|
||||
$env:ACDREAM_TEST_HOST = '127.0.0.1'
|
||||
$env:ACDREAM_TEST_PORT = '9000'
|
||||
$env:ACDREAM_TEST_USER = 'testaccount'
|
||||
$env:ACDREAM_TEST_PASS = 'testpassword'
|
||||
$env:ACDREAM_RETAIL_UI = '1'
|
||||
$env:ACDREAM_DEVTOOLS = '0'
|
||||
$env:ACDREAM_UI_PROBE_DUMP = '0'
|
||||
$env:ACDREAM_FRAME_PROF = '1'
|
||||
# The four determinism levers, forced. Without them a cycle-1-versus-cycle-N
|
||||
# capture measures the sun and the cloud sheet instead of the world.
|
||||
$env:ACDREAM_MSAA_SAMPLES = '0'
|
||||
$env:ACDREAM_DAY_GROUP = "$DayGroup"
|
||||
$env:ACDREAM_WORLD_TIME =
|
||||
$WorldDayFraction.ToString([System.Globalization.CultureInfo]::InvariantCulture)
|
||||
$env:ACDREAM_SKY_PHASE_SECONDS =
|
||||
$SkyPhaseSeconds.ToString([System.Globalization.CultureInfo]::InvariantCulture)
|
||||
$env:ACDREAM_RENDER_BACKEND = $Backend
|
||||
$env:ACDREAM_UI_PROBE_SCRIPT = $routePath
|
||||
$env:ACDREAM_AUTOMATION_ARTIFACT_DIR = $dir
|
||||
|
||||
Write-Step "launching $Backend"
|
||||
$proc = Start-Process -FilePath $exe -WorkingDirectory $repo `
|
||||
-RedirectStandardOutput $log -RedirectStandardError "$log.err" -PassThru
|
||||
|
||||
$samples = [System.Collections.Generic.List[object]]::new()
|
||||
$completed = $false
|
||||
$aborted = $null
|
||||
$deadline = (Get-Date).AddSeconds($RouteTimeoutSeconds)
|
||||
$lastReport = Get-Date
|
||||
while ((Get-Date) -lt $deadline) {
|
||||
if (Select-String -Path $log -SimpleMatch '[UI-PROBE] UI probe script complete' `
|
||||
-ErrorAction SilentlyContinue) { $completed = $true; break }
|
||||
if ($proc.HasExited) { break }
|
||||
try {
|
||||
$proc.Refresh()
|
||||
$samples.Add([pscustomobject][ordered]@{
|
||||
UtcTicks = [DateTime]::UtcNow.Ticks
|
||||
WorkingSetBytes = $proc.WorkingSet64
|
||||
PrivateBytes = $proc.PrivateMemorySize64
|
||||
HandleCount = $proc.HandleCount
|
||||
ThreadCount = $proc.Threads.Count
|
||||
})
|
||||
if ($proc.WorkingSet64 / 1MB -gt $AbortWorkingSetMiB) {
|
||||
$aborted = "working set crossed $AbortWorkingSetMiB MiB at sample $($samples.Count)"
|
||||
Write-Step "$Backend : ABORTING -- $aborted"
|
||||
break
|
||||
}
|
||||
}
|
||||
catch { }
|
||||
if (((Get-Date) - $lastReport).TotalSeconds -ge 60) {
|
||||
$lastReport = Get-Date
|
||||
$done = 0
|
||||
if (Test-Path -LiteralPath $timeline) {
|
||||
$done = @(Get-Content -LiteralPath $timeline -ErrorAction SilentlyContinue).Count
|
||||
}
|
||||
$ws = if ($samples.Count -gt 0) { [Math]::Round($samples[$samples.Count - 1].WorkingSetBytes / 1MB, 0) } else { 0 }
|
||||
Write-Step "$Backend : $done checkpoints, working set ${ws} MiB"
|
||||
}
|
||||
Start-Sleep -Seconds $SampleSeconds
|
||||
}
|
||||
|
||||
$grabBytes = 0
|
||||
if (-not $proc.HasExited) {
|
||||
try {
|
||||
$proc.Refresh()
|
||||
$h = $proc.MainWindowHandle
|
||||
if ($h -ne [IntPtr]::Zero) {
|
||||
[ChurnSoakSurface]::ShowWindow($h, 9) | Out-Null
|
||||
[ChurnSoakSurface]::SetForegroundWindow($h) | Out-Null
|
||||
Start-Sleep -Milliseconds 1200
|
||||
$r = New-Object ChurnSoakSurface+RECT
|
||||
[ChurnSoakSurface]::GetClientRect($h, [ref]$r) | Out-Null
|
||||
$p = New-Object ChurnSoakSurface+POINT
|
||||
[ChurnSoakSurface]::ClientToScreen($h, [ref]$p) | Out-Null
|
||||
$w = $r.R - $r.L; $ht = $r.B - $r.T
|
||||
if ($w -gt 0 -and $ht -gt 0) {
|
||||
$bmp = New-Object System.Drawing.Bitmap $w, $ht
|
||||
$g = [System.Drawing.Graphics]::FromImage($bmp)
|
||||
$g.CopyFromScreen($p.X, $p.Y, 0, 0, (New-Object System.Drawing.Size $w, $ht))
|
||||
$bmp.Save($grab, [System.Drawing.Imaging.ImageFormat]::Png)
|
||||
$g.Dispose(); $bmp.Dispose()
|
||||
$grabBytes = (Get-Item $grab).Length
|
||||
}
|
||||
}
|
||||
}
|
||||
catch { Write-Step "desktop grab failed on ${Backend}: $_" }
|
||||
}
|
||||
|
||||
# The peak matters as much as the end value: #257 reported a number reached,
|
||||
# not a number held. Take a final sample before the graceful close.
|
||||
try {
|
||||
$proc.Refresh()
|
||||
$samples.Add([pscustomobject][ordered]@{
|
||||
UtcTicks = [DateTime]::UtcNow.Ticks
|
||||
WorkingSetBytes = $proc.WorkingSet64
|
||||
PrivateBytes = $proc.PrivateMemorySize64
|
||||
HandleCount = $proc.HandleCount
|
||||
ThreadCount = $proc.Threads.Count
|
||||
})
|
||||
}
|
||||
catch { }
|
||||
|
||||
$graceful = $false
|
||||
$exitCode = $null
|
||||
$app = Get-Process -Name AcDream.App -ErrorAction SilentlyContinue
|
||||
if ($app) {
|
||||
$app.CloseMainWindow() | Out-Null
|
||||
if ($app.WaitForExit(45000)) { $graceful = $true; $exitCode = $app.ExitCode }
|
||||
else {
|
||||
Write-Step "WM_CLOSE timed out on $Backend; forcing"
|
||||
$app | Stop-Process -Force
|
||||
}
|
||||
}
|
||||
elseif ($proc.HasExited) { $exitCode = $proc.ExitCode }
|
||||
|
||||
foreach ($name in @('ACDREAM_RENDER_BACKEND', 'ACDREAM_MSAA_SAMPLES',
|
||||
'ACDREAM_SKY_PHASE_SECONDS', 'ACDREAM_WORLD_TIME',
|
||||
'ACDREAM_DAY_GROUP')) {
|
||||
Remove-Item "Env:\$name" -ErrorAction SilentlyContinue
|
||||
}
|
||||
|
||||
$samples | Export-Csv -NoTypeInformation -Encoding utf8 -LiteralPath (Join-Path $dir 'memory-samples.csv')
|
||||
|
||||
$checkpoints = @()
|
||||
if (Test-Path -LiteralPath $timeline) {
|
||||
$checkpoints = @(Get-Content -LiteralPath $timeline |
|
||||
Where-Object { $_.Trim().Length -gt 0 } |
|
||||
ForEach-Object { $_ | ConvertFrom-Json })
|
||||
}
|
||||
|
||||
$rows = [System.Collections.Generic.List[object]]::new()
|
||||
foreach ($cp in $checkpoints) {
|
||||
$ticks = ([DateTime]::Parse(
|
||||
$cp.timestampUtc,
|
||||
[System.Globalization.CultureInfo]::InvariantCulture,
|
||||
[System.Globalization.DateTimeStyles]::AdjustToUniversal -bor
|
||||
[System.Globalization.DateTimeStyles]::AssumeUniversal)).Ticks
|
||||
$nearest = $null
|
||||
$bestDelta = [long]::MaxValue
|
||||
foreach ($s in $samples) {
|
||||
$delta = [Math]::Abs($s.UtcTicks - $ticks)
|
||||
if ($delta -lt $bestDelta) { $bestDelta = $delta; $nearest = $s }
|
||||
}
|
||||
$r = $cp.resources
|
||||
$rows.Add([pscustomobject][ordered]@{
|
||||
Sequence = $cp.sequence
|
||||
Name = $cp.name
|
||||
Cycle = if ($cp.name -match '^c(\d+)_') { [int]$Matches[1] } else { -1 }
|
||||
Stop = if ($cp.name -match '^c\d+_(.+)$') { $Matches[1] } else { $cp.name }
|
||||
WorkingSetMiB = if ($nearest) { [Math]::Round($nearest.WorkingSetBytes / 1MB, 1) } else { $null }
|
||||
PrivateMiB = if ($nearest) { [Math]::Round($nearest.PrivateBytes / 1MB, 1) } else { $null }
|
||||
Handles = if ($nearest) { $nearest.HandleCount } else { $null }
|
||||
SampleSkewMs = if ($nearest) { [Math]::Round($bestDelta / 10000.0, 0) } else { $null }
|
||||
LiveEntities = $r.liveEntities
|
||||
Materialized = $r.materializedLiveEntities
|
||||
WorldEntities = $r.worldEntities
|
||||
LoadedLandblocks = $r.loadedLandblocks
|
||||
VisibleLandblocks = $r.visibleLandblocks
|
||||
MeshRenderData = $r.meshRenderData
|
||||
MeshMiB = [Math]::Round($r.meshEstimatedBytes / 1MB, 1)
|
||||
GpuMiB = [Math]::Round($r.trackedGpuBytes / 1MB, 1)
|
||||
ResidencyGpuMiB = [Math]::Round($r.residencyGpuBytes / 1MB, 1)
|
||||
GpuMinusResidencyMiB = [Math]::Round($r.gpuTrackerMinusResidencyBytes / 1MB, 1)
|
||||
GpuBuffers = $r.trackedGpuBuffers
|
||||
GpuTextures = $r.trackedGpuTextures
|
||||
CompositeTextures = $r.ownedCompositeTextures
|
||||
CompositeOwners = $r.compositeTextureOwners
|
||||
ParticleTextures = $r.activeParticleTextures
|
||||
ManagedMiB = [Math]::Round($r.managedBytes / 1MB, 1)
|
||||
CommittedMiB = [Math]::Round($r.managedCommittedBytes / 1MB, 1)
|
||||
LohMiB = [Math]::Round($r.lohSizeBytes / 1MB, 1)
|
||||
PendingTeardowns = $r.pendingLiveTeardowns
|
||||
PendingRetirements = $r.pendingLandblockRetirements
|
||||
StagedMeshUploads = $r.stagedMeshUploads
|
||||
Fps = [Math]::Round($r.fps, 1)
|
||||
})
|
||||
}
|
||||
$rows | Export-Csv -NoTypeInformation -Encoding utf8 -LiteralPath (Join-Path $dir 'per-checkpoint.csv')
|
||||
|
||||
$witness = if ($grabBytes -ge $MinRenderedBytes) { 'RENDERED' }
|
||||
elseif ($grabBytes -gt 0) { 'BLANK' }
|
||||
else { 'NO-CAPTURE' }
|
||||
|
||||
$errors = @(Select-String -Path $log -Pattern 'Unhandled exception|AccessViolation|OutOfMemoryException|event=invariant-failure|device removed|screenshot-failed' `
|
||||
-ErrorAction SilentlyContinue | Select-Object -First 10 | ForEach-Object { $_.Line })
|
||||
|
||||
return [pscustomobject][ordered]@{
|
||||
Backend = $Backend
|
||||
RouteCompleted = $completed
|
||||
Aborted = $aborted
|
||||
Graceful = $graceful
|
||||
ExitCode = $exitCode
|
||||
Witness = $witness
|
||||
GrabBytes = $grabBytes
|
||||
Checkpoints = $rows.Count
|
||||
Rows = $rows
|
||||
Samples = $samples.Count
|
||||
PeakWorkingSetMiB = if ($samples.Count -gt 0) { [Math]::Round((($samples | Measure-Object WorkingSetBytes -Maximum).Maximum) / 1MB, 1) } else { $null }
|
||||
PeakPrivateMiB = if ($samples.Count -gt 0) { [Math]::Round((($samples | Measure-Object PrivateBytes -Maximum).Maximum) / 1MB, 1) } else { $null }
|
||||
FinalWorkingSetMiB = if ($samples.Count -gt 0) { [Math]::Round($samples[$samples.Count - 1].WorkingSetBytes / 1MB, 1) } else { $null }
|
||||
ErrorLines = $errors
|
||||
Dir = $dir
|
||||
ScreenshotDir = Join-Path $dir 'screenshots'
|
||||
Log = $log
|
||||
}
|
||||
}
|
||||
|
||||
$arms = @()
|
||||
foreach ($backend in $Backends) {
|
||||
if ($arms.Count -gt 0) {
|
||||
Write-Step "cooldown ${CooldownSeconds}s so ACE clears the graceful logout"
|
||||
Start-Sleep -Seconds $CooldownSeconds
|
||||
}
|
||||
$arms += Invoke-Arm $backend
|
||||
}
|
||||
|
||||
# --- Within-arm capture comparison -------------------------------------------
|
||||
# The #256 half. First Holtburg capture against the last, inside one arm.
|
||||
$maskPath = $null
|
||||
$comparisons = @()
|
||||
foreach ($arm in $arms) {
|
||||
$shots = @(Get-ChildItem $arm.ScreenshotDir -Filter 'holtburg_c*.png' -ErrorAction SilentlyContinue |
|
||||
Sort-Object Name)
|
||||
if ($shots.Count -lt 2) {
|
||||
$comparisons += [pscustomobject][ordered]@{
|
||||
Backend = $arm.Backend; Pair = 'n/a'; Verdict = 'UNPAIRED'
|
||||
DifferentPixels = $null; Fraction = $null; MaxChannelDelta = $null
|
||||
}
|
||||
continue
|
||||
}
|
||||
if ($MaskTopPixels -gt 0 -and -not $maskPath) {
|
||||
$probe = [System.Drawing.Bitmap]::FromFile($shots[0].FullName)
|
||||
$mw = $probe.Width; $mh = $probe.Height
|
||||
$probe.Dispose()
|
||||
$mask = New-Object System.Drawing.Bitmap($mw, $mh, [System.Drawing.Imaging.PixelFormat]::Format32bppArgb)
|
||||
$graphics = [System.Drawing.Graphics]::FromImage($mask)
|
||||
$graphics.Clear([System.Drawing.Color]::FromArgb(0, 0, 0, 0))
|
||||
$opaque = New-Object System.Drawing.SolidBrush ([System.Drawing.Color]::FromArgb(255, 255, 0, 255))
|
||||
$graphics.FillRectangle($opaque, 0, 0, $mw, [Math]::Min($MaskTopPixels, $mh))
|
||||
$graphics.Dispose(); $opaque.Dispose()
|
||||
$maskPath = Join-Path $Out 'mask.png'
|
||||
$mask.Save($maskPath, [System.Drawing.Imaging.ImageFormat]::Png)
|
||||
$mask.Dispose()
|
||||
}
|
||||
$first = $shots[0]
|
||||
foreach ($later in $shots[1..($shots.Count - 1)]) {
|
||||
$report = Join-Path $Out "compare-$($arm.Backend)-$($first.BaseName)-vs-$($later.BaseName).json"
|
||||
if ($maskPath) {
|
||||
& dotnet $cli compare-screenshots $first.FullName $later.FullName $report $Tolerance $MaxDifferentFraction $maskPath | Out-Null
|
||||
}
|
||||
else {
|
||||
& dotnet $cli compare-screenshots $first.FullName $later.FullName $report $Tolerance $MaxDifferentFraction | Out-Null
|
||||
}
|
||||
$verdict = Get-Content $report -Raw | ConvertFrom-Json
|
||||
# A number cannot tell an absent object from a walking NPC. The map and
|
||||
# the row histogram can: a vanished sign is a contiguous blob in one
|
||||
# band, a creature is a thin moving silhouette, and phase noise is
|
||||
# scattered. Both are written beside the report.
|
||||
$mapPath = Join-Path $Out "diff-$($arm.Backend)-$($first.BaseName)-vs-$($later.BaseName).png"
|
||||
$bands = New-Object 'int[]' 18
|
||||
try { $bands = [ChurnDiffMap]::Write($first.FullName, $later.FullName, $mapPath, $Tolerance, $MaskTopPixels) }
|
||||
catch { Write-Step "diff map failed for $($later.BaseName): $_" }
|
||||
$comparisons += [pscustomobject][ordered]@{
|
||||
Backend = $arm.Backend
|
||||
Pair = "$($first.BaseName) vs $($later.BaseName)"
|
||||
Verdict = if ($verdict.passed) { 'MATCH' } else { 'DIVERGED' }
|
||||
DifferentPixels = $verdict.differentPixels
|
||||
Fraction = $verdict.differentPixelFraction
|
||||
MaxChannelDelta = $verdict.maximumChannelDelta
|
||||
RowBands40px = ($bands -join ',')
|
||||
DiffMap = $mapPath
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# --- Verdict ------------------------------------------------------------------
|
||||
function Get-StopTrend($rows, [string]$Stop, [string]$Column) {
|
||||
$series = @($rows | Where-Object { $_.Stop -eq $Stop -and $_.Cycle -ge 1 } | Sort-Object Cycle)
|
||||
if ($series.Count -lt 2) { return $null }
|
||||
$warm = if ($series.Count -ge 6) { $series[2..($series.Count - 1)] } else { $series }
|
||||
$firstValue = [double]$warm[0].$Column
|
||||
$lastValue = [double]$warm[-1].$Column
|
||||
$spanCycles = [double]($warm[-1].Cycle - $warm[0].Cycle)
|
||||
return [pscustomobject][ordered]@{
|
||||
Stop = $Stop
|
||||
Column = $Column
|
||||
FirstCycle = $warm[0].Cycle
|
||||
LastCycle = $warm[-1].Cycle
|
||||
First = $firstValue
|
||||
Last = $lastValue
|
||||
Delta = [Math]::Round($lastValue - $firstValue, 1)
|
||||
PerCycle = if ($spanCycles -gt 0) { [Math]::Round(($lastValue - $firstValue) / $spanCycles, 2) } else { $null }
|
||||
}
|
||||
}
|
||||
|
||||
$trends = @()
|
||||
foreach ($arm in $arms) {
|
||||
foreach ($stop in @('holtburg', 'hub', 'aerlinthe')) {
|
||||
foreach ($column in @('WorkingSetMiB', 'PrivateMiB', 'GpuMiB', 'ManagedMiB',
|
||||
'Materialized', 'WorldEntities', 'MeshRenderData',
|
||||
'CompositeTextures', 'GpuTextures', 'Handles')) {
|
||||
$t = Get-StopTrend $arm.Rows $stop $column
|
||||
if ($t) {
|
||||
$trends += [pscustomobject][ordered]@{
|
||||
Backend = $arm.Backend; Stop = $t.Stop; Column = $t.Column
|
||||
FirstCycle = $t.FirstCycle; LastCycle = $t.LastCycle
|
||||
First = $t.First; Last = $t.Last; Delta = $t.Delta; PerCycle = $t.PerCycle
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Write-Host ''
|
||||
foreach ($arm in $arms) {
|
||||
if ($arm.Aborted) { Write-Host "!!! $($arm.Backend) aborted: $($arm.Aborted)" -ForegroundColor Yellow }
|
||||
Write-Host "=== $($arm.Backend) : completed=$($arm.RouteCompleted) graceful=$($arm.Graceful) exit=$($arm.ExitCode) witness=$($arm.Witness) checkpoints=$($arm.Checkpoints) peakWS=$($arm.PeakWorkingSetMiB) MiB finalWS=$($arm.FinalWorkingSetMiB) MiB"
|
||||
$arm.Rows | Where-Object { $_.Stop -eq 'holtburg' } |
|
||||
Select-Object Cycle, WorkingSetMiB, PrivateMiB, GpuMiB, ManagedMiB, LiveEntities, Materialized, WorldEntities, MeshRenderData, CompositeTextures, GpuTextures, Handles |
|
||||
Format-Table -AutoSize | Out-String | Write-Host
|
||||
}
|
||||
Write-Host '=== within-arm capture comparison (the #256 half) ==='
|
||||
$comparisons | Format-Table -AutoSize | Out-String | Write-Host
|
||||
Write-Host '=== warm trends (cycle 3 -> last) ==='
|
||||
$trends | Where-Object { $_.Stop -eq 'holtburg' } | Format-Table -AutoSize | Out-String | Write-Host
|
||||
|
||||
$report = [pscustomobject][ordered]@{
|
||||
Commit = (& git -C $repo rev-parse HEAD).Trim()
|
||||
GeneratedUtc = [DateTime]::UtcNow.ToString('o')
|
||||
Cycles = $Cycles
|
||||
Transits = $transit
|
||||
Stops = @($stops | ForEach-Object { $_.Key })
|
||||
ShotCycles = @($ShotCycles)
|
||||
Route = $routePath
|
||||
DayGroup = $DayGroup
|
||||
WorldDayFraction = $WorldDayFraction
|
||||
SkyPhaseSeconds = $SkyPhaseSeconds
|
||||
MsaaSamples = 0
|
||||
MaskTopPixels = $MaskTopPixels
|
||||
Arms = @($arms | ForEach-Object {
|
||||
[pscustomobject][ordered]@{
|
||||
Backend = $_.Backend; RouteCompleted = $_.RouteCompleted; Aborted = $_.Aborted
|
||||
Graceful = $_.Graceful
|
||||
ExitCode = $_.ExitCode; Witness = $_.Witness; GrabBytes = $_.GrabBytes
|
||||
Checkpoints = $_.Checkpoints; Samples = $_.Samples
|
||||
PeakWorkingSetMiB = $_.PeakWorkingSetMiB; PeakPrivateMiB = $_.PeakPrivateMiB
|
||||
FinalWorkingSetMiB = $_.FinalWorkingSetMiB; ErrorLines = $_.ErrorLines
|
||||
Rows = $_.Rows
|
||||
}
|
||||
})
|
||||
Comparisons = @($comparisons)
|
||||
Trends = @($trends)
|
||||
}
|
||||
$reportPath = Join-Path $Out 'churn-soak.json'
|
||||
$report | ConvertTo-Json -Depth 8 | Set-Content -LiteralPath $reportPath -Encoding utf8
|
||||
Write-Step "report: $reportPath"
|
||||
Loading…
Add table
Add a link
Reference in a new issue