5852bdb877
Closes out the GL deletion by fixing the CI workflow and developer gate
scripts that still assumed a GL arm existed to compare against, build,
or select via ACDREAM_RENDER_BACKEND.
.github/workflows/headless-portability.yml: linux-graphical's "Verify
actionable unsupported-driver gate" step is deleted outright — it ran
the deleted `ui-studio` CLI verb (Studio was removed at Commit 1) to
prove the GL capability gate rejects Mesa's llvmpipe driver, and there
is no more GL capability gate for any driver to pass or fail. Its test
filter dropped two dead entries (GraphicalCapabilityRequirementsTests,
deleted at Commit 2; StudioWindowTests, already gone). Its package
contract check dropped the libcimgui.so assertion (ImGui's native
bridge, deleted at Commit 1). linux-vulkan's explanatory comment, which
described GL's rejection as the reason no cross-backend pixel diff runs
in CI, is rewritten to explain there is no GL arm left at all. Two dead
src/AcDream.UI.ImGui/** path triggers (that project no longer exists)
are removed from both the pull_request and push filters.
tools/run-backend-differential-gate.ps1 and its dedicated route file
tools/connected-backend-differential.route.txt are deleted: the whole
script's purpose was comparing a GL launch against a Vulkan launch of
the same route, and there is no second arm left to compare. Single-arm
regression checking already exists via run-offline-pixel-gate.ps1's
-Baseline mechanism.
tools/run-portal-churn-soak.ps1 is simplified rather than deleted: its
repeated-portal-churn methodology (within-arm capture comparison,
memory/entity/GPU trend analysis) has value independent of the
GL-versus-Vulkan question it was built to answer for issues #256/#257
before V11. -Backends now defaults to @('vulkan') alone; the doc
comments are rewritten from "step 0 discriminator, run before V11" to
an ongoing single-arm regression soak.
tools/run-offline-pixel-gate.ps1 drops its now-nonfunctional -Backend
parameter (ACDREAM_RENDER_BACKEND has read zero call sites since
RuntimeOptions.RenderBackend was removed at Commit 2 — passing -Backend
gl silently launched Vulkan anyway) along with its GL-escape-hatch
example and every comment that referenced the now-deleted differential
gate. tools/run-connected-world-lifecycle-gate.ps1,
tools/run-offline-vulkan-capture.ps1, and
tools/run-repeat-connected-gate.ps1 keep their (harmless, already
no-op) ACDREAM_RENDER_BACKEND set/clear lines but have their
now-inaccurate "escape hatch" / "GL run" comments corrected to state
plainly that the variable is unread and the line is kept only for the
historical record.
No .cs files touched; `dotnet build AcDream.slnx -c Release` unaffected
(0 warnings, 0 errors, matching the prior commit's build).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
617 lines
29 KiB
PowerShell
617 lines
29 KiB
PowerShell
<#
|
|
.SYNOPSIS
|
|
Repeated-portal-churn regression soak: memory growth and visual drift.
|
|
|
|
.DESCRIPTION
|
|
Originated as Campaign V slice V11 step 0: the #256/#257 discriminator, run
|
|
on both the GL and Vulkan backends before V11 deleted the former. Issues
|
|
#256 (server-spawned signs and portals go invisible after repeated portal
|
|
runs, while staying interactive) and #257 (working set grows to ~1.5 GB
|
|
over the same session) were both observed on the Vulkan backend in one long
|
|
live session, and running the same repeated-churn route on both arms told
|
|
the campaign whether either bug predated Vulkan or was new to it:
|
|
|
|
growth / drift on BOTH arms -> a pre-existing publication or resource
|
|
lifetime bug that the campaign merely
|
|
witnessed;
|
|
growth / drift on VULKAN ONLY -> the new arm's resource lifecycle, and
|
|
deleting its predecessor would destroy
|
|
the only reference implementation that
|
|
could answer the question.
|
|
|
|
Campaign V slice V11 deleted the GL backend, so that specific
|
|
before-and-after question can no longer be asked of any future bug — but
|
|
the soak itself does not depend on having two arms. Comparing a fresh
|
|
Holtburg capture against a much-later one from the SAME arm is still the
|
|
closest automated instrument to what a user's long session actually shows,
|
|
and the memory/entity/GPU trend analysis below never needed a second arm to
|
|
begin with. -Backends now defaults to Vulkan alone; pass additional values
|
|
only to compare arms that still exist (a future backend, a driver
|
|
variant) — 'gl' is no longer a valid launch and the app ignores it.
|
|
|
|
WHAT IT MEASURES, per stop, per cycle:
|
|
|
|
* working set and private bytes, sampled from the OS every -SampleSeconds
|
|
and joined to each checkpoint by timestamp. This is #257's quantity: the
|
|
client's own snapshot has no view of its own working set.
|
|
* the published-versus-live entity pair from the checkpoint JSON --
|
|
liveEntities / materializedLiveEntities against worldEntities and
|
|
meshRenderData. #256's symptom is "alive in the object table, gone from
|
|
the presentation", so a drift between those two halves at the SAME stop
|
|
across cycles is the instrument.
|
|
* the GPU accounting the checkpoint already carries: trackedGpuBytes,
|
|
residencyGpuBytes, their difference, tracked buffer and texture counts,
|
|
owned composite textures and their owners. GpuMemoryTracker's categories
|
|
reach the checkpoint as those counts.
|
|
* managed bytes, committed bytes, LOH size, and the streaming/retirement
|
|
backlogs, so a growth finding can be attributed to a heap rather than
|
|
left as "the process is bigger".
|
|
|
|
HOW THE ROUTE IS BUILT. -Cycles repetitions of three stops taken from the
|
|
two existing connected routes: Holtburg town (the dense outdoor stop the
|
|
differential route calls its smoke stop, and the one that has a portal in
|
|
frame), the Facility Hub interior EnvCell, and the Aerlinthe island. Every
|
|
teleloc carries the identity quaternion so the heading at a stop is the same
|
|
on every visit, which is what makes cycle 1 and cycle N comparable at all.
|
|
Each stop writes a checkpoint; Holtburg additionally captures a screenshot on
|
|
the cycles named by -ShotCycles.
|
|
|
|
THE PIXEL HALF. Within one arm, the Holtburg capture from cycle 1 and the
|
|
capture from the last shot cycle are compared against each other. Same
|
|
process, same position, same heading, same pinned clock: an object that has
|
|
stopped being drawn shows up as a contiguous mid-frame difference. This is
|
|
the closest automated instrument to what the user actually saw. The four
|
|
determinism levers the differential gate forces are forced here for the same
|
|
reason -- an unpinned sun or cloud sheet would swamp the signal.
|
|
|
|
NOT A PIXEL GATE. This script never compares two backends to each other —
|
|
even when -Backends names more than one arm, each is still compared only
|
|
with itself. The verdict is about growth and drift within an arm, not
|
|
parity between arms.
|
|
|
|
.PARAMETER Out
|
|
Directory for both arms' logs, checkpoints, captures, samples and the verdict.
|
|
|
|
.PARAMETER Cycles
|
|
Repetitions of the three-stop loop. Default 30 -- #257's acceptance criterion
|
|
is "a repeated-portal soak (>= 30 transits) holds working set flat after
|
|
warmup", and three stops per cycle makes 90 transits at the default.
|
|
|
|
.PARAMETER Backends
|
|
Which arms to run, in order. Default 'vulkan' alone — the only backend
|
|
that exists since Campaign V slice V11 deleted the GL arm.
|
|
|
|
.PARAMETER ShotCycles
|
|
Cycles on which Holtburg is captured. Default 1, 10, 20 and the last cycle.
|
|
|
|
.PARAMETER SettleMilliseconds
|
|
Stationary window between arrival and the checkpoint. Default 6000.
|
|
|
|
.EXAMPLE
|
|
tools/run-portal-churn-soak.ps1 -Out artifacts/v11-churn
|
|
tools/run-portal-churn-soak.ps1 -Out artifacts/v11-churn-short -Cycles 4 -SkipBuild
|
|
#>
|
|
[CmdletBinding()]
|
|
param(
|
|
[Parameter(Mandatory = $true)][string]$Out,
|
|
[int]$Cycles = 30,
|
|
[string[]]$Backends = @('vulkan'),
|
|
[int[]]$ShotCycles,
|
|
[int]$SettleMilliseconds = 6000,
|
|
[int]$ShotSettleMilliseconds = 12000,
|
|
[int]$SampleSeconds = 2,
|
|
[int]$DayGroup = 0,
|
|
[double]$WorldDayFraction = 0.5,
|
|
[double]$SkyPhaseSeconds = 0,
|
|
[int]$MaskTopPixels = 280,
|
|
[int]$Tolerance = 2,
|
|
[double]$MaxDifferentFraction = 0.001,
|
|
[int]$MinRenderedBytes = 500000,
|
|
# A soak whose subject is unbounded growth must not be allowed to take the
|
|
# machine down with it. Crossing this stops the arm early and keeps every
|
|
# sample and checkpoint taken so far -- which is the finding, not a failure.
|
|
[int]$AbortWorkingSetMiB = 10000,
|
|
[int]$CooldownSeconds = 20,
|
|
[int]$RouteTimeoutSeconds = 2400,
|
|
[switch]$SkipBuild
|
|
)
|
|
|
|
$ErrorActionPreference = 'Stop'
|
|
Set-StrictMode -Version Latest
|
|
$repo = Split-Path -Parent $PSScriptRoot
|
|
$exe = Join-Path $repo 'src\AcDream.App\bin\Release\net10.0\AcDream.App.exe'
|
|
$cli = Join-Path $repo 'src\AcDream.Cli\bin\Release\net10.0\AcDream.Cli.dll'
|
|
|
|
function Write-Step($message) { Write-Host "[churn] $message" }
|
|
|
|
if (-not $ShotCycles) {
|
|
$ShotCycles = @(1, 10, 20, $Cycles) | Where-Object { $_ -le $Cycles } | Sort-Object -Unique
|
|
}
|
|
|
|
# --- Preconditions ------------------------------------------------------------
|
|
if (Get-Process -Name AcDream.App -ErrorAction SilentlyContinue) {
|
|
throw 'AcDream.App is already running. This soak uses the shared test account and must not steal its session.'
|
|
}
|
|
if (@(Get-NetUDPEndpoint -LocalPort 9000 -ErrorAction SilentlyContinue).Count -eq 0) {
|
|
throw 'local ACE is not listening on UDP port 9000.'
|
|
}
|
|
if (-not $SkipBuild) {
|
|
Write-Step 'building Release'
|
|
& dotnet build (Join-Path $repo 'AcDream.slnx') -c Release --nologo -v q | Out-Null
|
|
if ($LASTEXITCODE -ne 0) { throw "Release build failed with exit code $LASTEXITCODE." }
|
|
}
|
|
if (-not (Test-Path $exe)) { throw "Client not found at $exe." }
|
|
if (-not (Test-Path $cli)) { throw "AcDream.Cli not found at $cli." }
|
|
|
|
if (Test-Path $Out) { Remove-Item -Recurse -Force $Out }
|
|
New-Item -ItemType Directory -Force -Path $Out | Out-Null
|
|
|
|
# --- The route ----------------------------------------------------------------
|
|
# Three stops per cycle, all three taken from routes that already exist. The
|
|
# identity quaternion on every teleloc is what pins the heading across visits.
|
|
$stops = @(
|
|
[pscustomobject]@{ Key = 'holtburg'; Teleloc = '0xA9B40019 84.0 7.1 94.005 1 0 0 0' }
|
|
[pscustomobject]@{ Key = 'hub'; Teleloc = '0x8A020164 70.35 -40.66 -5.9 1 0 0 0' }
|
|
[pscustomobject]@{ Key = 'aerlinthe'; Teleloc = '0x09040008 11.4 188.6 87.705 1 0 0 0' }
|
|
)
|
|
|
|
$lines = [System.Collections.Generic.List[string]]::new()
|
|
$lines.Add('# GENERATED by tools/run-portal-churn-soak.ps1 -- the portal-churn regression soak.')
|
|
$lines.Add("# $Cycles cycles x $($stops.Count) stops. Do not hand-edit; regenerate.")
|
|
$lines.Add('')
|
|
$lines.Add('wait world-ready 90000')
|
|
$lines.Add('wait world-visible 30000')
|
|
$lines.Add('sleep 15000')
|
|
$lines.Add('checkpoint c00_login')
|
|
$lines.Add('')
|
|
|
|
$transit = 0
|
|
foreach ($cycle in 1..$Cycles) {
|
|
$tag = '{0:d2}' -f $cycle
|
|
$shot = $ShotCycles -contains $cycle
|
|
foreach ($stop in $stops) {
|
|
$transit++
|
|
$settle = if ($shot -and $stop.Key -eq 'holtburg') { $ShotSettleMilliseconds } else { $SettleMilliseconds }
|
|
$lines.Add("command /teleloc $($stop.Teleloc)")
|
|
$lines.Add("wait materialized $transit 90000")
|
|
$lines.Add('wait world-visible 30000')
|
|
$lines.Add("sleep $settle")
|
|
$lines.Add("checkpoint c${tag}_$($stop.Key)")
|
|
if ($shot -and $stop.Key -eq 'holtburg') {
|
|
$lines.Add("screenshot holtburg_c$tag 15000")
|
|
}
|
|
}
|
|
$lines.Add('')
|
|
}
|
|
$lines.Add('sleep 5000')
|
|
|
|
$routePath = Join-Path $Out 'route.txt'
|
|
Set-Content -Encoding utf8 -LiteralPath $routePath -Value $lines
|
|
Write-Step "route: $Cycles cycles, $transit transits, $($lines.Count) lines -> $routePath"
|
|
|
|
Add-Type -AssemblyName System.Drawing
|
|
Add-Type @'
|
|
using System;
|
|
using System.Runtime.InteropServices;
|
|
public static class ChurnSoakSurface {
|
|
[DllImport("user32.dll")] public static extern bool SetForegroundWindow(IntPtr hWnd);
|
|
[DllImport("user32.dll")] public static extern bool ShowWindow(IntPtr hWnd, int nCmdShow);
|
|
[DllImport("user32.dll")] public static extern bool GetClientRect(IntPtr hWnd, out RECT r);
|
|
[DllImport("user32.dll")] public static extern bool ClientToScreen(IntPtr hWnd, ref POINT p);
|
|
[StructLayout(LayoutKind.Sequential)] public struct RECT { public int L, T, R, B; }
|
|
[StructLayout(LayoutKind.Sequential)] public struct POINT { public int X, Y; }
|
|
}
|
|
'@
|
|
|
|
# A number cannot tell an absent object from a walking NPC. The map and the row
|
|
# histogram can: a vanished sign is a contiguous blob in one band, a creature is
|
|
# a thin moving silhouette, and phase noise is scattered.
|
|
Add-Type -ReferencedAssemblies System.Drawing @'
|
|
using System;
|
|
using System.Drawing;
|
|
using System.Drawing.Imaging;
|
|
using System.Runtime.InteropServices;
|
|
public static class ChurnDiffMap {
|
|
public static int[] Write(string expected, string actual, string outPath, int tolerance, int maskTop) {
|
|
int[] bands = new int[18];
|
|
using (Bitmap a = new Bitmap(expected))
|
|
using (Bitmap b = new Bitmap(actual))
|
|
using (Bitmap map = new Bitmap(a.Width, a.Height, PixelFormat.Format32bppArgb)) {
|
|
Rectangle rect = new Rectangle(0, 0, a.Width, a.Height);
|
|
BitmapData da = a.LockBits(rect, ImageLockMode.ReadOnly, PixelFormat.Format32bppArgb);
|
|
BitmapData db = b.LockBits(rect, ImageLockMode.ReadOnly, PixelFormat.Format32bppArgb);
|
|
BitmapData dm = map.LockBits(rect, ImageLockMode.WriteOnly, PixelFormat.Format32bppArgb);
|
|
int count = a.Width * a.Height;
|
|
int[] pa = new int[count], pb = new int[count], pm = new int[count];
|
|
Marshal.Copy(da.Scan0, pa, 0, count);
|
|
Marshal.Copy(db.Scan0, pb, 0, count);
|
|
for (int y = 0; y < a.Height; y++) {
|
|
for (int x = 0; x < a.Width; x++) {
|
|
int i = y * a.Width + x;
|
|
int ca = pa[i], cb = pb[i];
|
|
int d = Math.Max(Math.Abs(((ca >> 16) & 255) - ((cb >> 16) & 255)),
|
|
Math.Max(Math.Abs(((ca >> 8) & 255) - ((cb >> 8) & 255)),
|
|
Math.Abs((ca & 255) - (cb & 255))));
|
|
if (y >= maskTop && d > tolerance) {
|
|
pm[i] = unchecked((int)0xFFFF00FF);
|
|
bands[Math.Min(17, y / 40)]++;
|
|
} else {
|
|
int grey = (int)((((cb >> 16) & 255) * 0.3 + ((cb >> 8) & 255) * 0.59 + (cb & 255) * 0.11) / 4);
|
|
pm[i] = unchecked((int)0xFF000000) | (grey << 16) | (grey << 8) | grey;
|
|
}
|
|
}
|
|
}
|
|
Marshal.Copy(pm, 0, dm.Scan0, count);
|
|
a.UnlockBits(da); b.UnlockBits(db); map.UnlockBits(dm);
|
|
map.Save(outPath, ImageFormat.Png);
|
|
}
|
|
return bands;
|
|
}
|
|
}
|
|
'@
|
|
|
|
function Invoke-Arm([string]$Backend) {
|
|
$dir = Join-Path $Out $Backend
|
|
New-Item -ItemType Directory -Force -Path $dir | Out-Null
|
|
$log = Join-Path $dir 'client.log'
|
|
$grab = Join-Path $dir 'desktop-grab.png'
|
|
$timeline = Join-Path $dir 'world-lifecycle.checkpoints.jsonl'
|
|
|
|
$env:ACDREAM_DAT_DIR = Join-Path $env:USERPROFILE "Documents\Asheron's Call"
|
|
$env:ACDREAM_LIVE = '1'
|
|
$env:ACDREAM_TEST_HOST = '127.0.0.1'
|
|
$env:ACDREAM_TEST_PORT = '9000'
|
|
$env:ACDREAM_TEST_USER = 'testaccount'
|
|
$env:ACDREAM_TEST_PASS = 'testpassword'
|
|
$env:ACDREAM_RETAIL_UI = '1'
|
|
$env:ACDREAM_DEVTOOLS = '0'
|
|
$env:ACDREAM_UI_PROBE_DUMP = '0'
|
|
$env:ACDREAM_FRAME_PROF = '1'
|
|
# The four determinism levers, forced. Without them a cycle-1-versus-cycle-N
|
|
# capture measures the sun and the cloud sheet instead of the world.
|
|
$env:ACDREAM_MSAA_SAMPLES = '0'
|
|
$env:ACDREAM_DAY_GROUP = "$DayGroup"
|
|
$env:ACDREAM_WORLD_TIME =
|
|
$WorldDayFraction.ToString([System.Globalization.CultureInfo]::InvariantCulture)
|
|
$env:ACDREAM_SKY_PHASE_SECONDS =
|
|
$SkyPhaseSeconds.ToString([System.Globalization.CultureInfo]::InvariantCulture)
|
|
$env:ACDREAM_RENDER_BACKEND = $Backend
|
|
$env:ACDREAM_UI_PROBE_SCRIPT = $routePath
|
|
$env:ACDREAM_AUTOMATION_ARTIFACT_DIR = $dir
|
|
|
|
Write-Step "launching $Backend"
|
|
$proc = Start-Process -FilePath $exe -WorkingDirectory $repo `
|
|
-RedirectStandardOutput $log -RedirectStandardError "$log.err" -PassThru
|
|
|
|
$samples = [System.Collections.Generic.List[object]]::new()
|
|
$completed = $false
|
|
$aborted = $null
|
|
$deadline = (Get-Date).AddSeconds($RouteTimeoutSeconds)
|
|
$lastReport = Get-Date
|
|
while ((Get-Date) -lt $deadline) {
|
|
if (Select-String -Path $log -SimpleMatch '[UI-PROBE] UI probe script complete' `
|
|
-ErrorAction SilentlyContinue) { $completed = $true; break }
|
|
if ($proc.HasExited) { break }
|
|
try {
|
|
$proc.Refresh()
|
|
$samples.Add([pscustomobject][ordered]@{
|
|
UtcTicks = [DateTime]::UtcNow.Ticks
|
|
WorkingSetBytes = $proc.WorkingSet64
|
|
PrivateBytes = $proc.PrivateMemorySize64
|
|
HandleCount = $proc.HandleCount
|
|
ThreadCount = $proc.Threads.Count
|
|
})
|
|
if ($proc.WorkingSet64 / 1MB -gt $AbortWorkingSetMiB) {
|
|
$aborted = "working set crossed $AbortWorkingSetMiB MiB at sample $($samples.Count)"
|
|
Write-Step "$Backend : ABORTING -- $aborted"
|
|
break
|
|
}
|
|
}
|
|
catch { }
|
|
if (((Get-Date) - $lastReport).TotalSeconds -ge 60) {
|
|
$lastReport = Get-Date
|
|
$done = 0
|
|
if (Test-Path -LiteralPath $timeline) {
|
|
$done = @(Get-Content -LiteralPath $timeline -ErrorAction SilentlyContinue).Count
|
|
}
|
|
$ws = if ($samples.Count -gt 0) { [Math]::Round($samples[$samples.Count - 1].WorkingSetBytes / 1MB, 0) } else { 0 }
|
|
Write-Step "$Backend : $done checkpoints, working set ${ws} MiB"
|
|
}
|
|
Start-Sleep -Seconds $SampleSeconds
|
|
}
|
|
|
|
$grabBytes = 0
|
|
if (-not $proc.HasExited) {
|
|
try {
|
|
$proc.Refresh()
|
|
$h = $proc.MainWindowHandle
|
|
if ($h -ne [IntPtr]::Zero) {
|
|
[ChurnSoakSurface]::ShowWindow($h, 9) | Out-Null
|
|
[ChurnSoakSurface]::SetForegroundWindow($h) | Out-Null
|
|
Start-Sleep -Milliseconds 1200
|
|
$r = New-Object ChurnSoakSurface+RECT
|
|
[ChurnSoakSurface]::GetClientRect($h, [ref]$r) | Out-Null
|
|
$p = New-Object ChurnSoakSurface+POINT
|
|
[ChurnSoakSurface]::ClientToScreen($h, [ref]$p) | Out-Null
|
|
$w = $r.R - $r.L; $ht = $r.B - $r.T
|
|
if ($w -gt 0 -and $ht -gt 0) {
|
|
$bmp = New-Object System.Drawing.Bitmap $w, $ht
|
|
$g = [System.Drawing.Graphics]::FromImage($bmp)
|
|
$g.CopyFromScreen($p.X, $p.Y, 0, 0, (New-Object System.Drawing.Size $w, $ht))
|
|
$bmp.Save($grab, [System.Drawing.Imaging.ImageFormat]::Png)
|
|
$g.Dispose(); $bmp.Dispose()
|
|
$grabBytes = (Get-Item $grab).Length
|
|
}
|
|
}
|
|
}
|
|
catch { Write-Step "desktop grab failed on ${Backend}: $_" }
|
|
}
|
|
|
|
# The peak matters as much as the end value: #257 reported a number reached,
|
|
# not a number held. Take a final sample before the graceful close.
|
|
try {
|
|
$proc.Refresh()
|
|
$samples.Add([pscustomobject][ordered]@{
|
|
UtcTicks = [DateTime]::UtcNow.Ticks
|
|
WorkingSetBytes = $proc.WorkingSet64
|
|
PrivateBytes = $proc.PrivateMemorySize64
|
|
HandleCount = $proc.HandleCount
|
|
ThreadCount = $proc.Threads.Count
|
|
})
|
|
}
|
|
catch { }
|
|
|
|
$graceful = $false
|
|
$exitCode = $null
|
|
$app = Get-Process -Name AcDream.App -ErrorAction SilentlyContinue
|
|
if ($app) {
|
|
$app.CloseMainWindow() | Out-Null
|
|
if ($app.WaitForExit(45000)) { $graceful = $true; $exitCode = $app.ExitCode }
|
|
else {
|
|
Write-Step "WM_CLOSE timed out on $Backend; forcing"
|
|
$app | Stop-Process -Force
|
|
}
|
|
}
|
|
elseif ($proc.HasExited) { $exitCode = $proc.ExitCode }
|
|
|
|
foreach ($name in @('ACDREAM_RENDER_BACKEND', 'ACDREAM_MSAA_SAMPLES',
|
|
'ACDREAM_SKY_PHASE_SECONDS', 'ACDREAM_WORLD_TIME',
|
|
'ACDREAM_DAY_GROUP')) {
|
|
Remove-Item "Env:\$name" -ErrorAction SilentlyContinue
|
|
}
|
|
|
|
$samples | Export-Csv -NoTypeInformation -Encoding utf8 -LiteralPath (Join-Path $dir 'memory-samples.csv')
|
|
|
|
$checkpoints = @()
|
|
if (Test-Path -LiteralPath $timeline) {
|
|
$checkpoints = @(Get-Content -LiteralPath $timeline |
|
|
Where-Object { $_.Trim().Length -gt 0 } |
|
|
ForEach-Object { $_ | ConvertFrom-Json })
|
|
}
|
|
|
|
$rows = [System.Collections.Generic.List[object]]::new()
|
|
foreach ($cp in $checkpoints) {
|
|
$ticks = ([DateTime]::Parse(
|
|
$cp.timestampUtc,
|
|
[System.Globalization.CultureInfo]::InvariantCulture,
|
|
[System.Globalization.DateTimeStyles]::AdjustToUniversal -bor
|
|
[System.Globalization.DateTimeStyles]::AssumeUniversal)).Ticks
|
|
$nearest = $null
|
|
$bestDelta = [long]::MaxValue
|
|
foreach ($s in $samples) {
|
|
$delta = [Math]::Abs($s.UtcTicks - $ticks)
|
|
if ($delta -lt $bestDelta) { $bestDelta = $delta; $nearest = $s }
|
|
}
|
|
$r = $cp.resources
|
|
$rows.Add([pscustomobject][ordered]@{
|
|
Sequence = $cp.sequence
|
|
Name = $cp.name
|
|
Cycle = if ($cp.name -match '^c(\d+)_') { [int]$Matches[1] } else { -1 }
|
|
Stop = if ($cp.name -match '^c\d+_(.+)$') { $Matches[1] } else { $cp.name }
|
|
WorkingSetMiB = if ($nearest) { [Math]::Round($nearest.WorkingSetBytes / 1MB, 1) } else { $null }
|
|
PrivateMiB = if ($nearest) { [Math]::Round($nearest.PrivateBytes / 1MB, 1) } else { $null }
|
|
Handles = if ($nearest) { $nearest.HandleCount } else { $null }
|
|
SampleSkewMs = if ($nearest) { [Math]::Round($bestDelta / 10000.0, 0) } else { $null }
|
|
LiveEntities = $r.liveEntities
|
|
Materialized = $r.materializedLiveEntities
|
|
WorldEntities = $r.worldEntities
|
|
LoadedLandblocks = $r.loadedLandblocks
|
|
VisibleLandblocks = $r.visibleLandblocks
|
|
MeshRenderData = $r.meshRenderData
|
|
MeshMiB = [Math]::Round($r.meshEstimatedBytes / 1MB, 1)
|
|
GpuMiB = [Math]::Round($r.trackedGpuBytes / 1MB, 1)
|
|
ResidencyGpuMiB = [Math]::Round($r.residencyGpuBytes / 1MB, 1)
|
|
GpuMinusResidencyMiB = [Math]::Round($r.gpuTrackerMinusResidencyBytes / 1MB, 1)
|
|
GpuBuffers = $r.trackedGpuBuffers
|
|
GpuTextures = $r.trackedGpuTextures
|
|
CompositeTextures = $r.ownedCompositeTextures
|
|
CompositeOwners = $r.compositeTextureOwners
|
|
ParticleTextures = $r.activeParticleTextures
|
|
ManagedMiB = [Math]::Round($r.managedBytes / 1MB, 1)
|
|
CommittedMiB = [Math]::Round($r.managedCommittedBytes / 1MB, 1)
|
|
LohMiB = [Math]::Round($r.lohSizeBytes / 1MB, 1)
|
|
PendingTeardowns = $r.pendingLiveTeardowns
|
|
PendingRetirements = $r.pendingLandblockRetirements
|
|
StagedMeshUploads = $r.stagedMeshUploads
|
|
Fps = [Math]::Round($r.fps, 1)
|
|
})
|
|
}
|
|
$rows | Export-Csv -NoTypeInformation -Encoding utf8 -LiteralPath (Join-Path $dir 'per-checkpoint.csv')
|
|
|
|
$witness = if ($grabBytes -ge $MinRenderedBytes) { 'RENDERED' }
|
|
elseif ($grabBytes -gt 0) { 'BLANK' }
|
|
else { 'NO-CAPTURE' }
|
|
|
|
$errors = @(Select-String -Path $log -Pattern 'Unhandled exception|AccessViolation|OutOfMemoryException|event=invariant-failure|device removed|screenshot-failed' `
|
|
-ErrorAction SilentlyContinue | Select-Object -First 10 | ForEach-Object { $_.Line })
|
|
|
|
return [pscustomobject][ordered]@{
|
|
Backend = $Backend
|
|
RouteCompleted = $completed
|
|
Aborted = $aborted
|
|
Graceful = $graceful
|
|
ExitCode = $exitCode
|
|
Witness = $witness
|
|
GrabBytes = $grabBytes
|
|
Checkpoints = $rows.Count
|
|
Rows = $rows
|
|
Samples = $samples.Count
|
|
PeakWorkingSetMiB = if ($samples.Count -gt 0) { [Math]::Round((($samples | Measure-Object WorkingSetBytes -Maximum).Maximum) / 1MB, 1) } else { $null }
|
|
PeakPrivateMiB = if ($samples.Count -gt 0) { [Math]::Round((($samples | Measure-Object PrivateBytes -Maximum).Maximum) / 1MB, 1) } else { $null }
|
|
FinalWorkingSetMiB = if ($samples.Count -gt 0) { [Math]::Round($samples[$samples.Count - 1].WorkingSetBytes / 1MB, 1) } else { $null }
|
|
ErrorLines = $errors
|
|
Dir = $dir
|
|
ScreenshotDir = Join-Path $dir 'screenshots'
|
|
Log = $log
|
|
}
|
|
}
|
|
|
|
$arms = @()
|
|
foreach ($backend in $Backends) {
|
|
if ($arms.Count -gt 0) {
|
|
Write-Step "cooldown ${CooldownSeconds}s so ACE clears the graceful logout"
|
|
Start-Sleep -Seconds $CooldownSeconds
|
|
}
|
|
$arms += Invoke-Arm $backend
|
|
}
|
|
|
|
# --- Within-arm capture comparison -------------------------------------------
|
|
# The #256 half. First Holtburg capture against the last, inside one arm.
|
|
$maskPath = $null
|
|
$comparisons = @()
|
|
foreach ($arm in $arms) {
|
|
$shots = @(Get-ChildItem $arm.ScreenshotDir -Filter 'holtburg_c*.png' -ErrorAction SilentlyContinue |
|
|
Sort-Object Name)
|
|
if ($shots.Count -lt 2) {
|
|
$comparisons += [pscustomobject][ordered]@{
|
|
Backend = $arm.Backend; Pair = 'n/a'; Verdict = 'UNPAIRED'
|
|
DifferentPixels = $null; Fraction = $null; MaxChannelDelta = $null
|
|
}
|
|
continue
|
|
}
|
|
if ($MaskTopPixels -gt 0 -and -not $maskPath) {
|
|
$probe = [System.Drawing.Bitmap]::FromFile($shots[0].FullName)
|
|
$mw = $probe.Width; $mh = $probe.Height
|
|
$probe.Dispose()
|
|
$mask = New-Object System.Drawing.Bitmap($mw, $mh, [System.Drawing.Imaging.PixelFormat]::Format32bppArgb)
|
|
$graphics = [System.Drawing.Graphics]::FromImage($mask)
|
|
$graphics.Clear([System.Drawing.Color]::FromArgb(0, 0, 0, 0))
|
|
$opaque = New-Object System.Drawing.SolidBrush ([System.Drawing.Color]::FromArgb(255, 255, 0, 255))
|
|
$graphics.FillRectangle($opaque, 0, 0, $mw, [Math]::Min($MaskTopPixels, $mh))
|
|
$graphics.Dispose(); $opaque.Dispose()
|
|
$maskPath = Join-Path $Out 'mask.png'
|
|
$mask.Save($maskPath, [System.Drawing.Imaging.ImageFormat]::Png)
|
|
$mask.Dispose()
|
|
}
|
|
$first = $shots[0]
|
|
foreach ($later in $shots[1..($shots.Count - 1)]) {
|
|
$report = Join-Path $Out "compare-$($arm.Backend)-$($first.BaseName)-vs-$($later.BaseName).json"
|
|
if ($maskPath) {
|
|
& dotnet $cli compare-screenshots $first.FullName $later.FullName $report $Tolerance $MaxDifferentFraction $maskPath | Out-Null
|
|
}
|
|
else {
|
|
& dotnet $cli compare-screenshots $first.FullName $later.FullName $report $Tolerance $MaxDifferentFraction | Out-Null
|
|
}
|
|
$verdict = Get-Content $report -Raw | ConvertFrom-Json
|
|
# A number cannot tell an absent object from a walking NPC. The map and
|
|
# the row histogram can: a vanished sign is a contiguous blob in one
|
|
# band, a creature is a thin moving silhouette, and phase noise is
|
|
# scattered. Both are written beside the report.
|
|
$mapPath = Join-Path $Out "diff-$($arm.Backend)-$($first.BaseName)-vs-$($later.BaseName).png"
|
|
$bands = New-Object 'int[]' 18
|
|
try { $bands = [ChurnDiffMap]::Write($first.FullName, $later.FullName, $mapPath, $Tolerance, $MaskTopPixels) }
|
|
catch { Write-Step "diff map failed for $($later.BaseName): $_" }
|
|
$comparisons += [pscustomobject][ordered]@{
|
|
Backend = $arm.Backend
|
|
Pair = "$($first.BaseName) vs $($later.BaseName)"
|
|
Verdict = if ($verdict.passed) { 'MATCH' } else { 'DIVERGED' }
|
|
DifferentPixels = $verdict.differentPixels
|
|
Fraction = $verdict.differentPixelFraction
|
|
MaxChannelDelta = $verdict.maximumChannelDelta
|
|
RowBands40px = ($bands -join ',')
|
|
DiffMap = $mapPath
|
|
}
|
|
}
|
|
}
|
|
|
|
# --- Verdict ------------------------------------------------------------------
|
|
function Get-StopTrend($rows, [string]$Stop, [string]$Column) {
|
|
$series = @($rows | Where-Object { $_.Stop -eq $Stop -and $_.Cycle -ge 1 } | Sort-Object Cycle)
|
|
if ($series.Count -lt 2) { return $null }
|
|
$warm = if ($series.Count -ge 6) { $series[2..($series.Count - 1)] } else { $series }
|
|
$firstValue = [double]$warm[0].$Column
|
|
$lastValue = [double]$warm[-1].$Column
|
|
$spanCycles = [double]($warm[-1].Cycle - $warm[0].Cycle)
|
|
return [pscustomobject][ordered]@{
|
|
Stop = $Stop
|
|
Column = $Column
|
|
FirstCycle = $warm[0].Cycle
|
|
LastCycle = $warm[-1].Cycle
|
|
First = $firstValue
|
|
Last = $lastValue
|
|
Delta = [Math]::Round($lastValue - $firstValue, 1)
|
|
PerCycle = if ($spanCycles -gt 0) { [Math]::Round(($lastValue - $firstValue) / $spanCycles, 2) } else { $null }
|
|
}
|
|
}
|
|
|
|
$trends = @()
|
|
foreach ($arm in $arms) {
|
|
foreach ($stop in @('holtburg', 'hub', 'aerlinthe')) {
|
|
foreach ($column in @('WorkingSetMiB', 'PrivateMiB', 'GpuMiB', 'ManagedMiB',
|
|
'Materialized', 'WorldEntities', 'MeshRenderData',
|
|
'CompositeTextures', 'GpuTextures', 'Handles')) {
|
|
$t = Get-StopTrend $arm.Rows $stop $column
|
|
if ($t) {
|
|
$trends += [pscustomobject][ordered]@{
|
|
Backend = $arm.Backend; Stop = $t.Stop; Column = $t.Column
|
|
FirstCycle = $t.FirstCycle; LastCycle = $t.LastCycle
|
|
First = $t.First; Last = $t.Last; Delta = $t.Delta; PerCycle = $t.PerCycle
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
Write-Host ''
|
|
foreach ($arm in $arms) {
|
|
if ($arm.Aborted) { Write-Host "!!! $($arm.Backend) aborted: $($arm.Aborted)" -ForegroundColor Yellow }
|
|
Write-Host "=== $($arm.Backend) : completed=$($arm.RouteCompleted) graceful=$($arm.Graceful) exit=$($arm.ExitCode) witness=$($arm.Witness) checkpoints=$($arm.Checkpoints) peakWS=$($arm.PeakWorkingSetMiB) MiB finalWS=$($arm.FinalWorkingSetMiB) MiB"
|
|
$arm.Rows | Where-Object { $_.Stop -eq 'holtburg' } |
|
|
Select-Object Cycle, WorkingSetMiB, PrivateMiB, GpuMiB, ManagedMiB, LiveEntities, Materialized, WorldEntities, MeshRenderData, CompositeTextures, GpuTextures, Handles |
|
|
Format-Table -AutoSize | Out-String | Write-Host
|
|
}
|
|
Write-Host '=== within-arm capture comparison (the #256 half) ==='
|
|
$comparisons | Format-Table -AutoSize | Out-String | Write-Host
|
|
Write-Host '=== warm trends (cycle 3 -> last) ==='
|
|
$trends | Where-Object { $_.Stop -eq 'holtburg' } | Format-Table -AutoSize | Out-String | Write-Host
|
|
|
|
$report = [pscustomobject][ordered]@{
|
|
Commit = (& git -C $repo rev-parse HEAD).Trim()
|
|
GeneratedUtc = [DateTime]::UtcNow.ToString('o')
|
|
Cycles = $Cycles
|
|
Transits = $transit
|
|
Stops = @($stops | ForEach-Object { $_.Key })
|
|
ShotCycles = @($ShotCycles)
|
|
Route = $routePath
|
|
DayGroup = $DayGroup
|
|
WorldDayFraction = $WorldDayFraction
|
|
SkyPhaseSeconds = $SkyPhaseSeconds
|
|
MsaaSamples = 0
|
|
MaskTopPixels = $MaskTopPixels
|
|
Arms = @($arms | ForEach-Object {
|
|
[pscustomobject][ordered]@{
|
|
Backend = $_.Backend; RouteCompleted = $_.RouteCompleted; Aborted = $_.Aborted
|
|
Graceful = $_.Graceful
|
|
ExitCode = $_.ExitCode; Witness = $_.Witness; GrabBytes = $_.GrabBytes
|
|
Checkpoints = $_.Checkpoints; Samples = $_.Samples
|
|
PeakWorkingSetMiB = $_.PeakWorkingSetMiB; PeakPrivateMiB = $_.PeakPrivateMiB
|
|
FinalWorkingSetMiB = $_.FinalWorkingSetMiB; ErrorLines = $_.ErrorLines
|
|
Rows = $_.Rows
|
|
}
|
|
})
|
|
Comparisons = @($comparisons)
|
|
Trends = @($trends)
|
|
}
|
|
$reportPath = Join-Path $Out 'churn-soak.json'
|
|
$report | ConvertTo-Json -Depth 8 | Set-Content -LiteralPath $reportPath -Encoding utf8
|
|
Write-Step "report: $reportPath"
|