<# .SYNOPSIS Repeated-portal-churn regression soak: memory growth and visual drift. .DESCRIPTION Originated as Campaign V slice V11 step 0: the #256/#257 discriminator, run on both the GL and Vulkan backends before V11 deleted the former. Issues #256 (server-spawned signs and portals go invisible after repeated portal runs, while staying interactive) and #257 (working set grows to ~1.5 GB over the same session) were both observed on the Vulkan backend in one long live session, and running the same repeated-churn route on both arms told the campaign whether either bug predated Vulkan or was new to it: growth / drift on BOTH arms -> a pre-existing publication or resource lifetime bug that the campaign merely witnessed; growth / drift on VULKAN ONLY -> the new arm's resource lifecycle, and deleting its predecessor would destroy the only reference implementation that could answer the question. Campaign V slice V11 deleted the GL backend, so that specific before-and-after question can no longer be asked of any future bug — but the soak itself does not depend on having two arms. Comparing a fresh Holtburg capture against a much-later one from the SAME arm is still the closest automated instrument to what a user's long session actually shows, and the memory/entity/GPU trend analysis below never needed a second arm to begin with. -Backends now defaults to Vulkan alone; pass additional values only to compare arms that still exist (a future backend, a driver variant) — 'gl' is no longer a valid launch and the app ignores it. WHAT IT MEASURES, per stop, per cycle: * working set and private bytes, sampled from the OS every -SampleSeconds and joined to each checkpoint by timestamp. This is #257's quantity: the client's own snapshot has no view of its own working set. * the published-versus-live entity pair from the checkpoint JSON -- liveEntities / materializedLiveEntities against worldEntities and meshRenderData. #256's symptom is "alive in the object table, gone from the presentation", so a drift between those two halves at the SAME stop across cycles is the instrument. * the GPU accounting the checkpoint already carries: trackedGpuBytes, residencyGpuBytes, their difference, tracked buffer and texture counts, owned composite textures and their owners. GpuMemoryTracker's categories reach the checkpoint as those counts. * managed bytes, committed bytes, LOH size, and the streaming/retirement backlogs, so a growth finding can be attributed to a heap rather than left as "the process is bigger". HOW THE ROUTE IS BUILT. -Cycles repetitions of three stops taken from the two existing connected routes: Holtburg town (the dense outdoor stop the differential route calls its smoke stop, and the one that has a portal in frame), the Facility Hub interior EnvCell, and the Aerlinthe island. Every teleloc carries the identity quaternion so the heading at a stop is the same on every visit, which is what makes cycle 1 and cycle N comparable at all. Each stop writes a checkpoint; Holtburg additionally captures a screenshot on the cycles named by -ShotCycles. THE PIXEL HALF. Within one arm, the Holtburg capture from cycle 1 and the capture from the last shot cycle are compared against each other. Same process, same position, same heading, same pinned clock: an object that has stopped being drawn shows up as a contiguous mid-frame difference. This is the closest automated instrument to what the user actually saw. The four determinism levers the differential gate forces are forced here for the same reason -- an unpinned sun or cloud sheet would swamp the signal. NOT A PIXEL GATE. This script never compares two backends to each other — even when -Backends names more than one arm, each is still compared only with itself. The verdict is about growth and drift within an arm, not parity between arms. .PARAMETER Out Directory for both arms' logs, checkpoints, captures, samples and the verdict. .PARAMETER Cycles Repetitions of the three-stop loop. Default 30 -- #257's acceptance criterion is "a repeated-portal soak (>= 30 transits) holds working set flat after warmup", and three stops per cycle makes 90 transits at the default. .PARAMETER Backends Which arms to run, in order. Default 'vulkan' alone — the only backend that exists since Campaign V slice V11 deleted the GL arm. .PARAMETER ShotCycles Cycles on which Holtburg is captured. Default 1, 10, 20 and the last cycle. .PARAMETER SettleMilliseconds Stationary window between arrival and the checkpoint. Default 6000. .EXAMPLE tools/run-portal-churn-soak.ps1 -Out artifacts/v11-churn tools/run-portal-churn-soak.ps1 -Out artifacts/v11-churn-short -Cycles 4 -SkipBuild #> [CmdletBinding()] param( [Parameter(Mandatory = $true)][string]$Out, [int]$Cycles = 30, [string[]]$Backends = @('vulkan'), [int[]]$ShotCycles, [int]$SettleMilliseconds = 6000, [int]$ShotSettleMilliseconds = 12000, [int]$SampleSeconds = 2, [int]$DayGroup = 0, [double]$WorldDayFraction = 0.5, [double]$SkyPhaseSeconds = 0, [int]$MaskTopPixels = 280, [int]$Tolerance = 2, [double]$MaxDifferentFraction = 0.001, [int]$MinRenderedBytes = 500000, # A soak whose subject is unbounded growth must not be allowed to take the # machine down with it. Crossing this stops the arm early and keeps every # sample and checkpoint taken so far -- which is the finding, not a failure. [int]$AbortWorkingSetMiB = 10000, [int]$CooldownSeconds = 20, [int]$RouteTimeoutSeconds = 2400, [switch]$SkipBuild ) $ErrorActionPreference = 'Stop' Set-StrictMode -Version Latest $repo = Split-Path -Parent $PSScriptRoot $exe = Join-Path $repo 'src\AcDream.App\bin\Release\net10.0\AcDream.App.exe' $cli = Join-Path $repo 'src\AcDream.Cli\bin\Release\net10.0\AcDream.Cli.dll' function Write-Step($message) { Write-Host "[churn] $message" } if (-not $ShotCycles) { $ShotCycles = @(1, 10, 20, $Cycles) | Where-Object { $_ -le $Cycles } | Sort-Object -Unique } # --- Preconditions ------------------------------------------------------------ if (Get-Process -Name AcDream.App -ErrorAction SilentlyContinue) { throw 'AcDream.App is already running. This soak uses the shared test account and must not steal its session.' } if (@(Get-NetUDPEndpoint -LocalPort 9000 -ErrorAction SilentlyContinue).Count -eq 0) { throw 'local ACE is not listening on UDP port 9000.' } if (-not $SkipBuild) { Write-Step 'building Release' & dotnet build (Join-Path $repo 'AcDream.slnx') -c Release --nologo -v q | Out-Null if ($LASTEXITCODE -ne 0) { throw "Release build failed with exit code $LASTEXITCODE." } } if (-not (Test-Path $exe)) { throw "Client not found at $exe." } if (-not (Test-Path $cli)) { throw "AcDream.Cli not found at $cli." } if (Test-Path $Out) { Remove-Item -Recurse -Force $Out } New-Item -ItemType Directory -Force -Path $Out | Out-Null # --- The route ---------------------------------------------------------------- # Three stops per cycle, all three taken from routes that already exist. The # identity quaternion on every teleloc is what pins the heading across visits. $stops = @( [pscustomobject]@{ Key = 'holtburg'; Teleloc = '0xA9B40019 84.0 7.1 94.005 1 0 0 0' } [pscustomobject]@{ Key = 'hub'; Teleloc = '0x8A020164 70.35 -40.66 -5.9 1 0 0 0' } [pscustomobject]@{ Key = 'aerlinthe'; Teleloc = '0x09040008 11.4 188.6 87.705 1 0 0 0' } ) $lines = [System.Collections.Generic.List[string]]::new() $lines.Add('# GENERATED by tools/run-portal-churn-soak.ps1 -- the portal-churn regression soak.') $lines.Add("# $Cycles cycles x $($stops.Count) stops. Do not hand-edit; regenerate.") $lines.Add('') $lines.Add('wait world-ready 90000') $lines.Add('wait world-visible 30000') $lines.Add('sleep 15000') $lines.Add('checkpoint c00_login') $lines.Add('') $transit = 0 foreach ($cycle in 1..$Cycles) { $tag = '{0:d2}' -f $cycle $shot = $ShotCycles -contains $cycle foreach ($stop in $stops) { $transit++ $settle = if ($shot -and $stop.Key -eq 'holtburg') { $ShotSettleMilliseconds } else { $SettleMilliseconds } $lines.Add("command /teleloc $($stop.Teleloc)") $lines.Add("wait materialized $transit 90000") $lines.Add('wait world-visible 30000') $lines.Add("sleep $settle") $lines.Add("checkpoint c${tag}_$($stop.Key)") if ($shot -and $stop.Key -eq 'holtburg') { $lines.Add("screenshot holtburg_c$tag 15000") } } $lines.Add('') } $lines.Add('sleep 5000') $routePath = Join-Path $Out 'route.txt' Set-Content -Encoding utf8 -LiteralPath $routePath -Value $lines Write-Step "route: $Cycles cycles, $transit transits, $($lines.Count) lines -> $routePath" Add-Type -AssemblyName System.Drawing Add-Type @' using System; using System.Runtime.InteropServices; public static class ChurnSoakSurface { [DllImport("user32.dll")] public static extern bool SetForegroundWindow(IntPtr hWnd); [DllImport("user32.dll")] public static extern bool ShowWindow(IntPtr hWnd, int nCmdShow); [DllImport("user32.dll")] public static extern bool GetClientRect(IntPtr hWnd, out RECT r); [DllImport("user32.dll")] public static extern bool ClientToScreen(IntPtr hWnd, ref POINT p); [StructLayout(LayoutKind.Sequential)] public struct RECT { public int L, T, R, B; } [StructLayout(LayoutKind.Sequential)] public struct POINT { public int X, Y; } } '@ # A number cannot tell an absent object from a walking NPC. The map and the row # histogram can: a vanished sign is a contiguous blob in one band, a creature is # a thin moving silhouette, and phase noise is scattered. Add-Type -ReferencedAssemblies System.Drawing @' using System; using System.Drawing; using System.Drawing.Imaging; using System.Runtime.InteropServices; public static class ChurnDiffMap { public static int[] Write(string expected, string actual, string outPath, int tolerance, int maskTop) { int[] bands = new int[18]; using (Bitmap a = new Bitmap(expected)) using (Bitmap b = new Bitmap(actual)) using (Bitmap map = new Bitmap(a.Width, a.Height, PixelFormat.Format32bppArgb)) { Rectangle rect = new Rectangle(0, 0, a.Width, a.Height); BitmapData da = a.LockBits(rect, ImageLockMode.ReadOnly, PixelFormat.Format32bppArgb); BitmapData db = b.LockBits(rect, ImageLockMode.ReadOnly, PixelFormat.Format32bppArgb); BitmapData dm = map.LockBits(rect, ImageLockMode.WriteOnly, PixelFormat.Format32bppArgb); int count = a.Width * a.Height; int[] pa = new int[count], pb = new int[count], pm = new int[count]; Marshal.Copy(da.Scan0, pa, 0, count); Marshal.Copy(db.Scan0, pb, 0, count); for (int y = 0; y < a.Height; y++) { for (int x = 0; x < a.Width; x++) { int i = y * a.Width + x; int ca = pa[i], cb = pb[i]; int d = Math.Max(Math.Abs(((ca >> 16) & 255) - ((cb >> 16) & 255)), Math.Max(Math.Abs(((ca >> 8) & 255) - ((cb >> 8) & 255)), Math.Abs((ca & 255) - (cb & 255)))); if (y >= maskTop && d > tolerance) { pm[i] = unchecked((int)0xFFFF00FF); bands[Math.Min(17, y / 40)]++; } else { int grey = (int)((((cb >> 16) & 255) * 0.3 + ((cb >> 8) & 255) * 0.59 + (cb & 255) * 0.11) / 4); pm[i] = unchecked((int)0xFF000000) | (grey << 16) | (grey << 8) | grey; } } } Marshal.Copy(pm, 0, dm.Scan0, count); a.UnlockBits(da); b.UnlockBits(db); map.UnlockBits(dm); map.Save(outPath, ImageFormat.Png); } return bands; } } '@ function Invoke-Arm([string]$Backend) { $dir = Join-Path $Out $Backend New-Item -ItemType Directory -Force -Path $dir | Out-Null $log = Join-Path $dir 'client.log' $grab = Join-Path $dir 'desktop-grab.png' $timeline = Join-Path $dir 'world-lifecycle.checkpoints.jsonl' $env:ACDREAM_DAT_DIR = Join-Path $env:USERPROFILE "Documents\Asheron's Call" $env:ACDREAM_LIVE = '1' $env:ACDREAM_TEST_HOST = '127.0.0.1' $env:ACDREAM_TEST_PORT = '9000' $env:ACDREAM_TEST_USER = 'testaccount' $env:ACDREAM_TEST_PASS = 'testpassword' $env:ACDREAM_RETAIL_UI = '1' $env:ACDREAM_DEVTOOLS = '0' $env:ACDREAM_UI_PROBE_DUMP = '0' $env:ACDREAM_FRAME_PROF = '1' # The four determinism levers, forced. Without them a cycle-1-versus-cycle-N # capture measures the sun and the cloud sheet instead of the world. $env:ACDREAM_MSAA_SAMPLES = '0' $env:ACDREAM_DAY_GROUP = "$DayGroup" $env:ACDREAM_WORLD_TIME = $WorldDayFraction.ToString([System.Globalization.CultureInfo]::InvariantCulture) $env:ACDREAM_SKY_PHASE_SECONDS = $SkyPhaseSeconds.ToString([System.Globalization.CultureInfo]::InvariantCulture) $env:ACDREAM_RENDER_BACKEND = $Backend $env:ACDREAM_UI_PROBE_SCRIPT = $routePath $env:ACDREAM_AUTOMATION_ARTIFACT_DIR = $dir Write-Step "launching $Backend" $proc = Start-Process -FilePath $exe -WorkingDirectory $repo ` -RedirectStandardOutput $log -RedirectStandardError "$log.err" -PassThru $samples = [System.Collections.Generic.List[object]]::new() $completed = $false $aborted = $null $deadline = (Get-Date).AddSeconds($RouteTimeoutSeconds) $lastReport = Get-Date while ((Get-Date) -lt $deadline) { if (Select-String -Path $log -SimpleMatch '[UI-PROBE] UI probe script complete' ` -ErrorAction SilentlyContinue) { $completed = $true; break } if ($proc.HasExited) { break } try { $proc.Refresh() $samples.Add([pscustomobject][ordered]@{ UtcTicks = [DateTime]::UtcNow.Ticks WorkingSetBytes = $proc.WorkingSet64 PrivateBytes = $proc.PrivateMemorySize64 HandleCount = $proc.HandleCount ThreadCount = $proc.Threads.Count }) if ($proc.WorkingSet64 / 1MB -gt $AbortWorkingSetMiB) { $aborted = "working set crossed $AbortWorkingSetMiB MiB at sample $($samples.Count)" Write-Step "$Backend : ABORTING -- $aborted" break } } catch { } if (((Get-Date) - $lastReport).TotalSeconds -ge 60) { $lastReport = Get-Date $done = 0 if (Test-Path -LiteralPath $timeline) { $done = @(Get-Content -LiteralPath $timeline -ErrorAction SilentlyContinue).Count } $ws = if ($samples.Count -gt 0) { [Math]::Round($samples[$samples.Count - 1].WorkingSetBytes / 1MB, 0) } else { 0 } Write-Step "$Backend : $done checkpoints, working set ${ws} MiB" } Start-Sleep -Seconds $SampleSeconds } $grabBytes = 0 if (-not $proc.HasExited) { try { $proc.Refresh() $h = $proc.MainWindowHandle if ($h -ne [IntPtr]::Zero) { [ChurnSoakSurface]::ShowWindow($h, 9) | Out-Null [ChurnSoakSurface]::SetForegroundWindow($h) | Out-Null Start-Sleep -Milliseconds 1200 $r = New-Object ChurnSoakSurface+RECT [ChurnSoakSurface]::GetClientRect($h, [ref]$r) | Out-Null $p = New-Object ChurnSoakSurface+POINT [ChurnSoakSurface]::ClientToScreen($h, [ref]$p) | Out-Null $w = $r.R - $r.L; $ht = $r.B - $r.T if ($w -gt 0 -and $ht -gt 0) { $bmp = New-Object System.Drawing.Bitmap $w, $ht $g = [System.Drawing.Graphics]::FromImage($bmp) $g.CopyFromScreen($p.X, $p.Y, 0, 0, (New-Object System.Drawing.Size $w, $ht)) $bmp.Save($grab, [System.Drawing.Imaging.ImageFormat]::Png) $g.Dispose(); $bmp.Dispose() $grabBytes = (Get-Item $grab).Length } } } catch { Write-Step "desktop grab failed on ${Backend}: $_" } } # The peak matters as much as the end value: #257 reported a number reached, # not a number held. Take a final sample before the graceful close. try { $proc.Refresh() $samples.Add([pscustomobject][ordered]@{ UtcTicks = [DateTime]::UtcNow.Ticks WorkingSetBytes = $proc.WorkingSet64 PrivateBytes = $proc.PrivateMemorySize64 HandleCount = $proc.HandleCount ThreadCount = $proc.Threads.Count }) } catch { } $graceful = $false $exitCode = $null $app = Get-Process -Name AcDream.App -ErrorAction SilentlyContinue if ($app) { $app.CloseMainWindow() | Out-Null if ($app.WaitForExit(45000)) { $graceful = $true; $exitCode = $app.ExitCode } else { Write-Step "WM_CLOSE timed out on $Backend; forcing" $app | Stop-Process -Force } } elseif ($proc.HasExited) { $exitCode = $proc.ExitCode } foreach ($name in @('ACDREAM_RENDER_BACKEND', 'ACDREAM_MSAA_SAMPLES', 'ACDREAM_SKY_PHASE_SECONDS', 'ACDREAM_WORLD_TIME', 'ACDREAM_DAY_GROUP')) { Remove-Item "Env:\$name" -ErrorAction SilentlyContinue } $samples | Export-Csv -NoTypeInformation -Encoding utf8 -LiteralPath (Join-Path $dir 'memory-samples.csv') $checkpoints = @() if (Test-Path -LiteralPath $timeline) { $checkpoints = @(Get-Content -LiteralPath $timeline | Where-Object { $_.Trim().Length -gt 0 } | ForEach-Object { $_ | ConvertFrom-Json }) } $rows = [System.Collections.Generic.List[object]]::new() foreach ($cp in $checkpoints) { $ticks = ([DateTime]::Parse( $cp.timestampUtc, [System.Globalization.CultureInfo]::InvariantCulture, [System.Globalization.DateTimeStyles]::AdjustToUniversal -bor [System.Globalization.DateTimeStyles]::AssumeUniversal)).Ticks $nearest = $null $bestDelta = [long]::MaxValue foreach ($s in $samples) { $delta = [Math]::Abs($s.UtcTicks - $ticks) if ($delta -lt $bestDelta) { $bestDelta = $delta; $nearest = $s } } $r = $cp.resources $rows.Add([pscustomobject][ordered]@{ Sequence = $cp.sequence Name = $cp.name Cycle = if ($cp.name -match '^c(\d+)_') { [int]$Matches[1] } else { -1 } Stop = if ($cp.name -match '^c\d+_(.+)$') { $Matches[1] } else { $cp.name } WorkingSetMiB = if ($nearest) { [Math]::Round($nearest.WorkingSetBytes / 1MB, 1) } else { $null } PrivateMiB = if ($nearest) { [Math]::Round($nearest.PrivateBytes / 1MB, 1) } else { $null } Handles = if ($nearest) { $nearest.HandleCount } else { $null } SampleSkewMs = if ($nearest) { [Math]::Round($bestDelta / 10000.0, 0) } else { $null } LiveEntities = $r.liveEntities Materialized = $r.materializedLiveEntities WorldEntities = $r.worldEntities LoadedLandblocks = $r.loadedLandblocks VisibleLandblocks = $r.visibleLandblocks MeshRenderData = $r.meshRenderData MeshMiB = [Math]::Round($r.meshEstimatedBytes / 1MB, 1) GpuMiB = [Math]::Round($r.trackedGpuBytes / 1MB, 1) ResidencyGpuMiB = [Math]::Round($r.residencyGpuBytes / 1MB, 1) GpuMinusResidencyMiB = [Math]::Round($r.gpuTrackerMinusResidencyBytes / 1MB, 1) GpuBuffers = $r.trackedGpuBuffers GpuTextures = $r.trackedGpuTextures CompositeTextures = $r.ownedCompositeTextures CompositeOwners = $r.compositeTextureOwners ParticleTextures = $r.activeParticleTextures ManagedMiB = [Math]::Round($r.managedBytes / 1MB, 1) CommittedMiB = [Math]::Round($r.managedCommittedBytes / 1MB, 1) LohMiB = [Math]::Round($r.lohSizeBytes / 1MB, 1) PendingTeardowns = $r.pendingLiveTeardowns PendingRetirements = $r.pendingLandblockRetirements StagedMeshUploads = $r.stagedMeshUploads Fps = [Math]::Round($r.fps, 1) }) } $rows | Export-Csv -NoTypeInformation -Encoding utf8 -LiteralPath (Join-Path $dir 'per-checkpoint.csv') $witness = if ($grabBytes -ge $MinRenderedBytes) { 'RENDERED' } elseif ($grabBytes -gt 0) { 'BLANK' } else { 'NO-CAPTURE' } $errors = @(Select-String -Path $log -Pattern 'Unhandled exception|AccessViolation|OutOfMemoryException|event=invariant-failure|device removed|screenshot-failed' ` -ErrorAction SilentlyContinue | Select-Object -First 10 | ForEach-Object { $_.Line }) return [pscustomobject][ordered]@{ Backend = $Backend RouteCompleted = $completed Aborted = $aborted Graceful = $graceful ExitCode = $exitCode Witness = $witness GrabBytes = $grabBytes Checkpoints = $rows.Count Rows = $rows Samples = $samples.Count PeakWorkingSetMiB = if ($samples.Count -gt 0) { [Math]::Round((($samples | Measure-Object WorkingSetBytes -Maximum).Maximum) / 1MB, 1) } else { $null } PeakPrivateMiB = if ($samples.Count -gt 0) { [Math]::Round((($samples | Measure-Object PrivateBytes -Maximum).Maximum) / 1MB, 1) } else { $null } FinalWorkingSetMiB = if ($samples.Count -gt 0) { [Math]::Round($samples[$samples.Count - 1].WorkingSetBytes / 1MB, 1) } else { $null } ErrorLines = $errors Dir = $dir ScreenshotDir = Join-Path $dir 'screenshots' Log = $log } } $arms = @() foreach ($backend in $Backends) { if ($arms.Count -gt 0) { Write-Step "cooldown ${CooldownSeconds}s so ACE clears the graceful logout" Start-Sleep -Seconds $CooldownSeconds } $arms += Invoke-Arm $backend } # --- Within-arm capture comparison ------------------------------------------- # The #256 half. First Holtburg capture against the last, inside one arm. $maskPath = $null $comparisons = @() foreach ($arm in $arms) { $shots = @(Get-ChildItem $arm.ScreenshotDir -Filter 'holtburg_c*.png' -ErrorAction SilentlyContinue | Sort-Object Name) if ($shots.Count -lt 2) { $comparisons += [pscustomobject][ordered]@{ Backend = $arm.Backend; Pair = 'n/a'; Verdict = 'UNPAIRED' DifferentPixels = $null; Fraction = $null; MaxChannelDelta = $null } continue } if ($MaskTopPixels -gt 0 -and -not $maskPath) { $probe = [System.Drawing.Bitmap]::FromFile($shots[0].FullName) $mw = $probe.Width; $mh = $probe.Height $probe.Dispose() $mask = New-Object System.Drawing.Bitmap($mw, $mh, [System.Drawing.Imaging.PixelFormat]::Format32bppArgb) $graphics = [System.Drawing.Graphics]::FromImage($mask) $graphics.Clear([System.Drawing.Color]::FromArgb(0, 0, 0, 0)) $opaque = New-Object System.Drawing.SolidBrush ([System.Drawing.Color]::FromArgb(255, 255, 0, 255)) $graphics.FillRectangle($opaque, 0, 0, $mw, [Math]::Min($MaskTopPixels, $mh)) $graphics.Dispose(); $opaque.Dispose() $maskPath = Join-Path $Out 'mask.png' $mask.Save($maskPath, [System.Drawing.Imaging.ImageFormat]::Png) $mask.Dispose() } $first = $shots[0] foreach ($later in $shots[1..($shots.Count - 1)]) { $report = Join-Path $Out "compare-$($arm.Backend)-$($first.BaseName)-vs-$($later.BaseName).json" if ($maskPath) { & dotnet $cli compare-screenshots $first.FullName $later.FullName $report $Tolerance $MaxDifferentFraction $maskPath | Out-Null } else { & dotnet $cli compare-screenshots $first.FullName $later.FullName $report $Tolerance $MaxDifferentFraction | Out-Null } $verdict = Get-Content $report -Raw | ConvertFrom-Json # A number cannot tell an absent object from a walking NPC. The map and # the row histogram can: a vanished sign is a contiguous blob in one # band, a creature is a thin moving silhouette, and phase noise is # scattered. Both are written beside the report. $mapPath = Join-Path $Out "diff-$($arm.Backend)-$($first.BaseName)-vs-$($later.BaseName).png" $bands = New-Object 'int[]' 18 try { $bands = [ChurnDiffMap]::Write($first.FullName, $later.FullName, $mapPath, $Tolerance, $MaskTopPixels) } catch { Write-Step "diff map failed for $($later.BaseName): $_" } $comparisons += [pscustomobject][ordered]@{ Backend = $arm.Backend Pair = "$($first.BaseName) vs $($later.BaseName)" Verdict = if ($verdict.passed) { 'MATCH' } else { 'DIVERGED' } DifferentPixels = $verdict.differentPixels Fraction = $verdict.differentPixelFraction MaxChannelDelta = $verdict.maximumChannelDelta RowBands40px = ($bands -join ',') DiffMap = $mapPath } } } # --- Verdict ------------------------------------------------------------------ function Get-StopTrend($rows, [string]$Stop, [string]$Column) { $series = @($rows | Where-Object { $_.Stop -eq $Stop -and $_.Cycle -ge 1 } | Sort-Object Cycle) if ($series.Count -lt 2) { return $null } $warm = if ($series.Count -ge 6) { $series[2..($series.Count - 1)] } else { $series } $firstValue = [double]$warm[0].$Column $lastValue = [double]$warm[-1].$Column $spanCycles = [double]($warm[-1].Cycle - $warm[0].Cycle) return [pscustomobject][ordered]@{ Stop = $Stop Column = $Column FirstCycle = $warm[0].Cycle LastCycle = $warm[-1].Cycle First = $firstValue Last = $lastValue Delta = [Math]::Round($lastValue - $firstValue, 1) PerCycle = if ($spanCycles -gt 0) { [Math]::Round(($lastValue - $firstValue) / $spanCycles, 2) } else { $null } } } $trends = @() foreach ($arm in $arms) { foreach ($stop in @('holtburg', 'hub', 'aerlinthe')) { foreach ($column in @('WorkingSetMiB', 'PrivateMiB', 'GpuMiB', 'ManagedMiB', 'Materialized', 'WorldEntities', 'MeshRenderData', 'CompositeTextures', 'GpuTextures', 'Handles')) { $t = Get-StopTrend $arm.Rows $stop $column if ($t) { $trends += [pscustomobject][ordered]@{ Backend = $arm.Backend; Stop = $t.Stop; Column = $t.Column FirstCycle = $t.FirstCycle; LastCycle = $t.LastCycle First = $t.First; Last = $t.Last; Delta = $t.Delta; PerCycle = $t.PerCycle } } } } } Write-Host '' foreach ($arm in $arms) { if ($arm.Aborted) { Write-Host "!!! $($arm.Backend) aborted: $($arm.Aborted)" -ForegroundColor Yellow } Write-Host "=== $($arm.Backend) : completed=$($arm.RouteCompleted) graceful=$($arm.Graceful) exit=$($arm.ExitCode) witness=$($arm.Witness) checkpoints=$($arm.Checkpoints) peakWS=$($arm.PeakWorkingSetMiB) MiB finalWS=$($arm.FinalWorkingSetMiB) MiB" $arm.Rows | Where-Object { $_.Stop -eq 'holtburg' } | Select-Object Cycle, WorkingSetMiB, PrivateMiB, GpuMiB, ManagedMiB, LiveEntities, Materialized, WorldEntities, MeshRenderData, CompositeTextures, GpuTextures, Handles | Format-Table -AutoSize | Out-String | Write-Host } Write-Host '=== within-arm capture comparison (the #256 half) ===' $comparisons | Format-Table -AutoSize | Out-String | Write-Host Write-Host '=== warm trends (cycle 3 -> last) ===' $trends | Where-Object { $_.Stop -eq 'holtburg' } | Format-Table -AutoSize | Out-String | Write-Host $report = [pscustomobject][ordered]@{ Commit = (& git -C $repo rev-parse HEAD).Trim() GeneratedUtc = [DateTime]::UtcNow.ToString('o') Cycles = $Cycles Transits = $transit Stops = @($stops | ForEach-Object { $_.Key }) ShotCycles = @($ShotCycles) Route = $routePath DayGroup = $DayGroup WorldDayFraction = $WorldDayFraction SkyPhaseSeconds = $SkyPhaseSeconds MsaaSamples = 0 MaskTopPixels = $MaskTopPixels Arms = @($arms | ForEach-Object { [pscustomobject][ordered]@{ Backend = $_.Backend; RouteCompleted = $_.RouteCompleted; Aborted = $_.Aborted Graceful = $_.Graceful ExitCode = $_.ExitCode; Witness = $_.Witness; GrabBytes = $_.GrabBytes Checkpoints = $_.Checkpoints; Samples = $_.Samples PeakWorkingSetMiB = $_.PeakWorkingSetMiB; PeakPrivateMiB = $_.PeakPrivateMiB FinalWorkingSetMiB = $_.FinalWorkingSetMiB; ErrorLines = $_.ErrorLines Rows = $_.Rows } }) Comparisons = @($comparisons) Trends = @($trends) } $reportPath = Join-Path $Out 'churn-soak.json' $report | ConvertTo-Json -Depth 8 | Set-Content -LiteralPath $reportPath -Encoding utf8 Write-Step "report: $reportPath"