<# .SYNOPSIS Campaign V slice V11 step 0: the #256/#257 discriminator, run on both backends. .DESCRIPTION Issues #256 (server-spawned signs and portals go invisible after repeated portal runs, while staying interactive) and #257 (working set grows to ~1.5 GB over the same session) were both observed on the Vulkan backend in one long live session. Both filings demand the SAME discriminator, and both demand it BEFORE V11 deletes the OpenGL backend: run the same repeated-churn route on GL and on Vulkan. growth / drift on BOTH arms -> a pre-existing publication or resource lifetime bug that the campaign merely witnessed; growth / drift on VULKAN ONLY -> the new arm's resource lifecycle, and deleting its predecessor would destroy the only reference implementation that can answer the question. After V11 the GL arm does not exist and this question costs far more to answer, which is why this script runs first and why its verdict can stop the slice. WHAT IT MEASURES, per stop, per cycle: * working set and private bytes, sampled from the OS every -SampleSeconds and joined to each checkpoint by timestamp. This is #257's quantity: the client's own snapshot has no view of its own working set. * the published-versus-live entity pair from the checkpoint JSON -- liveEntities / materializedLiveEntities against worldEntities and meshRenderData. #256's symptom is "alive in the object table, gone from the presentation", so a drift between those two halves at the SAME stop across cycles is the instrument. * the GPU accounting the checkpoint already carries: trackedGpuBytes, residencyGpuBytes, their difference, tracked buffer and texture counts, owned composite textures and their owners. GpuMemoryTracker's categories reach the checkpoint as those counts. * managed bytes, committed bytes, LOH size, and the streaming/retirement backlogs, so a growth finding can be attributed to a heap rather than left as "the process is bigger". HOW THE ROUTE IS BUILT. -Cycles repetitions of three stops taken from the two existing connected routes: Holtburg town (the dense outdoor stop the differential route calls its smoke stop, and the one that has a portal in frame), the Facility Hub interior EnvCell, and the Aerlinthe island. Every teleloc carries the identity quaternion so the heading at a stop is the same on every visit, which is what makes cycle 1 and cycle N comparable at all. Each stop writes a checkpoint; Holtburg additionally captures a screenshot on the cycles named by -ShotCycles. THE PIXEL HALF. Within one arm, the Holtburg capture from cycle 1 and the capture from the last shot cycle are compared against each other. Same process, same position, same heading, same pinned clock: an object that has stopped being drawn shows up as a contiguous mid-frame difference. This is the closest automated instrument to what the user actually saw. The four determinism levers the differential gate forces are forced here for the same reason -- an unpinned sun or cloud sheet would swamp the signal. NOT A PIXEL GATE. This script never compares the two backends to each other; that is run-backend-differential-gate.ps1's job and its thresholds are the campaign's. Here each arm is compared only with itself, and the verdict is about growth and drift. .PARAMETER Out Directory for both arms' logs, checkpoints, captures, samples and the verdict. .PARAMETER Cycles Repetitions of the three-stop loop. Default 30 -- #257's acceptance criterion is "a repeated-portal soak (>= 30 transits) holds working set flat after warmup", and three stops per cycle makes 90 transits at the default. .PARAMETER Backends Which arms to run, in order. Default gl then vulkan, so the arm under suspicion runs second and cannot be blamed for warming the machine. .PARAMETER ShotCycles Cycles on which Holtburg is captured. Default 1, 10, 20 and the last cycle. .PARAMETER SettleMilliseconds Stationary window between arrival and the checkpoint. Default 6000. .EXAMPLE tools/run-portal-churn-soak.ps1 -Out artifacts/v11-churn tools/run-portal-churn-soak.ps1 -Out artifacts/v11-churn-short -Cycles 4 -SkipBuild #> [CmdletBinding()] param( [Parameter(Mandatory = $true)][string]$Out, [int]$Cycles = 30, [string[]]$Backends = @('gl', 'vulkan'), [int[]]$ShotCycles, [int]$SettleMilliseconds = 6000, [int]$ShotSettleMilliseconds = 12000, [int]$SampleSeconds = 2, [int]$DayGroup = 0, [double]$WorldDayFraction = 0.5, [double]$SkyPhaseSeconds = 0, [int]$MaskTopPixels = 280, [int]$Tolerance = 2, [double]$MaxDifferentFraction = 0.001, [int]$MinRenderedBytes = 500000, # A soak whose subject is unbounded growth must not be allowed to take the # machine down with it. Crossing this stops the arm early and keeps every # sample and checkpoint taken so far -- which is the finding, not a failure. [int]$AbortWorkingSetMiB = 10000, [int]$CooldownSeconds = 20, [int]$RouteTimeoutSeconds = 2400, [switch]$SkipBuild ) $ErrorActionPreference = 'Stop' Set-StrictMode -Version Latest $repo = Split-Path -Parent $PSScriptRoot $exe = Join-Path $repo 'src\AcDream.App\bin\Release\net10.0\AcDream.App.exe' $cli = Join-Path $repo 'src\AcDream.Cli\bin\Release\net10.0\AcDream.Cli.dll' function Write-Step($message) { Write-Host "[churn] $message" } if (-not $ShotCycles) { $ShotCycles = @(1, 10, 20, $Cycles) | Where-Object { $_ -le $Cycles } | Sort-Object -Unique } # --- Preconditions ------------------------------------------------------------ if (Get-Process -Name AcDream.App -ErrorAction SilentlyContinue) { throw 'AcDream.App is already running. This soak uses the shared test account and must not steal its session.' } if (@(Get-NetUDPEndpoint -LocalPort 9000 -ErrorAction SilentlyContinue).Count -eq 0) { throw 'local ACE is not listening on UDP port 9000.' } if (-not $SkipBuild) { Write-Step 'building Release' & dotnet build (Join-Path $repo 'AcDream.slnx') -c Release --nologo -v q | Out-Null if ($LASTEXITCODE -ne 0) { throw "Release build failed with exit code $LASTEXITCODE." } } if (-not (Test-Path $exe)) { throw "Client not found at $exe." } if (-not (Test-Path $cli)) { throw "AcDream.Cli not found at $cli." } if (Test-Path $Out) { Remove-Item -Recurse -Force $Out } New-Item -ItemType Directory -Force -Path $Out | Out-Null # --- The route ---------------------------------------------------------------- # Three stops per cycle, all three taken from routes that already exist. The # identity quaternion on every teleloc is what pins the heading across visits. $stops = @( [pscustomobject]@{ Key = 'holtburg'; Teleloc = '0xA9B40019 84.0 7.1 94.005 1 0 0 0' } [pscustomobject]@{ Key = 'hub'; Teleloc = '0x8A020164 70.35 -40.66 -5.9 1 0 0 0' } [pscustomobject]@{ Key = 'aerlinthe'; Teleloc = '0x09040008 11.4 188.6 87.705 1 0 0 0' } ) $lines = [System.Collections.Generic.List[string]]::new() $lines.Add('# GENERATED by tools/run-portal-churn-soak.ps1 -- the V11 step-0 discriminator.') $lines.Add("# $Cycles cycles x $($stops.Count) stops. Do not hand-edit; regenerate.") $lines.Add('') $lines.Add('wait world-ready 90000') $lines.Add('wait world-visible 30000') $lines.Add('sleep 15000') $lines.Add('checkpoint c00_login') $lines.Add('') $transit = 0 foreach ($cycle in 1..$Cycles) { $tag = '{0:d2}' -f $cycle $shot = $ShotCycles -contains $cycle foreach ($stop in $stops) { $transit++ $settle = if ($shot -and $stop.Key -eq 'holtburg') { $ShotSettleMilliseconds } else { $SettleMilliseconds } $lines.Add("command /teleloc $($stop.Teleloc)") $lines.Add("wait materialized $transit 90000") $lines.Add('wait world-visible 30000') $lines.Add("sleep $settle") $lines.Add("checkpoint c${tag}_$($stop.Key)") if ($shot -and $stop.Key -eq 'holtburg') { $lines.Add("screenshot holtburg_c$tag 15000") } } $lines.Add('') } $lines.Add('sleep 5000') $routePath = Join-Path $Out 'route.txt' Set-Content -Encoding utf8 -LiteralPath $routePath -Value $lines Write-Step "route: $Cycles cycles, $transit transits, $($lines.Count) lines -> $routePath" Add-Type -AssemblyName System.Drawing Add-Type @' using System; using System.Runtime.InteropServices; public static class ChurnSoakSurface { [DllImport("user32.dll")] public static extern bool SetForegroundWindow(IntPtr hWnd); [DllImport("user32.dll")] public static extern bool ShowWindow(IntPtr hWnd, int nCmdShow); [DllImport("user32.dll")] public static extern bool GetClientRect(IntPtr hWnd, out RECT r); [DllImport("user32.dll")] public static extern bool ClientToScreen(IntPtr hWnd, ref POINT p); [StructLayout(LayoutKind.Sequential)] public struct RECT { public int L, T, R, B; } [StructLayout(LayoutKind.Sequential)] public struct POINT { public int X, Y; } } '@ # A number cannot tell an absent object from a walking NPC. The map and the row # histogram can: a vanished sign is a contiguous blob in one band, a creature is # a thin moving silhouette, and phase noise is scattered. Add-Type -ReferencedAssemblies System.Drawing @' using System; using System.Drawing; using System.Drawing.Imaging; using System.Runtime.InteropServices; public static class ChurnDiffMap { public static int[] Write(string expected, string actual, string outPath, int tolerance, int maskTop) { int[] bands = new int[18]; using (Bitmap a = new Bitmap(expected)) using (Bitmap b = new Bitmap(actual)) using (Bitmap map = new Bitmap(a.Width, a.Height, PixelFormat.Format32bppArgb)) { Rectangle rect = new Rectangle(0, 0, a.Width, a.Height); BitmapData da = a.LockBits(rect, ImageLockMode.ReadOnly, PixelFormat.Format32bppArgb); BitmapData db = b.LockBits(rect, ImageLockMode.ReadOnly, PixelFormat.Format32bppArgb); BitmapData dm = map.LockBits(rect, ImageLockMode.WriteOnly, PixelFormat.Format32bppArgb); int count = a.Width * a.Height; int[] pa = new int[count], pb = new int[count], pm = new int[count]; Marshal.Copy(da.Scan0, pa, 0, count); Marshal.Copy(db.Scan0, pb, 0, count); for (int y = 0; y < a.Height; y++) { for (int x = 0; x < a.Width; x++) { int i = y * a.Width + x; int ca = pa[i], cb = pb[i]; int d = Math.Max(Math.Abs(((ca >> 16) & 255) - ((cb >> 16) & 255)), Math.Max(Math.Abs(((ca >> 8) & 255) - ((cb >> 8) & 255)), Math.Abs((ca & 255) - (cb & 255)))); if (y >= maskTop && d > tolerance) { pm[i] = unchecked((int)0xFFFF00FF); bands[Math.Min(17, y / 40)]++; } else { int grey = (int)((((cb >> 16) & 255) * 0.3 + ((cb >> 8) & 255) * 0.59 + (cb & 255) * 0.11) / 4); pm[i] = unchecked((int)0xFF000000) | (grey << 16) | (grey << 8) | grey; } } } Marshal.Copy(pm, 0, dm.Scan0, count); a.UnlockBits(da); b.UnlockBits(db); map.UnlockBits(dm); map.Save(outPath, ImageFormat.Png); } return bands; } } '@ function Invoke-Arm([string]$Backend) { $dir = Join-Path $Out $Backend New-Item -ItemType Directory -Force -Path $dir | Out-Null $log = Join-Path $dir 'client.log' $grab = Join-Path $dir 'desktop-grab.png' $timeline = Join-Path $dir 'world-lifecycle.checkpoints.jsonl' $env:ACDREAM_DAT_DIR = Join-Path $env:USERPROFILE "Documents\Asheron's Call" $env:ACDREAM_LIVE = '1' $env:ACDREAM_TEST_HOST = '127.0.0.1' $env:ACDREAM_TEST_PORT = '9000' $env:ACDREAM_TEST_USER = 'testaccount' $env:ACDREAM_TEST_PASS = 'testpassword' $env:ACDREAM_RETAIL_UI = '1' $env:ACDREAM_DEVTOOLS = '0' $env:ACDREAM_UI_PROBE_DUMP = '0' $env:ACDREAM_FRAME_PROF = '1' # The four determinism levers, forced. Without them a cycle-1-versus-cycle-N # capture measures the sun and the cloud sheet instead of the world. $env:ACDREAM_MSAA_SAMPLES = '0' $env:ACDREAM_DAY_GROUP = "$DayGroup" $env:ACDREAM_WORLD_TIME = $WorldDayFraction.ToString([System.Globalization.CultureInfo]::InvariantCulture) $env:ACDREAM_SKY_PHASE_SECONDS = $SkyPhaseSeconds.ToString([System.Globalization.CultureInfo]::InvariantCulture) $env:ACDREAM_RENDER_BACKEND = $Backend $env:ACDREAM_UI_PROBE_SCRIPT = $routePath $env:ACDREAM_AUTOMATION_ARTIFACT_DIR = $dir Write-Step "launching $Backend" $proc = Start-Process -FilePath $exe -WorkingDirectory $repo ` -RedirectStandardOutput $log -RedirectStandardError "$log.err" -PassThru $samples = [System.Collections.Generic.List[object]]::new() $completed = $false $aborted = $null $deadline = (Get-Date).AddSeconds($RouteTimeoutSeconds) $lastReport = Get-Date while ((Get-Date) -lt $deadline) { if (Select-String -Path $log -SimpleMatch '[UI-PROBE] UI probe script complete' ` -ErrorAction SilentlyContinue) { $completed = $true; break } if ($proc.HasExited) { break } try { $proc.Refresh() $samples.Add([pscustomobject][ordered]@{ UtcTicks = [DateTime]::UtcNow.Ticks WorkingSetBytes = $proc.WorkingSet64 PrivateBytes = $proc.PrivateMemorySize64 HandleCount = $proc.HandleCount ThreadCount = $proc.Threads.Count }) if ($proc.WorkingSet64 / 1MB -gt $AbortWorkingSetMiB) { $aborted = "working set crossed $AbortWorkingSetMiB MiB at sample $($samples.Count)" Write-Step "$Backend : ABORTING -- $aborted" break } } catch { } if (((Get-Date) - $lastReport).TotalSeconds -ge 60) { $lastReport = Get-Date $done = 0 if (Test-Path -LiteralPath $timeline) { $done = @(Get-Content -LiteralPath $timeline -ErrorAction SilentlyContinue).Count } $ws = if ($samples.Count -gt 0) { [Math]::Round($samples[$samples.Count - 1].WorkingSetBytes / 1MB, 0) } else { 0 } Write-Step "$Backend : $done checkpoints, working set ${ws} MiB" } Start-Sleep -Seconds $SampleSeconds } $grabBytes = 0 if (-not $proc.HasExited) { try { $proc.Refresh() $h = $proc.MainWindowHandle if ($h -ne [IntPtr]::Zero) { [ChurnSoakSurface]::ShowWindow($h, 9) | Out-Null [ChurnSoakSurface]::SetForegroundWindow($h) | Out-Null Start-Sleep -Milliseconds 1200 $r = New-Object ChurnSoakSurface+RECT [ChurnSoakSurface]::GetClientRect($h, [ref]$r) | Out-Null $p = New-Object ChurnSoakSurface+POINT [ChurnSoakSurface]::ClientToScreen($h, [ref]$p) | Out-Null $w = $r.R - $r.L; $ht = $r.B - $r.T if ($w -gt 0 -and $ht -gt 0) { $bmp = New-Object System.Drawing.Bitmap $w, $ht $g = [System.Drawing.Graphics]::FromImage($bmp) $g.CopyFromScreen($p.X, $p.Y, 0, 0, (New-Object System.Drawing.Size $w, $ht)) $bmp.Save($grab, [System.Drawing.Imaging.ImageFormat]::Png) $g.Dispose(); $bmp.Dispose() $grabBytes = (Get-Item $grab).Length } } } catch { Write-Step "desktop grab failed on ${Backend}: $_" } } # The peak matters as much as the end value: #257 reported a number reached, # not a number held. Take a final sample before the graceful close. try { $proc.Refresh() $samples.Add([pscustomobject][ordered]@{ UtcTicks = [DateTime]::UtcNow.Ticks WorkingSetBytes = $proc.WorkingSet64 PrivateBytes = $proc.PrivateMemorySize64 HandleCount = $proc.HandleCount ThreadCount = $proc.Threads.Count }) } catch { } $graceful = $false $exitCode = $null $app = Get-Process -Name AcDream.App -ErrorAction SilentlyContinue if ($app) { $app.CloseMainWindow() | Out-Null if ($app.WaitForExit(45000)) { $graceful = $true; $exitCode = $app.ExitCode } else { Write-Step "WM_CLOSE timed out on $Backend; forcing" $app | Stop-Process -Force } } elseif ($proc.HasExited) { $exitCode = $proc.ExitCode } foreach ($name in @('ACDREAM_RENDER_BACKEND', 'ACDREAM_MSAA_SAMPLES', 'ACDREAM_SKY_PHASE_SECONDS', 'ACDREAM_WORLD_TIME', 'ACDREAM_DAY_GROUP')) { Remove-Item "Env:\$name" -ErrorAction SilentlyContinue } $samples | Export-Csv -NoTypeInformation -Encoding utf8 -LiteralPath (Join-Path $dir 'memory-samples.csv') $checkpoints = @() if (Test-Path -LiteralPath $timeline) { $checkpoints = @(Get-Content -LiteralPath $timeline | Where-Object { $_.Trim().Length -gt 0 } | ForEach-Object { $_ | ConvertFrom-Json }) } $rows = [System.Collections.Generic.List[object]]::new() foreach ($cp in $checkpoints) { $ticks = ([DateTime]::Parse( $cp.timestampUtc, [System.Globalization.CultureInfo]::InvariantCulture, [System.Globalization.DateTimeStyles]::AdjustToUniversal -bor [System.Globalization.DateTimeStyles]::AssumeUniversal)).Ticks $nearest = $null $bestDelta = [long]::MaxValue foreach ($s in $samples) { $delta = [Math]::Abs($s.UtcTicks - $ticks) if ($delta -lt $bestDelta) { $bestDelta = $delta; $nearest = $s } } $r = $cp.resources $rows.Add([pscustomobject][ordered]@{ Sequence = $cp.sequence Name = $cp.name Cycle = if ($cp.name -match '^c(\d+)_') { [int]$Matches[1] } else { -1 } Stop = if ($cp.name -match '^c\d+_(.+)$') { $Matches[1] } else { $cp.name } WorkingSetMiB = if ($nearest) { [Math]::Round($nearest.WorkingSetBytes / 1MB, 1) } else { $null } PrivateMiB = if ($nearest) { [Math]::Round($nearest.PrivateBytes / 1MB, 1) } else { $null } Handles = if ($nearest) { $nearest.HandleCount } else { $null } SampleSkewMs = if ($nearest) { [Math]::Round($bestDelta / 10000.0, 0) } else { $null } LiveEntities = $r.liveEntities Materialized = $r.materializedLiveEntities WorldEntities = $r.worldEntities LoadedLandblocks = $r.loadedLandblocks VisibleLandblocks = $r.visibleLandblocks MeshRenderData = $r.meshRenderData MeshMiB = [Math]::Round($r.meshEstimatedBytes / 1MB, 1) GpuMiB = [Math]::Round($r.trackedGpuBytes / 1MB, 1) ResidencyGpuMiB = [Math]::Round($r.residencyGpuBytes / 1MB, 1) GpuMinusResidencyMiB = [Math]::Round($r.gpuTrackerMinusResidencyBytes / 1MB, 1) GpuBuffers = $r.trackedGpuBuffers GpuTextures = $r.trackedGpuTextures CompositeTextures = $r.ownedCompositeTextures CompositeOwners = $r.compositeTextureOwners ParticleTextures = $r.activeParticleTextures ManagedMiB = [Math]::Round($r.managedBytes / 1MB, 1) CommittedMiB = [Math]::Round($r.managedCommittedBytes / 1MB, 1) LohMiB = [Math]::Round($r.lohSizeBytes / 1MB, 1) PendingTeardowns = $r.pendingLiveTeardowns PendingRetirements = $r.pendingLandblockRetirements StagedMeshUploads = $r.stagedMeshUploads Fps = [Math]::Round($r.fps, 1) }) } $rows | Export-Csv -NoTypeInformation -Encoding utf8 -LiteralPath (Join-Path $dir 'per-checkpoint.csv') $witness = if ($grabBytes -ge $MinRenderedBytes) { 'RENDERED' } elseif ($grabBytes -gt 0) { 'BLANK' } else { 'NO-CAPTURE' } $errors = @(Select-String -Path $log -Pattern 'Unhandled exception|AccessViolation|OutOfMemoryException|event=invariant-failure|device removed|screenshot-failed' ` -ErrorAction SilentlyContinue | Select-Object -First 10 | ForEach-Object { $_.Line }) return [pscustomobject][ordered]@{ Backend = $Backend RouteCompleted = $completed Aborted = $aborted Graceful = $graceful ExitCode = $exitCode Witness = $witness GrabBytes = $grabBytes Checkpoints = $rows.Count Rows = $rows Samples = $samples.Count PeakWorkingSetMiB = if ($samples.Count -gt 0) { [Math]::Round((($samples | Measure-Object WorkingSetBytes -Maximum).Maximum) / 1MB, 1) } else { $null } PeakPrivateMiB = if ($samples.Count -gt 0) { [Math]::Round((($samples | Measure-Object PrivateBytes -Maximum).Maximum) / 1MB, 1) } else { $null } FinalWorkingSetMiB = if ($samples.Count -gt 0) { [Math]::Round($samples[$samples.Count - 1].WorkingSetBytes / 1MB, 1) } else { $null } ErrorLines = $errors Dir = $dir ScreenshotDir = Join-Path $dir 'screenshots' Log = $log } } $arms = @() foreach ($backend in $Backends) { if ($arms.Count -gt 0) { Write-Step "cooldown ${CooldownSeconds}s so ACE clears the graceful logout" Start-Sleep -Seconds $CooldownSeconds } $arms += Invoke-Arm $backend } # --- Within-arm capture comparison ------------------------------------------- # The #256 half. First Holtburg capture against the last, inside one arm. $maskPath = $null $comparisons = @() foreach ($arm in $arms) { $shots = @(Get-ChildItem $arm.ScreenshotDir -Filter 'holtburg_c*.png' -ErrorAction SilentlyContinue | Sort-Object Name) if ($shots.Count -lt 2) { $comparisons += [pscustomobject][ordered]@{ Backend = $arm.Backend; Pair = 'n/a'; Verdict = 'UNPAIRED' DifferentPixels = $null; Fraction = $null; MaxChannelDelta = $null } continue } if ($MaskTopPixels -gt 0 -and -not $maskPath) { $probe = [System.Drawing.Bitmap]::FromFile($shots[0].FullName) $mw = $probe.Width; $mh = $probe.Height $probe.Dispose() $mask = New-Object System.Drawing.Bitmap($mw, $mh, [System.Drawing.Imaging.PixelFormat]::Format32bppArgb) $graphics = [System.Drawing.Graphics]::FromImage($mask) $graphics.Clear([System.Drawing.Color]::FromArgb(0, 0, 0, 0)) $opaque = New-Object System.Drawing.SolidBrush ([System.Drawing.Color]::FromArgb(255, 255, 0, 255)) $graphics.FillRectangle($opaque, 0, 0, $mw, [Math]::Min($MaskTopPixels, $mh)) $graphics.Dispose(); $opaque.Dispose() $maskPath = Join-Path $Out 'mask.png' $mask.Save($maskPath, [System.Drawing.Imaging.ImageFormat]::Png) $mask.Dispose() } $first = $shots[0] foreach ($later in $shots[1..($shots.Count - 1)]) { $report = Join-Path $Out "compare-$($arm.Backend)-$($first.BaseName)-vs-$($later.BaseName).json" if ($maskPath) { & dotnet $cli compare-screenshots $first.FullName $later.FullName $report $Tolerance $MaxDifferentFraction $maskPath | Out-Null } else { & dotnet $cli compare-screenshots $first.FullName $later.FullName $report $Tolerance $MaxDifferentFraction | Out-Null } $verdict = Get-Content $report -Raw | ConvertFrom-Json # A number cannot tell an absent object from a walking NPC. The map and # the row histogram can: a vanished sign is a contiguous blob in one # band, a creature is a thin moving silhouette, and phase noise is # scattered. Both are written beside the report. $mapPath = Join-Path $Out "diff-$($arm.Backend)-$($first.BaseName)-vs-$($later.BaseName).png" $bands = New-Object 'int[]' 18 try { $bands = [ChurnDiffMap]::Write($first.FullName, $later.FullName, $mapPath, $Tolerance, $MaskTopPixels) } catch { Write-Step "diff map failed for $($later.BaseName): $_" } $comparisons += [pscustomobject][ordered]@{ Backend = $arm.Backend Pair = "$($first.BaseName) vs $($later.BaseName)" Verdict = if ($verdict.passed) { 'MATCH' } else { 'DIVERGED' } DifferentPixels = $verdict.differentPixels Fraction = $verdict.differentPixelFraction MaxChannelDelta = $verdict.maximumChannelDelta RowBands40px = ($bands -join ',') DiffMap = $mapPath } } } # --- Verdict ------------------------------------------------------------------ function Get-StopTrend($rows, [string]$Stop, [string]$Column) { $series = @($rows | Where-Object { $_.Stop -eq $Stop -and $_.Cycle -ge 1 } | Sort-Object Cycle) if ($series.Count -lt 2) { return $null } $warm = if ($series.Count -ge 6) { $series[2..($series.Count - 1)] } else { $series } $firstValue = [double]$warm[0].$Column $lastValue = [double]$warm[-1].$Column $spanCycles = [double]($warm[-1].Cycle - $warm[0].Cycle) return [pscustomobject][ordered]@{ Stop = $Stop Column = $Column FirstCycle = $warm[0].Cycle LastCycle = $warm[-1].Cycle First = $firstValue Last = $lastValue Delta = [Math]::Round($lastValue - $firstValue, 1) PerCycle = if ($spanCycles -gt 0) { [Math]::Round(($lastValue - $firstValue) / $spanCycles, 2) } else { $null } } } $trends = @() foreach ($arm in $arms) { foreach ($stop in @('holtburg', 'hub', 'aerlinthe')) { foreach ($column in @('WorkingSetMiB', 'PrivateMiB', 'GpuMiB', 'ManagedMiB', 'Materialized', 'WorldEntities', 'MeshRenderData', 'CompositeTextures', 'GpuTextures', 'Handles')) { $t = Get-StopTrend $arm.Rows $stop $column if ($t) { $trends += [pscustomobject][ordered]@{ Backend = $arm.Backend; Stop = $t.Stop; Column = $t.Column FirstCycle = $t.FirstCycle; LastCycle = $t.LastCycle First = $t.First; Last = $t.Last; Delta = $t.Delta; PerCycle = $t.PerCycle } } } } } Write-Host '' foreach ($arm in $arms) { if ($arm.Aborted) { Write-Host "!!! $($arm.Backend) aborted: $($arm.Aborted)" -ForegroundColor Yellow } Write-Host "=== $($arm.Backend) : completed=$($arm.RouteCompleted) graceful=$($arm.Graceful) exit=$($arm.ExitCode) witness=$($arm.Witness) checkpoints=$($arm.Checkpoints) peakWS=$($arm.PeakWorkingSetMiB) MiB finalWS=$($arm.FinalWorkingSetMiB) MiB" $arm.Rows | Where-Object { $_.Stop -eq 'holtburg' } | Select-Object Cycle, WorkingSetMiB, PrivateMiB, GpuMiB, ManagedMiB, LiveEntities, Materialized, WorldEntities, MeshRenderData, CompositeTextures, GpuTextures, Handles | Format-Table -AutoSize | Out-String | Write-Host } Write-Host '=== within-arm capture comparison (the #256 half) ===' $comparisons | Format-Table -AutoSize | Out-String | Write-Host Write-Host '=== warm trends (cycle 3 -> last) ===' $trends | Where-Object { $_.Stop -eq 'holtburg' } | Format-Table -AutoSize | Out-String | Write-Host $report = [pscustomobject][ordered]@{ Commit = (& git -C $repo rev-parse HEAD).Trim() GeneratedUtc = [DateTime]::UtcNow.ToString('o') Cycles = $Cycles Transits = $transit Stops = @($stops | ForEach-Object { $_.Key }) ShotCycles = @($ShotCycles) Route = $routePath DayGroup = $DayGroup WorldDayFraction = $WorldDayFraction SkyPhaseSeconds = $SkyPhaseSeconds MsaaSamples = 0 MaskTopPixels = $MaskTopPixels Arms = @($arms | ForEach-Object { [pscustomobject][ordered]@{ Backend = $_.Backend; RouteCompleted = $_.RouteCompleted; Aborted = $_.Aborted Graceful = $_.Graceful ExitCode = $_.ExitCode; Witness = $_.Witness; GrabBytes = $_.GrabBytes Checkpoints = $_.Checkpoints; Samples = $_.Samples PeakWorkingSetMiB = $_.PeakWorkingSetMiB; PeakPrivateMiB = $_.PeakPrivateMiB FinalWorkingSetMiB = $_.FinalWorkingSetMiB; ErrorLines = $_.ErrorLines Rows = $_.Rows } }) Comparisons = @($comparisons) Trends = @($trends) } $reportPath = Join-Path $Out 'churn-soak.json' $report | ConvertTo-Json -Depth 8 | Set-Content -LiteralPath $reportPath -Encoding utf8 Write-Step "report: $reportPath"