diff --git a/docs/ISSUES.md b/docs/ISSUES.md index 6e656fb9..9a9dee1b 100644 --- a/docs/ISSUES.md +++ b/docs/ISSUES.md @@ -211,6 +211,50 @@ question costs far more to answer. sign/portal visible; a soak instrument counts published-vs-live entities per transit and holds at zero drift. +### Discriminator result (2026-07-28, V11 step 0) — NEGATIVE ON BOTH ARMS + +The discriminator this issue demanded before V11 was built and run: +`tools/run-portal-churn-soak.ps1`, 30 cycles x 3 portal-bearing stops +(Holtburg town / Facility Hub interior / Aerlinthe island, all taken from the +existing connected routes, every teleloc carrying the identity quaternion so the +heading repeats), **90 transits per arm**, once on `gl` and once on `vulkan` +from the same binary at `122fe8a7`. Both arms: 91 checkpoints, **zero** error +lines, graceful exit, desktop witness `RENDERED`. + +The pixel half is a *within-arm* comparison — the Holtburg capture at cycle 1 +against cycles 10, 20 and 30 from the same process at a pinned viewpoint with +MSAA off and all four determinism levers forced. An object that stopped being +drawn would appear as a contiguous blob in the difference map. + +| Arm | c01 vs c10 | c01 vs c20 | c01 vs c30 | +|---|---|---|---| +| gl | 12,366 px | 14,492 px | 17,082 px | +| vulkan | 12,774 px | 15,234 px | 16,183 px | + +**The two arms are the same to within noise, and the difference maps show +nothing missing.** Every building, the portal cone, the statue, the distant +treeline, the scenery and the NPC render identically at cycle 30 and cycle 1 on +both backends. What differs is: the player avatar's animation phase, the +portal's own scrolling texture, one corpse, the stamina readout — plus, on GL +only, a scatter of single-pixel alpha fringes across the ground that Vulkan does +not produce. Publication counters corroborate: `worldEntities` is **10,382 at +every one of the 30 cycles on both arms**, `ownedCompositeTextures` is 471, +`trackedGpuTextures` is 437 (GL) / 372 (VK) — all three constant from warmup to +the last transit, with zero drift. + +**Verdict: the "Vulkan-arm-only lifecycle bug" hypothesis is refuted for this +workload, so V11 is not blocked by this issue.** But neither arm *reproduced* +the symptom, so this does not identify the pre-existing bug either — it rules +out the one outcome that would have made deleting GL wrong. + +**This issue stays OPEN.** The next attempt must differ from this route the way +the user's session did: real portal *use* (walk into a portal, F751 wormhole, +materialization) rather than `/teleloc`, and Coldeve's town portal network +rather than these three stops. `/teleloc` and a walked portal do not take the +same path into the transit state machine, and that difference is now the leading +suspect. Artifacts: `artifacts/v11-churn/` (per-checkpoint CSVs, memory sample +CSVs, difference maps, `churn-soak.json`). + --- ## #257 — Working set balloons to ~1.5 GB over a live portal-churn session @@ -240,6 +284,52 @@ are the model to hold this to. **Acceptance:** a repeated-portal soak (≥30 transits) holds working set flat after warmup, with per-category GPU accounting drift at zero. +### Discriminator result (2026-07-28, V11 step 0) — NEGATIVE ON BOTH ARMS + +Same run as #256's: `tools/run-portal-churn-soak.ps1`, 30 cycles x 3 stops = +**90 transits per arm**, `gl` then `vulkan` from one binary at `122fe8a7`, with +the OS sampling working set and private bytes every 2 s (547 / 548 samples per +arm) and each checkpoint joined to its nearest sample by timestamp. + +Holtburg, every fifth cycle: + +| Cycle | GL WS | GL priv | GL GPU | VK WS | VK priv | VK GPU | +|---|---|---|---|---|---|---| +| 1 | 1878.1 | 1949.6 | 321.5 | 1893.3 | 1848.7 | 58.6 | +| 5 | 1728.6 | 1783.3 | 332.3 | 1905.7 | 1843.7 | 58.6 | +| 10 | 1902.7 | 1963.6 | 332.3 | 1928.5 | 1867.6 | 58.6 | +| 15 | 1911.7 | 1972.9 | 332.3 | 1933.8 | 1870.6 | 58.6 | +| 20 | 1976.1 | 2042.5 | 332.3 | 1757.8 | 1693.7 | 58.6 | +| 25 | 1874.3 | 1940.7 | 332.3 | 1875.4 | 1812.0 | 58.6 | +| 30 | 1920.4 | 1994.5 | 332.3 | 1774.5 | 1709.1 | 58.6 | + +(MiB. The GL/VK GPU-byte levels are not comparable to each other — the two +backends' `GpuMemoryTracker` coverage differs — but each is comparable to +itself, and **each is exactly constant**: GL pins at 332.3 MiB from cycle 2 +onward and never moves again; Vulkan sits at 58.6 MiB for all 30 cycles.) + +Whole-run statistics over every OS sample: + +| Arm | min | mean | max | peak private | warm 1st-half mean | warm 2nd-half mean | drift | +|---|---|---|---|---|---|---|---| +| gl | 1713 | **1864** | 2210 | 2282 | 1927 | 1880 | **-48.0** | +| vulkan | 1734 | **1863** | 2173 | 2112 | 1894 | 1867 | **-27.0** | + +**Neither arm grows.** The warm-half drift is *negative* on both, the two means +agree to 1 MiB, and handle counts are flat (GL +1, VK -4 over 30 cycles). + +**Verdict: not reproduced, on either backend, so V11 is not blocked.** The +~1.9 GB band is a *level*, not a leak: this route holds three very different +landblocks' content live, which is a different residency profile from V8's +stationary 877 MiB (VK) / 944 MiB (GL) vehicles. The 1.5 GB the user saw is +inside this band, so the reported number may simply be what a multi-region +session costs — but that is a hypothesis this run cannot confirm, because it +did not reproduce unbounded growth to explain. + +**This issue stays OPEN**, coupled to #256 and blocked on the same follow-up: +re-run with walked portal transits rather than `/teleloc`. Artifacts: +`artifacts/v11-churn/`. + --- ## #253 — Attribute/skill icons: not centered in their cells, and fully opaque diff --git a/tools/run-portal-churn-soak.ps1 b/tools/run-portal-churn-soak.ps1 new file mode 100644 index 00000000..d74027a9 --- /dev/null +++ b/tools/run-portal-churn-soak.ps1 @@ -0,0 +1,611 @@ +<# +.SYNOPSIS + Campaign V slice V11 step 0: the #256/#257 discriminator, run on both backends. + +.DESCRIPTION + Issues #256 (server-spawned signs and portals go invisible after repeated + portal runs, while staying interactive) and #257 (working set grows to + ~1.5 GB over the same session) were both observed on the Vulkan backend in + one long live session. Both filings demand the SAME discriminator, and both + demand it BEFORE V11 deletes the OpenGL backend: + + run the same repeated-churn route on GL and on Vulkan. + + growth / drift on BOTH arms -> a pre-existing publication or resource + lifetime bug that the campaign merely + witnessed; + growth / drift on VULKAN ONLY -> the new arm's resource lifecycle, and + deleting its predecessor would destroy + the only reference implementation that + can answer the question. + + After V11 the GL arm does not exist and this question costs far more to + answer, which is why this script runs first and why its verdict can stop the + slice. + + WHAT IT MEASURES, per stop, per cycle: + + * working set and private bytes, sampled from the OS every -SampleSeconds + and joined to each checkpoint by timestamp. This is #257's quantity: the + client's own snapshot has no view of its own working set. + * the published-versus-live entity pair from the checkpoint JSON -- + liveEntities / materializedLiveEntities against worldEntities and + meshRenderData. #256's symptom is "alive in the object table, gone from + the presentation", so a drift between those two halves at the SAME stop + across cycles is the instrument. + * the GPU accounting the checkpoint already carries: trackedGpuBytes, + residencyGpuBytes, their difference, tracked buffer and texture counts, + owned composite textures and their owners. GpuMemoryTracker's categories + reach the checkpoint as those counts. + * managed bytes, committed bytes, LOH size, and the streaming/retirement + backlogs, so a growth finding can be attributed to a heap rather than + left as "the process is bigger". + + HOW THE ROUTE IS BUILT. -Cycles repetitions of three stops taken from the + two existing connected routes: Holtburg town (the dense outdoor stop the + differential route calls its smoke stop, and the one that has a portal in + frame), the Facility Hub interior EnvCell, and the Aerlinthe island. Every + teleloc carries the identity quaternion so the heading at a stop is the same + on every visit, which is what makes cycle 1 and cycle N comparable at all. + Each stop writes a checkpoint; Holtburg additionally captures a screenshot on + the cycles named by -ShotCycles. + + THE PIXEL HALF. Within one arm, the Holtburg capture from cycle 1 and the + capture from the last shot cycle are compared against each other. Same + process, same position, same heading, same pinned clock: an object that has + stopped being drawn shows up as a contiguous mid-frame difference. This is + the closest automated instrument to what the user actually saw. The four + determinism levers the differential gate forces are forced here for the same + reason -- an unpinned sun or cloud sheet would swamp the signal. + + NOT A PIXEL GATE. This script never compares the two backends to each other; + that is run-backend-differential-gate.ps1's job and its thresholds are the + campaign's. Here each arm is compared only with itself, and the verdict is + about growth and drift. + +.PARAMETER Out + Directory for both arms' logs, checkpoints, captures, samples and the verdict. + +.PARAMETER Cycles + Repetitions of the three-stop loop. Default 30 -- #257's acceptance criterion + is "a repeated-portal soak (>= 30 transits) holds working set flat after + warmup", and three stops per cycle makes 90 transits at the default. + +.PARAMETER Backends + Which arms to run, in order. Default gl then vulkan, so the arm under + suspicion runs second and cannot be blamed for warming the machine. + +.PARAMETER ShotCycles + Cycles on which Holtburg is captured. Default 1, 10, 20 and the last cycle. + +.PARAMETER SettleMilliseconds + Stationary window between arrival and the checkpoint. Default 6000. + +.EXAMPLE + tools/run-portal-churn-soak.ps1 -Out artifacts/v11-churn + tools/run-portal-churn-soak.ps1 -Out artifacts/v11-churn-short -Cycles 4 -SkipBuild +#> +[CmdletBinding()] +param( + [Parameter(Mandatory = $true)][string]$Out, + [int]$Cycles = 30, + [string[]]$Backends = @('gl', 'vulkan'), + [int[]]$ShotCycles, + [int]$SettleMilliseconds = 6000, + [int]$ShotSettleMilliseconds = 12000, + [int]$SampleSeconds = 2, + [int]$DayGroup = 0, + [double]$WorldDayFraction = 0.5, + [double]$SkyPhaseSeconds = 0, + [int]$MaskTopPixels = 280, + [int]$Tolerance = 2, + [double]$MaxDifferentFraction = 0.001, + [int]$MinRenderedBytes = 500000, + # A soak whose subject is unbounded growth must not be allowed to take the + # machine down with it. Crossing this stops the arm early and keeps every + # sample and checkpoint taken so far -- which is the finding, not a failure. + [int]$AbortWorkingSetMiB = 10000, + [int]$CooldownSeconds = 20, + [int]$RouteTimeoutSeconds = 2400, + [switch]$SkipBuild +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest +$repo = Split-Path -Parent $PSScriptRoot +$exe = Join-Path $repo 'src\AcDream.App\bin\Release\net10.0\AcDream.App.exe' +$cli = Join-Path $repo 'src\AcDream.Cli\bin\Release\net10.0\AcDream.Cli.dll' + +function Write-Step($message) { Write-Host "[churn] $message" } + +if (-not $ShotCycles) { + $ShotCycles = @(1, 10, 20, $Cycles) | Where-Object { $_ -le $Cycles } | Sort-Object -Unique +} + +# --- Preconditions ------------------------------------------------------------ +if (Get-Process -Name AcDream.App -ErrorAction SilentlyContinue) { + throw 'AcDream.App is already running. This soak uses the shared test account and must not steal its session.' +} +if (@(Get-NetUDPEndpoint -LocalPort 9000 -ErrorAction SilentlyContinue).Count -eq 0) { + throw 'local ACE is not listening on UDP port 9000.' +} +if (-not $SkipBuild) { + Write-Step 'building Release' + & dotnet build (Join-Path $repo 'AcDream.slnx') -c Release --nologo -v q | Out-Null + if ($LASTEXITCODE -ne 0) { throw "Release build failed with exit code $LASTEXITCODE." } +} +if (-not (Test-Path $exe)) { throw "Client not found at $exe." } +if (-not (Test-Path $cli)) { throw "AcDream.Cli not found at $cli." } + +if (Test-Path $Out) { Remove-Item -Recurse -Force $Out } +New-Item -ItemType Directory -Force -Path $Out | Out-Null + +# --- The route ---------------------------------------------------------------- +# Three stops per cycle, all three taken from routes that already exist. The +# identity quaternion on every teleloc is what pins the heading across visits. +$stops = @( + [pscustomobject]@{ Key = 'holtburg'; Teleloc = '0xA9B40019 84.0 7.1 94.005 1 0 0 0' } + [pscustomobject]@{ Key = 'hub'; Teleloc = '0x8A020164 70.35 -40.66 -5.9 1 0 0 0' } + [pscustomobject]@{ Key = 'aerlinthe'; Teleloc = '0x09040008 11.4 188.6 87.705 1 0 0 0' } +) + +$lines = [System.Collections.Generic.List[string]]::new() +$lines.Add('# GENERATED by tools/run-portal-churn-soak.ps1 -- the V11 step-0 discriminator.') +$lines.Add("# $Cycles cycles x $($stops.Count) stops. Do not hand-edit; regenerate.") +$lines.Add('') +$lines.Add('wait world-ready 90000') +$lines.Add('wait world-visible 30000') +$lines.Add('sleep 15000') +$lines.Add('checkpoint c00_login') +$lines.Add('') + +$transit = 0 +foreach ($cycle in 1..$Cycles) { + $tag = '{0:d2}' -f $cycle + $shot = $ShotCycles -contains $cycle + foreach ($stop in $stops) { + $transit++ + $settle = if ($shot -and $stop.Key -eq 'holtburg') { $ShotSettleMilliseconds } else { $SettleMilliseconds } + $lines.Add("command /teleloc $($stop.Teleloc)") + $lines.Add("wait materialized $transit 90000") + $lines.Add('wait world-visible 30000') + $lines.Add("sleep $settle") + $lines.Add("checkpoint c${tag}_$($stop.Key)") + if ($shot -and $stop.Key -eq 'holtburg') { + $lines.Add("screenshot holtburg_c$tag 15000") + } + } + $lines.Add('') +} +$lines.Add('sleep 5000') + +$routePath = Join-Path $Out 'route.txt' +Set-Content -Encoding utf8 -LiteralPath $routePath -Value $lines +Write-Step "route: $Cycles cycles, $transit transits, $($lines.Count) lines -> $routePath" + +Add-Type -AssemblyName System.Drawing +Add-Type @' +using System; +using System.Runtime.InteropServices; +public static class ChurnSoakSurface { + [DllImport("user32.dll")] public static extern bool SetForegroundWindow(IntPtr hWnd); + [DllImport("user32.dll")] public static extern bool ShowWindow(IntPtr hWnd, int nCmdShow); + [DllImport("user32.dll")] public static extern bool GetClientRect(IntPtr hWnd, out RECT r); + [DllImport("user32.dll")] public static extern bool ClientToScreen(IntPtr hWnd, ref POINT p); + [StructLayout(LayoutKind.Sequential)] public struct RECT { public int L, T, R, B; } + [StructLayout(LayoutKind.Sequential)] public struct POINT { public int X, Y; } +} +'@ + +# A number cannot tell an absent object from a walking NPC. The map and the row +# histogram can: a vanished sign is a contiguous blob in one band, a creature is +# a thin moving silhouette, and phase noise is scattered. +Add-Type -ReferencedAssemblies System.Drawing @' +using System; +using System.Drawing; +using System.Drawing.Imaging; +using System.Runtime.InteropServices; +public static class ChurnDiffMap { + public static int[] Write(string expected, string actual, string outPath, int tolerance, int maskTop) { + int[] bands = new int[18]; + using (Bitmap a = new Bitmap(expected)) + using (Bitmap b = new Bitmap(actual)) + using (Bitmap map = new Bitmap(a.Width, a.Height, PixelFormat.Format32bppArgb)) { + Rectangle rect = new Rectangle(0, 0, a.Width, a.Height); + BitmapData da = a.LockBits(rect, ImageLockMode.ReadOnly, PixelFormat.Format32bppArgb); + BitmapData db = b.LockBits(rect, ImageLockMode.ReadOnly, PixelFormat.Format32bppArgb); + BitmapData dm = map.LockBits(rect, ImageLockMode.WriteOnly, PixelFormat.Format32bppArgb); + int count = a.Width * a.Height; + int[] pa = new int[count], pb = new int[count], pm = new int[count]; + Marshal.Copy(da.Scan0, pa, 0, count); + Marshal.Copy(db.Scan0, pb, 0, count); + for (int y = 0; y < a.Height; y++) { + for (int x = 0; x < a.Width; x++) { + int i = y * a.Width + x; + int ca = pa[i], cb = pb[i]; + int d = Math.Max(Math.Abs(((ca >> 16) & 255) - ((cb >> 16) & 255)), + Math.Max(Math.Abs(((ca >> 8) & 255) - ((cb >> 8) & 255)), + Math.Abs((ca & 255) - (cb & 255)))); + if (y >= maskTop && d > tolerance) { + pm[i] = unchecked((int)0xFFFF00FF); + bands[Math.Min(17, y / 40)]++; + } else { + int grey = (int)((((cb >> 16) & 255) * 0.3 + ((cb >> 8) & 255) * 0.59 + (cb & 255) * 0.11) / 4); + pm[i] = unchecked((int)0xFF000000) | (grey << 16) | (grey << 8) | grey; + } + } + } + Marshal.Copy(pm, 0, dm.Scan0, count); + a.UnlockBits(da); b.UnlockBits(db); map.UnlockBits(dm); + map.Save(outPath, ImageFormat.Png); + } + return bands; + } +} +'@ + +function Invoke-Arm([string]$Backend) { + $dir = Join-Path $Out $Backend + New-Item -ItemType Directory -Force -Path $dir | Out-Null + $log = Join-Path $dir 'client.log' + $grab = Join-Path $dir 'desktop-grab.png' + $timeline = Join-Path $dir 'world-lifecycle.checkpoints.jsonl' + + $env:ACDREAM_DAT_DIR = Join-Path $env:USERPROFILE "Documents\Asheron's Call" + $env:ACDREAM_LIVE = '1' + $env:ACDREAM_TEST_HOST = '127.0.0.1' + $env:ACDREAM_TEST_PORT = '9000' + $env:ACDREAM_TEST_USER = 'testaccount' + $env:ACDREAM_TEST_PASS = 'testpassword' + $env:ACDREAM_RETAIL_UI = '1' + $env:ACDREAM_DEVTOOLS = '0' + $env:ACDREAM_UI_PROBE_DUMP = '0' + $env:ACDREAM_FRAME_PROF = '1' + # The four determinism levers, forced. Without them a cycle-1-versus-cycle-N + # capture measures the sun and the cloud sheet instead of the world. + $env:ACDREAM_MSAA_SAMPLES = '0' + $env:ACDREAM_DAY_GROUP = "$DayGroup" + $env:ACDREAM_WORLD_TIME = + $WorldDayFraction.ToString([System.Globalization.CultureInfo]::InvariantCulture) + $env:ACDREAM_SKY_PHASE_SECONDS = + $SkyPhaseSeconds.ToString([System.Globalization.CultureInfo]::InvariantCulture) + $env:ACDREAM_RENDER_BACKEND = $Backend + $env:ACDREAM_UI_PROBE_SCRIPT = $routePath + $env:ACDREAM_AUTOMATION_ARTIFACT_DIR = $dir + + Write-Step "launching $Backend" + $proc = Start-Process -FilePath $exe -WorkingDirectory $repo ` + -RedirectStandardOutput $log -RedirectStandardError "$log.err" -PassThru + + $samples = [System.Collections.Generic.List[object]]::new() + $completed = $false + $aborted = $null + $deadline = (Get-Date).AddSeconds($RouteTimeoutSeconds) + $lastReport = Get-Date + while ((Get-Date) -lt $deadline) { + if (Select-String -Path $log -SimpleMatch '[UI-PROBE] UI probe script complete' ` + -ErrorAction SilentlyContinue) { $completed = $true; break } + if ($proc.HasExited) { break } + try { + $proc.Refresh() + $samples.Add([pscustomobject][ordered]@{ + UtcTicks = [DateTime]::UtcNow.Ticks + WorkingSetBytes = $proc.WorkingSet64 + PrivateBytes = $proc.PrivateMemorySize64 + HandleCount = $proc.HandleCount + ThreadCount = $proc.Threads.Count + }) + if ($proc.WorkingSet64 / 1MB -gt $AbortWorkingSetMiB) { + $aborted = "working set crossed $AbortWorkingSetMiB MiB at sample $($samples.Count)" + Write-Step "$Backend : ABORTING -- $aborted" + break + } + } + catch { } + if (((Get-Date) - $lastReport).TotalSeconds -ge 60) { + $lastReport = Get-Date + $done = 0 + if (Test-Path -LiteralPath $timeline) { + $done = @(Get-Content -LiteralPath $timeline -ErrorAction SilentlyContinue).Count + } + $ws = if ($samples.Count -gt 0) { [Math]::Round($samples[$samples.Count - 1].WorkingSetBytes / 1MB, 0) } else { 0 } + Write-Step "$Backend : $done checkpoints, working set ${ws} MiB" + } + Start-Sleep -Seconds $SampleSeconds + } + + $grabBytes = 0 + if (-not $proc.HasExited) { + try { + $proc.Refresh() + $h = $proc.MainWindowHandle + if ($h -ne [IntPtr]::Zero) { + [ChurnSoakSurface]::ShowWindow($h, 9) | Out-Null + [ChurnSoakSurface]::SetForegroundWindow($h) | Out-Null + Start-Sleep -Milliseconds 1200 + $r = New-Object ChurnSoakSurface+RECT + [ChurnSoakSurface]::GetClientRect($h, [ref]$r) | Out-Null + $p = New-Object ChurnSoakSurface+POINT + [ChurnSoakSurface]::ClientToScreen($h, [ref]$p) | Out-Null + $w = $r.R - $r.L; $ht = $r.B - $r.T + if ($w -gt 0 -and $ht -gt 0) { + $bmp = New-Object System.Drawing.Bitmap $w, $ht + $g = [System.Drawing.Graphics]::FromImage($bmp) + $g.CopyFromScreen($p.X, $p.Y, 0, 0, (New-Object System.Drawing.Size $w, $ht)) + $bmp.Save($grab, [System.Drawing.Imaging.ImageFormat]::Png) + $g.Dispose(); $bmp.Dispose() + $grabBytes = (Get-Item $grab).Length + } + } + } + catch { Write-Step "desktop grab failed on ${Backend}: $_" } + } + + # The peak matters as much as the end value: #257 reported a number reached, + # not a number held. Take a final sample before the graceful close. + try { + $proc.Refresh() + $samples.Add([pscustomobject][ordered]@{ + UtcTicks = [DateTime]::UtcNow.Ticks + WorkingSetBytes = $proc.WorkingSet64 + PrivateBytes = $proc.PrivateMemorySize64 + HandleCount = $proc.HandleCount + ThreadCount = $proc.Threads.Count + }) + } + catch { } + + $graceful = $false + $exitCode = $null + $app = Get-Process -Name AcDream.App -ErrorAction SilentlyContinue + if ($app) { + $app.CloseMainWindow() | Out-Null + if ($app.WaitForExit(45000)) { $graceful = $true; $exitCode = $app.ExitCode } + else { + Write-Step "WM_CLOSE timed out on $Backend; forcing" + $app | Stop-Process -Force + } + } + elseif ($proc.HasExited) { $exitCode = $proc.ExitCode } + + foreach ($name in @('ACDREAM_RENDER_BACKEND', 'ACDREAM_MSAA_SAMPLES', + 'ACDREAM_SKY_PHASE_SECONDS', 'ACDREAM_WORLD_TIME', + 'ACDREAM_DAY_GROUP')) { + Remove-Item "Env:\$name" -ErrorAction SilentlyContinue + } + + $samples | Export-Csv -NoTypeInformation -Encoding utf8 -LiteralPath (Join-Path $dir 'memory-samples.csv') + + $checkpoints = @() + if (Test-Path -LiteralPath $timeline) { + $checkpoints = @(Get-Content -LiteralPath $timeline | + Where-Object { $_.Trim().Length -gt 0 } | + ForEach-Object { $_ | ConvertFrom-Json }) + } + + $rows = [System.Collections.Generic.List[object]]::new() + foreach ($cp in $checkpoints) { + $ticks = ([DateTime]::Parse( + $cp.timestampUtc, + [System.Globalization.CultureInfo]::InvariantCulture, + [System.Globalization.DateTimeStyles]::AdjustToUniversal -bor + [System.Globalization.DateTimeStyles]::AssumeUniversal)).Ticks + $nearest = $null + $bestDelta = [long]::MaxValue + foreach ($s in $samples) { + $delta = [Math]::Abs($s.UtcTicks - $ticks) + if ($delta -lt $bestDelta) { $bestDelta = $delta; $nearest = $s } + } + $r = $cp.resources + $rows.Add([pscustomobject][ordered]@{ + Sequence = $cp.sequence + Name = $cp.name + Cycle = if ($cp.name -match '^c(\d+)_') { [int]$Matches[1] } else { -1 } + Stop = if ($cp.name -match '^c\d+_(.+)$') { $Matches[1] } else { $cp.name } + WorkingSetMiB = if ($nearest) { [Math]::Round($nearest.WorkingSetBytes / 1MB, 1) } else { $null } + PrivateMiB = if ($nearest) { [Math]::Round($nearest.PrivateBytes / 1MB, 1) } else { $null } + Handles = if ($nearest) { $nearest.HandleCount } else { $null } + SampleSkewMs = if ($nearest) { [Math]::Round($bestDelta / 10000.0, 0) } else { $null } + LiveEntities = $r.liveEntities + Materialized = $r.materializedLiveEntities + WorldEntities = $r.worldEntities + LoadedLandblocks = $r.loadedLandblocks + VisibleLandblocks = $r.visibleLandblocks + MeshRenderData = $r.meshRenderData + MeshMiB = [Math]::Round($r.meshEstimatedBytes / 1MB, 1) + GpuMiB = [Math]::Round($r.trackedGpuBytes / 1MB, 1) + ResidencyGpuMiB = [Math]::Round($r.residencyGpuBytes / 1MB, 1) + GpuMinusResidencyMiB = [Math]::Round($r.gpuTrackerMinusResidencyBytes / 1MB, 1) + GpuBuffers = $r.trackedGpuBuffers + GpuTextures = $r.trackedGpuTextures + CompositeTextures = $r.ownedCompositeTextures + CompositeOwners = $r.compositeTextureOwners + ParticleTextures = $r.activeParticleTextures + ManagedMiB = [Math]::Round($r.managedBytes / 1MB, 1) + CommittedMiB = [Math]::Round($r.managedCommittedBytes / 1MB, 1) + LohMiB = [Math]::Round($r.lohSizeBytes / 1MB, 1) + PendingTeardowns = $r.pendingLiveTeardowns + PendingRetirements = $r.pendingLandblockRetirements + StagedMeshUploads = $r.stagedMeshUploads + Fps = [Math]::Round($r.fps, 1) + }) + } + $rows | Export-Csv -NoTypeInformation -Encoding utf8 -LiteralPath (Join-Path $dir 'per-checkpoint.csv') + + $witness = if ($grabBytes -ge $MinRenderedBytes) { 'RENDERED' } + elseif ($grabBytes -gt 0) { 'BLANK' } + else { 'NO-CAPTURE' } + + $errors = @(Select-String -Path $log -Pattern 'Unhandled exception|AccessViolation|OutOfMemoryException|event=invariant-failure|device removed|screenshot-failed' ` + -ErrorAction SilentlyContinue | Select-Object -First 10 | ForEach-Object { $_.Line }) + + return [pscustomobject][ordered]@{ + Backend = $Backend + RouteCompleted = $completed + Aborted = $aborted + Graceful = $graceful + ExitCode = $exitCode + Witness = $witness + GrabBytes = $grabBytes + Checkpoints = $rows.Count + Rows = $rows + Samples = $samples.Count + PeakWorkingSetMiB = if ($samples.Count -gt 0) { [Math]::Round((($samples | Measure-Object WorkingSetBytes -Maximum).Maximum) / 1MB, 1) } else { $null } + PeakPrivateMiB = if ($samples.Count -gt 0) { [Math]::Round((($samples | Measure-Object PrivateBytes -Maximum).Maximum) / 1MB, 1) } else { $null } + FinalWorkingSetMiB = if ($samples.Count -gt 0) { [Math]::Round($samples[$samples.Count - 1].WorkingSetBytes / 1MB, 1) } else { $null } + ErrorLines = $errors + Dir = $dir + ScreenshotDir = Join-Path $dir 'screenshots' + Log = $log + } +} + +$arms = @() +foreach ($backend in $Backends) { + if ($arms.Count -gt 0) { + Write-Step "cooldown ${CooldownSeconds}s so ACE clears the graceful logout" + Start-Sleep -Seconds $CooldownSeconds + } + $arms += Invoke-Arm $backend +} + +# --- Within-arm capture comparison ------------------------------------------- +# The #256 half. First Holtburg capture against the last, inside one arm. +$maskPath = $null +$comparisons = @() +foreach ($arm in $arms) { + $shots = @(Get-ChildItem $arm.ScreenshotDir -Filter 'holtburg_c*.png' -ErrorAction SilentlyContinue | + Sort-Object Name) + if ($shots.Count -lt 2) { + $comparisons += [pscustomobject][ordered]@{ + Backend = $arm.Backend; Pair = 'n/a'; Verdict = 'UNPAIRED' + DifferentPixels = $null; Fraction = $null; MaxChannelDelta = $null + } + continue + } + if ($MaskTopPixels -gt 0 -and -not $maskPath) { + $probe = [System.Drawing.Bitmap]::FromFile($shots[0].FullName) + $mw = $probe.Width; $mh = $probe.Height + $probe.Dispose() + $mask = New-Object System.Drawing.Bitmap($mw, $mh, [System.Drawing.Imaging.PixelFormat]::Format32bppArgb) + $graphics = [System.Drawing.Graphics]::FromImage($mask) + $graphics.Clear([System.Drawing.Color]::FromArgb(0, 0, 0, 0)) + $opaque = New-Object System.Drawing.SolidBrush ([System.Drawing.Color]::FromArgb(255, 255, 0, 255)) + $graphics.FillRectangle($opaque, 0, 0, $mw, [Math]::Min($MaskTopPixels, $mh)) + $graphics.Dispose(); $opaque.Dispose() + $maskPath = Join-Path $Out 'mask.png' + $mask.Save($maskPath, [System.Drawing.Imaging.ImageFormat]::Png) + $mask.Dispose() + } + $first = $shots[0] + foreach ($later in $shots[1..($shots.Count - 1)]) { + $report = Join-Path $Out "compare-$($arm.Backend)-$($first.BaseName)-vs-$($later.BaseName).json" + if ($maskPath) { + & dotnet $cli compare-screenshots $first.FullName $later.FullName $report $Tolerance $MaxDifferentFraction $maskPath | Out-Null + } + else { + & dotnet $cli compare-screenshots $first.FullName $later.FullName $report $Tolerance $MaxDifferentFraction | Out-Null + } + $verdict = Get-Content $report -Raw | ConvertFrom-Json + # A number cannot tell an absent object from a walking NPC. The map and + # the row histogram can: a vanished sign is a contiguous blob in one + # band, a creature is a thin moving silhouette, and phase noise is + # scattered. Both are written beside the report. + $mapPath = Join-Path $Out "diff-$($arm.Backend)-$($first.BaseName)-vs-$($later.BaseName).png" + $bands = New-Object 'int[]' 18 + try { $bands = [ChurnDiffMap]::Write($first.FullName, $later.FullName, $mapPath, $Tolerance, $MaskTopPixels) } + catch { Write-Step "diff map failed for $($later.BaseName): $_" } + $comparisons += [pscustomobject][ordered]@{ + Backend = $arm.Backend + Pair = "$($first.BaseName) vs $($later.BaseName)" + Verdict = if ($verdict.passed) { 'MATCH' } else { 'DIVERGED' } + DifferentPixels = $verdict.differentPixels + Fraction = $verdict.differentPixelFraction + MaxChannelDelta = $verdict.maximumChannelDelta + RowBands40px = ($bands -join ',') + DiffMap = $mapPath + } + } +} + +# --- Verdict ------------------------------------------------------------------ +function Get-StopTrend($rows, [string]$Stop, [string]$Column) { + $series = @($rows | Where-Object { $_.Stop -eq $Stop -and $_.Cycle -ge 1 } | Sort-Object Cycle) + if ($series.Count -lt 2) { return $null } + $warm = if ($series.Count -ge 6) { $series[2..($series.Count - 1)] } else { $series } + $firstValue = [double]$warm[0].$Column + $lastValue = [double]$warm[-1].$Column + $spanCycles = [double]($warm[-1].Cycle - $warm[0].Cycle) + return [pscustomobject][ordered]@{ + Stop = $Stop + Column = $Column + FirstCycle = $warm[0].Cycle + LastCycle = $warm[-1].Cycle + First = $firstValue + Last = $lastValue + Delta = [Math]::Round($lastValue - $firstValue, 1) + PerCycle = if ($spanCycles -gt 0) { [Math]::Round(($lastValue - $firstValue) / $spanCycles, 2) } else { $null } + } +} + +$trends = @() +foreach ($arm in $arms) { + foreach ($stop in @('holtburg', 'hub', 'aerlinthe')) { + foreach ($column in @('WorkingSetMiB', 'PrivateMiB', 'GpuMiB', 'ManagedMiB', + 'Materialized', 'WorldEntities', 'MeshRenderData', + 'CompositeTextures', 'GpuTextures', 'Handles')) { + $t = Get-StopTrend $arm.Rows $stop $column + if ($t) { + $trends += [pscustomobject][ordered]@{ + Backend = $arm.Backend; Stop = $t.Stop; Column = $t.Column + FirstCycle = $t.FirstCycle; LastCycle = $t.LastCycle + First = $t.First; Last = $t.Last; Delta = $t.Delta; PerCycle = $t.PerCycle + } + } + } + } +} + +Write-Host '' +foreach ($arm in $arms) { + if ($arm.Aborted) { Write-Host "!!! $($arm.Backend) aborted: $($arm.Aborted)" -ForegroundColor Yellow } + Write-Host "=== $($arm.Backend) : completed=$($arm.RouteCompleted) graceful=$($arm.Graceful) exit=$($arm.ExitCode) witness=$($arm.Witness) checkpoints=$($arm.Checkpoints) peakWS=$($arm.PeakWorkingSetMiB) MiB finalWS=$($arm.FinalWorkingSetMiB) MiB" + $arm.Rows | Where-Object { $_.Stop -eq 'holtburg' } | + Select-Object Cycle, WorkingSetMiB, PrivateMiB, GpuMiB, ManagedMiB, LiveEntities, Materialized, WorldEntities, MeshRenderData, CompositeTextures, GpuTextures, Handles | + Format-Table -AutoSize | Out-String | Write-Host +} +Write-Host '=== within-arm capture comparison (the #256 half) ===' +$comparisons | Format-Table -AutoSize | Out-String | Write-Host +Write-Host '=== warm trends (cycle 3 -> last) ===' +$trends | Where-Object { $_.Stop -eq 'holtburg' } | Format-Table -AutoSize | Out-String | Write-Host + +$report = [pscustomobject][ordered]@{ + Commit = (& git -C $repo rev-parse HEAD).Trim() + GeneratedUtc = [DateTime]::UtcNow.ToString('o') + Cycles = $Cycles + Transits = $transit + Stops = @($stops | ForEach-Object { $_.Key }) + ShotCycles = @($ShotCycles) + Route = $routePath + DayGroup = $DayGroup + WorldDayFraction = $WorldDayFraction + SkyPhaseSeconds = $SkyPhaseSeconds + MsaaSamples = 0 + MaskTopPixels = $MaskTopPixels + Arms = @($arms | ForEach-Object { + [pscustomobject][ordered]@{ + Backend = $_.Backend; RouteCompleted = $_.RouteCompleted; Aborted = $_.Aborted + Graceful = $_.Graceful + ExitCode = $_.ExitCode; Witness = $_.Witness; GrabBytes = $_.GrabBytes + Checkpoints = $_.Checkpoints; Samples = $_.Samples + PeakWorkingSetMiB = $_.PeakWorkingSetMiB; PeakPrivateMiB = $_.PeakPrivateMiB + FinalWorkingSetMiB = $_.FinalWorkingSetMiB; ErrorLines = $_.ErrorLines + Rows = $_.Rows + } + }) + Comparisons = @($comparisons) + Trends = @($trends) +} +$reportPath = Join-Path $Out 'churn-soak.json' +$report | ConvertTo-Json -Depth 8 | Set-Content -LiteralPath $reportPath -Encoding utf8 +Write-Step "report: $reportPath"