From 777f60708d889159061723a45c934ef168d33f9c Mon Sep 17 00:00:00 2001 From: Erik Date: Tue, 28 Jul 2026 21:37:32 +0200 Subject: [PATCH] ci(render): make V9's first CI run green on both operating systems The lavapipe job did the thing it was built to do on its first attempt. It accepted a Cpu device at API 1.4, created a device and read pixels back, captured a real frame, and exited 4 when a feature was forced unsupported. Three other things were red, and none of them were the Vulkan backend. The shader-freshness step aborted for two separate Linux faults in the compiler tool. Disposing the Silk.NET API container unloads the native module, and dlclose-ing libshaderc_shared.so leaves glslang's process-level teardown running against unmapped code. Bisected with a four-mode probe on Ubuntu 24.04: GetApi, CompilerInitialize and CompilerRelease each exit 0, and adding only the container Dispose turns the exit into SIGSEGV. That is the 134 CI reported. shaderc's own handles are still released; the container is not, because the module's lifetime is the process's and the process is one statement from returning. Separately, a portable dotnet build leaves the native under runtimes/linux-x64/native/ and makes reaching it Silk.NET's probing problem, which it solved on a local Ubuntu 24.04 and did not solve on the runner. The script now publishes the tool for the host RID, so the native sits beside the assembly where AppContext.BaseDirectory finds it, and checks for it by name so a regression says which file is missing rather than which names failed. With both fixed, the question section 5.5.20 left open has an answer: Linux shaderc and Windows shaderc agree byte-for-byte at the pinned Silk 2.23.0. Eighteen of eighteen .spv identical, manifest identical. The byte comparison stays a byte comparison. The Windows leg of portable-headless was running sudo apt-get. That step is older than this campaign - it is red in the 2026-07-27 main run too - and it was misplaced rather than mis-conditioned. Nothing in that job opens a display or links GL, and the graphical jobs that do call xvfb-run take it from the runner image, so the step is deleted rather than guarded. Every remaining step in the two-operating-system matrix is pwsh; every bash step now lives in an ubuntu-only job. The last failure was ours in a quieter way. WaitForCharacterLogOff- Confirmation expressed its deadline only as a CancellationTokenSource, whose timeout is published from a thread-pool timer callback, so on a saturated pool the token stays unsignalled past the deadline while the loop keeps draining items that are already queued. That is the case the method exists to bound. Reproduced by pinning the suite to two CPUs on Linux, which failed 2 of 6 where four CPUs and sixteen were clean, and where CI failed 3 of 3. The drain now reads the deadline off the monotonic clock as well; the token still bounds the asynchronous wait. Ten of ten clean under the same pin. The test is untouched. Filed as Release build green. App tests 4,152 / 3 skipped against the same 4,152 / 3 measured at base 32f9bcfa. Core.Net 600 / 600. Co-Authored-By: Claude Fable 5 --- .github/workflows/headless-portability.yml | 19 ++++--- docs/ISSUES.md | 44 +++++++++++++++ docs/plans/2026-07-27-vulkan-campaign.md | 62 ++++++++++++++++++++++ src/AcDream.Core.Net/WorldSession.cs | 16 +++++- tools/ShaderCompiler/Program.cs | 12 ++++- tools/compile-shaders.ps1 | 53 +++++++++++++++--- 6 files changed, 187 insertions(+), 19 deletions(-) diff --git a/.github/workflows/headless-portability.yml b/.github/workflows/headless-portability.yml index e764b216..828c17f1 100644 --- a/.github/workflows/headless-portability.yml +++ b/.github/workflows/headless-portability.yml @@ -67,16 +67,15 @@ jobs: with: dotnet-version: "10.0.x" - - name: Install Linux graphical smoke dependencies - shell: bash - run: | - sudo apt-get update - sudo apt-get install -y --no-install-recommends \ - libgl1-mesa-dri \ - mesa-utils \ - xauth \ - xvfb - + # No apt step here on purpose. This job's whole claim is that the closure + # below is presentation-free: it builds Plugin.Abstractions, Core, + # Core.Net, Content, Runtime and Headless, runs their tests, and invokes + # the Headless CLI. Nothing in it opens a display, links GL, or calls + # xvfb-run, so an "install the graphical smoke dependencies" step here was + # both unnecessary and, being unconditional `sudo apt-get` in a + # two-operating-system matrix, fatal on the windows-latest leg (exit 127, + # `sudo: command not found`). The graphical jobs that do use xvfb-run and + # jq run only on ubuntu-latest and take both from the runner image. - name: Build presentation-free closure shell: pwsh run: | diff --git a/docs/ISSUES.md b/docs/ISSUES.md index 23741108..1f09c0ab 100644 --- a/docs/ISSUES.md +++ b/docs/ISSUES.md @@ -97,6 +97,50 @@ Copy this block when adding a new issue: --- +## #254 — Logout confirmation wait overran its timeout on a starved thread pool + +**Status:** DONE — 2026-07-28; monotonic deadline added to the synchronous drain +**Severity:** LOW (shutdown-path timing; no user-visible defect observed) +**Filed:** 2026-07-28 +**Component:** net / session shutdown + +**Description:** `AcDream.Core.Net.Tests.WorldSessionShutdownTests.WaitForConfirmation_TimeoutWinsDuringContinuousUnrelatedDrain` +failed on the `portable-headless (ubuntu-latest)` leg of the Headless +portability workflow in **three CI runs out of three** (30259857711 on `main`, +30386410215 and 30389334662 on the campaign branch), 599/600 each time, while +the windows-latest leg never saw it. The assertion that fell was +`Assert.True(processed < 100)`: the method drained all 100 queued items — about +200 ms of work behind a 25 ms timeout — before noticing the deadline. + +**Root cause:** `WorldSession.WaitForCharacterLogOffConfirmation` expressed its +deadline **only** as `CancellationTokenSource(TimeSpan)` and polled +`IsCancellationRequested`. A CTS timeout is published from a thread-pool timer +callback, so when the pool is saturated the token stays unsignalled well past +the deadline while the drain loop keeps consuming already-queued items — exactly +the case the method exists to bound. This is a real (if small) product defect, +not a test artifact: on a loaded machine the client's logout wait could overrun +its own timeout by the length of whatever is already sitting in the inbound +queue. + +**Evidence:** not reproducible on Windows (5/5 suite runs clean) nor on native +Linux with 16 cores (8/8 clean). Reproduced by pinning the suite to two CPUs on +Ubuntu 24.04 — `taskset -c 0,1` — which failed 2 runs out of 6; the same pin at +four CPUs was clean 6/6. GitHub-hosted `ubuntu-latest` is 4-core and slower than +the local pin, which is why CI saw it every time. + +**Fix:** the drain loop now also compares `Stopwatch.GetElapsedTime` against the +requested timeout, so the deadline is read off the monotonic clock rather than +only off a thread-pool callback. The CTS is kept — it still bounds the +asynchronous `WaitToReadAsync`. A negative timeout keeps its framework meaning +of "infinite". Ten of ten suite runs clean under the same two-CPU pin +afterwards. **The test was not modified.** + +**Files:** `src/AcDream.Core.Net/WorldSession.cs` (the internal generic +`WaitForCharacterLogOffConfirmation`), asserted by +`tests/AcDream.Core.Net.Tests/WorldSessionShutdownTests.cs:203`. + +--- + ## #253 — Attribute/skill icons: not centered in their cells, and fully opaque **Status:** OPEN diff --git a/docs/plans/2026-07-27-vulkan-campaign.md b/docs/plans/2026-07-27-vulkan-campaign.md index 3a55513b..426e1585 100644 --- a/docs/plans/2026-07-27-vulkan-campaign.md +++ b/docs/plans/2026-07-27-vulkan-campaign.md @@ -3257,6 +3257,68 @@ should name its measurement vehicle, because the obvious candidate is the wrong one and cost this slice an hour to establish; and the founding numbers should be labelled with their scene, because Caul and Aerlinthe differ by more than the entire GL-versus-Vulkan CPU gap. +#### 5.5.22 V9's first CI runs: the answer on determinism, and three failures + +The `linux-vulkan` job did what it was built to do on its first attempt. Steps +(a), (b) and (c) all passed on lavapipe: the gate accepted a `Cpu` device at API +1.4, the active probe created the device and read pixels back, the captured PNG +was a real frame, and the forced-unsupported run exited 4 with a report naming +the feature. **The first CI job in the project's history that renders a frame +rendered one.** + +Step (d) aborted, and two failures elsewhere came with it. + +**The determinism verdict: byte-identical, yes.** §5.5.20 left one thing +asserted-but-unobserved — whether Linux shaderc and Windows shaderc agree +byte-for-byte at the same pinned Silk.NET 2.23.0. **They do.** Measured directly +rather than inferred: the same GLSL sources compiled on Ubuntu 24.04 through the +package's `linux-x64` native produce **18/18 `.spv` byte-identical** to the +committed Windows-produced artifacts, and a parsed-JSON manifest compare is +clean. So the byte comparison in step (d) is the right instrument and needs no +softening — neither pinning a single compiler build nor falling back to a +structural `spirv-dis` compare. The pinned NuGet native is already the pin. + +**Why (d) aborted anyway: two Linux faults in the tool, not in the shaders.** + +1. **`Shaderc.Dispose()` kills the process on Linux.** Disposing the Silk.NET API + container unloads the native module, and `dlclose`-ing `libshaderc_shared.so` + leaves glslang's process-level teardown to run against unmapped code. Bisected + with a four-mode probe on Ubuntu 24.04: `GetApi`, `CompilerInitialize` and + `CompilerRelease` each exit 0, and adding **only** the container `Dispose` + turns the exit into SIGSEGV. That is what CI reported as exit 134. shaderc's + own handles are still released; the container is not, because the module's + lifetime is the process's and the process is one statement from returning. + +2. **A portable `dotnet build` does not reliably put the native where it can be + found.** It leaves `libshaderc_shared.so` under `runtimes/linux-x64/native/` + and makes reaching it the job of Silk.NET's probing chain, which resolved it + on a local Ubuntu 24.04 and did **not** on the ubuntu-24.04 runner — + `Could not load from any of the possible library names!` at `GetApi`. The + script now **publishes** the tool for the host RID, which flattens the native + beside the assembly where `AppContext.BaseDirectory`, the first candidate + Silk.NET tries, always finds it, and then checks the file is there by name so + a future regression says which file is missing instead of which names failed. + +**The two failures outside the Vulkan job.** + +3. **The `portable-headless` matrix ran `sudo apt-get` on windows-latest** and + exited 127. Pre-existing since L1 (`11501d52`) rather than introduced here — + the same step is red in the `main` run of 2026-07-27 — and it was misplaced + rather than mis-conditioned: that job builds the presentation-free closure and + the Headless CLI, nothing in it opens a display or links GL, and the graphical + jobs that *do* call `xvfb-run` take it from the runner image. Deleted. + +4. **`WaitForConfirmation_TimeoutWinsDuringContinuousUnrelatedDrain` failed on + the ubuntu leg**, 3 CI runs out of 3, while passing on Windows. Not a flake + and not the campaign's: `CancellationTokenSource(TimeSpan)` publishes its + cancellation from a thread-pool timer callback, so on a saturated pool the + token stays unsignalled past the deadline while the drain loop keeps + consuming already-queued items — the precise case the method exists to bound. + Reproduced by pinning the suite to two CPUs on Linux (2 failures in 6; clean + at four and at sixteen, and clean on Windows). Fixed at the cause by reading + the deadline off the monotonic clock in the synchronous drain as well as off + the token, which still bounds the asynchronous wait. Ten of ten clean under + the same two-CPU pin afterwards. The test was not touched. ### 5.4 The null-target `BeginPass` divergence (V4c) — ✅ DISCHARGED at V6k diff --git a/src/AcDream.Core.Net/WorldSession.cs b/src/AcDream.Core.Net/WorldSession.cs index ef3affca..38602a84 100644 --- a/src/AcDream.Core.Net/WorldSession.cs +++ b/src/AcDream.Core.Net/WorldSession.cs @@ -2422,13 +2422,25 @@ public sealed class WorldSession : IDisposable ArgumentNullException.ThrowIfNull(processAndCheckConfirmation); using var timeoutSource = new CancellationTokenSource(timeout); + // The deadline is also read straight off the monotonic clock, not only + // off the token. CancellationTokenSource(TimeSpan) publishes its + // cancellation from a thread-pool timer callback, so when the pool is + // saturated the token can stay unsignalled well past the deadline while + // the loop below keeps draining a queue that already has items in it - + // exactly the case this method exists to bound. The token still bounds + // the asynchronous wait; the clock bounds the synchronous drain. A + // negative timeout is the framework's "infinite" and keeps its meaning. + long started = Stopwatch.GetTimestamp(); + bool bounded = timeout >= TimeSpan.Zero; + bool Expired() => bounded && Stopwatch.GetElapsedTime(started) >= timeout; + try { - while (!timeoutSource.IsCancellationRequested) + while (!timeoutSource.IsCancellationRequested && !Expired()) { while (reader.TryRead(out T? item)) { - if (timeoutSource.IsCancellationRequested) + if (timeoutSource.IsCancellationRequested || Expired()) { release?.Invoke(item); return false; diff --git a/tools/ShaderCompiler/Program.cs b/tools/ShaderCompiler/Program.cs index 584b3906..f273c110 100644 --- a/tools/ShaderCompiler/Program.cs +++ b/tools/ShaderCompiler/Program.cs @@ -122,9 +122,19 @@ internal static class Program } finally { + // shaderc's own handles are released; the Silk.NET API container + // deliberately is not. Disposing the container unloads the native + // module, and on Linux dlclose'ing libshaderc_shared.so kills the + // process during exit: glslang registers process-level teardown + // that runs after the module's code has been unmapped. Bisected on + // Ubuntu 24.04 with a four-mode probe - GetApi, CompilerInitialize + // and CompilerRelease all exit 0, and adding only the container + // Dispose turns the exit into SIGSEGV (139), which reached CI as + // "shader compilation failed with exit code 134". Nothing is leaked + // by omitting it: the module's lifetime is the process's, and the + // process is one statement from returning. shaderc.CompileOptionsRelease(options); shaderc.CompilerRelease(compiler); - shaderc.Dispose(); } var manifest = new ShaderManifest( diff --git a/tools/compile-shaders.ps1 b/tools/compile-shaders.ps1 index dad4b154..2a352eee 100644 --- a/tools/compile-shaders.ps1 +++ b/tools/compile-shaders.ps1 @@ -98,15 +98,56 @@ else { $tool = [System.IO.Path]::Combine( $repo, 'tools', 'ShaderCompiler', 'ShaderCompiler.csproj') -Write-Step 'building the shader compiler' -& dotnet build $tool -c Release --nologo -v q | Out-Null -if ($LASTEXITCODE -ne 0) { throw "Shader compiler build failed with exit code $LASTEXITCODE." } -$binary = [System.IO.Path]::Combine( - $repo, 'tools', 'ShaderCompiler', 'bin', 'Release', 'net10.0', - 'AcDream.Tools.ShaderCompiler.dll') +# Campaign V slice V9 follow-up: the tool is PUBLISHED for the host runtime +# identifier rather than built portable. A portable `dotnet build` leaves +# shaderc's native module under runtimes//native/ and makes loading it the +# job of Silk.NET's probing chain, which reaches it on some hosts and not +# others - it resolved on Ubuntu 24.04 locally and failed on the ubuntu-24.04 +# CI runner with "Could not load from any of the possible library names!". +# Publishing with an explicit RID flattens the native next to the assembly, +# where AppContext.BaseDirectory - the first candidate Silk.NET tries - always +# finds it. Same managed code, same pinned shaderc 2.23.0; only the lookup +# changes. +$architecture = [System.Runtime.InteropServices.RuntimeInformation]::ProcessArchitecture +$ridArchitecture = switch ($architecture) { + 'X64' { 'x64' } + 'Arm64' { 'arm64' } + 'X86' { 'x86' } + default { throw "No shaderc native is published for processor architecture $architecture." } +} +# $IsWindows and friends only exist in PowerShell 6+; 5.1 is Windows by +# definition, and -or short-circuits before touching the undefined variable. +$ridOs = + if ($PSVersionTable.PSVersion.Major -lt 6 -or $IsWindows) { 'win' } + elseif ($IsMacOS) { 'osx' } + elseif ($IsLinux) { 'linux' } + else { throw 'Unrecognised operating system; cannot choose a shaderc native.' } +$rid = "$ridOs-$ridArchitecture" + +$toolDirectory = [System.IO.Path]::Combine( + $repo, 'tools', 'ShaderCompiler', 'bin', 'publish', $rid) +Write-Step "publishing the shader compiler for $rid" +& dotnet publish $tool -c Release -r $rid --self-contained false -o $toolDirectory --nologo -v q | Out-Null +if ($LASTEXITCODE -ne 0) { throw "Shader compiler publish failed with exit code $LASTEXITCODE." } + +$binary = Join-Path $toolDirectory 'AcDream.Tools.ShaderCompiler.dll' if (-not (Test-Path $binary)) { throw "Shader compiler not found at $binary." } +# Name the missing file rather than letting Silk.NET report the generic +# "could not load from any of the possible library names" three steps later. +$nativeName = + if ($ridOs -eq 'win') { 'shaderc_shared.dll' } + elseif ($ridOs -eq 'osx') { 'libshaderc_shared.dylib' } + else { 'libshaderc_shared.so' } +$native = Join-Path $toolDirectory $nativeName +if (-not (Test-Path $native)) { + throw ("The shaderc native $nativeName is not beside the shader compiler at " + + "$toolDirectory. The Silk.NET.Shaderc.Native package did not publish a " + + "$rid asset.") +} +Write-Step "shaderc native: $native" + Write-Step "compiling $ShadersDirectory -> $OutputDirectory" & dotnet $binary $ShadersDirectory $OutputDirectory if ($LASTEXITCODE -ne 0) { throw "Shader compilation failed with exit code $LASTEXITCODE." }