acdream/tests/AcDream.App.Tests/ZeroAllocationProbe.cs

using Xunit;

namespace AcDream.App.Tests;

/// <summary>
/// Measurement apparatus for the "this warmed path allocates nothing" assertions
/// (docs/ISSUES.md #250). The bound it enforces is strict — zero managed bytes,
/// never a tolerance — and the whole point of the class is that the number it
/// hands to that bound is the steady-state allocation rather than a one-time
/// startup cost the runtime happened to charge to the measuring thread.
///
/// <para>
/// The family flaked because in every member the measured window was *not* the
/// warmed path. Two distinct shapes produced the same symptom:
/// </para>
///
/// <list type="number">
/// <item>
/// <b>The window contained a hot loop.</b> `UiDatFontTests` measured a
/// 10,000-iteration loop and `RenderFrameProductTests` a 1,000-iteration one,
/// both written directly in the test method. A test method is JIT-compiled at
/// tier 0 like anything else, and a long-running loop inside tier-0 code is
/// replaced mid-flight by on-stack replacement. OSR compiles on the thread
/// running the loop — the measuring thread — so its bookkeeping lands inside
/// the window being measured.
/// </item>
/// <item>
/// <b>The warmup exercised a different branch.</b> `ArchRenderSceneTests` warmed
/// with `Apply(registrations)` and then measured `Apply(updates)`, which is the
/// `ApplyUpdate` arm rather than the `ApplyRegister` one — so the measured call
/// was the first-ever call into that half of the switch, and its tier-0 JIT,
/// type loads and static initialisation were all charged to the window.
/// `CurrentRenderSceneOracleTests` warmed once and measured once, which is the
/// same problem with a shorter fuse.
/// </item>
/// </list>
///
/// <para>
/// Both explain the signature the issue recorded — clean in isolation, failing
/// roughly one full-suite run in three. Running alone, the process is quiet and
/// the runtime's tiering work is finished before the assertion arrives. Running
/// alongside eight other test assemblies, tier-0 compilation never stops, the
/// call-counting delay is re-armed continually, and the work slides into the
/// measured window.
/// </para>
///
/// <para>
/// So the probe invokes the delegate many times before it measures anything,
/// and every measured window then runs the same already-warmed loop over the
/// same already-taken code path. Nothing is left for OSR to replace and nothing
/// is left to JIT.
/// </para>
///
/// <para>
/// Each window is a <i>batch</i> of invocations rather than a single one, and
/// the probe reports the <b>minimum</b> across several such batches. Both halves
/// of that matter, and the batch is the half that is easy to get wrong:
/// </para>
///
/// <list type="bullet">
/// <item>
/// The <b>minimum</b> is what excludes one-time costs. A promotion enqueue or a
/// lazily-grown internal buffer happens in at most one batch, so some other
/// batch is clean and the floor is zero. This is not a retry and not a
/// tolerance — it is the ordinary way to estimate a steady-state floor.
/// </item>
/// <item>
/// The <b>batch</b> is what keeps the assertion as strong as the loops it
/// replaced. Minimising over <i>single</i> invocations would report zero for a
/// path that allocates every tenth call, because nine windows in ten would be
/// clean — a real regression, invisible. Summing a batch first means any cost
/// recurring at a period the batch covers appears in <i>every</i> batch, so the
/// minimum is above zero and the assertion fails. That restores what the
/// original 1,000- and 10,000-iteration loops were measuring, without putting
/// the loop somewhere the runtime will rewrite it mid-measurement.
/// </item>
/// </list>
///
/// <para>
/// A cost recurring less often than once per batch is the one thing this cannot
/// see. <see cref="DefaultBatchSize"/> is chosen well above any plausible cache
/// or ring period in the paths under test, and a test guarding a longer cycle
/// should pass a batch size that covers it.
/// </para>
/// </summary>
internal static class ZeroAllocationProbe
{
    /// <summary>
    /// Invocations per measured window. Large enough that any per-call or
    /// short-period allocation lands inside every window, and therefore in the
    /// minimum across them.
    /// </summary>
    internal const int DefaultBatchSize = 32;

    /// <summary>
    /// Batches run before the first measurement. Their invocation count is
    /// comfortably past the tier-0 call-counting threshold of 30, so the callee
    /// tree is promoted and its one-time costs are paid while nobody is
    /// watching — and the batch loop itself is warm by the time it is measured.
    /// </summary>
    internal const int DefaultWarmupBatches = 4;

    /// <summary>
    /// Measured windows to take the minimum over. Small: the warmup does the
    /// real work, and this only has to outlast a one-time cost that landed
    /// unluckily inside the first batch.
    /// </summary>
    internal const int DefaultSamples = 4;

    /// <summary>
    /// Returns the smallest number of managed bytes that a batch of warmed
    /// invocations of <paramref name="step"/> charged to the current thread.
    /// Zero means the path allocates nothing in steady state.
    /// </summary>
    /// <param name="step">
    /// One complete unit of the work under test. Must be safe to invoke
    /// repeatedly and must take the same code path every time — if it needs
    /// fresh input per invocation (a monotonic sequence number, say), it is
    /// responsible for producing that itself, and for doing so on every
    /// invocation including the warm ones, so the cost is never novel inside a
    /// measured window.
    /// </param>
    /// <param name="batchSize">Invocations summed into one window.</param>
    /// <param name="warmupBatches">Batches run before measuring.</param>
    /// <param name="samples">Measured windows; the minimum is returned.</param>
    internal static long MeasureWarmed(
        Action step,
        int batchSize = DefaultBatchSize,
        int warmupBatches = DefaultWarmupBatches,
        int samples = DefaultSamples)
    {
        ArgumentNullException.ThrowIfNull(step);
        ArgumentOutOfRangeException.ThrowIfLessThan(batchSize, 1);
        ArgumentOutOfRangeException.ThrowIfNegative(warmupBatches);
        ArgumentOutOfRangeException.ThrowIfLessThan(samples, 1);

        for (int batch = 0; batch < warmupBatches; batch++)
            RunBatch(step, batchSize);

        long smallest = long.MaxValue;
        for (int sample = 0; sample < samples; sample++)
        {
            // The window holds one call to an already-jitted RunBatch, running
            // an already-warmed loop over already-warmed code. Both reads
            // bracket that call and nothing else.
            long before = GC.GetAllocatedBytesForCurrentThread();
            RunBatch(step, batchSize);
            long allocated = GC.GetAllocatedBytesForCurrentThread() - before;

            if (allocated < smallest)
                smallest = allocated;

            // Nothing can go below the floor, so stop once it is reached.
            if (smallest == 0)
                break;
        }

        return smallest;
    }

    private static void RunBatch(Action step, int batchSize)
    {
        for (int invocation = 0; invocation < batchSize; invocation++)
            step();
    }

    /// <summary>
    /// Asserts that a warmed invocation of <paramref name="step"/> allocates no
    /// managed bytes at all. The bound is exact and deliberately has no slack.
    /// </summary>
    /// <param name="what">
    /// What the path is, for the failure message — e.g.
    /// "ArchRenderScene.Apply(updates)".
    /// </param>
    internal static void AssertAllocatesNothing(
        string what,
        Action step,
        int batchSize = DefaultBatchSize,
        int warmupBatches = DefaultWarmupBatches,
        int samples = DefaultSamples)
    {
        long allocated = MeasureWarmed(step, batchSize, warmupBatches, samples);

        Assert.True(
            allocated == 0,
            $"{what} allocated {allocated:N0} managed bytes per warmed batch of "
            + $"{batchSize} invocations, expected 0. Measured as the minimum of "
            + $"{samples} such batches after {warmupBatches} warmup batches, so "
            + "this is a steady-state cost rather than a one-time startup cost "
            + "(see docs/ISSUES.md #250).");
    }
}