PAK v2 resource campaign

Status: CLOSED — SHIPPED (2026-08-27)

Objective and release gates

Ship one crash-safe prepared-asset format migration that:

reduces the complete installed package from 29,908,271,024 bytes to at most 5 GiB;
preserves decoded geometry, material metadata, texture bytes, deterministic baking, corruption isolation, and random-access loading;
does not regress cold or warm world-reveal latency or frame-time percentiles;
reduces offline bake time and live client CPU/GPU memory where the data permits it, without changing the rendered result; the one-time bake remains bounded for the user-confirmed 16-32 GiB target machines;
gives launcher users a clear one-time update message and progress, while retaining the last verified package until the replacement is validated;
passes two different worker-count bakes with identical SHA-256, the complete installed-DAT bake, content equivalence, performance, solution, Windows CI, and release gates.

Measured format-1 baseline

The installed package was parsed from its actual TOC, not estimated:

Partition	Physical blobs	Physical bytes
GfxObj render meshes	15,318	9,306,115,868
Setup render meshes	5,935	4,078,139
EnvCell render meshes	17,117	20,232,745,510
All collision payloads	12,938	30,392,894
EnvCell topology	729,888	255,512,622
TOC	2,232,170 rows	53,572,080

Total: 29,908,271,024 bytes. Render payloads account for approximately 29.54 GB and 99% of physical payload bytes. The collision and index data are not the size problem. Format 1 already aliases duplicate complete EnvCell blobs, but each remaining mesh embeds another copy of every decoded RGBA texture it uses.

Historical complete-bake baseline: 80.5-107.5 seconds, 4.43-4.89 GB peak working set, and 3.81-4.21 GB peak private bytes.

Format 2 contract

The 64-byte header and 24-byte sorted TOC row remain fixed. Format version is 2 and bake recipe is 6.

A new TexturePayload key partition (type 8) owns globally shared texture byte arrays. Mesh payloads store the texture payload key while retaining their own exact dimensions, format, upload metadata, surface identity, translucency, culling, and index data.
Texture payload keys are the first 56 bits of SHA-256 under the type-8 namespace. The writer retains the full SHA-256 digest and length for every unique texture, making even a truncated-key collision a loud bake failure rather than silent substitution without pinning another copy of all bytes.
Every physical blob is independently encoded. The high bit of the TOC length marks compression; the low 31 bits are the stored length. A compressed blob contains a four-byte decoded-length prefix followed by Brotli. Small or insufficiently compressible blobs remain exactly raw. CRC-32 covers stored bytes, then decompression is independently validated.
Random access remains one binary search plus one mmap copy for raw blobs. Compressed blobs add decompression only when the writer proved a material size win. Texture payloads use a bounded, thread-safe 64 MiB / 1,024-entry LRU; concurrently decoded meshes converge on one shared array instance.
Whole-file compression is forbidden. It would destroy random access and make a small world reveal depend on unrelated content.
Unedited DAT DXT1/3/5 surfaces retain their exact BC1/2/3 source blocks through bake, mmap, and Vulkan upload. Clip maps and surfaces with authored translucency still decode to RGBA8 because their per-surface alpha edits require pixels. This is smaller and closer to retail's hardware DXT path than the former unconditional software decode.

Determinism and publication

Asset traversal and mesh serialization remain sorted. A texture is emitted at its first deterministic encounter, so its physical order is independent of worker completion order. Aliases preserve the source row's exact offset, encoded length/flags, and CRC.

Recipe 5 to 6 is a mandatory full rebuild. The launcher builds acdream.pak.candidate beside the active package, validates format, recipe, DAT iterations, TOC counts, size, completion protocol, and SHA-256, then uses the existing atomic promotion/backup transaction. Cancellation or failure keeps the verified format-1 package. No overlay may cross this format change.

Complete installed-DAT evidence

The first installed-DAT mixed sample (four GfxObj, three Setup, three EnvCell, all corresponding collision/topology payloads) produced 58 keys, 29 globally deduplicated texture payloads, and 57 physical blobs. Decoded payload was 3.5 MiB and stored payload 1.0 MiB (3.62x); output was 1.0 MiB. Eight-worker and three-worker bakes had the identical SHA-256 78886DFA28A3EDF9368A1E25C9B02A3B69ADC5DFCC01358D64A073B549B5B532.

The final complete four-worker and nine-worker bakes are byte-identical:

2,237,866 logical keys and 786,892 physical blobs;
5,696 globally shared texture payloads;
zero extraction/validation failures;
597,229,424 bytes (569.6 MiB), down 98.0% from 29,908,271,024 bytes;
SHA-256 37BC0EA1778F899AF9E3B2397937D373F69D615B15D3E32041BF389D93B624BF;
28.7 seconds for the final four-worker bake versus 79.1 seconds for the same-machine recipe-5 format-1 baseline (63.7% faster).

The exact before/after connected lifecycle route used the same machine, server, 1280x720 Vulkan presentation, retail render pack, and matching client code. Both runs passed fresh login, multi-world portal travel, same-location revisit, fresh-process reconnect, and graceful teardown.

Matching live measurement	Format 1	Format 2	Change
Heavy-route final working set	2,621.9 MiB	1,363.9 MiB	-48.0%
Heavy-route final private bytes	2,470.5 MiB	1,825.6 MiB	-26.1%
Holtburg prepared-mesh GPU bytes	229.6 MiB	170.1 MiB	-25.9%
Fresh reconnect working set	974.1 MiB	869.0 MiB	-10.8%
Fresh reconnect private bytes	1,253.8 MiB	1,166.6 MiB	-7.0%
Fresh reconnect to checkpoint	59.3 s	59.3 s	equal

Rynthid and Facility transition times were equal; Holtburg completed 2.9 s faster; Aerlinthe revisit differed by 0.1 s. Matching stable CPU p50/p95 rows were equal or slightly faster and GPU time remained within 0.1 ms. The first raw login samples were excluded from comparison because the server started them in different cells with 6,671 versus 11,799 world entities. Matching screenshots preserve geometry, materials, lighting, transparency, texture detail, and mip behavior.

Automated gates at this checkpoint: 181/181 installed-DAT Content tests, 16,151/16,151 CI-filtered Windows tests, Release build with zero warnings, and the authoritative connected lifecycle/reconnect gate all pass.

Release closeout

Main fast-forwarded cleanly to 45ba42a3. Gitea Actions run 206 passed the Windows gate, Linux portable gate, and release job. The published alpha is 0.1.0-build.202608271848; the stable latest pointer carries the 45 MiB Windows client, 78 MiB Windows launcher, and update manifest.

The first remote run exposed one locale-only contract failure: the launcher's disk-space guidance rendered 2,0 GiB under the runner culture. The shipped fix formats the value invariantly and exercises the failure path under sv-SE; the exact follow-up local Release gate again passed 16,151/16,151. The exact shipped App binary also passed the connected lifecycle/reconnect route (connected-world-gate-20260827-204249) with only the 25 expected world-edge misses.

Work ledger

Measure the format-1 package by TOC partition.
Implement and unit-test format-2 external texture references, adaptive independent compression, corruption handling, bounded sharing, and byte determinism.
Integrate format-2 accounting and strict validation into the bake.
Publish the recipe-6 mandatory full-rebuild launcher migration.
Add launcher disk-space preflight and explicit long-work detail.
Complete installed-DAT equivalence and dual-worker full bakes.
Measure/tune package size, bake time, read CPU/allocations, cold and warm reveal latency, and frame-time percentiles.
Evaluate source-native BC texture retention only if it remains visually exact and does not shift mip-generation work into the reveal frame.
Pass complete local tests and authoritative connected gates.
Pass Gitea Windows CI, merge, push, and release gates.

8.3 KiB Raw Blame History