Optimize prepared asset package v2

docs/plans/2026-08-27-pak-v2-resource-campaign.md

@@ -0,0 +1,109 @@
+# PAK v2 resource campaign
+
+Status: ACTIVE (2026-08-27)
+
+## Objective and release gates
+
+Ship one crash-safe prepared-asset format migration that:
+
+- reduces the complete installed package from 29,908,271,024 bytes to at
+  most 5 GiB;
+- preserves decoded geometry, material metadata, texture bytes, deterministic
+  baking, corruption isolation, and random-access loading;
+- does not regress cold or warm world-reveal latency or frame-time percentiles;
+- reduces bake CPU/memory and client retained texture memory where the data
+  permits it, without changing the rendered result;
+- gives launcher users a clear one-time update message and progress, while
+  retaining the last verified package until the replacement is validated;
+- passes two different worker-count bakes with identical SHA-256, the complete
+  installed-DAT bake, content equivalence, performance, solution, Windows CI,
+  and release gates.
+
+## Measured format-1 baseline
+
+The installed package was parsed from its actual TOC, not estimated:
+
+| Partition | Physical blobs | Physical bytes |
+|---|---:|---:|
+| GfxObj render meshes | 15,318 | 9,306,115,868 |
+| Setup render meshes | 5,935 | 4,078,139 |
+| EnvCell render meshes | 17,117 | 20,232,745,510 |
+| All collision payloads | 12,938 | 30,392,894 |
+| EnvCell topology | 729,888 | 255,512,622 |
+| TOC | 2,232,170 rows | 53,572,080 |
+
+Total: 29,908,271,024 bytes. Render payloads account for approximately
+29.54 GB and 99% of physical payload bytes. The collision and index data are
+not the size problem. Format 1 already aliases duplicate complete EnvCell
+blobs, but each remaining mesh embeds another copy of every decoded RGBA
+texture it uses.
+
+Historical complete-bake baseline: 80.5-107.5 seconds, 4.43-4.89 GB peak
+working set, and 3.81-4.21 GB peak private bytes.
+
+## Format 2 contract
+
+The 64-byte header and 24-byte sorted TOC row remain fixed. Format version is
+2 and bake recipe is 6.
+
+1. A new `TexturePayload` key partition (type 8) owns globally shared texture
+   byte arrays. Mesh payloads store the texture payload key while retaining
+   their own exact dimensions, format, upload metadata, surface identity,
+   translucency, culling, and index data.
+2. Texture payload keys are the first 56 bits of SHA-256 under the type-8
+   namespace. The writer retains and compares the full bytes for the complete
+   bake, making even a truncated-hash collision a loud bake failure rather
+   than silent substitution.
+3. Every physical blob is independently encoded. The high bit of the TOC
+   length marks compression; the low 31 bits are the stored length. A
+   compressed blob contains a four-byte decoded-length prefix followed by
+   Brotli. Small or insufficiently compressible blobs remain exactly raw.
+   CRC-32 covers stored bytes, then decompression is independently validated.
+4. Random access remains one binary search plus one mmap copy for raw blobs.
+   Compressed blobs add decompression only when the writer proved a material
+   size win. Texture payloads use a bounded, thread-safe 256 MiB / 2,048-entry
+   LRU; concurrently decoded meshes converge on one shared array instance.
+5. Whole-file compression is forbidden. It would destroy random access and
+   make a small world reveal depend on unrelated content.
+
+## Determinism and publication
+
+Asset traversal and mesh serialization remain sorted. A texture is emitted at
+its first deterministic encounter, so its physical order is independent of
+worker completion order. Aliases preserve the source row's exact offset,
+encoded length/flags, and CRC.
+
+Recipe 5 to 6 is a mandatory full rebuild. The launcher builds
+`acdream.pak.candidate` beside the active package, validates format, recipe,
+DAT iterations, TOC counts, size, completion protocol, and SHA-256, then uses
+the existing atomic promotion/backup transaction. Cancellation or failure
+keeps the verified format-1 package. No overlay may cross this format change.
+
+## Checkpoint evidence
+
+The first installed-DAT mixed sample (four GfxObj, three Setup, three EnvCell,
+all corresponding collision/topology payloads) produced 58 keys, 29 globally
+deduplicated texture payloads, and 57 physical blobs. Decoded payload was
+3.5 MiB and stored payload 1.0 MiB (3.62x); output was 1.0 MiB. Eight-worker
+and three-worker bakes had the identical SHA-256
+`78886DFA28A3EDF9368A1E25C9B02A3B69ADC5DFCC01358D64A073B549B5B532`.
+
+This sample is a checkpoint only. The size and performance release gates are
+decided by the complete installed-DAT package and connected world-loading
+measurements.
+
+## Work ledger
+
+- [x] Measure the format-1 package by TOC partition.
+- [x] Implement and unit-test format-2 external texture references, adaptive
+  independent compression, corruption handling, bounded sharing, and byte
+  determinism.
+- [x] Integrate format-2 accounting and strict validation into the bake.
+- [x] Publish the recipe-6 mandatory full-rebuild launcher migration.
+- [ ] Add launcher disk-space preflight and explicit long-work detail.
+- [ ] Complete installed-DAT equivalence and dual-worker full bakes.
+- [ ] Measure/tune package size, bake CPU/memory, read CPU/allocations, cold
+  and warm reveal latency, and frame-time percentiles.
+- [ ] Evaluate source-native BC texture retention only if it remains visually
+  exact and does not shift mip-generation work into the reveal frame.
+- [ ] Pass complete tests, Windows CI, merge, push, and release gates.