164 lines
8.3 KiB
Markdown
164 lines
8.3 KiB
Markdown
# PAK v2 resource campaign
|
|
|
|
Status: CLOSED — SHIPPED (2026-08-27)
|
|
|
|
## Objective and release gates
|
|
|
|
Ship one crash-safe prepared-asset format migration that:
|
|
|
|
- reduces the complete installed package from 29,908,271,024 bytes to at
|
|
most 5 GiB;
|
|
- preserves decoded geometry, material metadata, texture bytes, deterministic
|
|
baking, corruption isolation, and random-access loading;
|
|
- does not regress cold or warm world-reveal latency or frame-time percentiles;
|
|
- reduces offline bake time and live client CPU/GPU memory where the data
|
|
permits it, without changing the rendered result; the one-time bake remains
|
|
bounded for the user-confirmed 16-32 GiB target machines;
|
|
- gives launcher users a clear one-time update message and progress, while
|
|
retaining the last verified package until the replacement is validated;
|
|
- passes two different worker-count bakes with identical SHA-256, the complete
|
|
installed-DAT bake, content equivalence, performance, solution, Windows CI,
|
|
and release gates.
|
|
|
|
## Measured format-1 baseline
|
|
|
|
The installed package was parsed from its actual TOC, not estimated:
|
|
|
|
| Partition | Physical blobs | Physical bytes |
|
|
|---|---:|---:|
|
|
| GfxObj render meshes | 15,318 | 9,306,115,868 |
|
|
| Setup render meshes | 5,935 | 4,078,139 |
|
|
| EnvCell render meshes | 17,117 | 20,232,745,510 |
|
|
| All collision payloads | 12,938 | 30,392,894 |
|
|
| EnvCell topology | 729,888 | 255,512,622 |
|
|
| TOC | 2,232,170 rows | 53,572,080 |
|
|
|
|
Total: 29,908,271,024 bytes. Render payloads account for approximately
|
|
29.54 GB and 99% of physical payload bytes. The collision and index data are
|
|
not the size problem. Format 1 already aliases duplicate complete EnvCell
|
|
blobs, but each remaining mesh embeds another copy of every decoded RGBA
|
|
texture it uses.
|
|
|
|
Historical complete-bake baseline: 80.5-107.5 seconds, 4.43-4.89 GB peak
|
|
working set, and 3.81-4.21 GB peak private bytes.
|
|
|
|
## Format 2 contract
|
|
|
|
The 64-byte header and 24-byte sorted TOC row remain fixed. Format version is
|
|
2 and bake recipe is 6.
|
|
|
|
1. A new `TexturePayload` key partition (type 8) owns globally shared texture
|
|
byte arrays. Mesh payloads store the texture payload key while retaining
|
|
their own exact dimensions, format, upload metadata, surface identity,
|
|
translucency, culling, and index data.
|
|
2. Texture payload keys are the first 56 bits of SHA-256 under the type-8
|
|
namespace. The writer retains the full SHA-256 digest and length for every
|
|
unique texture, making even a truncated-key collision a loud bake failure
|
|
rather than silent substitution without pinning another copy of all bytes.
|
|
3. Every physical blob is independently encoded. The high bit of the TOC
|
|
length marks compression; the low 31 bits are the stored length. A
|
|
compressed blob contains a four-byte decoded-length prefix followed by
|
|
Brotli. Small or insufficiently compressible blobs remain exactly raw.
|
|
CRC-32 covers stored bytes, then decompression is independently validated.
|
|
4. Random access remains one binary search plus one mmap copy for raw blobs.
|
|
Compressed blobs add decompression only when the writer proved a material
|
|
size win. Texture payloads use a bounded, thread-safe 64 MiB / 1,024-entry
|
|
LRU; concurrently decoded meshes converge on one shared array instance.
|
|
5. Whole-file compression is forbidden. It would destroy random access and
|
|
make a small world reveal depend on unrelated content.
|
|
6. Unedited DAT DXT1/3/5 surfaces retain their exact BC1/2/3 source blocks
|
|
through bake, mmap, and Vulkan upload. Clip maps and surfaces with authored
|
|
translucency still decode to RGBA8 because their per-surface alpha edits
|
|
require pixels. This is smaller and closer to retail's hardware DXT path
|
|
than the former unconditional software decode.
|
|
|
|
## Determinism and publication
|
|
|
|
Asset traversal and mesh serialization remain sorted. A texture is emitted at
|
|
its first deterministic encounter, so its physical order is independent of
|
|
worker completion order. Aliases preserve the source row's exact offset,
|
|
encoded length/flags, and CRC.
|
|
|
|
Recipe 5 to 6 is a mandatory full rebuild. The launcher builds
|
|
`acdream.pak.candidate` beside the active package, validates format, recipe,
|
|
DAT iterations, TOC counts, size, completion protocol, and SHA-256, then uses
|
|
the existing atomic promotion/backup transaction. Cancellation or failure
|
|
keeps the verified format-1 package. No overlay may cross this format change.
|
|
|
|
## Complete installed-DAT evidence
|
|
|
|
The first installed-DAT mixed sample (four GfxObj, three Setup, three EnvCell,
|
|
all corresponding collision/topology payloads) produced 58 keys, 29 globally
|
|
deduplicated texture payloads, and 57 physical blobs. Decoded payload was
|
|
3.5 MiB and stored payload 1.0 MiB (3.62x); output was 1.0 MiB. Eight-worker
|
|
and three-worker bakes had the identical SHA-256
|
|
`78886DFA28A3EDF9368A1E25C9B02A3B69ADC5DFCC01358D64A073B549B5B532`.
|
|
|
|
The final complete four-worker and nine-worker bakes are byte-identical:
|
|
|
|
- 2,237,866 logical keys and 786,892 physical blobs;
|
|
- 5,696 globally shared texture payloads;
|
|
- zero extraction/validation failures;
|
|
- 597,229,424 bytes (569.6 MiB), down 98.0% from 29,908,271,024 bytes;
|
|
- SHA-256
|
|
`37BC0EA1778F899AF9E3B2397937D373F69D615B15D3E32041BF389D93B624BF`;
|
|
- 28.7 seconds for the final four-worker bake versus 79.1 seconds for the
|
|
same-machine recipe-5 format-1 baseline (63.7% faster).
|
|
|
|
The exact before/after connected lifecycle route used the same machine,
|
|
server, 1280x720 Vulkan presentation, retail render pack, and matching client
|
|
code. Both runs passed fresh login, multi-world portal travel, same-location
|
|
revisit, fresh-process reconnect, and graceful teardown.
|
|
|
|
| Matching live measurement | Format 1 | Format 2 | Change |
|
|
|---|---:|---:|---:|
|
|
| Heavy-route final working set | 2,621.9 MiB | 1,363.9 MiB | -48.0% |
|
|
| Heavy-route final private bytes | 2,470.5 MiB | 1,825.6 MiB | -26.1% |
|
|
| Holtburg prepared-mesh GPU bytes | 229.6 MiB | 170.1 MiB | -25.9% |
|
|
| Fresh reconnect working set | 974.1 MiB | 869.0 MiB | -10.8% |
|
|
| Fresh reconnect private bytes | 1,253.8 MiB | 1,166.6 MiB | -7.0% |
|
|
| Fresh reconnect to checkpoint | 59.3 s | 59.3 s | equal |
|
|
|
|
Rynthid and Facility transition times were equal; Holtburg completed 2.9 s
|
|
faster; Aerlinthe revisit differed by 0.1 s. Matching stable CPU p50/p95 rows
|
|
were equal or slightly faster and GPU time remained within 0.1 ms. The first
|
|
raw login samples were excluded from comparison because the server started
|
|
them in different cells with 6,671 versus 11,799 world entities. Matching
|
|
screenshots preserve geometry, materials, lighting, transparency, texture
|
|
detail, and mip behavior.
|
|
|
|
Automated gates at this checkpoint: 181/181 installed-DAT Content tests,
|
|
16,151/16,151 CI-filtered Windows tests, Release build with zero warnings, and
|
|
the authoritative connected lifecycle/reconnect gate all pass.
|
|
|
|
## Release closeout
|
|
|
|
Main fast-forwarded cleanly to `45ba42a3`. Gitea Actions run 206 passed the
|
|
Windows gate, Linux portable gate, and release job. The published alpha is
|
|
`0.1.0-build.202608271848`; the stable `latest` pointer carries the 45 MiB
|
|
Windows client, 78 MiB Windows launcher, and update manifest.
|
|
|
|
The first remote run exposed one locale-only contract failure: the launcher's
|
|
disk-space guidance rendered `2,0 GiB` under the runner culture. The shipped
|
|
fix formats the value invariantly and exercises the failure path under
|
|
`sv-SE`; the exact follow-up local Release gate again passed 16,151/16,151.
|
|
The exact shipped App binary also passed the connected lifecycle/reconnect
|
|
route (`connected-world-gate-20260827-204249`) with only the 25 expected
|
|
world-edge misses.
|
|
|
|
## Work ledger
|
|
|
|
- [x] Measure the format-1 package by TOC partition.
|
|
- [x] Implement and unit-test format-2 external texture references, adaptive
|
|
independent compression, corruption handling, bounded sharing, and byte
|
|
determinism.
|
|
- [x] Integrate format-2 accounting and strict validation into the bake.
|
|
- [x] Publish the recipe-6 mandatory full-rebuild launcher migration.
|
|
- [x] Add launcher disk-space preflight and explicit long-work detail.
|
|
- [x] Complete installed-DAT equivalence and dual-worker full bakes.
|
|
- [x] Measure/tune package size, bake time, read CPU/allocations, cold
|
|
and warm reveal latency, and frame-time percentiles.
|
|
- [x] Evaluate source-native BC texture retention only if it remains visually
|
|
exact and does not shift mip-generation work into the reveal frame.
|
|
- [x] Pass complete local tests and authoritative connected gates.
|
|
- [x] Pass Gitea Windows CI, merge, push, and release gates.
|