diff --git a/CLAUDE.md b/CLAUDE.md index 96449ed..499f978 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,10 +8,17 @@ with a plugin API the original never had. **The code is modern. The behavior is retail.** -Every AC-specific algorithm is ported from the decompiled retail client -(`docs/research/decompiled/`, 22,225 functions, 688K lines of C). The code -around those algorithms is modern C# with clean architecture. The plugin API -exposes game state through well-defined interfaces. +Every AC-specific algorithm is ported from the **named retail decomp** +at `docs/research/named-retail/` — Sept 2013 EoR build PDB (18,366 +named functions, 5,371 named struct/class types) + Binary Ninja +pseudo-C with 99.6% function-name recovery + verbatim retail header +struct definitions. The older Ghidra `FUN_xxx` chunks under +`docs/research/decompiled/` (22,225 functions, 688K lines) remain a +fallback for chunk-by-chunk address-range navigation. **Grep +`named-retail/acclient_2013_pseudo_c.txt` by `class::method` BEFORE +decompiling fresh.** The code around those algorithms is modern C# +with clean architecture. The plugin API exposes game state through +well-defined interfaces. **Architecture:** `docs/architecture/acdream-architecture.md` is the single source of truth for how the client is structured. All work must @@ -84,19 +91,33 @@ a phase just landed, and move to the next todo item. always yes — keep going.** The single exception is visual verification; otherwise, act. -## Development workflow: decompile → verify → port +## Development workflow: grep named → decompile → verify → port **This is the mandatory workflow for implementing ANY AC-specific behavior.** The triangle-boundary Z bug cost 5 failed fix attempts from guessing. The animation frame-swap bug cost 4 failed attempts. Every time we checked the decompiled code first, we got it right on the first try. +**Now we have named retail symbols too — Step 0 cuts most lookups +from 30 minutes to 5 seconds.** ### For each new feature or bug fix: -1. **DECOMPILE FIRST.** Before writing any AC-specific code, find the - matching function in the decompiled client (`docs/research/decompiled/`) - or decompile a new region using `tools/decompile_acclient.py`. Use - the function map at `docs/research/acclient_function_map.md` to find +0. **GREP NAMED FIRST.** Before any decompilation work, search + `docs/research/named-retail/acclient_2013_pseudo_c.txt` by + `class::method` name. 99.6% of functions have real names from the + Sept 2013 EoR build PDB. `docs/research/named-retail/acclient.h` + has every retail struct verbatim. `docs/research/named-retail/symbols.json` + is greppable by name or address (regenerate via + `py tools/pdb-extract/pdb_extract.py refs/acclient.pdb`). Only fall + back to Step 1 below if the named pseudo-C lacks a function (rare — + covers only the obfuscated/packed minority). + +1. **DECOMPILE FIRST (fallback).** Only when grep-named-first returned + nothing. Find the matching function in the older Ghidra chunks at + `docs/research/decompiled/` or decompile a new region using + `tools/decompile_acclient.py`. Use the function map at + `docs/research/acclient_function_map.md` (cross-port index) + + `docs/research/named-retail/symbols.json` (raw PDB names) to find known functions. If the function isn't mapped yet, search by characteristic constants (motion commands, magic numbers, string literals). @@ -132,7 +153,11 @@ checked the decompiled code first, we got it right on the first try. ### What NOT to do: - **Do not guess** at AC-specific algorithms, formulas, constants, wire - formats, or coordinate conventions. Ever. + formats, or coordinate conventions. Ever. **The named retail decomp + has the answer for almost everything; guessing is no longer a + recoverable error, it's negligence.** If you can't find it in + `docs/research/named-retail/`, file a research note and ASK before + writing. - **Do not "fix" the decompiled code.** If the retail client does something that looks wrong, it's probably right. Verify before changing. @@ -149,7 +174,8 @@ checked the decompiled code first, we got it right on the first try. Before marking any phase as done: - [ ] Every AC-specific algorithm has a decompiled reference cited in - comments (function address + chunk file) + comments (named symbol + address from `named-retail/symbols.json`, + OR function address + chunk file from older `decompiled/` chunks) - [ ] Conformance tests exist for the critical paths - [ ] The code was cross-referenced against at least 2 reference repos - [ ] `dotnet build` green, `dotnet test` green @@ -430,6 +456,7 @@ decompiled client code and would have fixed it in minutes. | Domain | Primary Oracle | Secondary | Notes | |--------|---------------|-----------|-------| +| **Any AC-specific algorithm** | **`docs/research/named-retail/`** (PDB-named decomp + verbatim retail header structs from Sept 2013 EoR build) | the existing references below | The retail client itself, fully named. 18,366 functions + 5,371 struct types + 1.4 M lines of pseudo-C in one searchable tree. Beats every other reference for "what does the real client do." | | **Terrain** (split direction, height sampling, palCode, vertex position, normals) | **ACME `ClientReference.cs`** — decompiled retail client with exact offsets | ACME `TerrainGeometryGenerator.cs` (matches the mesh index buffer) | WorldBuilder original is SUPERSEDED for terrain algorithms. AC2D confirms the same formula. | | **Terrain blending** (texture atlas, alpha masks, road overlays) | **ACME `LandSurfaceManager.cs`** | WorldBuilder original `LandSurfaceManager.cs` (same code, less tested) | Both use the same TexMerge pipeline. ACME has conformance tests. | | **GfxObj / Setup rendering** (mesh extraction, multi-part assembly, ObjDesc) | **ACME `StaticObjectManager.cs`** — includes CreaturePalette, GfxObjRemapping, HiddenParts | ACViewer `Render/` namespace | ACME has the complete creature appearance pipeline in one file. | diff --git a/docs/architecture/acdream-architecture.md b/docs/architecture/acdream-architecture.md index 80536a1..56b888d 100644 --- a/docs/architecture/acdream-architecture.md +++ b/docs/architecture/acdream-architecture.md @@ -17,10 +17,15 @@ A modern C# .NET 10 Asheron's Call client that: **The code is modern. The behavior is retail.** -Every AC-specific algorithm is ported faithfully from the decompiled retail -client (docs/research/decompiled/, 688K lines). The code AROUND those -algorithms is modern C# with clean architecture. The plugin API exposes -game state through well-defined interfaces that the retail client never had. +Every AC-specific algorithm is ported faithfully from the **named retail +decomp** at `docs/research/named-retail/` — Sept 2013 EoR build PDB +(18,366 named functions, 5,371 named struct types) + Binary Ninja +pseudo-C with 99.6% function-name recovery + verbatim retail header +struct definitions. The older Ghidra `FUN_xxx` chunks at +`docs/research/decompiled/` (688K lines) remain a fallback for the +obfuscated/packed minority. The code AROUND those algorithms is modern +C# with clean architecture. The plugin API exposes game state through +well-defined interfaces that the retail client never had. --- diff --git a/docs/plans/2026-04-11-roadmap.md b/docs/plans/2026-04-11-roadmap.md index aabe568..cea4b88 100644 --- a/docs/plans/2026-04-11-roadmap.md +++ b/docs/plans/2026-04-11-roadmap.md @@ -223,6 +223,41 @@ Not detailed here; each gets its own brainstorm when it becomes relevant. --- +### Phase R — Retail research infrastructure + +**Goal:** sustainable, scalable access to retail-client decompilation — +named symbols, struct layouts, wire schemas — so every future port is +a 5-second grep instead of 30-minute archaeology. + +**Sub-pieces:** + +- **✓ SHIPPED — R.1 — Named-retail corpus committed.** Shipped 2026-04-25 + (commit `a9a01d8`). `docs/research/named-retail/{acclient_2013_pseudo_c.txt, + acclient.h, acclient.c}` + `docs/research/data/spells.csv` + vendored + `references/actestclient/`. 1.4 M lines of named pseudo-C (99.6% function + naming) + 70K lines of verbatim retail headers + 3,956 spells with `Family` + for buff stacking + machine-readable wire schema in `messages.xml`. +- **✓ SHIPPED — R.2 — pdb-extract tool + JSON sidecars.** Shipped 2026-04-25 + (commit `69d884a`). `tools/pdb-extract/pdb_extract.py` reads + `refs/acclient.pdb` (Sept 2013 EoR build) and emits `symbols.json` + (18,366 named functions) + `types.json` (5,371 named struct types) to + `docs/research/named-retail/`. Pure Python, no deps, runs in <1 s. +- **✓ SHIPPED — R.3 — actestclient vendored.** Shipped 2026-04-25 alongside + R.1. `references/actestclient/` (covered by `references/` gitignore) + includes the canonical `messages.xml` AC-protocol wire schema. + +**Acceptance:** Step 0 of `CLAUDE.md`'s development workflow points at +`named-retail/` first; subsequent issue closures (#6 / #7 / #9 / #11) +all consume this foundation. + +**Effects on other phases:** Issue closures unblocked by Phase R land +under their existing letter phases (#6 enchantment buffs → Phase D / +F.5 maintenance; #7 PlayerDescription trailer → Phase H.1 / F.2 +maintenance). The PDB symbols + headers also accelerate any future +port in any phase — no separate listing here. + +--- + ## Cross-cutting work tracked in parallel - **Test coverage.** Each phase lands with unit + integration tests in `tests/`. Current count: 98 Core + 96 Core.Net = 194. Keep the ratio as new phases land. diff --git a/memory/project_collision_port.md b/memory/project_collision_port.md index c94771d..dc1fcd1 100644 --- a/memory/project_collision_port.md +++ b/memory/project_collision_port.md @@ -62,10 +62,19 @@ ground truth. ACE has the ENTIRE system already in C#: - `references/ACE/Source/ACE.Server/Physics/Collision/CollisionInfo.cs` - `references/ACE/Source/ACE.Server/Physics/Collision/ObjectInfo.cs` -### Decompiled ground truth: -- `docs/research/decompiled/chunk_00530000.c` — BSP, Polygon, Sphere collision -- `docs/research/decompiled/chunk_00500000.c` — PhysicsObj, transition callers -- `docs/research/acclient_function_map.md` — mapped functions +### Decompiled ground truth (named-retail is now primary, 2026-04-25): +- **`docs/research/named-retail/acclient_2013_pseudo_c.txt`** — grep for + `BSPTree::`, `BSPNode::`, `BSPLeaf::`, `CPolygon::`, `CCylSphere::`, + `Transition::`, `CPhysicsObj::`, `SpherePath::` to find named bodies. +- **`docs/research/named-retail/acclient.h`** — verbatim retail struct + layouts for the BSP / Sphere / Transition types. +- **`docs/research/named-retail/symbols.json`** — name↔address lookup. +- `docs/research/decompiled/chunk_00530000.c` — older Ghidra fallback for + BSP / Polygon / Sphere collision (FUN_xxx names). +- `docs/research/decompiled/chunk_00500000.c` — older Ghidra fallback for + PhysicsObj / transition callers. +- `docs/research/acclient_function_map.md` — hand-curated cross-port index + (ACE / ACME mappings + struct-offset notes). ### Pseudocode (already written): - `docs/research/transition_pseudocode.md` — full system documented @@ -90,10 +99,12 @@ ground truth. ACE has the ENTIRE system already in C#: For EVERY function: -1. **DECOMPILE FIRST.** Find the matching function in the decompiled - client (`docs/research/decompiled/`). Use the function map at - `docs/research/acclient_function_map.md`. If not mapped, search - by characteristic constants or struct offsets. +1. **GREP NAMED FIRST, then DECOMPILE FALLBACK.** Search the named + retail decomp first: `grep -n "ClassName::Method" docs/research/named-retail/acclient_2013_pseudo_c.txt`. + For struct layouts: `grep -n "^struct ClassName" docs/research/named-retail/acclient.h`. + Only if the named pseudo-C lacks a function (rare), fall back to the + older `docs/research/decompiled/` chunks via the function map at + `docs/research/acclient_function_map.md`. 2. **CROSS-REFERENCE ACE.** Read ACE's C# port of the same function. ACE provides naming and structure. Note any differences. diff --git a/memory/project_named_decompilation.md b/memory/project_named_decompilation.md new file mode 100644 index 0000000..8c12f5f --- /dev/null +++ b/memory/project_named_decompilation.md @@ -0,0 +1,84 @@ +# Project: named-retail decompilation foundation + +**Agreed 2026-04-25.** Read once per session. Foundation commits: +`a9a01d8` (corpus), `69d884a` (pdb-extract tool). + +## What changed + +The retail-client decompilation has gone from "FUN_xxx Ghidra chunks + +71 hand-mapped functions" to **"18,366 named retail functions + 5,371 +named struct types + verbatim retail headers"**, all committed under +`docs/research/named-retail/`. The `acclient.pdb` (Sept 2013 EoR build, +MSVC 7.00) is the source of truth. + +## Files at `docs/research/named-retail/` + +| File | What | When to use | +|---|---|---| +| `acclient_2013_pseudo_c.txt` (62 MB / 1.44 M lines) | Binary Ninja pseudo-C with 99.6% naming. Address-prefixed lines. | **Primary symbol lookup.** Grep by `class::method`. | +| `acclient.h` (1.7 MB / 70K lines) | IDA-decompiled retail headers. Every struct verbatim. | **Struct field names + offsets.** Grep by struct name. | +| `acclient.c` (46 MB / 1.3 M lines) | IDA full-binary export. Mixed FUN_/named, but has named struct fields. | **Cross-reference when pseudo-C body is corrupt.** | +| `symbols.json` (3 MB) | 18,366 entries: `address` + `name` (demangled `Class::Method`) + `mangled` (raw MSVC ABI). | **Programmatic name↔address lookup via jq.** | +| `types.json` (506 KB) | 5,371 named class/struct records with size + kind. | **Programmatic type-layout queries.** | + +## Workflow change — STEP 0 GREP NAMED FIRST + +`CLAUDE.md` workflow now starts with **Step 0** before the older +DECOMPILE step: + +```bash +# Find a function body +grep -n "CEnchantmentRegistry::EnchantAttribute" \ + docs/research/named-retail/acclient_2013_pseudo_c.txt + +# Find a struct +grep -n "^struct.*CEnchantmentRegistry" \ + docs/research/named-retail/acclient.h + +# Programmatic +cat docs/research/named-retail/symbols.json | \ + jq '.[] | select(.name == "CEnchantmentRegistry::EnchantAttribute")' +``` + +The older `docs/research/decompiled/` Ghidra chunks remain a fallback +for the obfuscated/packed minority that pseudo-C lacks. ~5 second +grep replaces ~30 minute archaeology for typical lookups. + +## Hard rules + +1. **Grep `named-retail/` first.** Always. The "Do not guess" rule in + CLAUDE.md is upgraded: with the PDB available, guessing is no + longer recoverable error — it's negligence. +2. **Match by name, not raw address.** The PDB build has a ~0xC00 + byte delta vs the binary that produced our older chunks. Address + lookups in `symbols.json` are correct; the older + `acclient_function_map.md` had several mid-body addresses (issue + #9 sweep corrected them). +3. **Regenerate JSON sidecars when needed:** + ```powershell + py tools\pdb-extract\pdb_extract.py refs\acclient.pdb + ``` +4. **`refs/` stays gitignored.** It's per-developer download cache; + the committed extracts at `docs/research/named-retail/` are what + subagents and post-compaction sessions inherit. + +## Counts (Sept 2013 EoR build) + +- 18,366 named public function symbols (`S_PUB32`, code flag set) +- 5,371 unique named class/struct types (filtered for non-forward-declared) +- 1,053 .obj compilation units recorded in DBI +- Build root preserved in PDB: `d:\ac1_sep13\` + +## Why this matters + +Closes / unblocks issues #6 (enchantment buffs in vital max — uses +`CEnchantmentRegistry::EnchantAttribute` at `0x594570`), #7 +(PlayerDescription trailer — uses `CPlayerSystem::Handle_PlayerDescription` +at `0x5636A0`), plus reduces the cost of every future port from "30 +min archaeology" to "5 sec grep". + +## Links + +- Plan: `C:/Users/erikn/.claude/plans/ticklish-conjuring-cake.md` (8-phase foundation) +- Tool: `tools/pdb-extract/` (pure Python, no deps) +- Reference: `docs/research/named-retail/README.md` diff --git a/memory/project_retail_research_index.md b/memory/project_retail_research_index.md index bdc21db..c48e80c 100644 --- a/memory/project_retail_research_index.md +++ b/memory/project_retail_research_index.md @@ -8,6 +8,13 @@ all Opus-4.7 high-effort). subsystem. Open the slice for that subsystem BEFORE writing the first line — every slice has pseudocode + C# port sketch + citations. +> **2026-04-25 update — named-retail foundation.** First stop for any +> AC-specific question is now `docs/research/named-retail/` (PDB-named +> Sept 2013 EoR decomp; 18,366 named functions, 5,371 named struct +> types, 1.4 M lines of pseudo-C). The slices below remain useful for +> their pseudocode + C# port sketches and the deeper research notes +> they synthesize. See `memory/project_named_decompilation.md`. + --- ## UI layer (`docs/research/retail-ui/`) — 6 slices, ~30,000 words