Wires the validated item-processor into the ingestion endpoints, writing to an
isolated inventory-go-db (never production):
- schema.go: faithful 7-table replica of inventory-service/database.py.
- ingest.go: /process-inventory (full replace), POST/DELETE single item, with the
exact delete-then-insert flow, dynamic INSERT builder (quotes reserved "unique"),
spell union (is_active), and item_raw_data verbatim. enhancements always inserts.
- compose: isolated inventory-go-db (postgres:14, 127.0.0.1:5435) + read-write
inventory-go-shadow (:8773) that owns it; schema init on boot.
Validated by ingesting a recently-ingested character's items (from production's
original_json) into the shadow DB and diffing vs production: byte-identical —
items 243, combat 243, enhancements 243, ratings 6, requirements 19, spells 52
all match; 0 per-column mismatches across 243 items.
Finding: older production normalized rows can be STALE (predate the code reading
Decal keys 218103832/218103835); Go matches the CURRENT Python code, so validate
ingestion against recently-ingested characters.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First slice of the inventory-service port, running in parallel READ-ONLY against
the production inventory_db (never written):
- main.go/store.go: pgx pool (forced read-only), enum-DB loader extracting
AttributeSetInfo for set-name resolution, /health, /sets/list, /characters/list.
- Dockerfile + compose service inventory-go (127.0.0.1:8772, enum JSON mounted).
Validated vs the Python service on the same DB: /characters/list 167 chars exact
counts; /sets/list 76 sets EXACT match (ids, names, counts).
Remaining (large): /search/items (40+ filters + enrich_db_item), inventory
fetch, item-processing ingestion (extract_item_properties), and the suitbuilder
solver.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Completes the Go tracker as a cutover-ready drop-in:
- wslive.go: browser broadcast hub with per-client subscribe filters (nil=all),
request_dungeon_map replies, and command routing; auth = internal-trust or
session cookie. The ingestor broadcasts every handled event to it.
- wsposition.go: plugin ingest server with X-Plugin-Secret/SHARED_SECRET auth
(constant-time, fails closed, legacy fallback), register -> plugin_conns, and
dispatch into the shared Ingestor. plugin registry for backend->plugin commands.
- main.go: statusRecorder.Unwrap() so coder/websocket can hijack through the
logging middleware (WS handshakes failed without it); /ws/ bypasses HTTP auth.
Shadow consumer robustness (the harness was being evicted under the full
firehose): decouple socket read from processing — the read loop only copies raw
frames to a queue; a worker unmarshals + dispatches. JSON parsing in the read
loop was slowing it enough that Python's broadcast send errored and evicted us
(Read then blocked forever). Added a 25s read-deadline watchdog to self-heal.
Validated live: shadow /live online = 73 = production; telemetry sustained ~12/s,
0 drops, no eviction; and the shadow's /ws/live re-broadcast stream is IDENTICAL
to production's (TOTAL 2150=2150, every event type exact).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implements the plugin event handlers (the /ws/position write logic) as a shared
Ingestor, validated against real traffic by replaying Python's /ws/live firehose
into an isolated dereth_go DB (no production write, no plugin stolen).
- ingest.go: faithful ports of telemetry (kill-delta -> char_stats, server
received_at stamp), rare (rare_stats/rare_stats_sessions/rare_events), portal
(coord upsert), character_stats (stats_data JSONB subset + upsert), spawn, and
the memory-only handlers (vitals/quest/equipment_cantrip/nearby/dungeon). In
-memory live state + read-side overlay accessors.
- shadow.go: coder/websocket consumer of /ws/live -> Ingestor.dispatch (telemetry
matched by shape since its broadcast has no type field).
- main.go/store.go: ingest mode (READ_ONLY=false + SHADOW_INGEST_WS) wires the
ingestor; read handlers (/character-stats, /equipment-cantrip, /quest-status)
now consult the live overlay first, like Python.
- compose: shadow instance ingests ws://dereth-tracker:8765/ws/live.
Validated live: dereth_go has 73 distinct telemetry chars; shadow /live online
set == production (73=73); character_stats 5/5 exact byte-match (0 mismatch);
char_stats kill-deltas + portals accumulating. compare/compare_ingest.py.
Deferred to next pass: combat_stats (delta/merge), share_*, the /ws/position +
/ws/live servers (for cutover).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Stands up the shadow-ingest substrate without touching production:
- schema.go: faithful replica of db_async.init_db_async (idempotent DDL),
run only when an instance OWNS its DB (READ_ONLY=false). Fixes for a fresh DB:
spawn_events has no sole-id PK (so it can be a hypertable), telemetry_events
compression is enabled before its policy, and the portal unique index uses
ROUND(..,1) to match main.py's ON CONFLICT. 35/35 statements OK.
- store.go: read-only transaction enforcement is now conditional (on for
production read parity, off for ingest).
- main.go: READ_ONLY + SHADOW_INGEST_WS config; schema init on boot when owning
the DB.
- compose override: a SEPARATE TimescaleDB `dereth-go-db` (isolated volume,
127.0.0.1:5434) and a `dereth-tracker-go-shadow` instance (image reused via
dereth-tracker-go:local) that owns it. Production DB never written.
Verified: dereth_go has all 13 tables; telemetry_events + spawn_events are
hypertables; the read-side instance still serves production read-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Consumes the Python tracker's /ws/live firehose (subscribed to rare+chat),
classifies rares common/great, posts embeds + relays allegiance chat.
- classify.go: the 74-name common-rares set, extracted verbatim from the Python
COMMON_RARES_PATTERN (not hand-transcribed). go test runs at build time; a
server-side dump-rares vs the Python regex confirms the sets are IDENTICAL.
- poster.go: a `poster` interface with a real discordgo impl (REST sends by
channel id; gold/blue embeds, location/time fields, icon attachment) and a
dry-run log impl.
- ws.go: coder/websocket client to /ws/live with subscribe, ping keepalive,
exponential-backoff reconnect; rare/chat dispatch incl. vortex-warning + the
MONITOR_CHARACTER filter.
- SAFE BY DEFAULT: dry-run unless a token AND DRY_RUN=0 are set, so it can never
double-post to production. Deployed via the compose override
(discord-rare-monitor-go), running dry-run against the same live firehose.
Validated on the server: connects, subscribes, relays a real chat in dry-run;
classifier parity 74/74 vs the Python regex.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replicates main.py's AuthMiddleware so /go/ can be exposed safely:
- internal-trust: private source IP AND no X-Forwarded-For => skip auth
(loopback/compose callers; nginx adds XFF to all internet traffic).
- session cookie: byte-compatible itsdangerous URLSafeTimedSerializer verify
(HMAC-SHA1, django-concat key derivation sha1("itsdangerous"+"signer"+key),
Unix-epoch timestamp, urlsafe-b64 no pad, optional zlib payload), keyed on the
same SECRET_KEY. 30-day max-age. Public allowlist (/login,/logout,login assets,
/icons/,/health); 302->/login for html, 401 JSON otherwise.
Validated on the server: internal-trust loopback 200; external no-cookie 401;
html 302; valid cookie 200; tampered 401; /health public 200; and the SAME
Python-issued cookie authenticates BOTH services (cross-compat proof).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the rest of the read-side endpoints to the Go tracker, all parity-checked
against the live Python service:
- DB reads: /stats/{c}, /portals, /spawns/heatmap, /server-health,
/character-stats/{c} (stats_data JSONB merged to top level),
/combat-stats[/{c}], /inventories, /inventory/{c}/search.
- 5-minute totals cache + /total-rares, /total-kills.
- Ingest-only state returned as Python's empty/default shapes (/quest-status,
/vital-sharing/peers, /equipment-cantrip-state/{c}); /issues (flat file),
/me (401 until cookie verification lands).
- Streaming reverse proxy to inventory-service (/inventory/{c},
/inventory-characters, /search/*, /sets/list, /inv/{path...} incl. the SSE
suitbuilder stream).
- compare/compare_endpoints.py: structural parity for all read endpoints +
exact-match check for /character-stats and /combat-stats on OFFLINE chars
(online chars legitimately differ — Python serves a richer live overlay that
Phase-1 Go lacks until ingest).
Verified live: 14/14 endpoints structural-match, 8/8 rich offline chars
exact-match on /character-stats.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Parallel Go reimplementation of the dereth-tracker read side, deployed
loopback-only (:8770) and reading the dereth TimescaleDB read-only. The live
Python stack is untouched (added via a compose override, not by editing the
tracked docker-compose.yml).
- Phase 0 scaffold: stdlib net/http server (Go 1.22+ method+path routing),
/health + /api-version, multi-stage distroless Docker build, and
go-services/docker-compose.go.yml override (loopback :8770).
- Phase 1: pgx v5 pool forced into read-only transactions, a 5s /live + /trails
cache loop using the exact main.py:837 SQL, and Python-isoformat timestamps
so output matches FastAPI's jsonable_encoder.
- compare/compare_live.py: parity harness vs the live Python service. Uses the
server-stamped received_at to prove same-row full-field equality and to make
the online-set diff boundary-aware.
Verified on live traffic (73 players): identical online set + 23-key schema,
identity/type parity for all, every same-row pair matches on every field, and
diff-row pairs differ only by the ~6s two-cache refresh skew.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
