diff --git a/.gitignore b/.gitignore index 0696fc7f..b523c6d7 100644 --- a/.gitignore +++ b/.gitignore @@ -3,6 +3,12 @@ __pycache__ static/v2/ frontend/node_modules/ +# Secrets — the server-side env files hold SHARED_SECRET, SECRET_KEY, DB +# passwords, and the Discord token. This repo is PUBLIC — never commit them. +# .env.example stays tracked as the template. +.env +.env.bak-* + # Claude Code config — never commit. The production agent's strict # permissions live server-side at /var/lib/overlord-agent/.claude/ # (and via CLI flags in agent/claude_wrapper.py). The repo stays diff --git a/CLAUDE.md b/CLAUDE.md index 3fa68e55..e825bcb9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -5,22 +5,41 @@ Cross-repo workflows (plugin coupling, deploy commands, nginx) live in the works ## Project Overview -Dereth Tracker is a real-time telemetry platform for Asheron's Call world tracking. A FastAPI WebSocket/HTTP service (`main.py`, single file ~4200 lines) ingests player data from the MosswartMassacre DECAL plugin and serves a live React dashboard, with TimescaleDB persistence, a separate inventory microservice, Grafana dashboards, a Discord rare bot, and a host-side Claude-powered assistant. +Dereth Tracker is a real-time telemetry platform for Asheron's Call world tracking. **The production backend is Go** (`go-services/`): a tracker service (`tracker-go/`) ingests player data from the MosswartMassacre DECAL plugin over `/ws/position`, serves the React dashboard + login/admin + the read API, and writes TimescaleDB; an inventory service (`inventory-go/`) handles item search, the suitbuilder solver, and inventory ingestion. Plus Grafana, a (Python) Discord rare bot, and a host-side Claude-powered assistant. + +The original Python/FastAPI implementation (`main.py` ~4200 lines, `inventory-service/`) is preserved on the **`python-legacy`** branch; the Go services were validated byte-identical against it in a parallel "strangler-fig" run, then production was cut over. ⚠ **The behavioral contracts below (WS, auth, DB, routes, suitbuilder) describe what Go honors. Where they cite `main.py` / `inventory-service/`, that's the legacy source that defined the contract — the live implementation is the corresponding Go handler.** ## Components | Component | Where | Runs as | |---|---|---| -| Tracker API (`main.py`) | repo root | Docker `dereth-tracker`, 127.0.0.1:8765 | -| Telemetry DB (TimescaleDB) | `db_async.py` schema | Docker `dereth-db`, port 5432 | -| Inventory service + DB | `inventory-service/` | Docker `inventory-service` (127.0.0.1:8766) + `inventory-db` (5433) | -| React frontend | `frontend/` → built into `static/` | served by tracker (FastAPI StaticFiles) | -| Classic v1 frontend | `static/classic/` | served at `/classic` | -| Legacy vanilla pages | `static/inventory.html`, `static/suitbuilder.html` | still live | +| **Tracker** (ingest + website + read API + WS) | `go-services/tracker-go/` | Docker `dereth-tracker-go`, 127.0.0.1:8770 | +| **Inventory** (search + suitbuilder + ingestion) | `go-services/inventory-go/` | Docker `inventory-go`, 127.0.0.1:8772 | +| Telemetry DB (TimescaleDB) | schema in `tracker-go/schema.go` (replica of legacy `db_async.py`) | Docker `dereth-db`, port 5432 | +| Inventory DB | schema in `inventory-go/schema.go` | Docker `inventory-db`, 5433 | +| React frontend | `frontend/` → built into `static/` | served by `tracker-go` (static file server, SPA fallback) | +| Classic v1 / legacy pages | `static/classic/`, `static/*.html` | served by `tracker-go` | | Grafana | compose service `dereth-grafana` | 127.0.0.1:3000, anonymous Viewer auth, proxied at `/grafana/` | -| Discord rare bot | `discord-rare-monitor/` | Docker, connects to `/ws/live` internally | +| Discord rare bot | `discord-rare-monitor/` (Python) | Docker, reads the Go `/ws/live` | | Overlord Agent (assistant) | `agent/` | **host-side systemd service** `overlord-agent`, 127.0.0.1:8767 | +### Go services — build, deploy, gotchas + +- **Build on the server, no host Go needed** (multi-stage distroless images). Go 1.25, `pgx/v5`, `coder/websocket`, `bwmarrin/discordgo`, `x/crypto/bcrypt`. Sync + build + recreate: + ```bash + tar czf - go-services | ssh erik@overlord.snakedesert.se "tar xzf - -C /home/erik/MosswartOverlord/" + ssh erik@overlord.snakedesert.se 'cd /home/erik/MosswartOverlord && \ + export BUILD_VERSION="$(date -u +%Y.%-m.%-d.%H%M)-$(git rev-parse --short HEAD)" && \ + docker compose -f docker-compose.yml -f go-services/docker-compose.go.yml build dereth-tracker-go inventory-go && \ + docker compose -f docker-compose.yml -f go-services/docker-compose.go.yml -f go-services/docker-compose.cutover.yml \ + up -d --no-deps dereth-tracker-go inventory-go' + ``` +- **`docker-compose.cutover.yml`** is what makes the Go services production: `READ_ONLY=false` (write the prod DBs), `SKIP_SCHEMA_INIT=true` (trust the existing schema, run NO DDL), `SHARED_SECRET`/`DISCORD_ACLOG_WEBHOOK` for the tracker, and the Discord bot repointed at `ws://dereth-tracker-go:8770/ws/live`. Drop it to revert to read-only parallel mode. +- **Rollback** = `docker compose ... up -d` WITHOUT the cutover override (Go → read-only) + start the Python `dereth-tracker`/`inventory-service` + revert the nginx `http://tracker_go/` lines to `http://tracker/`. +- ⚠ **Plugin sends some numeric fields as STRINGS** (`kills_per_hour`, `deaths`, `total_deaths`, `prismatic_taper_count`). Go coerces via `coerceNum` (`tracker-go/reads.go`) — pydantic did this implicitly; a plain number cast would write null/0. +- ⚠ **Telemetry must be broadcast TYPELESS** to `/ws/live` (`stripType` in `tracker-go/ingest.go`). The browser ignores typeless messages and uses the 5 s `/live` poll for player data; broadcasting telemetry WITH a type makes the UI overwrite the /live-derived counters and flap them 0↔value. +- ⚠ `inventory-go` `slot_names=Trinket` must exclude `%bracelet%` or bracelets duplicate the Wrist buckets in the suitbuilder. + ## WebSocket endpoints - `/ws/position` — plugin ingest (telemetry, inventory, portal, rare, combat, share_*, …). Authenticated by `X-Plugin-Secret` header against the `SHARED_SECRET` env var; fails closed (refuses all plugins) when unset or left at the old placeholder. Constant-time compare. @@ -63,12 +82,14 @@ Dereth Tracker is a real-time telemetry platform for Asheron's Call world tracki ## Suitbuilder -Production equipment-optimization engine (`inventory-service/suitbuilder.py`): multi-character search, armor set constraints, cantrip overlap detection, SSE streaming. UI at `/suitbuilder.html`. Architecture doc: `docs/plans/2026-02-09-suitbuilder-architecture.md`. -Known limitations: no slot-aware spell filtering, equal spell weighting. The legacy `/optimize/*` solver in inventory-service/main.py is a near-duplicate — `suitbuilder.py` is the production path. +Production equipment-optimization engine, ported to Go in `inventory-go/suit_*.go` (constraint-satisfaction DFS: multi-character search, armor set constraints, cantrip overlap, SSE streaming) — validated byte-identical against the legacy `inventory-service/suitbuilder.py`. Live endpoint: `POST /suitbuilder/search` (the tracker proxies `/inv/suitbuilder/search`); the `/optimize/*` solver in the legacy `inventory-service/main.py` was a near-duplicate and is NOT the live path. UI at `/suitbuilder.html`. Known limitations: no slot-aware spell filtering, equal spell weighting. ## Deploying -See workspace `../CLAUDE.md` "Build & Deploy Instructions" — quick deploy (git pull + `docker compose restart dereth-tracker` for Python; nothing for static), `deploy-frontend.sh` for React, full `--no-cache` rebuild only for Dockerfile/pip/version-stamp changes. Bind mounts: `main.py`, `db_async.py`, `static/`, `alembic/` only. +- **Go backend changes** → see "Go services — build, deploy, gotchas" above (sync `go-services/`, build, recreate with the cutover override). `BUILD_VERSION` (CalVer `YYYY.M.D.HHMM-gitshorthash`) shows in the frontend sidebar. +- **Frontend** → `bash deploy-frontend.sh` (complete build+copy into `static/`); the tracker serves `static/` from a bind mount, no restart needed. +- **Overlord Agent** → unchanged (host-side Python systemd): `git pull && sudo systemctl restart overlord-agent`. +- `README.md` has the full build/run reference. The legacy Python deploy lives on the `python-legacy` branch. ## Operational notes diff --git a/README.md b/README.md index 0864e4d0..381cd775 100644 --- a/README.md +++ b/README.md @@ -1,424 +1,155 @@ # Mosswart Overlord (Dereth Tracker) -Real-time telemetry, inventory, and analytics platform for Asheron's Call. -FastAPI backend + React frontend + PostgreSQL (TimescaleDB) + Discord integrations, -all driven by WebSocket events from the companion [MosswartMassacre](https://github.com/SawatoMosswartsEnjoyersClub/MosswartMassacre) DECAL plugin. +Real-time telemetry, inventory, and analytics platform for Asheron's Call — +driven by a firehose of WebSocket events from the companion +[MosswartMassacre](https://github.com/SawatoMosswartsEnjoyersClub/MosswartMassacre) +DECAL plugin running on 60+ characters. + +**The production backend is written in Go** (`go-services/`). It replaced the +original Python/FastAPI implementation via a strangler-fig migration: the Go +services ran in parallel against live traffic until every endpoint was proven +byte-identical, then production was cut over. The Python implementation is +preserved on the `python-legacy` branch. --- -## Table of Contents -- [Overview](#overview) -- [Architecture](#architecture) -- [Features](#features) -- [Requirements](#requirements) -- [Installation](#installation) -- [Configuration](#configuration) -- [Deploying Changes](#deploying-changes) -- [WebSocket Contract](#websocket-contract) -- [HTTP API Reference](#http-api-reference) -- [Frontend](#frontend) -- [AI Assistant (Overlord Agent)](#ai-assistant-overlord-agent) -- [Database Schema](#database-schema) -- [Operations & Health](#operations--health) -- [Contributing](#contributing) - ---- - -## Overview - -Mosswart Overlord is the backend that consumes a firehose of telemetry, vitals, inventory, combat, and chat events from 60+ characters running the `MosswartMassacre` plugin. It stores selected data in TimescaleDB, runs analytics (combat stats, idle/death detection), and broadcasts live updates to connected browser clients. - -The frontend is a React + Vite app served at `/` with a live map, draggable windows (inventory, chat, combat, radar, etc.), and a server uptime sidebar. The previous vanilla JS frontend is preserved at `/classic`. - ## Architecture ``` - ┌─────────────────────────┐ - │ MosswartMassacre (C#) │ ← plugin per game client - └────────────┬────────────┘ - │ WebSocket /ws/position (authenticated) - ▼ -┌────────────────────────────────────────────────────────┐ -│ dereth-tracker (FastAPI, Docker) │ -│ • main.py — WS routing, analytics, broadcasts │ -│ • idle/death detection → Discord webhook │ -│ • combat stats delta/lifetime accumulation │ -│ • vital sharing relay (cross-machine) │ -└──┬──────────────────┬────────────────────┬────────────┘ - │ │ │ - │ WS /ws/live │ HTTP │ HTTP - ▼ ▼ ▼ -┌──────────┐ ┌──────────────────┐ ┌──────────────────┐ -│ Browsers │ │ inventory-svc │ │ Discord bot │ -│ (React) │ │ (FastAPI, Docker)│ │ (rare monitor) │ -└────┬─────┘ └────────┬─────────┘ └──────────────────┘ - │ ▼ - │ ┌──────────────┐ - │ │ inventory-db │ - │ └──────────────┘ - │ - │ /api/agent/* (host-side, OUTSIDE Docker) - ▼ -┌────────────────────────────────────────┐ -│ overlord-agent (FastAPI, systemd) │ ← runs as dedicated unprivileged user -│ • shells out to `claude -p ...` │ /var/lib/overlord-agent home, -│ • MCP server: live-state Q&A tools │ strict settings, no /home/erik -└────────────────────────────────────────┘ - - ┌──────────────┐ - │ dereth-db │ ← TimescaleDB (telemetry, spawns, rares, portals) - └──────────────┘ + MosswartMassacre plugin ──wss──> nginx ──> Go tracker (tracker-go) ──> dereth (TimescaleDB) + (60+ game clients) │ │ + │ ├──HTTP──> Go inventory (inventory-go) ──> inventory_db + Browsers ──https──────────────────> nginx │ + │ └──/ws/live──> Discord rare bot (relays rares + chat) + └──> Grafana (/grafana/) death/idle alerts → Discord webhook ``` -Most services run via Docker Compose. **`overlord-agent` is host-side** -(systemd) because it shells out to the `claude` CLI which depends on -host-side credentials — see [AI Assistant](#ai-assistant-overlord-agent). - -## Features - -### Live Data -- **Live Map** — real-time player positions, dots, trails, portals, heatmap -- **WebSocket firehose** (`/ws/live`) — broadcasts every incoming event to browsers -- **Per-client subscriptions** — clients can send `{"type":"subscribe","message_types":[...]}` to receive only specific event types (the Discord rare monitor bot uses this to filter the 82GB/day firehose down to just `rare` and `chat`) - -### Inventory -- Full inventory snapshot on login + incremental `inventory_delta` updates (add/update/remove) -- Per-character live refresh in the browser (debounced 2s) -- Advanced search with filters: material, set, armor level, spells, tinks, workmanship, etc. -- **Suitbuilder** at `/suitbuilder.html` — constraint-based armor optimization across multiple mule inventories with primary/secondary set support, cantrip overlap detection, and real-time SSE streaming - -### Combat Stats (Mag-Tools style) -- Plugin parses combat chat into session deltas -- Backend accumulates lifetime totals from per-session snapshots -- Offense/defense broken out per damage element -- Browser combat window shows monster-by-monster damage - -### Cross-Machine Vital Sharing -- WebSocket relay replaces UtilityBelt's localhost-only `VTankFellowHeals` -- Plugin broadcasts its own vitals and consumes peer vitals -- In-game `DxHud` overlay shows peer health/stamina/mana bars with direction arrows - -### AI Assistant -- 🤖 chat window in the dashboard backed by `claude -p` running headless on the server -- Read-only access to live game state via 12 MCP tools (live players, inventory cross-search, combat stats, quests, suitbuilder, read-only SQL, etc.) -- Per-browser persistent session, "New Chat" button, history rehydration on reload -- Hardened: dedicated unprivileged Linux user, systemd lockdown, strict tool whitelist, audit log, rate limit. See [AI Assistant section](#ai-assistant-overlord-agent) for the full security stack. - -### Discord Integration -- **Rare Monitor Bot** — posts rares (split by common/great) to configured channels -- **Death Alerts** — webhook to `#alerts` when a character's vitae goes from 0 → >0 (rate-limited to one per character per 5 min) -- **Idle Alerts** — webhook after 5 minutes of continuous idle state (caught portals, stuck nav, etc.). The grace period prevents false positives on brief idle blips. -- **Vortex Warning** — bot watches for "whirlwind of vortexes" chat and posts a warning embed - -### Portals -- Automatic discovery + 1-hour retention -- Coordinate-deduplicated (rounded to 0.1 precision) - -### Stats -- Per-character lifetime kills, deaths, rares, taper counts -- Grafana dashboards (2x2 iframe grid in the stats window) - -### Health & Monitoring -- Server uptime + latency + player count from TreeStats.net (checked every 30s) -- Only current state is kept — no historical `server_health_checks` table (removed April 2026 as write-only bloat) - -## Requirements - -- Docker & Docker Compose (recommended) -- OR: Python 3.11+, Node.js 20+, and a PostgreSQL 14+ with TimescaleDB - -## Installation - -```bash -git clone git@git.snakedesert.se:SawatoMosswartsEnjoyersClub/MosswartOverlord.git -cd MosswartOverlord -cp .env.example .env # fill in secrets (see Configuration below) -docker compose up -d -``` - -### Frontend development loop - -```bash -cd frontend -npm install -npm run dev # local Vite server -# ...edit files, hot reload... -cd .. -bash deploy-frontend.sh # builds + copies to static/ for production serving -``` - -⚠️ **`npm run build` writes to `static/_build/` but the web server serves from `static/`.** You must run `deploy-frontend.sh` to copy `_build/ → static/`. Otherwise the browser keeps loading the previous bundle. - -## Configuration - -All secrets go in `.env`: - -| Variable | Purpose | -|---|---| -| `POSTGRES_PASSWORD` | Telemetry DB password | -| `INVENTORY_DB_PASSWORD` | Inventory DB password | -| `SHARED_SECRET` | Plugin auth for `/ws/position` | -| `SECRET_KEY` | Session cookie signing | -| `DISCORD_RARE_BOT_TOKEN` | Bot token for rare monitor | -| `DISCORD_ACLOG_WEBHOOK` | Webhook URL for death/idle alerts | -| `GF_SECURITY_ADMIN_PASSWORD` | Grafana admin | -| `COMMON_RARE_CHANNEL_ID` | Discord channel ID for common rares | -| `GREAT_RARE_CHANNEL_ID` | Discord channel ID for great rares | -| `ACLOG_CHANNEL_ID` | Discord channel ID for the rare bot's status/vortex messages | -| `MONITOR_CHARACTER` | Which character's chat the bot monitors | - -The Overlord Agent has its own env file at `/etc/overlord/agent.env` (root:overlord-agent 0640) so it doesn't share the tracker's secrets: - -| Variable | Purpose | -|---|---| -| `SECRET_KEY` | Same value as the tracker — validates browser session cookies | -| `AGENT_DB_DSN` | Read-only connection string `postgresql://overlord_agent_ro:@127.0.0.1:5432/dereth` | -| `TRACKER_URL` | Loopback to the tracker container (default `http://127.0.0.1:8765`) | -| `AGENT_RATE_MAX` | Per-user rate limit (default 60/hour) | -| `AGENT_RATE_WINDOW_S` | Rate-limit window in seconds (default 3600) | -| `AGENT_AUDIT_LOG` | Path to audit JSONL (default `/var/log/overlord-agent/audit.jsonl`) | -| `CLAUDE_TIMEOUT_S` | Max seconds per `claude -p` invocation (default 240) | - -## Deploying Changes - -Live backend host: `overlord.snakedesert.se` (SSH user `erik`, key-based auth). - -### Quick deploy — Python / static file changes - -```bash -ssh erik@overlord.snakedesert.se \ - "cd /home/erik/MosswartOverlord && git pull --ff-only origin master" -# Python changes require a restart: -ssh erik@overlord.snakedesert.se "docker compose restart dereth-tracker" -# Static files (JS/CSS/HTML) are served from the bind-mounted static/ — no restart. -``` - -⚠️ Uvicorn runs **without** `--reload` in production. Do not add it back — without the `watchfiles` package it falls back to a polling reloader that busy-loops at 100% CPU and eats a whole core. - -### React frontend deploy - -```bash -cd frontend && npm run build && cd .. -bash deploy-frontend.sh -git add static/ && git commit -m "deploy frontend" && git push -ssh erik@overlord.snakedesert.se "cd /home/erik/MosswartOverlord && git pull" -# No container restart needed. -``` - -### Full rebuild — Dockerfile / pip package / version stamp changes - -```bash -ssh erik@overlord.snakedesert.se "cd /home/erik/MosswartOverlord && \ - git pull --ff-only origin master && \ - export BUILD_VERSION=\"\$(date -u +%Y.%-m.%-d.%H%M)-\$(git rev-parse --short HEAD)\" && \ - docker compose build --no-cache --build-arg BUILD_VERSION=\$BUILD_VERSION dereth-tracker && \ - docker compose up -d dereth-tracker" -``` - -`BUILD_VERSION` is displayed in the sidebar of the live frontend. Format is CalVer: `YYYY.M.D.HHMM-gitshorthash`. - -### Overlord Agent deploy - -Code changes to `agent/` only: -```bash -ssh erik@overlord.snakedesert.se "cd /home/erik/MosswartOverlord && \ - git pull --ff-only origin master && \ - sudo systemctl restart overlord-agent" -journalctl -u overlord-agent -f # tail logs to verify -``` - -`agent/requirements.txt` changed (new pip deps): -```bash -ssh erik@overlord.snakedesert.se "cd /home/erik/MosswartOverlord && \ - git pull --ff-only origin master && \ - agent/.venv/bin/pip install -r agent/requirements.txt && \ - sudo systemctl restart overlord-agent" -``` - -systemd unit changed: -```bash -ssh erik@overlord.snakedesert.se "cd /home/erik/MosswartOverlord && \ - git pull --ff-only origin master && \ - sudo cp agent/overlord-agent.service /etc/systemd/system/ && \ - sudo systemctl daemon-reload && sudo systemctl restart overlord-agent" -``` - -First-time install: `bash agent/install.sh` — see `agent/README.md` for the full bootstrap procedure (creating the `overlord-agent` user, copying claude auth, granting filesystem access, populating `/etc/overlord/agent.env`). - -## WebSocket Contract - -### `/ws/position` (plugin → backend) - -Authenticated via `?secret=` or `X-Plugin-Secret` header. Accepts JSON frames with a `type` discriminator: - -| `type` | Purpose | -|---|---| -| `telemetry` | Position, kills, session metrics (every 2s per character) | -| `vitals` | Health/stamina/mana/vitae percentages | -| `character_stats` | Full attributes/skills/allegiance (every 10 min) | -| `inventory` / `full_inventory` | Complete inventory dump on login | -| `inventory_delta` | Incremental add/update/remove of a single item | -| `equipment_cantrip_state` | Equipped spell effects | -| `portal` | Discovered portal with coordinates | -| `spawn` | Monster spawn observation | -| `chat` | In-game chat line (any channel) | -| `quest` | Quest timer / progress | -| `rare` | Rare item find notification | -| `nearby_objects` | On-demand radar data (nearby entities) | -| `combat_stats` | Session combat snapshot (Mag-Tools parser output) | -| `share_*` | Cross-machine vital/debuff sharing envelopes | -| `dungeon_map` | Dungeon floor tile data for radar overlay | - -See `EVENT_FORMATS.json` for exact per-type schemas. - -### `/ws/live` (browser → backend) - -Session-cookie authenticated (except for internal Docker network clients, which are trusted by IP). Clients can: - -- Send `{"type":"subscribe","message_types":["rare","chat"]}` to filter which events they receive. Without subscribing, all types are forwarded (browser default). -- Send `{"player_name":"Larsson","command":"/radar start"}` to route a command to that character's plugin client. -- Send `{"type":"request_dungeon_map","landblock":"..."}` to pull cached dungeon tile data. - -Backend pushes the same firehose (subject to subscription filter) to every browser client. - -## HTTP API Reference - -See `EVENT_FORMATS.json` for event schemas. Major HTTP endpoints: - -- `GET /live` — active players seen in the last 30s -- `GET /history?from=…&to=…` — historical telemetry snapshots -- `GET /trails` — recent player trails for the map -- `GET /spawns/heatmap?hours=N` — aggregated spawn density -- `GET /portals` — discovered portals within retention window -- `GET /inventory/{character}` — current inventory (proxied to inventory-service) -- `GET /character-stats/{character}` — full character attributes/skills -- `GET /combat-stats/{character}` — session + lifetime combat stats -- `GET /vital-sharing/peers` — currently-registered vital sharing peers -- `GET /api-version` — build version stamp -- `GET /server-health` — current Coldeve server status + player count - -## Frontend - -### React v2 (primary, at `/`) -- Map-first layout with draggable/resizable windows -- Code-split bundles: one chunk per window type, lazy-loaded on open -- Window types: Chat, Stats, Inventory, Character, Radar, CombatStats, CombatPicker, Issues, VitalSharing, QuestStatus, PlayerDashboard -- Per-character inventory version counter — an open inventory window refreshes 2s after its own character's last `inventory_delta`, ignoring unrelated traffic -- Direct DOM pan/zoom on the map (no React state per frame) -- Service worker caches a small whitelist of static assets -- Version badge in the sidebar confirms which build is loaded - -### Classic v1 (preserved at `/classic`) -The original vanilla JS frontend with element-pooling optimization is kept for fallback and reference. - -## AI Assistant (Overlord Agent) - -A draggable chat window in the dashboard (🤖 Assistant button). Powered by `claude -p` running headless on the server, with read-only access to live game state via an MCP server. - -### Architecture -- **Host-side service** (`agent/`, systemd unit `overlord-agent`) runs OUTSIDE Docker because the `claude` CLI binary lives on the host (`/home/erik/.local/bin/claude`) and depends on host-side authentication credentials. -- **Dedicated UNIX user** (`overlord-agent`, system account, `/var/lib/overlord-agent` home, no shell) — kernel-level isolation from the operator's `erik` account. Cannot read `/home/erik/.claude`, `~/.ssh`, `.bash_history`, `.env`, etc. -- **MCP stdio server** (`agent/mcp_overlord.py`) exposes 12 tools that wrap the tracker's HTTP endpoints + read-only DB queries. Claude only sees these tools; no `Bash`, `Read`, `Write`, etc. -- **Frontend** (`AgentWindow.tsx`) — per-browser session UUID in localStorage, "New Chat" button, on-mount rehydration from `/agent/sessions/{id}/history`. - -### MCP tools available to the assistant -`get_live_players`, `get_player_state`, `get_combat_stats`, `get_equipment_cantrips`, `get_inventory`, `get_inventory_search`, `search_items` (cross-character), `get_recent_rares`, `get_quest_status`, `get_server_health`, `query_telemetry_db` (read-only SQL via sqlglot parser + GRANT-SELECT-only PG role), `suitbuilder_search`. Plus `WebFetch(domain:acpedia.org)` for AC info lookups. - -### Security stack (defense-in-depth) -1. **Cookie auth** on `/agent/ask` (same session cookie the tracker issues) -2. **Per-user rate limit** (60 req/h default) and **concurrency cap** (1 in-flight) -3. **JSONL audit log** at `/var/log/overlord-agent/audit.jsonl` (every prompt + result) -4. **CLI flags** — `--allowed-tools` (just our 12 MCP tools), `--disallowed-tools` (Bash, Write, Read, Edit, Agent, ToolSearch, Monitor, scheduling, Gmail/Drive/Calendar, etc.), `--permission-mode dontAsk` -5. **`/var/lib/overlord-agent/.claude/settings.json`** — strict deny rules (server-side only, NOT in repo) -6. **System-prompt scope rules** in `CLAUDE.md` — instruct the model not to probe, not to suggest workarounds -7. **SQL parser** (`sqlglot`) rejects any non-SELECT statement on `query_telemetry_db` -8. **Read-only PG role** `overlord_agent_ro` (GRANT SELECT only) — even a parser bypass can't mutate -9. **systemd hardening** — `ProtectSystem=strict`, `ProtectHome=read-only`, `InaccessiblePaths=/etc/shadow,/root,~/.ssh,…`, `NoNewPrivileges=true`, `CapabilityBoundingSet=` (empty), `PrivateTmp=true`, `PrivateDevices=true`, `RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6`, `SystemCallFilter=@system-service ~@privileged ~@reboot ~@mount`, `MemoryMax=512M`, `TasksMax=128` -10. **Secrets out of /home** — `/etc/overlord/agent.env` (root:overlord-agent 0640) for SECRET_KEY + AGENT_DB_DSN - -### Files - -| Path | What | -|------|------| -| `agent/service.py` | FastAPI app: `/agent/health`, `/agent/sessions/new`, `/agent/ask`, `/agent/sessions/{id}/history` | -| `agent/auth.py` | Session cookie validation (mirrors `main.py:1013-1019`) | -| `agent/claude_wrapper.py` | `asyncio.create_subprocess_exec("claude", "-p", …)` with allowed/disallowed-tools | -| `agent/tools.py` | Pure tool implementations | -| `agent/mcp_overlord.py` | MCP stdio server registering tools | -| `agent/sql/0001_overlord_agent_ro.sql` | Read-only PG role | -| `agent/overlord-agent.service` | systemd unit (the hardening directives) | -| `agent/install.sh` | venv + systemd setup | -| `agent/README.md` | Operator's deeper reference | -| `.mcp.json` (repo root) | Project-level MCP config Claude Code auto-loads | -| `CLAUDE.md` "Overlord Assistant Mode" section | System-prompt briefing | - -### Routing -nginx forwards `/api/agent/*` to `127.0.0.1:8767` (the host-side service) with a 300s read/send timeout (suitbuilder runs can be slow). Other `/api/*` continues to the dereth-tracker container at `127.0.0.1:8765`. - -### Cost / quota -Subscription auth (no API key); per-call cost is informational only. Each `/agent/ask` invocation = one `claude -p` subprocess with shared session cache. Reactive only — no background polling, no scheduled tasks. - -## Database Schema - -### Telemetry DB (`dereth`, TimescaleDB) - -| Table | Type | Retention | Purpose | +| Component | Path | Runs as | Notes | |---|---|---|---| -| `telemetry_events` | hypertable | 30 days | Position/stats snapshots | -| `spawn_events` | hypertable | 7 days | Monster spawn observations (heatmap source) | -| `rare_events` | regular | forever | Rare find history | -| `portals` | regular | 1 hour | Discovered portals, dedup by rounded coords | -| `char_stats` | regular | forever | Per-character lifetime kill total | -| `rare_stats` | regular | forever | Per-character lifetime rare total | -| `rare_stats_sessions` | regular | forever | Per-session rare count | -| `combat_stats` | regular | forever | Lifetime combat accumulator | -| `combat_stats_sessions` | regular | forever | Per-session combat snapshots | -| `character_stats` | regular | forever | Latest full stats JSON per character | -| `server_status` | regular | forever | Current Coldeve server state (single row) | +| **Tracker** (ingest + website + read API + WS) | `go-services/tracker-go/` | Docker `dereth-tracker-go`, 127.0.0.1:8770 | serves the React frontend, login/admin, the plugin `/ws/position`, browser `/ws/live`, and the full read API; writes the `dereth` DB | +| **Inventory** (search + suitbuilder + ingestion) | `go-services/inventory-go/` | Docker `inventory-go`, 127.0.0.1:8772 | normalized item search, the suitbuilder solver (SSE), inventory ingestion; writes `inventory_db` | +| Telemetry DB | TimescaleDB | Docker `dereth-db`, 5432 | hypertables `telemetry_events`, `spawn_events` | +| Inventory DB | postgres:14 | Docker `inventory-db`, 5433 | 7-table normalized item schema | +| React frontend | `frontend/` → `static/` | served by `tracker-go` | unchanged by the migration — same paths, same API | +| Classic v1 / legacy pages | `static/classic/`, `static/*.html` | served by `tracker-go` | `/classic`, `/suitbuilder.html`, `/inventory.html` | +| Grafana | compose `dereth-grafana` | 127.0.0.1:3000 | anonymous Viewer auth, proxied at `/grafana/` | +| Discord rare bot | `discord-rare-monitor/` (Python) | Docker, reads Go `/ws/live` | posts rares + relays allegiance chat | +| Overlord Agent (assistant) | `agent/` | host-side systemd `overlord-agent`, 127.0.0.1:8767 | shells out to `claude -p`; outside Docker by design | -### Inventory DB (`inventory_db`, PostgreSQL) +**Stack:** Go 1.25 (stdlib `net/http` with 1.22 method+path routing, `pgx/v5`, +`coder/websocket`, `bwmarrin/discordgo`, `golang.org/x/crypto/bcrypt`), distroless +multi-stage images. React 19 + Vite + TypeScript. PostgreSQL/TimescaleDB. nginx +reverse proxy (host-side). Unlike the old single-worker Python service, the Go +tracker uses `GOMAXPROCS` = all available cores, so traffic bursts parallelize +instead of bottlenecking on one core. -Normalized schema: `items`, `item_combat_stats`, `item_requirements`, `item_enhancements`, `item_ratings`, `item_spells`, `item_raw_data`. +--- -`items.container_id` stores the in-game ID of the container holding the item (0 = character body). The frontend groups items into packs by this ID. +## Build & run -## Operations & Health +Everything builds and runs in Docker — **no host Go toolchain needed** (the +multi-stage images compile from source). The production stack is the base compose +(databases, Grafana, Discord bot) plus two override files for the Go services and +the cutover wiring. -### PostgreSQL tuning -`dereth-db` runs with explicit memory overrides in `docker-compose.yml`: -- `shared_buffers=8GB` (was 96GB via auto-tune on a 32GB host — caused thrashing) -- `effective_cache_size=16GB` -- `work_mem=16MB`, `maintenance_work_mem=1GB` -- `max_wal_size=4GB` - -### Retention policies -- `telemetry_events`: 30-day drop, daily -- `spawn_events`: 7-day drop, daily -- `portals`: 1-hour cleanup (background task in `main.py`) -- `server_health_checks`: **removed** — was write-only, 850K rows of nothing - -### Log levels -Both `dereth-tracker` and `inventory-service` run at `LOG_LEVEL=INFO`. Do not set to `DEBUG` in production — it dumps full inventory_delta payloads for every item update (hundreds of KB/sec). - -### Host (Proxmox VM) -- 6 vCPU, 32 GiB RAM (of which ~30 GiB is normally free under current load) -- Live host: `overlord.snakedesert.se` -- Reverse proxy: Nginx on the host terminates TLS and strips the `/api/` prefix before forwarding to port 8765 - -### Debug commands ```bash -docker ps -docker logs mosswartoverlord-dereth-tracker-1 --tail 100 -docker logs mosswartoverlord-inventory-service-1 --tail 100 -docker logs mosswartoverlord-discord-rare-monitor-1 --tail 100 -docker exec dereth-db psql -U postgres -d dereth +# --- build the Go service images --- +export BUILD_VERSION="$(date -u +%Y.%-m.%-d.%H%M)-$(git rev-parse --short HEAD)" +docker compose -f docker-compose.yml -f go-services/docker-compose.go.yml \ + build dereth-tracker-go inventory-go + +# --- production: Go services in write mode, serving the site + ingest --- +docker compose -f docker-compose.yml \ + -f go-services/docker-compose.go.yml \ + -f go-services/docker-compose.cutover.yml \ + up -d --no-deps dereth-tracker-go inventory-go ``` -## Contributing +- `docker-compose.go.yml` defines the Go services (plus the isolated shadow DBs used during the parallel run). +- `docker-compose.cutover.yml` flips the Go services to **write mode** against the production DBs (`READ_ONLY=false`, `SKIP_SCHEMA_INIT=true` so they run no DDL and trust the existing schema) and points the Discord bot at the Go `/ws/live`. Drop this file to return the Go services to read-only parallel mode. +- `BUILD_VERSION` is shown in the frontend sidebar (CalVer: `YYYY.M.D.HHMM-gitshorthash`). +- Required env (server `.env`, **never committed**): `SHARED_SECRET`, `SECRET_KEY`, `POSTGRES_PASSWORD`, `INVENTORY_DB_PASSWORD`, `DISCORD_ACLOG_WEBHOOK`, `DISCORD_RARE_BOT_TOKEN`, the Discord channel IDs, and Grafana admin. See `.env.example`. -Contributions welcome. Please: -- Keep cross-repo protocol changes additive (new optional fields > renames/removes) -- Update both this README and `CLAUDE.md` when workflows change -- Test end-to-end: plugin → backend → browser for any new event type +### Frontend (unchanged by the migration) -For detailed architecture notes and ongoing investigations, see `CLAUDE.md` and `docs/plans/`. +The React app and the legacy static pages call the same absolute paths +(`/api/...`, `/inv/...`, `/live`, …) — the Go tracker answers them, so the +frontend ships as-is. + +```bash +cd frontend && npm run dev # local dev, port 5173, /api → :8770 +bash deploy-frontend.sh # complete build + copy into static/ (runs npm run build itself) +``` + +The tracker serves `static/` directly (bind-mounted), so static/JS/CSS changes +need no restart. ⚠️ `npm run build` writes to `static/_build/`; only +`deploy-frontend.sh` copies it into the served `static/`. + +### nginx + +The live config is host-side at `/etc/nginx/sites-enabled/overlord` (source copy +in `nginx/overlord.conf`); the `tracker_go` upstream is in +`/etc/nginx/conf.d/tracker_go.conf` (`server 127.0.0.1:8770;`). Production routes +`/`, `/api/`, `/websocket/` to the Go tracker. Every location that proxies to the +tracker **must** set `X-Forwarded-For` — it drives the internal-trust auth rule. + +### Overlord Agent + +Unchanged by the migration — it's a host-side Python systemd service. Code change: +`git pull && sudo systemctl restart overlord-agent`. Its env lives separately at +`/etc/overlord/agent.env`. See `agent/` and `CLAUDE.md`. + +--- + +## WebSocket contract + +- **`/ws/position`** — plugin → backend. Telemetry, vitals, inventory, portal, rare, combat, quest, chat, share_*, … Authenticated by the `X-Plugin-Secret` header against `SHARED_SECRET` (constant-time; fails closed when unset). The tracker forwards inventory to `inventory-go`, accumulates kill/combat stats, and re-broadcasts to browsers. +- **`/ws/live`** — browser ↔ backend. Session-cookie (or internal-trust) authenticated. Accepts `subscribe`, `request_dungeon_map`, and `{player_name, command}` envelopes routed to the matching plugin socket. **Telemetry is broadcast typeless** so the browser ignores it and takes player data from the 5 s `/live` poll (matching the original design — broadcasting it typed flaps the per-player counters). +- **Internal-trust rule:** a request skips cookie auth only when its source is private/loopback **and** carries no `X-Forwarded-For`. nginx sets XFF on all internet traffic, so only host-side / compose-network callers qualify. + +### Payload note + +Payloads are snake_case JSON; keep field names and shapes stable across plugin + +backend. The plugin sends several numeric telemetry fields as **strings** +(`kills_per_hour`, `deaths`, `total_deaths`, `prismatic_taper_count`) — the backend +coerces them (`coerceNum` in `tracker-go/reads.go`). + +## Auth & users + +Session cookies are signed with `SECRET_KEY` via an itsdangerous-compatible +`URLSafeTimedSerializer` (HMAC-SHA1, 30-day expiry) — cookies interoperate with +the legacy Python service. Login at `/login` (bcrypt against the `users` table), +admin user CRUD at `/api-admin/users`, current user at `/me`. + +## Databases + +Two separate Postgres databases, both schema-from-code: + +- **`dereth`** (TimescaleDB, `dereth-db`): hypertables `telemetry_events` + `spawn_events`, plus `char_stats`, `combat_stats(_sessions)`, `rare_*`, `portals`, `character_stats`, `users`. Persisted event types: telemetry, spawn, rare, portal, character_stats, combat_stats. Everything else (vitals, quest, cantrips, nearby_objects, dungeon_map, share_*) is memory-only. +- **`inventory_db`** (postgres:14, `inventory-db`): 7 normalized tables (`items` + combat/requirements/enhancements/ratings/spells/raw_data). + +In cutover mode the Go services reuse these production databases directly; the +shadow DBs in `docker-compose.go.yml` exist only for isolated parallel-run +validation. **Backups:** `pg_dump -Fc` of both DBs; TimescaleDB restore needs +`timescaledb_pre_restore()` / `post_restore()` around `pg_restore`. + +## Route conventions + +- nginx strips `/api/` before proxying, so backend routes do **not** start with `/api/`. +- Hyphenated routes (`/api-version`, `/api-admin/...`) deliberately bypass the strip (they fall through nginx's `location /`). +- The static SPA is the catch-all (`GET /`), registered after the API routes, with `index.html` fallback for client-side routing. +- `/inv/*` reverse-proxies to the inventory service; `/api/agent/*` is proxied by nginx (not the tracker) to the host-side agent. + +## Operational notes + +- Discord: the rare bot posts rares + relays allegiance chat; **death/idle alerts come from the tracker itself** via `DISCORD_ACLOG_WEBHOOK`. +- Issue board persists to the flat file `static/openissues.json` (web-served, mounted read-write). +- Logs: `docker logs dereth-tracker-go`, `docker logs inventory-go`. Read-only psql: `docker exec dereth-db psql -U postgres -d dereth`, `docker exec inventory-db psql -U inventory_user -d inventory_db`. +- **This repo is PUBLIC** on git.snakedesert.se — never commit secrets. `.env` is gitignored; `.env.example` is the template. + +## Branches + +- **`master`** — the Go production backend (this). +- **`python-legacy`** — the original Python/FastAPI implementation, preserved for reference and rollback. + +See [`CLAUDE.md`](CLAUDE.md) for contributor/agent guidance and deeper internals. diff --git a/docs/plans/2026-06-25-suitbuilder-cd-tier-filter-design.md b/docs/plans/2026-06-25-suitbuilder-cd-tier-filter-design.md new file mode 100644 index 00000000..a17585c7 --- /dev/null +++ b/docs/plans/2026-06-25-suitbuilder-cd-tier-filter-design.md @@ -0,0 +1,85 @@ +# Suitbuilder CD-tier filter — design + +**Date:** 2026-06-25 +**Status:** Approved (pending spec review) +**Scope:** Live Go suitbuilder only (`go-services/inventory-go/`) + the static suitbuilder page (`static/suitbuilder.{html,js}`). **No changes** to the frozen `inventory-service/suitbuilder.py` (legacy rollback reference). + +## Goal + +Let the user restrict which **crit-damage tiers** (CD0 / CD1 / CD2) are allowed on **armor** pieces in a suit search, so they can build, e.g., all-CD1 suits or CD1/CD0-only suits. Among whatever tiers are allowed, the solver still prefers the highest (existing behavior) — so this is fundamentally a **filter**, not a scoring change. + +## Background — current state + +- The live suitbuilder is the Go solver (`suit_solver.go` / `suit_model.go` / `suit_http.go`), reached via browser → tracker `/inv/suitbuilder/search` → inventory-go `/suitbuilder/search`. Python is frozen on `python-legacy`. +- There is **no crit-damage filtering today.** CD0/CD1/CD2 armor all flows into the search. The only thing distinguishing tiers is scoring (`CritDamage1: +10`, `CritDamage2: +20`) and the CD-descending armor sort — which is why CD2 always wins. +- The UI already shows **Crit Damage min/max** number inputs (`suitbuilder.html:54-57`), and the JS already sends `min_crit_damage`/`max_crit_damage` (`suitbuilder.js:310-311, 386-387`). The Go solver receives them into `SearchConstraints.MinCritDamage`/`MaxCritDamage` but **never references them** — dead, half-wired scaffold. This feature replaces that dead control. + +## Behavior contract + +- A new per-search filter selects which CD tiers are **allowed on armor**: independent CD0 / CD1 / CD2 toggles. +- **A checked tier = "allowed."** "Prefer higher, fall back lower" happens automatically among the allowed tiers via the existing scoring/sort — no scoring change. +- **Default = all three allowed.** Because the solver prefers the highest allowed tier, the default naturally leads with CD2 — i.e. identical to today's behavior. This is the "default CD2" state. +- **Empty / none-selected = treated as the default** (all allowed). A search can never be forced into an armorless state by this control. +- **Jewelry and clothing are never filtered by CD** — they are categorized separately in `loadItems` and the filter only touches armor. +- **Tier mapping** (handles rare high-crit gear): `CD0 = rating ≤ 0`, `CD1 = rating == 1`, **`CD2 = rating ≥ 2`**. A CD3+ gear piece counts as CD2 and is not silently dropped. + +### Worked examples + +| Allowed set | Result | +|---|---| +| `{0,1,2}` (default / empty) | Unchanged from today — prefer CD2, fall back CD1, CD0 | +| `{0,1}` | No CD2 armor; prefer CD1, fall back CD0 | +| `{1}` | All-CD1 suits; a slot with no CD1 piece is left empty | +| `{1,2}` | No CD0 armor; prefer CD2, fall back CD1 | + +## Backend design — `go-services/inventory-go` + +### 1. Constraint field (`suit_model.go`) +- Add `AllowedCritDamage []int \`json:"allowed_crit_damage"\`` to `SearchConstraints`. +- **Remove** the dead `MinCritDamage *int` / `MaxCritDamage *int` fields (never wired; their UI is being replaced). Leave the other unrelated dead fields (`MinArmor`/`MaxArmor`/`MinDamageRating`/`MaxDamageRating`) untouched — out of scope. + +### 2. Precompute the allowed set (`newSolver`, `suit_solver.go`) +- Build `allowedCD map[int]bool` by normalizing each value in `AllowedCritDamage` to a tier in `{0,1,2}` (clamp ≥2 to 2, ≤0 to 0). +- **Filter inactive** (no-op) when the resulting set is empty **or** already contains all of `{0,1,2}`. This makes "all checked", "none checked", and "field absent" all mean *no filter* — and guarantees the default path is byte-identical to current output. + +### 3. Apply the filter in `loadItems` (`suit_solver.go`) +- **Location & ordering are load-bearing:** filter armor items **after** the raw `items` slice is built (~line 254) and **before `removeSurpassedItems`** (line 262). If the CD filter ran after domination, a CD2 piece could dominate and remove an allowed CD1 piece, which we'd then exclude — leaving the slot needlessly empty. Filtering first keeps domination confined to allowed items. +- An item is "armor" iff its slot matches `armorSlotSet` (including comma-joined multi-coverage slots like `"Chest, Abdomen"`). Factor a small package-level helper `isArmorSlot(slot string) bool` (mirrors the existing `matches(it.Slot, armorSlotSet, nil)` logic) so it can be used both here and in the existing categorization pass. Non-armor items (jewelry/clothing/unknown) are never dropped by this filter. +- When the filter is active, drop armor items whose normalized tier ∉ `allowedCD`. +- Tailored/reduced armor inherits its CD from the origin piece (already filtered upstream), so reductions of excluded pieces never appear — no extra handling needed. + +### Regression safety +- The default (no `allowed_crit_damage`, or all three) path must produce **identical** output to the current solver. The no-op guard in step 2 ensures this. + +## Frontend design — `static/suitbuilder.{html,js}` + +(Vanilla static page served from the bind-mounted `static/` — no build step, no container restart.) + +### 1. `suitbuilder.html` (~lines 53-58) +- Replace the `Crit Damage [Min]-[Max]` number inputs (`#minCritDmg`, `#maxCritDmg`) with three checkboxes inside the existing `filter-group`: `#allowCD0`, `#allowCD1`, `#allowCD2`, labelled CD0 / CD1 / CD2, **all `checked` by default.** Keep the surrounding `filter-row`/`filter-group`/`constraint-section` layout. + +### 2. `suitbuilder.js` +- **`gatherConstraints()` (lines 310-311):** remove the `min_crit_damage`/`max_crit_damage` reads; add `allowed_crit_damage`, an array of the checked tiers, e.g. `[0,1,2]`. +- **`validateConstraints()` (line 360):** remove the now-deleted `!constraints.min_crit_damage` term from the "at least one constraint" check. (A CD restriction is not a valid *standalone* search — armor is only loaded for the chosen primary/secondary set, so a set/cantrip/ward/rating-min is still required. The CD filter is a refinement on top.) +- **`streamOptimalSuits()` (lines 386-387):** remove `min_crit_damage`/`max_crit_damage` from `requestBody`; add `allowed_crit_damage: constraints.allowed_crit_damage`. + +## Testing + +- **Regression (Go):** a default search (no `allowed_crit_damage`) yields output identical to baseline — assert the no-op path. Where existing suitbuilder validation/golden harnesses exist (`compare/`), the default case must stay byte-identical; filtered cases are intentionally Python-divergent and are validated by the new tests below, not against Python. +- **New unit test (Go):** + - `allowed=[1]` ⇒ every armor piece in every returned suit has tier CD1; jewelry/clothing still present. + - `allowed=[0,1]` ⇒ no CD2 armor appears in any suit. + - `allowed=[1,2]` ⇒ no CD0 armor appears. + - `allowed=[]` / `[0,1,2]` ⇒ identical to baseline. +- **Manual:** on the server, run a real CD1-only search and confirm all-CD1 armor and sane fallback/empty-slot behavior. + +## Deploy + +- **Backend:** rebuild `inventory-go` on the server (sync `go-services/`, build, recreate with the cutover override) — see MosswartOverlord CLAUDE.md "Go services — build, deploy, gotchas". +- **Frontend:** edit `static/suitbuilder.{html,js}`; a normal `git pull` on the host picks them up via the bind mount — no build, no restart. + +## Out of scope + +- `inventory-service/suitbuilder.py` (frozen/legacy) — intentionally left to diverge. +- The other dead constraint fields (`min/max_armor`, `min/max_damage_rating`) — separate follow-up if wanted. +- No scoring-weight changes; no new scoring knobs. diff --git a/docs/plans/2026-06-25-suitbuilder-cd-tier-filter-plan.md b/docs/plans/2026-06-25-suitbuilder-cd-tier-filter-plan.md new file mode 100644 index 00000000..c9336f7d --- /dev/null +++ b/docs/plans/2026-06-25-suitbuilder-cd-tier-filter-plan.md @@ -0,0 +1,522 @@ +# Suitbuilder CD-tier filter — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Let a suitbuilder search restrict which crit-damage tiers (CD0/CD1/CD2) are allowed on armor pieces, so the user can build e.g. all-CD1 suits — while the default (all allowed) stays byte-identical to today. + +**Architecture:** Add an `allowed_crit_damage` constraint. In the live Go solver (`inventory-go`), drop armor items whose CD tier isn't allowed during item loading, before the domination pre-filter. "Prefer highest allowed tier" needs no new code — it falls out of the existing scoring and CD-descending armor sort. Frontend swaps the dead Crit-Damage min/max inputs for three CD checkboxes. + +**Tech Stack:** Go 1.25 (`go-services/inventory-go`), vanilla JS/HTML (`static/suitbuilder.*`), Docker on the server (no local Go toolchain). + +**Spec:** `docs/plans/2026-06-25-suitbuilder-cd-tier-filter-design.md` + +--- + +## Conventions for this plan + +- **Source-of-truth edits** happen in the local repo at `C:/Users/erikn/source/repos/dereth-workspace/MosswartOverlord`, on branch `suitbuilder-cd-tier-filter`. Commit there. +- **No local Go toolchain.** Build & test run on the server (`overlord.snakedesert.se`) inside Docker. +- **Fast unit-test loop** (run from the local MosswartOverlord dir after copying changed files to the host — see Task 6 for the copy command): + ```bash + ssh erik@overlord.snakedesert.se "docker run --rm \ + -v /home/erik/MosswartOverlord/go-services/inventory-go:/src -w /src \ + golang:1.25-bookworm sh -c 'go mod tidy >/dev/null 2>&1 && go test ./... -v'" + ``` + (Mounts the host's inventory-go source into a throwaway golang container. `go mod tidy` writes go.sum into that untracked dir — harmless.) +- The live container is `inventory-go` (image `inventory-go:local`, `127.0.0.1:8772`). + +--- + +## File structure + +- `go-services/inventory-go/suit_model.go` — **modify**: constraint field. +- `go-services/inventory-go/suit_cd.go` — **create**: pure CD-tier helpers (one responsibility, DB-free, unit-testable). +- `go-services/inventory-go/suit_cd_test.go` — **create**: unit tests for the helpers. +- `go-services/inventory-go/suit_solver.go` — **modify**: solver field + wire filter into `loadItems`. +- `go-services/inventory-go/Dockerfile` — **modify**: add a `go test` build gate (mirrors tracker-go). +- `static/suitbuilder.html` — **modify**: CD checkboxes replace min/max inputs. +- `static/suitbuilder.js` — **modify**: gather/validate/send `allowed_crit_damage`. +- `static/suitbuilder.css` — **modify**: minor styling for the toggles. + +--- + +## Task 1: Add the `allowed_crit_damage` constraint field + +**Files:** Modify `go-services/inventory-go/suit_model.go` + +- [ ] **Step 1: Replace the dead crit min/max fields** + +In `SearchConstraints`, replace these two lines: + +```go + MinCritDamage *int `json:"min_crit_damage"` + MaxCritDamage *int `json:"max_crit_damage"` +``` + +with: + +```go + AllowedCritDamage []int `json:"allowed_crit_damage"` +``` + +(The `Min/MaxCritDamage` fields were never referenced by the solver — confirmed by grep. The other `Min/Max*` fields stay untouched.) + +- [ ] **Step 2: Commit** + +```bash +cd /c/Users/erikn/source/repos/dereth-workspace/MosswartOverlord +git add go-services/inventory-go/suit_model.go +git commit -m "feat(suitbuilder): add allowed_crit_damage constraint field" +``` + +--- + +## Task 2: CD-tier helpers + unit tests (TDD) + +**Files:** +- Create: `go-services/inventory-go/suit_cd.go` +- Create: `go-services/inventory-go/suit_cd_test.go` + +- [ ] **Step 1: Write the failing tests** + +Create `go-services/inventory-go/suit_cd_test.go`: + +```go +package main + +import "testing" + +func TestCritTier(t *testing.T) { + cases := []struct { + rating, want int + }{{-1, 0}, {0, 0}, {1, 1}, {2, 2}, {3, 2}, {5, 2}} + for _, c := range cases { + if got := critTier(c.rating); got != c.want { + t.Errorf("critTier(%d) = %d, want %d", c.rating, got, c.want) + } + } +} + +func TestAllowedCritSet(t *testing.T) { + for _, vals := range [][]int{nil, {}, {0, 1, 2}, {0, 1, 3}} { + if allowedCritSet(vals) != nil { + t.Errorf("allowedCritSet(%v) should be nil (inactive)", vals) + } + } + if s := allowedCritSet([]int{1}); s == nil || !s[1] || s[0] || s[2] { + t.Errorf("allowedCritSet({1}) = %v, want only tier 1", s) + } + if s := allowedCritSet([]int{0, 1}); s == nil || !s[0] || !s[1] || s[2] { + t.Errorf("allowedCritSet({0,1}) = %v, want tiers 0,1", s) + } + if s := allowedCritSet([]int{3}); s == nil || !s[2] || s[0] || s[1] { + t.Errorf("allowedCritSet({3}) = %v, want only tier 2 (normalized)", s) + } +} + +func TestIsArmorSlot(t *testing.T) { + for _, s := range []string{"Chest", "Head", "Feet", "Chest, Abdomen", "Upper Legs, Lower Legs"} { + if !isArmorSlot(s) { + t.Errorf("isArmorSlot(%q) = false, want true", s) + } + } + for _, s := range []string{"Neck", "Left Ring", "Left Wrist", "Trinket", "Shirt", "Pants", "Unknown", ""} { + if isArmorSlot(s) { + t.Errorf("isArmorSlot(%q) = true, want false", s) + } + } +} + +func cdItem(slot string, cd int) *SuitItem { + return &SuitItem{Slot: slot, Ratings: map[string]int{"crit_damage_rating": cd}} +} + +func TestFilterArmorByCD(t *testing.T) { + items := []*SuitItem{ + cdItem("Chest", 0), cdItem("Head", 1), cdItem("Feet", 2), + cdItem("Chest, Abdomen", 2), // multi-coverage armor, CD2 + cdItem("Neck", 0), // jewelry — never filtered + cdItem("Shirt", 0), // clothing — never filtered + } + + if got := filterArmorByCD(items, nil); len(got) != len(items) { + t.Errorf("nil filter dropped items: got %d, want %d", len(got), len(items)) + } + + got := filterArmorByCD(items, map[int]bool{1: true}) + keep := map[string]bool{"Head": true, "Neck": true, "Shirt": true} + if len(got) != 3 { + t.Fatalf("allowed{1}: got %d items, want 3", len(got)) + } + for _, it := range got { + if !keep[it.Slot] { + t.Errorf("allowed{1}: unexpected slot %q survived", it.Slot) + } + } + + got = filterArmorByCD(items, map[int]bool{0: true, 1: true}) + if len(got) != 4 { // Chest(0), Head(1), Neck, Shirt + t.Errorf("allowed{0,1}: got %d items, want 4", len(got)) + } + for _, it := range got { + if isArmorSlot(it.Slot) && it.Ratings["crit_damage_rating"] >= 2 { + t.Errorf("allowed{0,1}: CD2 armor %q should have been dropped", it.Slot) + } + } +} +``` + +- [ ] **Step 2: Run the tests to confirm they fail to build** + +Copy only the test file to the host (the implementation doesn't exist yet): + +```bash +cd /c/Users/erikn/source/repos/dereth-workspace/MosswartOverlord +scp go-services/inventory-go/suit_cd_test.go \ + erik@overlord.snakedesert.se:/home/erik/MosswartOverlord/go-services/inventory-go/ +``` + +Then run the fast test loop (see Conventions). +Expected: FAIL — `undefined: critTier`, `allowedCritSet`, `isArmorSlot`, `filterArmorByCD`. + +- [ ] **Step 3: Write the implementation** + +Create `go-services/inventory-go/suit_cd.go`: + +```go +package main + +import "strings" + +// CD-tier filtering for the suitbuilder. The allowed_crit_damage constraint +// restricts which crit-damage tiers are permitted on ARMOR pieces; jewelry and +// clothing are never affected. "Prefer the highest allowed tier" is NOT done +// here — it falls out of the existing scoring (CritDamage2 > CritDamage1) and +// the CD-descending armor sort once disallowed tiers are removed. + +// critTier normalizes a raw crit_damage_rating into a tier in {0,1,2}. Rare +// high-crit gear (rating >= 2, including 3+) collapses to tier 2 so it counts +// as "CD2" rather than being silently excluded. +func critTier(rating int) int { + switch { + case rating <= 0: + return 0 + case rating == 1: + return 1 + default: + return 2 + } +} + +// isArmorSlot reports whether a slot name denotes an armor coverage slot, +// including comma-joined multi-coverage slots like "Chest, Abdomen". +func isArmorSlot(slot string) bool { + if armorSlotSet[slot] { + return true + } + if strings.Contains(slot, ", ") { + for _, p := range strings.Split(slot, ", ") { + if armorSlotSet[strings.TrimSpace(p)] { + return true + } + } + } + return false +} + +// allowedCritSet normalizes the constraint's allowed crit-damage tiers into a +// set, or returns nil when the filter is INACTIVE: no values, or all three +// tiers {0,1,2} present (== default). A nil result means "no filter" and keeps +// the default search path byte-identical to the unfiltered solver. +func allowedCritSet(vals []int) map[int]bool { + if len(vals) == 0 { + return nil + } + set := map[int]bool{} + for _, v := range vals { + set[critTier(v)] = true + } + if set[0] && set[1] && set[2] { + return nil + } + return set +} + +// filterArmorByCD drops armor items whose crit-damage tier is not in allowed. +// Non-armor items (jewelry, clothing, unknown) always pass through. When +// allowed is nil the input is returned unchanged. +func filterArmorByCD(items []*SuitItem, allowed map[int]bool) []*SuitItem { + if allowed == nil { + return items + } + out := make([]*SuitItem, 0, len(items)) + for _, it := range items { + if isArmorSlot(it.Slot) && !allowed[critTier(it.Ratings["crit_damage_rating"])] { + continue + } + out = append(out, it) + } + return out +} +``` + +- [ ] **Step 4: Run the tests to confirm they pass** + +```bash +scp go-services/inventory-go/suit_cd.go \ + erik@overlord.snakedesert.se:/home/erik/MosswartOverlord/go-services/inventory-go/ +``` + +Run the fast test loop. Expected: PASS (`ok` — 4 tests). + +- [ ] **Step 5: Add the `go test` build gate to the Dockerfile** + +In `go-services/inventory-go/Dockerfile`, after `RUN go mod tidy` add: + +```dockerfile +RUN go test ./... +``` + +(Mirrors `tracker-go/Dockerfile`; from now on every image build runs the tests.) + +- [ ] **Step 6: Commit** + +```bash +git add go-services/inventory-go/suit_cd.go go-services/inventory-go/suit_cd_test.go go-services/inventory-go/Dockerfile +git commit -m "feat(suitbuilder): CD-tier filter helpers + tests; gate inventory-go build on go test" +``` + +--- + +## Task 3: Wire the filter into the solver + +**Files:** Modify `go-services/inventory-go/suit_solver.go` + +- [ ] **Step 1: Add the precomputed set to the Solver struct** + +In the `Solver` struct, after `armorBucketsItems int`, add: + +```go + allowedCD map[int]bool // nil == no CD filter (default / all tiers) +``` + +- [ ] **Step 2: Populate it in `newSolver`** + +In `newSolver`, after the line `sv.neededSpellBitmap = sv.spellIndex.getBitmap(c.RequiredSpells)`, add: + +```go + sv.allowedCD = allowedCritSet(c.AllowedCritDamage) +``` + +- [ ] **Step 3: Apply the filter in `loadItems` before domination** + +In `loadItems`, find: + +```go + filtered := removeSurpassedItems(items) +``` + +and immediately ABOVE it insert: + +```go + // Drop armor whose CD tier is disallowed BEFORE domination, so a CD2 piece + // can't surpass-and-remove an allowed CD1 piece we'd then exclude. + items = filterArmorByCD(items, sv.allowedCD) +``` + +- [ ] **Step 4: Verify it still builds and all tests pass** + +Copy the changed solver file and run the test loop: + +```bash +scp go-services/inventory-go/suit_solver.go \ + erik@overlord.snakedesert.se:/home/erik/MosswartOverlord/go-services/inventory-go/ +``` + +Run the fast test loop. Expected: PASS, and the package compiles (the wiring type-checks; `go test` builds the whole `main` package). + +- [ ] **Step 5: Commit** + +```bash +git add go-services/inventory-go/suit_solver.go +git commit -m "feat(suitbuilder): apply CD-tier filter in loadItems (before domination)" +``` + +--- + +## Task 4: Frontend — CD checkboxes + +**Files:** Modify `static/suitbuilder.html`, `static/suitbuilder.js`, `static/suitbuilder.css` + +- [ ] **Step 1: Replace the Crit Damage inputs with checkboxes** + +In `static/suitbuilder.html`, replace this block: + +```html +
+ + + - + +
+``` + +with: + +```html +
+ + + + +
+``` + +- [ ] **Step 2: Build `allowed_crit_damage` in `gatherConstraints()`** + +In `static/suitbuilder.js`, replace these two lines: + +```js + min_crit_damage: document.getElementById('minCritDmg').value || null, + max_crit_damage: document.getElementById('maxCritDmg').value || null, +``` + +with: + +```js + allowed_crit_damage: [ + document.getElementById('allowCD0').checked ? 0 : null, + document.getElementById('allowCD1').checked ? 1 : null, + document.getElementById('allowCD2').checked ? 2 : null, + ].filter(v => v !== null), +``` + +- [ ] **Step 3: Drop the deleted field from validation** + +In `validateConstraints()`, change: + +```js + !constraints.min_armor && !constraints.min_crit_damage && !constraints.min_damage_rating) { +``` + +to: + +```js + !constraints.min_armor && !constraints.min_damage_rating) { +``` + +- [ ] **Step 4: Send `allowed_crit_damage` in the request body** + +In `streamOptimalSuits()`, replace these two lines: + +```js + min_crit_damage: constraints.min_crit_damage ? parseInt(constraints.min_crit_damage) : null, + max_crit_damage: constraints.max_crit_damage ? parseInt(constraints.max_crit_damage) : null, +``` + +with: + +```js + allowed_crit_damage: constraints.allowed_crit_damage, +``` + +- [ ] **Step 5: Style the toggles** + +Append to `static/suitbuilder.css`: + +```css +.cd-toggle { + display: inline-flex; + align-items: center; + gap: 4px; + margin-right: 10px; + font-weight: normal; + cursor: pointer; +} +.cd-toggle input { margin: 0; } +``` + +- [ ] **Step 6: Commit** + +```bash +git add static/suitbuilder.html static/suitbuilder.js static/suitbuilder.css +git commit -m "feat(suitbuilder): CD0/CD1/CD2 allowed-tier checkboxes (replace dead crit min/max)" +``` + +--- + +## Task 5: Deploy to the server & verify end-to-end + +- [ ] **Step 1: Copy changed backend files to the host build context** + +```bash +cd /c/Users/erikn/source/repos/dereth-workspace/MosswartOverlord +scp go-services/inventory-go/suit_model.go go-services/inventory-go/suit_cd.go \ + go-services/inventory-go/suit_cd_test.go go-services/inventory-go/suit_solver.go \ + go-services/inventory-go/Dockerfile \ + erik@overlord.snakedesert.se:/home/erik/MosswartOverlord/go-services/inventory-go/ +``` + +- [ ] **Step 2: Build the image (runs `go test` as part of the build)** + +```bash +ssh erik@overlord.snakedesert.se 'cd /home/erik/MosswartOverlord && \ + docker compose -f docker-compose.yml -f go-services/docker-compose.go.yml \ + build inventory-go' +``` +Expected: build succeeds; the `RUN go test ./...` layer passes. + +- [ ] **Step 3: Recreate the container with the cutover override** + +```bash +ssh erik@overlord.snakedesert.se 'cd /home/erik/MosswartOverlord && \ + docker compose -f docker-compose.yml -f go-services/docker-compose.go.yml \ + -f go-services/docker-compose.cutover.yml up -d --no-deps inventory-go' +``` +Expected: `inventory-go` recreated; `docker ps` shows it healthy on :8772. + +- [ ] **Step 4: Copy the changed static files (bind-mounted; live immediately)** + +```bash +scp static/suitbuilder.html static/suitbuilder.js static/suitbuilder.css \ + erik@overlord.snakedesert.se:/home/erik/MosswartOverlord/static/ +``` + +- [ ] **Step 5: Verify default search is unchanged + CD1-only works** + +Manual, in the browser at the suitbuilder page (hard-refresh to bust cache): +- With **all three CD boxes checked**, run a search (a primary set + a character with armor). Confirm results look like before. +- Check **only CD1**, run the same search. Confirm in the Network tab the request body has `"allowed_crit_damage":[1]`, and every armor piece in the returned suits shows **CD1** (jewelry/clothing unaffected; slots with no CD1 piece may be empty). +- Check **CD1 + CD0**, confirm no CD2 armor appears and CD1 is preferred where available. + +--- + +## Task 6: Finalize the local feature commit + +- [ ] **Step 1: Confirm the branch state** + +```bash +cd /c/Users/erikn/source/repos/dereth-workspace/MosswartOverlord +git log --oneline -6 +git status +``` +Expected: clean tree; the spec + plan + Tasks 1-4 feature commits on `suitbuilder-cd-tier-filter`. + +--- + +## Phase 2: Reconcile host git + push to Gitea (separate, after the feature is verified live) + +> ⚠ Pushing to the **public** Gitea is outward-facing and partly irreversible. Investigate state and decide a strategy BEFORE any push; surface the chosen strategy to the user first. Never `git add` the host's `.env` (secrets). + +- [ ] **Step 1: Establish the true state of all three gits** + - Local `MosswartOverlord` HEAD (`9911edbf`, has go-services committed). + - Host `/home/erik/MosswartOverlord` HEAD (`6a0bb9fe`, go-services untracked, has server-only commits like rickroll/midsummer). + - Gitea `origin/master` — fetch and inspect; determine whether local's go-services history and/or the host's server-only commits are already on the remote. + +- [ ] **Step 2: Decide a reconciliation strategy** (depends on Step 1 findings): + - Get the host's server-only commits into the canonical local history (cherry-pick or merge), and get the local go-services history onto the host — so a single `master` contains both, with this feature on top. + - Plan must avoid clobbering the host's untracked `.env`/backups and avoid a destructive force-push unless explicitly chosen. + +- [ ] **Step 3: Execute the chosen reconciliation, then `git pull` on the host** so the host runs tracked code, and push the unified `master` to Gitea. Confirm `docker compose build` still uses the now-tracked go-services. + +(Phase 2 steps are deliberately high-level — the exact git commands depend on Step 1's findings and a strategy choice. Do not pre-bake destructive commands.) diff --git a/go-services/.gitattributes b/go-services/.gitattributes new file mode 100644 index 00000000..b81224e3 --- /dev/null +++ b/go-services/.gitattributes @@ -0,0 +1,9 @@ +# Go services run on Linux; keep LF in the working tree on all platforms. +* text=auto eol=lf +*.go text eol=lf +*.mod text eol=lf +*.sum text eol=lf +*.py text eol=lf +*.yml text eol=lf +*.conf text eol=lf +Dockerfile text eol=lf diff --git a/go-services/compare/compare_endpoints.py b/go-services/compare/compare_endpoints.py new file mode 100644 index 00000000..d8a300c8 --- /dev/null +++ b/go-services/compare/compare_endpoints.py @@ -0,0 +1,110 @@ +#!/usr/bin/env python3 +"""Structural + value parity check for the tracker-go read API vs the Python service. + +Run on the server (loopback access to both, plus `docker exec dereth-db` for the +offline-character exact-match check): + python3 compare_endpoints.py + +Most live endpoints can't be value-equal byte-for-byte (the firehose updates +between fetches), so we assert: + * status code + top-level key-set parity for every read endpoint, and + * EXACT equality of /character-stats and /combat-stats for *offline* + characters (where Python also falls back to the DB, like Go). For online + characters Python serves a richer live in-memory overlay that Phase-1 Go + intentionally lacks (no ingest yet) — that difference is expected. +""" +import json +import subprocess +import sys +import urllib.error +import urllib.parse +import urllib.request + +PY = "http://127.0.0.1:8765" +GO = "http://127.0.0.1:8770" + + +def get(base, path): + try: + with urllib.request.urlopen(base + path, timeout=12) as r: + return r.status, json.load(r) + except urllib.error.HTTPError as e: + return e.code, None + except Exception as e: # noqa: BLE001 + return "ERR:" + str(e)[:40], None + + +def topkeys(d): + if isinstance(d, dict): + return sorted(d.keys()) + if isinstance(d, list): + return ["[list]"] + return [type(d).__name__] + + +def main(): + failures = 0 + _, live = get(PY, "/live") + ch = live["players"][0]["character_name"] if live and live.get("players") else "Nobody" + chq = urllib.parse.quote(ch) + print(f"sample online character: {ch}\n") + + endpoints = [ + "/total-rares", "/total-kills", "/server-health", "/portals", + "/spawns/heatmap?hours=2", "/combat-stats", "/inventories", + "/quest-status", "/vital-sharing/peers", + f"/stats/{chq}", f"/combat-stats/{chq}", + f"/inventory/{chq}/search", "/sets/list", "/inventory-characters", + ] + print(f"{'endpoint':<36} {'py':>5} {'go':>5} keys") + for ep in endpoints: + ps, pj = get(PY, ep) + gs, gj = get(GO, ep) + pk, gk = topkeys(pj), topkeys(gj) + ok = ps == gs and pk == gk + if not ok: + failures += 1 + print(f"{ep:<36} {str(ps):>5} {str(gs):>5} {'OK' if ok else 'MISMATCH py=%s go=%s' % (pk, gk)}") + + # Online-overlay endpoints: only structural note (expected to differ for online chars). + for ep in (f"/character-stats/{chq}", f"/equipment-cantrip-state/{chq}"): + ps, _ = get(PY, ep) + gs, _ = get(GO, ep) + print(f"{ep:<36} {str(ps):>5} {str(gs):>5} (online live-overlay; exact match only for offline chars)") + + # Offline-character exact-match check. + print("\n-- offline-character exact match (/character-stats, /combat-stats) --") + try: + online = {p["character_name"] for p in live["players"]} + names = subprocess.check_output( + ["docker", "exec", "dereth-db", "psql", "-U", "postgres", "-d", "dereth", + "-tA", "-c", "SELECT character_name FROM character_stats"], text=True) + off = [n for n in names.split("\n") if n.strip() and n not in online] + tested = matched = 0 + for ch in off: + q = urllib.parse.quote(ch) + _, py = get(PY, f"/character-stats/{q}") + _, go = get(GO, f"/character-stats/{q}") + if not (isinstance(py, dict) and len(py.keys()) >= 18): + continue + tested += 1 + same = py == go + matched += same + if not same: + failures += 1 + print(f" MISMATCH {ch}: keydiff={set(py) ^ set(go)}") + if tested >= 8: + break + print(f" {matched}/{tested} rich offline chars exact-match") + if tested == 0: + print(" (no offline rich characters available to test)") + except Exception as e: # noqa: BLE001 + print(f" (skipped DB-backed offline check: {e})") + + print("\n" + ("RESULT: read-API parity OK" if failures == 0 + else f"RESULT: {failures} mismatch(es)")) + return 1 if failures else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/go-services/compare/compare_ingest.py b/go-services/compare/compare_ingest.py new file mode 100644 index 00000000..faff8e44 --- /dev/null +++ b/go-services/compare/compare_ingest.py @@ -0,0 +1,64 @@ +#!/usr/bin/env python3 +"""Validate the Go shadow ingest (dereth_go) against production (dereth). + +Run on the server. The shadow tracker replays Python's /ws/live firehose into +its own dereth_go DB. Absolute counts differ (shadow started fresh; char_stats / +rare_stats accumulate deltas from connect time), so we validate the paths whose +writes are FULL upserts/inserts and therefore exactly comparable: + + * character_stats: a full-payload upsert. For a character whose row has the + SAME timestamp in both DBs, stats_data must be byte-identical. + * /live online set: telemetry end-to-end (compared separately by the caller). +""" +import json +import subprocess + +SEP = "\x1f" + + +def q(container, db, sql): + out = subprocess.check_output( + ["docker", "exec", container, "psql", "-U", "postgres", "-d", db, "-tA", "-F", SEP, "-c", sql], + text=True) + return [line.split(SEP) for line in out.splitlines() if line.strip()] + + +def main(): + print("=== dereth_go ingested row counts ===") + counts = q("dereth-go-db", "dereth_go", """ + SELECT 'telemetry_events', count(*)::text FROM telemetry_events + UNION ALL SELECT 'telemetry_distinct_chars', count(distinct character_name)::text FROM telemetry_events + UNION ALL SELECT 'character_stats', count(*)::text FROM character_stats + UNION ALL SELECT 'char_stats', count(*)::text FROM char_stats + UNION ALL SELECT 'rare_events', count(*)::text FROM rare_events + UNION ALL SELECT 'rare_stats', count(*)::text FROM rare_stats + UNION ALL SELECT 'portals', count(*)::text FROM portals + """) + for k, v in counts: + print(f" {k:26} {v}") + + print("\n=== character_stats exact match (same-timestamp rows) ===") + prod = {r[0]: (r[1], r[2]) for r in + q("dereth-db", "dereth", "SELECT character_name, timestamp::text, stats_data::text FROM character_stats")} + shadow = q("dereth-go-db", "dereth_go", + "SELECT character_name, timestamp::text, stats_data::text FROM character_stats") + match = mismatch = newer = 0 + for name, ts, sd in shadow: + if name not in prod: + continue + pts, psd = prod[name] + if ts != pts: + newer += 1 # one side got a newer character_stats message; not comparable + continue + if json.loads(sd) == json.loads(psd): + match += 1 + else: + mismatch += 1 + print(f" MISMATCH {name}") + print(f" exact match={match} mismatch={mismatch} skipped(diff timestamp)={newer}") + print("\nRESULT:", "ingest OK" if mismatch == 0 else f"{mismatch} character_stats mismatch(es)") + return 1 if mismatch else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/go-services/compare/compare_live.py b/go-services/compare/compare_live.py new file mode 100644 index 00000000..5208ac58 --- /dev/null +++ b/go-services/compare/compare_live.py @@ -0,0 +1,223 @@ +#!/usr/bin/env python3 +"""Compare the Go tracker's /live (and /trails) against the live Python service. + +Run on the server (or anywhere with loopback access to both): + python3 compare_live.py # default loopback ports + python3 compare_live.py --py http://127.0.0.1:8765 --go http://127.0.0.1:8770 + +Parity strategy for a live firehose +----------------------------------- +The two services rebuild their /live cache independently every 5s, so an +actively-updating character can legitimately show a newer telemetry row in one +than the other. We separate "is this a real divergence?" from "is this just +cache timing?" using the server-stamped received_at: + + * SAME ROW (py.received_at == go.received_at): both rendered the *same* + telemetry_events row -> every field MUST match (numbers within epsilon, + timestamps compared as instants). This is the rigorous render-parity proof. + * DIFFERENT ROW: a newer row arrived between the two refreshes -> we only + require identity + key-set + type/null-pattern parity, and report the + volatile-field skew (which should be small and recent). + +Exit code 0 if no real parity violations, 1 otherwise. +""" +import argparse +import json +import sys +import urllib.request +from datetime import datetime, timezone + +EPS = 1e-6 + +# Fields that identify the entity / join keys — must always match for a player +# present in both outputs. +IDENTITY = ("character_name", "char_tag", "session_id") +# Slowly-changing aggregates — informational when they differ on a same-row pair +# (a kill/rare recorded between refreshes can bump these even for the same +# telemetry row). +AGGREGATES = ("total_kills", "total_rares", "session_rares") +TIMESTAMP_FIELDS = ("timestamp", "received_at") + + +def fetch(base, path): + with urllib.request.urlopen(base.rstrip("/") + path, timeout=8) as r: + return json.load(r) + + +def jtype(v): + if v is None: + return "null" + if isinstance(v, bool): + return "bool" + if isinstance(v, (int, float)): + return "num" + if isinstance(v, str): + return "str" + return type(v).__name__ + + +def parse_ts(s): + if s is None: + return None + return datetime.fromisoformat(s.replace("Z", "+00:00")) + + +def values_equal(key, a, b): + """Semantic equality for a single field value.""" + if a is None or b is None: + return a is b or a == b + if key in TIMESTAMP_FIELDS and isinstance(a, str) and isinstance(b, str): + return parse_ts(a) == parse_ts(b) + an, bn = isinstance(a, (int, float)) and not isinstance(a, bool), isinstance(b, (int, float)) and not isinstance(b, bool) + if an and bn: + return abs(float(a) - float(b)) <= EPS + return a == b + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--py", default="http://127.0.0.1:8765") + ap.add_argument("--go", default="http://127.0.0.1:8770") + args = ap.parse_args() + + py = fetch(args.py, "/live")["players"] + go = fetch(args.go, "/live")["players"] + now = datetime.now(timezone.utc) + + pyi = {p["character_name"]: p for p in py} + goi = {p["character_name"]: p for p in go} + common = sorted(set(pyi) & set(goi)) + only_py = sorted(set(pyi) - set(goi)) + only_go = sorted(set(goi) - set(pyi)) + + print("=" * 72) + print("/live PARITY python(%s) vs go(%s)" % (args.py, args.go)) + print("=" * 72) + print(f"python players : {len(py)}") + print(f"go players : {len(go)}") + print(f"common : {len(common)}") + + violations = 0 + + # --- key-set parity (all players) --- + py_keys = set().union(*[set(p) for p in py]) if py else set() + go_keys = set().union(*[set(p) for p in go]) if go else set() + if py_keys == go_keys: + print(f"key set : IDENTICAL ({len(py_keys)} keys)") + else: + violations += 1 + print("key set : MISMATCH") + print(" only in python:", sorted(py_keys - go_keys)) + print(" only in go :", sorted(go_keys - py_keys)) + + # --- online-set parity (boundary-aware) --- + def age(p): + ts = parse_ts(p.get("received_at") or p.get("timestamp")) + return (now - ts).total_seconds() if ts else None + + print("\n-- online set --") + if not only_py and not only_go: + print("online set : IDENTICAL") + else: + # Players near the 30s boundary can flap between the two refreshes. + def explain(names, idx): + for n in names: + a = age(idx[n]) + tag = "boundary-flap (age %.1fs)" % a if a is not None and 22 <= a <= 38 else "age %s" % (None if a is None else round(a, 1)) + print(f" only_{('py' if idx is pyi else 'go')}: {n:<20} {tag}") + if only_py: + print(f"only in python : {len(only_py)}") + explain(only_py, pyi) + if only_go: + print(f"only in go : {len(only_go)}") + explain(only_go, goi) + unexplained = [n for n in (only_py + only_go) + if not (lambda a: a is not None and 22 <= a <= 38)(age((pyi.get(n) or goi.get(n))))] + if unexplained: + violations += 1 + print(" UNEXPLAINED set difference (not near 30s boundary):", unexplained) + else: + print(" (all set differences explained by the 30s online boundary)") + + # --- per-player field parity --- + same_row = [] # py.received_at == go.received_at -> must fully match + diff_row = [] # newer row arrived between refreshes + for n in common: + a, b = pyi[n], goi[n] + if a.get("received_at") is not None and a.get("received_at") == b.get("received_at"): + same_row.append(n) + else: + diff_row.append(n) + + print("\n-- per-player parity --") + print(f"same-row pairs (identical received_at, must fully match): {len(same_row)}") + print(f"diff-row pairs (newer telemetry between refreshes) : {len(diff_row)}") + + # Identity + type/null-pattern parity across ALL common players. + id_bad = type_bad = 0 + for n in common: + a, b = pyi[n], goi[n] + for k in IDENTITY: + if a.get(k) != b.get(k): + id_bad += 1 + print(f" IDENTITY mismatch {n}.{k}: py={a.get(k)!r} go={b.get(k)!r}") + for k in py_keys: + ta, tb = jtype(a.get(k)), jtype(b.get(k)) + if ta != tb: + # null vs num/str is a real null-pattern divergence; num-vs-num + # whole-float (0.0) vs int (0) is already unified under "num". + type_bad += 1 + print(f" TYPE mismatch {n}.{k}: py={ta}({a.get(k)!r}) go={tb}({b.get(k)!r})") + if id_bad: + violations += id_bad + if type_bad: + violations += type_bad + if not id_bad and not type_bad: + print("identity+type : IDENTICAL for all common players") + + # Rigorous: same-row pairs must match on every field. + sr_full_match = 0 + for n in same_row: + a, b = pyi[n], goi[n] + diffs = [] + for k in py_keys: + if not values_equal(k, a.get(k), b.get(k)): + diffs.append((k, a.get(k), b.get(k))) + if not diffs: + sr_full_match += 1 + else: + # Aggregate-only diffs are timing-explainable even on a same row. + non_agg = [d for d in diffs if d[0] not in AGGREGATES] + if non_agg: + violations += 1 + print(f" SAME-ROW FIELD divergence {n}: " + + ", ".join(f"{k}: py={pa!r} go={ga!r}" for k, pa, ga in non_agg)) + else: + print(f" (same-row {n}: only aggregate fields differ — kill/rare between refreshes: " + + ", ".join(f"{k} py={pa} go={ga}" for k, pa, ga in diffs) + ")") + print(f"same-row full-field matches: {sr_full_match}/{len(same_row)}") + + # Volatile-field skew on diff-row pairs (informational). + if diff_row: + ts_deltas = [] + for n in diff_row: + da, db = parse_ts(pyi[n].get("timestamp")), parse_ts(goi[n].get("timestamp")) + if da and db: + ts_deltas.append(abs((da - db).total_seconds())) + if ts_deltas: + ts_deltas.sort() + print(f"diff-row timestamp skew: min={ts_deltas[0]:.1f}s " + f"median={ts_deltas[len(ts_deltas)//2]:.1f}s max={ts_deltas[-1]:.1f}s " + "(bounded by the two 5s refresh cycles)") + + print("\n" + "=" * 72) + if violations == 0: + print("RESULT: PARITY OK — no structural or same-row divergences.") + else: + print(f"RESULT: {violations} PARITY VIOLATION(S) — see above.") + print("=" * 72) + return 1 if violations else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/go-services/discord-go/Dockerfile b/go-services/discord-go/Dockerfile new file mode 100644 index 00000000..b33d96e4 --- /dev/null +++ b/go-services/discord-go/Dockerfile @@ -0,0 +1,14 @@ +# Multi-stage build for the Go discord-rare-monitor port. The unit test (rare +# classification) runs at build time, so a classifier regression fails the build. +FROM golang:1.25-bookworm AS build +WORKDIR /src +COPY . . +RUN go mod tidy +RUN go test ./... +ARG BUILD_VERSION=dev +RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags "-s -w" -o /out/discord-go . + +# distroless/static includes CA certificates (needed for Discord's HTTPS REST API). +FROM gcr.io/distroless/static-debian12:nonroot +COPY --from=build /out/discord-go /discord-go +ENTRYPOINT ["/discord-go"] diff --git a/go-services/discord-go/classify.go b/go-services/discord-go/classify.go new file mode 100644 index 00000000..1893b83d --- /dev/null +++ b/go-services/discord-go/classify.go @@ -0,0 +1,93 @@ +package main + +// Rare classification — a faithful port of discord_rare_monitor.py's +// COMMON_RARES_PATTERN (an anchored exact-match regex of common-rare names). +// Because the regex is fully anchored with no wildcards, an exact-match set is +// equivalent. This list was extracted verbatim from the Python source, not +// hand-transcribed. classify_test.go asserts every entry maps to "common". +// +// classify returns "common" for an exact match, "great" otherwise — identical +// to classify_rare(). +func classify(rareName string) string { + if commonRares[rareName] { + return "common" + } + return "great" +} + +var commonRares = map[string]bool{ + "Alchemist's Crystal": true, + "Scholar's Crystal": true, + "Smithy's Crystal": true, + "Hunter's Crystal": true, + "Observer's Crystal": true, + "Thorsten's Crystal": true, + "Elysa's Crystal": true, + "Chef's Crystal": true, + "Enchanter's Crystal": true, + "Oswald's Crystal": true, + "Deceiver's Crystal": true, + "Fletcher's Crystal": true, + "Physician's Crystal": true, + "Artificer's Crystal": true, + "Tinker's Crystal": true, + "Vaulter's Crystal": true, + "Monarch's Crystal": true, + "Life Giver's Crystal": true, + "Thief's Crystal": true, + "Adherent's Crystal": true, + "Resister's Crystal": true, + "Imbuer's Crystal": true, + "Converter's Crystal": true, + "Evader's Crystal": true, + "Dodger's Crystal": true, + "Zefir's Crystal": true, + "Ben Ten's Crystal": true, + "Corruptor's Crystal": true, + "Artist's Crystal": true, + "T'ing's Crystal": true, + "Warrior's Crystal": true, + "Brawler's Crystal": true, + "Hieromancer's Crystal": true, + "Rogue's Crystal": true, + "Berzerker's Crystal": true, + "Knight's Crystal": true, + "Lugian's Pearl": true, + "Ursuin's Pearl": true, + "Wayfarer's Pearl": true, + "Sprinter's Pearl": true, + "Magus's Pearl": true, + "Lich's Pearl": true, + "Warrior's Jewel": true, + "Melee's Jewel": true, + "Mage's Jewel": true, + "Duelist's Jewel": true, + "Archer's Jewel": true, + "Tusker's Jewel": true, + "Olthoi's Jewel": true, + "Inferno's Jewel": true, + "Gelid's Jewel": true, + "Astyrrian's Jewel": true, + "Executor's Jewel": true, + "Pearl of Blood Drinking": true, + "Pearl of Heart Seeking": true, + "Pearl of Defending": true, + "Pearl of Swift Killing": true, + "Pearl of Spirit Drinking": true, + "Pearl of Hermetic Linking": true, + "Pearl of Blade Baning": true, + "Pearl of Pierce Baning": true, + "Pearl of Bludgeon Baning": true, + "Pearl of Acid Baning": true, + "Pearl of Flame Baning": true, + "Pearl of Frost Baning": true, + "Pearl of Lightning Baning": true, + "Pearl of Impenetrability": true, + "Refreshing Elixir": true, + "Invigorating Elixir": true, + "Miraculous Elixir": true, + "Medicated Health Kit": true, + "Medicated Stamina Kit": true, + "Medicated Mana Kit": true, + "Casino Exquisite Keyring": true, +} diff --git a/go-services/discord-go/classify_test.go b/go-services/discord-go/classify_test.go new file mode 100644 index 00000000..1e4a3e36 --- /dev/null +++ b/go-services/discord-go/classify_test.go @@ -0,0 +1,38 @@ +package main + +import "testing" + +// Every name in the common-rares set must classify as "common". +func TestClassifyCommon(t *testing.T) { + if len(commonRares) != 74 { + t.Fatalf("expected 74 common rares, got %d", len(commonRares)) + } + for name := range commonRares { + if got := classify(name); got != "common" { + t.Errorf("classify(%q) = %q, want common", name, got) + } + } +} + +// Names not in the set (including near-misses) must classify as "great". +func TestClassifyGreat(t *testing.T) { + greats := []string{ + "Shimmering Skeleton Key", + "Star of Tukal", + "Hieroglyph of the Bludgeoner", + "Infinite Phial of Pyreal Flux", + "Foolproof Hooks", + "Staff of All Aphus", + "Count Renari's Equctioneer", + "Gelidite Long Sword", + "Pearl of Blade Baning ", // trailing space — not an exact match + "alchemist's crystal", // wrong case — not an exact match + "Alchemist's Crystals", // plural — not an exact match + "", + } + for _, name := range greats { + if got := classify(name); got != "great" { + t.Errorf("classify(%q) = %q, want great", name, got) + } + } +} diff --git a/go-services/discord-go/go.mod b/go-services/discord-go/go.mod new file mode 100644 index 00000000..9b5552ab --- /dev/null +++ b/go-services/discord-go/go.mod @@ -0,0 +1,3 @@ +module git.snakedesert.se/SawatoMosswartsEnjoyersClub/MosswartOverlord/go-services/discord-go + +go 1.25 diff --git a/go-services/discord-go/main.go b/go-services/discord-go/main.go new file mode 100644 index 00000000..51cfa9ae --- /dev/null +++ b/go-services/discord-go/main.go @@ -0,0 +1,90 @@ +// Command discord-go is a Go port of discord-rare-monitor: it consumes the +// tracker's /ws/live firehose (subscribed to rare+chat), classifies rares +// common/great, posts embeds to Discord, and relays allegiance chat. +// +// SAFETY: it runs in dry-run (log only, no Discord posts) by default. Going live +// requires BOTH a bot token AND DRY_RUN=0 — so it can never accidentally +// double-post to the production channels during the parallel run. For a parallel +// test, set a TEST token + TEST channel IDs + DRY_RUN=0. +package main + +import ( + "context" + "fmt" + "log/slog" + "os" + "os/signal" + "sort" + "syscall" + + "github.com/bwmarrin/discordgo" +) + +func main() { + // `discord-go dump-rares` prints the common-rares set (for parity diffing + // against the Python regex). No network, no token. + if len(os.Args) > 1 && os.Args[1] == "dump-rares" { + names := make([]string, 0, len(commonRares)) + for n := range commonRares { + names = append(names, n) + } + sort.Strings(names) + for _, n := range names { + fmt.Println(n) + } + return + } + + logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo})) + slog.SetDefault(logger) + + token := os.Getenv("DISCORD_RARE_BOT_TOKEN") + // Dry-run unless a token is present AND DRY_RUN is explicitly "0". + dryRun := token == "" || os.Getenv("DRY_RUN") != "0" + + wsURL := envOr("DERETH_TRACKER_WS_URL", "ws://dereth-tracker:8765/ws/live") + monitorChar := envOr("MONITOR_CHARACTER", "Dunking Rares") + + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) + defer stop() + + var out poster + if dryRun { + reason := "no DISCORD_RARE_BOT_TOKEN" + if token != "" { + reason = "DRY_RUN != 0" + } + logger.Info("starting in DRY-RUN — classifying but NOT posting to Discord", "reason", reason, "ws", wsURL, "monitor", monitorChar) + out = &logPoster{log: logger} + } else { + dg, err := discordgo.New("Bot " + token) + if err != nil { + logger.Error("discord session init failed", "err", err) + os.Exit(1) + } + // REST-only: we send by channel ID, so no gateway Open()/intents needed. + logger.Info("starting LIVE — posting to Discord", "ws", wsURL, "monitor", monitorChar) + out = &discordPoster{ + dg: dg, + common: envOr("COMMON_RARE_CHANNEL_ID", "1355328792184226014"), + great: envOr("GREAT_RARE_CHANNEL_ID", "1353676584334131211"), + aclog: envOr("ACLOG_CHANNEL_ID", "1349649482786275328"), + sawato: envOr("SAWATOLIFE_CHANNEL_ID", "1387323032271327423"), + iconsDir: envOr("ICONS_DIR", "icons"), + log: logger, + } + } + + b := &bot{wsURL: wsURL, monitorChar: monitorChar, out: out, log: logger} + go b.run(ctx) + + <-ctx.Done() + logger.Info("shutdown signal received") +} + +func envOr(key, def string) string { + if v := os.Getenv(key); v != "" { + return v + } + return def +} diff --git a/go-services/discord-go/poster.go b/go-services/discord-go/poster.go new file mode 100644 index 00000000..7510f995 --- /dev/null +++ b/go-services/discord-go/poster.go @@ -0,0 +1,166 @@ +package main + +import ( + "fmt" + "log/slog" + "os" + "path/filepath" + "strings" + "time" + + "github.com/bwmarrin/discordgo" +) + +// Discord embed colors, matching discord.py's Color.gold()/Color.blue(). +const ( + colorGold = 0xf1c40f + colorBlue = 0x3498db + colorRed = 0xe74c3c +) + +type rareEvent struct { + Name string + CharacterName string + Timestamp string + EW, NS, Z *float64 +} + +// poster abstracts where messages go: a real Discord session, or a dry-run +// logger used for parallel validation without a bot token / live channels. +type poster interface { + postRare(ev rareEvent, tier string) + postChat(charName, text, ts string) + postVortex(speaker, text, ts string) + postStatus(text string) +} + +// ---- dry-run (log-only) ---- + +type logPoster struct{ log *slog.Logger } + +func (p *logPoster) postRare(ev rareEvent, tier string) { + p.log.Info("DRY-RUN would post rare", "tier", tier, "channel", tier, "name", ev.Name, "character", ev.CharacterName) +} +func (p *logPoster) postChat(charName, text, ts string) { + p.log.Info("DRY-RUN would relay chat", "character", charName, "text", text) +} +func (p *logPoster) postVortex(speaker, text, ts string) { + p.log.Warn("DRY-RUN would post vortex warning", "speaker", speaker, "text", text) +} +func (p *logPoster) postStatus(text string) { + p.log.Info("DRY-RUN would post status", "text", text) +} + +// ---- real Discord ---- + +type discordPoster struct { + dg *discordgo.Session + common string + great string + aclog string + sawato string + iconsDir string + log *slog.Logger +} + +func (p *discordPoster) postRare(ev rareEvent, tier string) { + embed := buildRareEmbed(ev, tier) + channel := p.common + if tier == "great" { + channel = p.great + } + if iconPath := p.iconPath(ev.Name); iconPath != "" { + if f, err := os.Open(iconPath); err == nil { + defer f.Close() + fn := filepath.Base(iconPath) + embed.Image = &discordgo.MessageEmbedImage{URL: "attachment://" + fn} + if _, err := p.dg.ChannelMessageSendComplex(channel, &discordgo.MessageSend{ + Embed: embed, + Files: []*discordgo.File{{Name: fn, Reader: f}}, + }); err != nil { + p.log.Error("send rare embed (with icon) failed", "err", err, "channel", channel) + } + return + } + } + if _, err := p.dg.ChannelMessageSendEmbed(channel, embed); err != nil { + p.log.Error("send rare embed failed", "err", err, "channel", channel) + } +} + +func (p *discordPoster) postChat(charName, text, ts string) { + t := parseTime(ts) + cleaned := strings.TrimPrefix(text, "Dunking Rares: ") + msg := fmt.Sprintf("`%s` %s", t.Format("15:04:05"), cleaned) + if _, err := p.dg.ChannelMessageSend(p.sawato, msg); err != nil { + p.log.Error("send chat failed", "err", err) + } +} + +func (p *discordPoster) postVortex(speaker, text, ts string) { + embed := &discordgo.MessageEmbed{ + Title: "🌪️ VORTEX WARNING", + Description: fmt.Sprintf("**%s**: %s", speaker, text), + Color: colorRed, + Timestamp: parseTime(ts).Format(time.RFC3339), + } + if _, err := p.dg.ChannelMessageSendEmbed(p.aclog, embed); err != nil { + p.log.Error("send vortex failed", "err", err) + } +} + +func (p *discordPoster) postStatus(text string) { + if _, err := p.dg.ChannelMessageSend(p.aclog, text); err != nil { + p.log.Error("send status failed", "err", err) + } +} + +func (p *discordPoster) iconPath(rareName string) string { + if p.iconsDir == "" { + return "" + } + fn := strings.NewReplacer("'", "", " ", "_", "-", "_").Replace(rareName) + "_Icon.png" + path := filepath.Join(p.iconsDir, fn) + if _, err := os.Stat(path); err == nil { + return path + } + return "" +} + +// buildRareEmbed mirrors post_rare_to_discord's embed. +func buildRareEmbed(ev rareEvent, tier string) *discordgo.MessageEmbed { + title, color := "🔸 Common Rare Discovery", colorBlue + if tier == "great" { + title, color = "💎 Great Rare Discovery!", colorGold + } + t := parseTime(ev.Timestamp) + embed := &discordgo.MessageEmbed{ + Title: title, + Description: fmt.Sprintf("**%s** has discovered the **%s**!", ev.CharacterName, ev.Name), + Color: color, + Timestamp: t.Format(time.RFC3339), + } + if ev.EW != nil && ev.NS != nil { + loc := fmt.Sprintf("%.1fE, %.1fN", *ev.EW, *ev.NS) + if ev.Z != nil { + loc += fmt.Sprintf(", %.1fZ", *ev.Z) + } + embed.Fields = append(embed.Fields, &discordgo.MessageEmbedField{Name: "📍 Location", Value: loc, Inline: true}) + } + embed.Fields = append(embed.Fields, &discordgo.MessageEmbedField{ + Name: "⏰ Time", Value: t.UTC().Format("15:04:05") + " UTC", Inline: true, + }) + return embed +} + +// parseTime accepts the plugin's ISO8601 (with or without 'Z'); falls back to now. +func parseTime(ts string) time.Time { + if ts != "" { + for _, layout := range []string{time.RFC3339Nano, time.RFC3339, "2006-01-02T15:04:05.999999", "2006-01-02T15:04:05"} { + if t, err := time.Parse(layout, strings.Replace(ts, "Z", "+00:00", 1)); err == nil { + return t + } + } + } + return time.Now().UTC() +} diff --git a/go-services/discord-go/ws.go b/go-services/discord-go/ws.go new file mode 100644 index 00000000..ed60e58a --- /dev/null +++ b/go-services/discord-go/ws.go @@ -0,0 +1,159 @@ +package main + +import ( + "context" + "encoding/json" + "log/slog" + "strings" + "time" + + "github.com/coder/websocket" +) + +// bot consumes the tracker's /ws/live firehose (subscribed to rare+chat) and +// routes events to a poster. It reconnects with exponential backoff, mirroring +// monitor_websocket(). +type bot struct { + wsURL string + monitorChar string + out poster + log *slog.Logger +} + +func (b *bot) run(ctx context.Context) { + backoff := time.Second + const maxBackoff = 60 * time.Second + for ctx.Err() == nil { + err := b.connectAndConsume(ctx) + if ctx.Err() != nil { + return + } + if err != nil { + b.log.Warn("ws disconnected; reconnecting", "err", err, "backoff", backoff.String()) + } + select { + case <-ctx.Done(): + return + case <-time.After(backoff): + } + backoff *= 2 + if backoff > maxBackoff { + backoff = maxBackoff + } + } +} + +func (b *bot) connectAndConsume(ctx context.Context) error { + b.log.Info("connecting to /ws/live", "url", b.wsURL) + c, _, err := websocket.Dial(ctx, b.wsURL, nil) + if err != nil { + return err + } + defer c.CloseNow() + c.SetReadLimit(8 << 20) // payloads (nearby_objects etc.) can be large; we only read rare/chat but the socket carries all + + // Subscribe to just rare + chat (server-side filtering), like the Python bot. + sub, _ := json.Marshal(map[string]any{"type": "subscribe", "message_types": []string{"rare", "chat"}}) + if err := c.Write(ctx, websocket.MessageText, sub); err != nil { + return err + } + b.log.Info("subscribed", "message_types", []string{"rare", "chat"}) + b.out.postStatus("🔗 (go) WebSocket connection established") + + // Keepalive pings, like ping_interval=20. + pingCtx, cancel := context.WithCancel(ctx) + defer cancel() + go func() { + t := time.NewTicker(20 * time.Second) + defer t.Stop() + for { + select { + case <-pingCtx.Done(): + return + case <-t.C: + pc, cc := context.WithTimeout(pingCtx, 10*time.Second) + _ = c.Ping(pc) + cc() + } + } + }() + + for { + _, data, err := c.Read(ctx) + if err != nil { + return err + } + b.handleMessage(data) + } +} + +func (b *bot) handleMessage(raw []byte) { + var data map[string]any + if err := json.Unmarshal(raw, &data); err != nil { + return // ignore invalid JSON, like the Python bot + } + switch asString(data["type"]) { + case "rare": + b.handleRare(data) + case "chat": + b.handleChat(data) + } +} + +func (b *bot) handleRare(data map[string]any) { + ev := rareEvent{ + Name: asStringOr(data["name"], "Unknown Rare"), + CharacterName: asStringOr(data["character_name"], "Unknown Character"), + Timestamp: asString(data["timestamp"]), + EW: asFloatPtr(data["ew"]), + NS: asFloatPtr(data["ns"]), + Z: asFloatPtr(data["z"]), + } + tier := classify(ev.Name) + b.log.Info("rare", "name", ev.Name, "character", ev.CharacterName, "tier", tier) + b.out.postRare(ev, tier) +} + +func (b *bot) handleChat(data map[string]any) { + charName := asString(data["character_name"]) + text := asString(data["text"]) + if charName != b.monitorChar { + return + } + if strings.Contains(text, "m in whirlwind of vortexes") { + b.out.postVortex(parseAllegianceSpeaker(text), text, asString(data["timestamp"])) + return + } + b.out.postChat(charName, text, asString(data["timestamp"])) +} + +// parseAllegianceSpeaker extracts from "[Allegiance] says, ...". +func parseAllegianceSpeaker(text string) string { + const prefix = "[Allegiance] " + if i := strings.Index(text, prefix); i >= 0 { + rest := text[i+len(prefix):] + if j := strings.Index(rest, " says,"); j >= 0 { + return rest[:j] + } + } + return "Unknown" +} + +func asString(v any) string { + s, _ := v.(string) + return s +} + +func asStringOr(v any, def string) string { + if s, ok := v.(string); ok && s != "" { + return s + } + return def +} + +func asFloatPtr(v any) *float64 { + if f, ok := v.(float64); ok { + return &f + } + return nil +} diff --git a/go-services/docker-compose.cutover.yml b/go-services/docker-compose.cutover.yml new file mode 100644 index 00000000..1f813918 --- /dev/null +++ b/go-services/docker-compose.cutover.yml @@ -0,0 +1,32 @@ +# Cutover override — flips the Go services from read-only parallel mode to +# PRODUCTION write mode, reusing the existing production databases (no data +# migration). Apply ON TOP of the base + go overrides: +# +# docker compose -f docker-compose.yml -f go-services/docker-compose.go.yml \ +# -f go-services/docker-compose.cutover.yml up -d --no-deps \ +# dereth-tracker-go inventory-go discord-rare-monitor +# +# Reversible: re-up WITHOUT this file to return the Go services to read-only +# parallel mode (and start the Python services back up for rollback). +# +# SKIP_SCHEMA_INIT=true makes the Go services trust the existing prod schema and +# run NO DDL. The Go tracker writes prod `dereth`; inventory-go writes prod +# `inventory_db`; the (still Python) rare/chat bot is repointed at the Go +# tracker's /ws/live (proven posting path, fed by Go data). +services: + dereth-tracker-go: + environment: + READ_ONLY: "false" + SKIP_SCHEMA_INIT: "true" + SHARED_SECRET: "${SHARED_SECRET}" + SHARED_SECRET_LEGACY: "${SHARED_SECRET_LEGACY:-}" + DISCORD_ACLOG_WEBHOOK: "${DISCORD_ACLOG_WEBHOOK}" + + inventory-go: + environment: + READ_ONLY: "false" + SKIP_SCHEMA_INIT: "true" + + discord-rare-monitor: + environment: + DERETH_TRACKER_WS_URL: "ws://dereth-tracker-go:8770/ws/live" diff --git a/go-services/docker-compose.go.yml b/go-services/docker-compose.go.yml new file mode 100644 index 00000000..2e64c737 --- /dev/null +++ b/go-services/docker-compose.go.yml @@ -0,0 +1,212 @@ +# Compose OVERRIDE that adds the Go services alongside the live Python stack. +# It only ADDS containers; it never modifies the tracked docker-compose.yml or +# any running Python service. +# +# Invoke from the repo root so the Compose project name resolves to +# "mosswartoverlord" (same as the live stack) and the new container joins the +# existing default network — letting it reach the `db` service by name: +# +# cd /home/erik/MosswartOverlord +# export BUILD_VERSION="$(date -u +%Y.%-m.%-d.%H%M)-$(git rev-parse --short HEAD)" +# docker compose -f docker-compose.yml -f go-services/docker-compose.go.yml \ +# build dereth-tracker-go +# docker compose -f docker-compose.yml -f go-services/docker-compose.go.yml \ +# up -d --no-deps dereth-tracker-go +# +# --no-deps keeps Compose from touching the already-running `db` (and anything +# else). The service is loopback-bound (127.0.0.1:8770); external reach is only +# ever via the host nginx `location /go/` block (added separately). +services: + dereth-tracker-go: + build: + context: ./go-services/tracker-go + args: + BUILD_VERSION: ${BUILD_VERSION:-dev} + image: dereth-tracker-go:local + container_name: dereth-tracker-go + ports: + - "127.0.0.1:8770:8770" + environment: + PORT: "8770" + # Read-only use of the same dereth TimescaleDB the Python tracker writes. + DATABASE_URL: "postgresql://postgres:${POSTGRES_PASSWORD}@db:5432/dereth" + # Point at the Go inventory service so the /go/ read stack is fully Go + # end-to-end (browser -> Go tracker -> Go inventory -> read-only prod DBs). + # inventory-go is read-only against the production inventory_db. + INVENTORY_SERVICE_URL: "http://inventory-go:8772" + # Same signing key as the Python tracker so the same login cookie verifies + # on both during the parallel run. + SECRET_KEY: "${SECRET_KEY}" + # Serve the (unchanged) frontend from the same static/ the Python tracker + # serves — needed for the full cutover (login, index.html, assets, icons). + STATIC_DIR: "/static" + LOG_LEVEL: "INFO" + volumes: + - ./static:/static:ro + # Issue board is a flat file the tracker writes; mount it read-write + # (more specific than the :ro static mount above, so it wins). + - ./static/openissues.json:/static/openissues.json + depends_on: + - db + restart: unless-stopped + logging: + driver: "json-file" + options: + max-size: "10m" + max-file: "3" + + # Go port of discord-rare-monitor. Consumes the SAME Python /ws/live firehose + # as the live Python bot. DRY-RUN by default (logs classifications, posts + # nothing) so it can't double-post. To parallel-test for real, set a TEST + # DISCORD_RARE_BOT_TOKEN + TEST channel IDs + DRY_RUN=0 here. + discord-rare-monitor-go: + build: + context: ./go-services/discord-go + args: + BUILD_VERSION: ${BUILD_VERSION:-dev} + container_name: discord-rare-monitor-go + environment: + DERETH_TRACKER_WS_URL: "ws://dereth-tracker:8765/ws/live" + MONITOR_CHARACTER: "Dunking Rares" + ICONS_DIR: "/icons" + LOG_LEVEL: "INFO" + # DISCORD_RARE_BOT_TOKEN: "" # set a TEST token to go live + # DRY_RUN: "0" # required (with a token) to actually post + # COMMON_RARE_CHANNEL_ID / GREAT_RARE_CHANNEL_ID / SAWATOLIFE_CHANNEL_ID / + # ACLOG_CHANNEL_ID: set TEST channels before going live + volumes: + - ./discord-rare-monitor/icons:/icons:ro + depends_on: + - dereth-tracker + restart: unless-stopped + logging: + driver: "json-file" + options: + max-size: "10m" + max-file: "3" + + # ---- Phase 2: shadow ingest (fully isolated; production never touched) ---- + + # A SEPARATE TimescaleDB the Go tracker owns for shadow ingest. Isolated + # volume + loopback port; the production dereth DB is never written. + dereth-go-db: + image: timescale/timescaledb:2.19.3-pg14 + container_name: dereth-go-db + ports: + - "127.0.0.1:5434:5432" + environment: + POSTGRES_DB: "dereth_go" + POSTGRES_USER: "postgres" + POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}" + volumes: + - dereth-go-data:/var/lib/postgresql/data + restart: unless-stopped + logging: + driver: "json-file" + options: + max-size: "10m" + max-file: "3" + + # Shadow tracker instance: same image, but OWNS dereth-go-db (read-write) and + # (once ingest lands) consumes the Python /ws/live firehose into it, so its + # ingest output can be compared against production without writing to it. + dereth-tracker-go-shadow: + image: dereth-tracker-go:local + container_name: dereth-tracker-go-shadow + ports: + - "127.0.0.1:8771:8771" + environment: + PORT: "8771" + DATABASE_URL: "postgresql://postgres:${POSTGRES_PASSWORD}@dereth-go-db:5432/dereth_go" + READ_ONLY: "false" # owns its DB; creates schema on boot + INVENTORY_SERVICE_URL: "http://inventory-service:8000" + SECRET_KEY: "${SECRET_KEY}" + SHARED_SECRET: "${SHARED_SECRET}" # /ws/position plugin auth (cutover-ready) + SHARED_SECRET_LEGACY: "${SHARED_SECRET_LEGACY:-}" + # Replay the Python /ws/live firehose into the ingest handlers (shadow). + SHADOW_INGEST_WS: "ws://dereth-tracker:8765/ws/live" + LOG_LEVEL: "INFO" + depends_on: + - dereth-go-db + restart: unless-stopped + logging: + driver: "json-file" + options: + max-size: "10m" + max-file: "3" + + # Go port of inventory-service. Phase A: read side, READ-ONLY against the + # production inventory_db, validated vs the Python service. Loopback :8772. + inventory-go: + build: + context: ./go-services/inventory-go + args: + BUILD_VERSION: ${BUILD_VERSION:-dev} + image: inventory-go:local + container_name: inventory-go + ports: + - "127.0.0.1:8772:8772" + environment: + PORT: "8772" + DATABASE_URL: "postgresql://inventory_user:${INVENTORY_DB_PASSWORD}@inventory-db:5432/inventory_db" + READ_ONLY: "true" + ENUM_DB_PATH: "/enums/comprehensive_enum_database_v2.json" + LOG_LEVEL: "INFO" + volumes: + - ./inventory-service/comprehensive_enum_database_v2.json:/enums/comprehensive_enum_database_v2.json:ro + depends_on: + - inventory-db + restart: unless-stopped + logging: + driver: "json-file" + options: + max-size: "10m" + max-file: "3" + + # Phase C: isolated inventory DB the Go ingestion writes to (never production). + inventory-go-db: + image: postgres:14 + container_name: inventory-go-db + ports: + - "127.0.0.1:5435:5432" + environment: + POSTGRES_DB: "inventory_db" + POSTGRES_USER: "inventory_user" + POSTGRES_PASSWORD: "${INVENTORY_DB_PASSWORD}" + volumes: + - inventory-go-data:/var/lib/postgresql/data + restart: unless-stopped + logging: + driver: "json-file" + options: + max-size: "10m" + max-file: "3" + + # Read-write inventory-go instance: owns inventory-go-db, exposes the ingestion + # endpoints. Used to validate ingestion (POST a character's items, compare the + # resulting normalized rows to production) without touching the production DB. + inventory-go-shadow: + image: inventory-go:local + container_name: inventory-go-shadow + ports: + - "127.0.0.1:8773:8773" + environment: + PORT: "8773" + DATABASE_URL: "postgresql://inventory_user:${INVENTORY_DB_PASSWORD}@inventory-go-db:5432/inventory_db" + READ_ONLY: "false" + ENUM_DB_PATH: "/enums/comprehensive_enum_database_v2.json" + LOG_LEVEL: "INFO" + volumes: + - ./inventory-service/comprehensive_enum_database_v2.json:/enums/comprehensive_enum_database_v2.json:ro + depends_on: + - inventory-go-db + restart: unless-stopped + logging: + driver: "json-file" + options: + max-size: "10m" + max-file: "3" + +volumes: + dereth-go-data: + inventory-go-data: diff --git a/go-services/inventory-go/Dockerfile b/go-services/inventory-go/Dockerfile new file mode 100644 index 00000000..3a9d8097 --- /dev/null +++ b/go-services/inventory-go/Dockerfile @@ -0,0 +1,13 @@ +FROM golang:1.25-bookworm AS build +WORKDIR /src +COPY . . +RUN go mod tidy +RUN go test ./... +ARG BUILD_VERSION=dev +RUN CGO_ENABLED=0 GOOS=linux go build \ + -trimpath -ldflags "-s -w -X main.buildVersion=${BUILD_VERSION}" -o /out/inventory-go . + +FROM gcr.io/distroless/static-debian12:nonroot +COPY --from=build /out/inventory-go /inventory-go +EXPOSE 8772 +ENTRYPOINT ["/inventory-go"] diff --git a/go-services/inventory-go/go.mod b/go-services/inventory-go/go.mod new file mode 100644 index 00000000..e0615528 --- /dev/null +++ b/go-services/inventory-go/go.mod @@ -0,0 +1,5 @@ +module git.snakedesert.se/SawatoMosswartsEnjoyersClub/MosswartOverlord/go-services/inventory-go + +go 1.25 + +require github.com/jackc/pgx/v5 v5.10.0 diff --git a/go-services/inventory-go/ingest.go b/go-services/inventory-go/ingest.go new file mode 100644 index 00000000..04ab99e3 --- /dev/null +++ b/go-services/inventory-go/ingest.go @@ -0,0 +1,266 @@ +package main + +import ( + "context" + "encoding/json" + "io" + "net/http" + "sort" + "strconv" + "strings" + "time" + + "github.com/jackc/pgx/v5" +) + +// Ingestion endpoints — port of process_inventory / upsert_inventory_item / +// delete_inventory_item. They write to THIS instance's own DB (ingest mode, +// READ_ONLY=false), reusing the validated item-processor. Production is never +// written: an isolated inventory-go-db backs the shadow instance. + +func quoteCol(k string) string { + if k == "unique" { + return `"unique"` + } + return k +} + +func buildInsert(table string, cols map[string]any, returningID bool) (string, []any) { + keys := make([]string, 0, len(cols)) + for k := range cols { + keys = append(keys, k) + } + sort.Strings(keys) + qc := make([]string, len(keys)) + ph := make([]string, len(keys)) + args := make([]any, len(keys)) + for i, k := range keys { + qc[i] = quoteCol(k) + ph[i] = "$" + strconv.Itoa(i+1) + args[i] = cols[k] + } + sql := "INSERT INTO " + table + " (" + strings.Join(qc, ", ") + ") VALUES (" + strings.Join(ph, ", ") + ")" + if returningID { + sql += " RETURNING id" + } + return sql, args +} + +var childTables = []struct{ table, key string }{ + {"item_combat_stats", "combat"}, + {"item_requirements", "requirements"}, + {"item_enhancements", "enhancements"}, + {"item_ratings", "ratings"}, +} + +// ingestItem processes one raw item and inserts it across the 7 tables. +func (s *Server) ingestItem(ctx context.Context, tx pgx.Tx, charName string, ts time.Time, raw map[string]any) error { + p := s.processItem(raw) + items := p["items"].(map[string]any) + items["character_name"] = charName + items["timestamp"] = ts + sql, args := buildInsert("items", items, true) + var id int + if err := tx.QueryRow(ctx, sql, args...).Scan(&id); err != nil { + return err + } + for _, ct := range childTables { + cols, _ := p[ct.key].(map[string]any) + if cols == nil { + continue // table skipped (all-sentinel) + } + cols["item_id"] = id + csql, cargs := buildInsert(ct.table, cols, false) + if _, err := tx.Exec(ctx, csql, cargs...); err != nil { + return err + } + } + if rows, ok := p["spells"].([]map[string]any); ok { + for _, sp := range rows { + if _, err := tx.Exec(ctx, + "INSERT INTO item_spells (item_id, spell_id, is_active) VALUES ($1,$2,$3) ON CONFLICT DO NOTHING", + id, sp["spell_id"], sp["is_active"]); err != nil { + return err + } + } + } + ivb, _ := json.Marshal(bag(raw, "IntValues")) + dvb, _ := json.Marshal(bag(raw, "DoubleValues")) + svb, _ := json.Marshal(bag(raw, "StringValues")) + bvb, _ := json.Marshal(bag(raw, "BoolValues")) + ojb, _ := json.Marshal(raw) + _, err := tx.Exec(ctx, + "INSERT INTO item_raw_data (item_id,int_values,double_values,string_values,bool_values,original_json) VALUES ($1,$2,$3,$4,$5,$6)", + id, ivb, dvb, svb, bvb, ojb) + return err +} + +// deleteCharItems removes a character's rows across all tables (children first). +func deleteCharItems(ctx context.Context, tx pgx.Tx, charName string) error { + var ids []int + rows, err := tx.Query(ctx, "SELECT id FROM items WHERE character_name=$1", charName) + if err != nil { + return err + } + for rows.Next() { + var id int + if err := rows.Scan(&id); err != nil { + rows.Close() + return err + } + ids = append(ids, id) + } + rows.Close() + if len(ids) > 0 { + for _, t := range []string{"item_raw_data", "item_combat_stats", "item_requirements", "item_enhancements", "item_ratings", "item_spells"} { + if _, err := tx.Exec(ctx, "DELETE FROM "+t+" WHERE item_id = ANY($1)", ids); err != nil { + return err + } + } + } + _, err = tx.Exec(ctx, "DELETE FROM items WHERE character_name=$1", charName) + return err +} + +func deleteOneItem(ctx context.Context, tx pgx.Tx, charName string, itemID int64) error { + var id int + err := tx.QueryRow(ctx, "SELECT id FROM items WHERE character_name=$1 AND item_id=$2", charName, itemID).Scan(&id) + if err == pgx.ErrNoRows { + return nil + } + if err != nil { + return err + } + for _, t := range []string{"item_raw_data", "item_combat_stats", "item_requirements", "item_enhancements", "item_ratings", "item_spells"} { + if _, err := tx.Exec(ctx, "DELETE FROM "+t+" WHERE item_id=$1", id); err != nil { + return err + } + } + _, err = tx.Exec(ctx, "DELETE FROM items WHERE id=$1", id) + return err +} + +// POST /process-inventory — full replacement of a character's inventory. +func (s *Server) handleProcessInventory(w http.ResponseWriter, r *http.Request) { + body, _ := io.ReadAll(io.LimitReader(r.Body, 64<<20)) + var inv struct { + CharacterName string `json:"character_name"` + Timestamp string `json:"timestamp"` + Items []map[string]any `json:"items"` + } + if json.Unmarshal(body, &inv) != nil || inv.CharacterName == "" { + writeJSON(w, http.StatusBadRequest, map[string]any{"detail": "invalid payload"}) + return + } + ts := parseNaiveTime(inv.Timestamp) + ctx, cancel := context.WithTimeout(r.Context(), 120*time.Second) + defer cancel() + tx, err := s.pool.Begin(ctx) + if err != nil { + s.dbErr(w, "process-inventory begin", err) + return + } + defer tx.Rollback(ctx) + if err := deleteCharItems(ctx, tx, inv.CharacterName); err != nil { + s.dbErr(w, "process-inventory delete", err) + return + } + processed, errs := 0, 0 + for _, raw := range inv.Items { + if raw["Id"] == nil && raw["id"] == nil { + errs++ + continue + } + if err := s.ingestItem(ctx, tx, inv.CharacterName, ts, raw); err != nil { + s.log.Error("ingest item failed", "err", err, "char", inv.CharacterName) + errs++ + continue + } + processed++ + } + if err := tx.Commit(ctx); err != nil { + s.dbErr(w, "process-inventory commit", err) + return + } + writeJSON(w, http.StatusOK, map[string]any{"processed_count": processed, "error_count": errs, "total_items": len(inv.Items)}) +} + +// POST /inventory/{character_name}/item — single-item upsert. +func (s *Server) handleUpsertItem(w http.ResponseWriter, r *http.Request) { + char := r.PathValue("character_name") + body, _ := io.ReadAll(io.LimitReader(r.Body, 16<<20)) + var raw map[string]any + if json.Unmarshal(body, &raw) != nil { + writeJSON(w, http.StatusBadRequest, map[string]any{"detail": "invalid JSON"}) + return + } + if raw["Id"] == nil && raw["id"] == nil { + writeJSON(w, http.StatusBadRequest, map[string]any{"detail": "item missing Id"}) + return + } + itemID := int64(toFloat(firstNonNil(raw["Id"], raw["id"]))) + ctx, cancel := context.WithTimeout(r.Context(), 30*time.Second) + defer cancel() + tx, err := s.pool.Begin(ctx) + if err != nil { + s.dbErr(w, "upsert begin", err) + return + } + defer tx.Rollback(ctx) + if err := deleteOneItem(ctx, tx, char, itemID); err != nil { + s.dbErr(w, "upsert delete", err) + return + } + if err := s.ingestItem(ctx, tx, char, time.Now().UTC(), raw); err != nil { + s.dbErr(w, "upsert insert", err) + return + } + if err := tx.Commit(ctx); err != nil { + s.dbErr(w, "upsert commit", err) + return + } + writeJSON(w, http.StatusOK, map[string]any{"status": "ok", "item_id": itemID}) +} + +// DELETE /inventory/{character_name}/item/{item_id} +func (s *Server) handleDeleteItem(w http.ResponseWriter, r *http.Request) { + char := r.PathValue("character_name") + itemID, _ := strconv.ParseInt(r.PathValue("item_id"), 10, 64) + ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second) + defer cancel() + tx, err := s.pool.Begin(ctx) + if err != nil { + s.dbErr(w, "delete begin", err) + return + } + defer tx.Rollback(ctx) + if err := deleteOneItem(ctx, tx, char, itemID); err != nil { + s.dbErr(w, "delete", err) + return + } + if err := tx.Commit(ctx); err != nil { + s.dbErr(w, "delete commit", err) + return + } + writeJSON(w, http.StatusOK, map[string]any{"status": "deleted", "item_id": itemID}) +} + +func parseNaiveTime(s string) time.Time { + if s == "" { + return time.Now().UTC() + } + s = strings.Replace(s, "Z", "+00:00", 1) + for _, l := range []string{time.RFC3339Nano, time.RFC3339, "2006-01-02T15:04:05.999999", "2006-01-02T15:04:05"} { + if t, err := time.Parse(l, s); err == nil { + return t.UTC() + } + } + return time.Now().UTC() +} + +func firstNonNil(a, b any) any { + if a != nil { + return a + } + return b +} diff --git a/go-services/inventory-go/inventory_char.go b/go-services/inventory-go/inventory_char.go new file mode 100644 index 00000000..f7a8bc57 --- /dev/null +++ b/go-services/inventory-go/inventory_char.go @@ -0,0 +1,92 @@ +package main + +import ( + "net/http" + "strings" +) + +// GET /inventory/{character_name} — full per-character inventory for the React +// Inventory window. Port of inventory-service get_character_inventory + +// enrich_db_item (main.py:2622 / 2338). The Go cutover omitted this endpoint +// (it was assumed unused), but the React InventoryWindow fetches it, so its +// absence (404) made the live inventory render empty. +// +// Returns {character_name, item_count, items:[...]} with the snake_case fields +// the frontend normalizeItem consumes: placement via current_wielded_location / +// container_id / items_capacity, the mana panel via current_mana / max_mana, +// icon overlays, plus tooltip combat/requirement/enhancement/rating stats. Mana +// and icon overlays are pulled straight from original_json IntValues (same keys +// the plugin/search path use); the rest come from the normalized join tables. +func (s *Server) handleCharacterInventory(w http.ResponseWriter, r *http.Request) { + name := r.PathValue("character_name") + limit := clampInt(qIntDefault(r.URL.Query(), "limit", 1000), 1, 5000) + offset := qIntDefault(r.URL.Query(), "offset", 0) + if offset < 0 { + offset = 0 + } + + const q = ` + SELECT + i.item_id, i.name, i.icon, i.object_class, i.value, i.burden, + i.current_wielded_location, i.container_id, i.items_capacity, i.stack_size, + cs.max_damage, cs.armor_level, cs.damage_bonus, cs.attack_bonus, + cs.melee_defense_bonus, cs.magic_defense_bonus, + r.wield_level, r.skill_level, r.equip_skill, r.lore_requirement, + e.material, e.imbue, e.item_set, e.tinks, e.workmanship, + rt.damage_rating, rt.crit_rating, rt.crit_damage_rating, rt.heal_boost_rating, + NULLIF((rd.original_json->'IntValues'->>'218103815')::int, 0) AS current_mana, + NULLIF((rd.original_json->'IntValues'->>'218103814')::int, 0) AS max_mana, + NULLIF((rd.original_json->'IntValues'->>'218103849')::int, 0) AS icon_overlay_id, + NULLIF((rd.original_json->'IntValues'->>'218103850')::int, 0) AS icon_underlay_id + FROM items i + LEFT JOIN item_combat_stats cs ON i.id = cs.item_id + LEFT JOIN item_requirements r ON i.id = r.item_id + LEFT JOIN item_enhancements e ON i.id = e.item_id + LEFT JOIN item_ratings rt ON i.id = rt.item_id + LEFT JOIN item_raw_data rd ON i.id = rd.item_id + WHERE i.character_name = $1 + ORDER BY i.name + LIMIT $2 OFFSET $3` + + rows, err := queryRowsAsMaps(r.Context(), s.pool, q, name, limit, offset) + if err != nil { + s.dbErr(w, "inventory/"+name, err) + return + } + + items := make([]map[string]any, 0, len(rows)) + for _, row := range rows { + items = append(items, enrichInventoryRow(row)) + } + + // Unlike the Python endpoint (404 on no rows), always return 200 with a + // possibly-empty list — the window treats both as empty, and 200 avoids the + // frontend's catch-all error path. + writeJSON(w, http.StatusOK, map[string]any{ + "character_name": name, + "item_count": len(items), + "items": items, + }) +} + +// enrichInventoryRow flattens a joined inventory row into the frontend item +// shape: drops NULL columns and applies the material-name prefix to the item +// name (enrich_db_item parity, e.g. "Pyreal" + "Chiran Helm" -> "Pyreal Chiran +// Helm"), preserving the un-prefixed name in original_name. +func enrichInventoryRow(row map[string]any) map[string]any { + out := make(map[string]any, len(row)+2) + for k, v := range row { + if v != nil { + out[k] = v + } + } + if mat, ok := out["material"].(string); ok && mat != "" { + out["material_name"] = mat + if name, ok := out["name"].(string); ok && name != "" && + !strings.HasPrefix(strings.ToLower(name), strings.ToLower(mat)) { + out["name"] = mat + " " + name + out["original_name"] = name + } + } + return out +} diff --git a/go-services/inventory-go/inventory_char_test.go b/go-services/inventory-go/inventory_char_test.go new file mode 100644 index 00000000..bc90d41c --- /dev/null +++ b/go-services/inventory-go/inventory_char_test.go @@ -0,0 +1,35 @@ +package main + +import "testing" + +func TestEnrichInventoryRow(t *testing.T) { + // NULL columns are dropped. + out := enrichInventoryRow(map[string]any{"name": "Helm", "armor_level": nil, "value": 100}) + if _, ok := out["armor_level"]; ok { + t.Errorf("nil column armor_level should be dropped, got %v", out["armor_level"]) + } + if out["value"] != 100 { + t.Errorf("value = %v, want 100", out["value"]) + } + + // Material prefix is applied and original_name preserved. + out = enrichInventoryRow(map[string]any{"name": "Chiran Helm", "material": "Pyreal"}) + if out["name"] != "Pyreal Chiran Helm" { + t.Errorf("name = %v, want %q", out["name"], "Pyreal Chiran Helm") + } + if out["original_name"] != "Chiran Helm" { + t.Errorf("original_name = %v, want %q", out["original_name"], "Chiran Helm") + } + if out["material_name"] != "Pyreal" { + t.Errorf("material_name = %v, want Pyreal", out["material_name"]) + } + + // Already-prefixed name is not doubled. + out = enrichInventoryRow(map[string]any{"name": "Pyreal Helm", "material": "Pyreal"}) + if out["name"] != "Pyreal Helm" { + t.Errorf("name = %v, want no double prefix", out["name"]) + } + if _, ok := out["original_name"]; ok { + t.Errorf("original_name should be unset when no prefix applied") + } +} diff --git a/go-services/inventory-go/main.go b/go-services/inventory-go/main.go new file mode 100644 index 00000000..51611dcc --- /dev/null +++ b/go-services/inventory-go/main.go @@ -0,0 +1,305 @@ +// Command inventory-go is a Go reimplementation of the MosswartOverlord +// inventory-service (FastAPI), deployed in parallel for comparison. +// +// Phase A: read side. Connects READ-ONLY to the existing inventory_db and +// reimplements the read endpoints, validated against the Python service on the +// same data. The heavy item-processing ingestion and the suitbuilder solver +// follow in later phases. +package main + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "log/slog" + "net/http" + "os" + "os/signal" + "sort" + "strconv" + "strings" + "syscall" + "time" + + "github.com/jackc/pgx/v5/pgxpool" +) + +var buildVersion = "dev" + +type Server struct { + pool *pgxpool.Pool + attributeSets map[string]string // AttributeSetInfo: set-id -> set name + objectClasses map[int]string // ObjectClass: id -> name + materials map[int]string // MaterialType: id -> name + spells map[int]map[string]any // SpellTable: spell-id -> raw spell value object + equipMaskMap map[int]string // EquipMask: mask -> technical name (exact lookup) + equipMaskOrdered []equipMaskEntry // EquipMask in ascending-mask order (bit-flag decode) + log *slog.Logger +} + +func main() { + logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo})) + slog.SetDefault(logger) + + addr := ":" + envOr("PORT", "8772") + dsn := os.Getenv("DATABASE_URL") + enumPath := envOr("ENUM_DB_PATH", "comprehensive_enum_database_v2.json") + readOnly := envOr("READ_ONLY", "true") != "false" + + logger.Info("starting inventory-go", "version", buildVersion, "addr", addr, "read_only", readOnly) + + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) + defer stop() + + srv := &Server{log: logger, attributeSets: map[string]string{}, objectClasses: map[int]string{}, materials: map[int]string{}, spells: map[int]map[string]any{}} + + if e, err := loadEnums(enumPath); err != nil { + logger.Warn("could not load enum DB (set/class/material/spell names will be unknown)", "err", err, "path", enumPath) + } else { + srv.attributeSets = e.sets + srv.objectClasses = e.objectClasses + srv.materials = e.materials + srv.spells = e.spells + srv.equipMaskMap = e.equipMaskMap + srv.equipMaskOrdered = e.equipMaskOrdered + logger.Info("loaded enum DB", "sets", len(e.sets), "object_classes", len(e.objectClasses), "materials", len(e.materials), "spells", len(e.spells), "equip_masks", len(e.equipMaskOrdered)) + } + + if dsn == "" { + logger.Error("DATABASE_URL is required") + os.Exit(1) + } + connectCtx, cancel := context.WithTimeout(ctx, 15*time.Second) + pool, err := newPool(connectCtx, dsn, readOnly) + cancel() + if err != nil { + logger.Error("db pool init failed", "err", err) + os.Exit(1) + } + defer pool.Close() + srv.pool = pool + + // Ingest mode owns its DB: create the schema on first run. In cutover + // (reusing the production inventory_db) SKIP_SCHEMA_INIT runs no DDL. + if !readOnly && envOr("SKIP_SCHEMA_INIT", "false") != "true" { + sctx, c := context.WithTimeout(ctx, 60*time.Second) + initSchema(sctx, pool, logger) + c() + } + + mux := http.NewServeMux() + mux.HandleFunc("GET /health", srv.handleHealth) + mux.HandleFunc("GET /sets/list", srv.handleSetsList) + mux.HandleFunc("GET /characters/list", srv.handleCharactersList) + mux.HandleFunc("GET /search/items", srv.handleSearchItems) + mux.HandleFunc("GET /inventory/{character_name}", srv.handleCharacterInventory) + mux.HandleFunc("POST /debug/process", srv.handleDebugProcess) + // Ingestion (works in read-write mode; on the read-only prod instance these + // fail the read-only transaction, which is the intended guard). + mux.HandleFunc("POST /process-inventory", srv.handleProcessInventory) + mux.HandleFunc("POST /inventory/{character_name}/item", srv.handleUpsertItem) + mux.HandleFunc("DELETE /inventory/{character_name}/item/{item_id}", srv.handleDeleteItem) + // Suitbuilder (port of suitbuilder.py router, mounted at /suitbuilder). + mux.HandleFunc("POST /suitbuilder/search", srv.handleSuitSearch) + mux.HandleFunc("GET /suitbuilder/characters", srv.handleSuitCharacters) + mux.HandleFunc("GET /suitbuilder/sets", srv.handleSuitSets) + + httpSrv := &http.Server{Addr: addr, Handler: withLogging(mux), ReadHeaderTimeout: 10 * time.Second} + go func() { + if err := httpSrv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) { + logger.Error("http server failed", "err", err) + os.Exit(1) + } + }() + logger.Info("listening", "addr", addr) + + <-ctx.Done() + shutdownCtx, c := context.WithTimeout(context.Background(), 10*time.Second) + defer c() + _ = httpSrv.Shutdown(shutdownCtx) + logger.Info("stopped") +} + +// GET /health (main.py:2674) +func (s *Server) handleHealth(w http.ResponseWriter, r *http.Request) { + ctx, cancel := context.WithTimeout(r.Context(), 3*time.Second) + defer cancel() + dbOK := s.pool.Ping(ctx) == nil + status := "healthy" + if !dbOK { + status = "degraded" + } + writeJSON(w, http.StatusOK, map[string]any{ + "status": status, + "timestamp": pyISO(time.Now()), + "database_connected": dbOK, + "version": "1.0.0", + }) +} + +// GET /sets/list (main.py:2712) +func (s *Server) handleSetsList(w http.ResponseWriter, r *http.Request) { + ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second) + defer cancel() + rows, err := queryRowsAsMaps(ctx, s.pool, ` + SELECT enh.item_set, COUNT(*) AS item_count + FROM item_enhancements enh + WHERE enh.item_set IS NOT NULL AND enh.item_set != '' + GROUP BY enh.item_set + ORDER BY item_count DESC, enh.item_set`) + if err != nil { + s.dbErr(w, "sets/list", err) + return + } + sets := make([]map[string]any, 0, len(rows)) + for _, row := range rows { + setID := toStr(row["item_set"]) + name, ok := s.attributeSets[setID] + if !ok { + name = "Unknown Set " + setID + } + sets = append(sets, map[string]any{"id": setID, "name": name, "item_count": row["item_count"]}) + } + writeJSON(w, http.StatusOK, map[string]any{"sets": sets}) +} + +// GET /characters/list (main.py:4291) +func (s *Server) handleCharactersList(w http.ResponseWriter, r *http.Request) { + ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second) + defer cancel() + rows, err := queryRowsAsMaps(ctx, s.pool, ` + SELECT character_name, COUNT(*) AS item_count, MAX(timestamp) AS last_updated + FROM items GROUP BY character_name ORDER BY character_name`) + if err != nil { + s.dbErr(w, "characters/list", err) + return + } + formatTimes(rows, "last_updated") + chars := make([]map[string]any, 0, len(rows)) + for _, row := range rows { + chars = append(chars, map[string]any{ + "character_name": row["character_name"], + "item_count": row["item_count"], + "last_updated": row["last_updated"], + }) + } + writeJSON(w, http.StatusOK, map[string]any{"characters": chars, "total_characters": len(chars)}) +} + +func (s *Server) dbErr(w http.ResponseWriter, where string, err error) { + s.log.Error("db query failed", "where", where, "err", err) + writeJSON(w, http.StatusInternalServerError, map[string]any{"detail": "Internal server error"}) +} + +type enumMaps struct { + sets map[string]string + objectClasses map[int]string + materials map[int]string + spells map[int]map[string]any + equipMaskMap map[int]string + equipMaskOrdered []equipMaskEntry +} + +// loadEnums reads the comprehensive enum DB and extracts AttributeSetInfo +// (set-id -> name), ObjectClass (id -> name), and MaterialType (id -> name), +// mirroring load_comprehensive_enums (dictionaries first, then enums). +func loadEnums(path string) (enumMaps, error) { + var em enumMaps + b, err := os.ReadFile(path) + if err != nil { + return em, err + } + type valmap struct { + Values map[string]string `json:"values"` + } + var db struct { + Dictionaries map[string]valmap `json:"dictionaries"` + Enums map[string]valmap `json:"enums"` + ObjectClasses valmap `json:"object_classes"` + Spells struct { + Values map[string]map[string]any `json:"values"` + } `json:"spells"` + } + if err := json.Unmarshal(b, &db); err != nil { + return em, err + } + em.sets = map[string]string{} + if d, ok := db.Dictionaries["AttributeSetInfo"]; ok && len(d.Values) > 0 { + em.sets = d.Values + } else if e, ok := db.Enums["AttributeSetInfo"]; ok { + em.sets = e.Values + } + intMap := func(v valmap) map[int]string { + m := map[int]string{} + for k, val := range v.Values { + if n, err := strconv.Atoi(k); err == nil { + m[n] = val + } + } + return m + } + em.objectClasses = intMap(db.ObjectClasses) + em.materials = intMap(db.Enums["MaterialType"]) + // SpellTable: spell-id -> raw value object (translate_spell reads .name etc.). + em.spells = map[int]map[string]any{} + for k, v := range db.Spells.Values { + if n, err := strconv.Atoi(k); err == nil { + em.spells[n] = v + } + } + // EquipMask: mask -> technical name. Skip EXPR: keys; order by ascending mask + // (the JSON order) so multi-bit bit-flag decode joins parts deterministically. + em.equipMaskMap = map[int]string{} + for k, v := range db.Enums["EquipMask"].Values { + if strings.HasPrefix(k, "EXPR:") { + continue + } + if n, err := strconv.Atoi(k); err == nil { + em.equipMaskMap[n] = v + em.equipMaskOrdered = append(em.equipMaskOrdered, equipMaskEntry{Mask: n, Name: v}) + } + } + sort.Slice(em.equipMaskOrdered, func(i, j int) bool { return em.equipMaskOrdered[i].Mask < em.equipMaskOrdered[j].Mask }) + return em, nil +} + +// translateSpell mirrors main.py translate_spell: returns the spell dict +// (id + name/description/school/difficulty/duration/mana/family), defaulting +// missing fields to "" and the name to Unknown_Spell_. +func (s *Server) translateSpell(id int) map[string]any { + raw := s.spells[id] + get := func(k string, def any) any { + if raw != nil { + if v, ok := raw[k]; ok { + return v + } + } + return def + } + return map[string]any{ + "id": id, + "name": get("name", fmt.Sprintf("Unknown_Spell_%d", id)), + "description": get("description", ""), + "school": get("school", ""), + "difficulty": get("difficulty", ""), + "duration": get("duration", ""), + "mana": get("mana", ""), + "family": get("family", ""), + } +} + +func envOr(key, def string) string { + if v := os.Getenv(key); v != "" { + return v + } + return def +} + +func withLogging(next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + start := time.Now() + next.ServeHTTP(w, r) + slog.Info("http", "method", r.Method, "path", r.URL.Path, "dur_ms", time.Since(start).Milliseconds()) + }) +} diff --git a/go-services/inventory-go/process.go b/go-services/inventory-go/process.go new file mode 100644 index 00000000..4a377309 --- /dev/null +++ b/go-services/inventory-go/process.go @@ -0,0 +1,434 @@ +package main + +import ( + "encoding/json" + "io" + "math" + "net/http" + "strconv" + "strings" +) + +// Item-processor: a faithful port of inventory-service extract_item_properties + +// the process_inventory column population. Given a raw item dict it produces the +// normalized rows for the 7 tables, applying the exact per-table sentinel->NULL +// rules. Validated against production's stored rows (read-only) via /debug/process. + +// --- value-bag accessors (JSON object keys are strings) --- + +func bag(raw map[string]any, name string) map[string]any { + if m, ok := raw[name].(map[string]any); ok { + return m + } + return map[string]any{} +} +func ivI(iv map[string]any, key string, def int) int { + if v, ok := iv[key]; ok { + if f, ok := v.(float64); ok { + return int(f) + } + } + return def +} +func dvF(dv map[string]any, key string, def float64) float64 { + if v, ok := dv[key]; ok { + if f, ok := v.(float64); ok { + return f + } + } + return def +} +func rawI(raw map[string]any, key string, def int) int { + if v, ok := raw[key]; ok { + if f, ok := v.(float64); ok { + return int(f) + } + } + return def +} +func rawF(raw map[string]any, key string, def float64) float64 { + if v, ok := raw[key]; ok { + if f, ok := v.(float64); ok { + return f + } + } + return def +} +func rawS(raw map[string]any, key string) string { + if s, ok := raw[key].(string); ok { + return s + } + return "" +} +func rawB(raw map[string]any, key string) bool { + b, _ := raw[key].(bool) + return b +} + +// IV first, else top-level field, else default (e.g. max_damage). +func ivElseTopI(iv, raw map[string]any, ivKey, topKey string, def int) int { + if v, ok := iv[ivKey]; ok { + if f, ok := v.(float64); ok { + return int(f) + } + } + return rawI(raw, topKey, def) +} +func dvElseTopF(dv, raw map[string]any, dvKey, topKey string, def float64) float64 { + if v, ok := dv[dvKey]; ok { + if f, ok := v.(float64); ok { + return f + } + } + return rawF(raw, topKey, def) +} + +// translateMaterial: materials[id] else "Unknown_Material_{id}". +func (s *Server) translateMaterial(id int) string { + if n, ok := s.materials[id]; ok { + return n + } + return "Unknown_Material_" + strconv.Itoa(id) +} + +func toIntList(v any) []int { + arr, ok := v.([]any) + if !ok { + return nil + } + out := make([]int, 0, len(arr)) + for _, e := range arr { + if f, ok := e.(float64); ok { + out = append(out, int(f)) + } + } + return out +} + +// processItem produces the normalized columns for all 7 tables, post-null. +func (s *Server) processItem(raw map[string]any) map[string]any { + iv := bag(raw, "IntValues") + dv := bag(raw, "DoubleValues") + + items := map[string]any{ + "item_id": rawValue(raw, "Id"), + "name": rawS(raw, "Name"), + "icon": rawI(raw, "Icon", 0), + "object_class": rawI(raw, "ObjectClass", 0), + "value": rawI(raw, "Value", 0), + "burden": rawI(raw, "Burden", 0), + "has_id_data": rawB(raw, "HasIdData"), + "last_id_time": rawI(raw, "LastIdTime", 0), + "current_wielded_location": ivI(iv, "10", 0), + "container_id": rawI(raw, "ContainerId", 0), + "slot": ivI(iv, "231735296", -1), + "bonded": ivI(iv, "33", 0), + "attuned": ivI(iv, "114", 0), + "unique": ivI(iv, "279", 0) != 0, + "stack_size": ivI(iv, "12", 1), + "max_stack_size": ivI(iv, "11", 1), + "items_capacity": nilNeg(ivI(iv, "6", -1)), + "containers_capacity": nilNeg(ivI(iv, "7", -1)), + "structure": nilNeg(ivI(iv, "92", -1)), + "max_structure": nilNeg(ivI(iv, "91", -1)), + "rare_id": nilNeg(ivI(iv, "17", -1)), + "lifespan": nilNeg(ivI(iv, "267", -1)), + "remaining_lifespan": nilNeg(ivI(iv, "268", -1)), + } + + // combat (sentinel defaults), then base values merged. + wt := ivI(iv, "218103835", -1) + if wt > 100 { + wt = 100 + } + combat := map[string]any{ + "max_damage": ivElseTopI(iv, raw, "218103842", "MaxDamage", -1), + "damage_type": ivI(iv, "218103832", -1), + "damage_bonus": dvElseTopF(dv, raw, "167772174", "DamageBonus", -1.0), + "elemental_damage_bonus": ivI(iv, "204", -1), + "elemental_damage_vs_monsters": dvF(dv, "152", -1.0), + "variance": dvF(dv, "167772171", -1.0), + "cleaving": ivI(iv, "292", -1), + "crit_damage_rating": ivI(iv, "314", -1), + "damage_over_time": ivI(iv, "318", -1), + "attack_bonus": dvElseTopF(dv, raw, "167772170", "AttackBonus", -1.0), + "weapon_time": wt, + "weapon_skill": ivI(iv, "218103840", -1), + "armor_level": topElseIvI(raw, iv, "ArmorLevel", "28", -1), + "melee_defense_bonus": dvF(dv, "29", -1.0), + "missile_defense_bonus": dvF(dv, "149", -1.0), + "magic_defense_bonus": dvF(dv, "150", -1.0), + "resist_magic": ivI(iv, "36", -1), + "crit_resist_rating": ivI(iv, "315", -1), + "crit_damage_resist_rating": ivI(iv, "316", -1), + "dot_resist_rating": ivI(iv, "350", -1), + "life_resist_rating": ivI(iv, "351", -1), + "nether_resist_rating": ivI(iv, "331", -1), + "heal_over_time": ivI(iv, "312", -1), + "healing_resist_rating": ivI(iv, "317", -1), + "mana_conversion_bonus": dvF(dv, "144", -1.0), + "pk_damage_rating": ivI(iv, "381", -1), + "pk_damage_resist_rating": ivI(iv, "382", -1), + "gear_pk_damage_rating": ivI(iv, "383", -1), + "gear_pk_damage_resist_rating": ivI(iv, "384", -1), + } + s.mergeBaseValues(raw, combat) + + requirements := map[string]any{ + "wield_level": rawI(raw, "WieldLevel", -1), + "skill_level": rawI(raw, "SkillLevel", -1), + "lore_requirement": rawI(raw, "LoreRequirement", -1), + "equip_skill": rawValueStr(raw, "EquipSkill"), + } + + // material + item_set translated strings. + var material any + if m := rawS(raw, "Material"); m != "" { + material = m + } else if v, ok := iv["131"]; ok { + if f, ok := v.(float64); ok && int(f) != 0 { + name := s.translateMaterial(int(f)) + if !strings.HasPrefix(name, "Unknown_Material_") { + material = name + } + } + } + var itemSet any + if v, ok := iv["265"]; ok { + if f, ok := v.(float64); ok && int(f) != 0 { + id := strconv.Itoa(int(f)) + if n, ok := s.attributeSets[id]; ok { + itemSet = n + } else { + itemSet = id + } + } + } + enhancements := map[string]any{ + "material": material, + "imbue": rawValueStr(raw, "Imbue"), + "tinks": rawI(raw, "Tinks", -1), + "workmanship": rawF(raw, "Workmanship", -1.0), + "num_times_tinkered": ivI(iv, "171", -1), + "free_tinkers_bitfield": ivI(iv, "264", -1), + "num_items_in_material": ivI(iv, "170", -1), + "imbue_attempts": ivI(iv, "205", -1), + "imbue_successes": ivI(iv, "206", -1), + "imbued_effect2": ivI(iv, "303", -1), + "imbued_effect3": ivI(iv, "304", -1), + "imbued_effect4": ivI(iv, "305", -1), + "imbued_effect5": ivI(iv, "306", -1), + "imbue_stacking_bits": ivI(iv, "311", -1), + "item_set": itemSet, + "equipment_set_extra": ivI(iv, "321", -1), + "aetheria_bitfield": ivI(iv, "322", -1), + "heritage_specific_armor": ivI(iv, "324", -1), + "shared_cooldown": ivI(iv, "280", -1), + } + + ratingKeys := map[string]string{ + "damage_rating": "307", "damage_resist_rating": "308", "crit_rating": "313", + "crit_resist_rating": "315", "crit_damage_rating": "314", "crit_damage_resist_rating": "316", + "heal_boost_rating": "323", "vitality_rating": "341", "healing_rating": "342", + "weakness_rating": "329", "nether_over_time": "330", "healing_resist_rating": "317", + "nether_resist_rating": "331", "dot_resist_rating": "350", "life_resist_rating": "351", + "sneak_attack_rating": "356", "recklessness_rating": "357", "deception_rating": "358", + "pk_damage_rating": "381", "pk_damage_resist_rating": "382", "gear_pk_damage_rating": "383", + "gear_pk_damage_resist_rating": "384", "gear_damage": "370", "gear_damage_resist": "371", + "gear_crit": "372", "gear_crit_resist": "373", "gear_crit_damage": "374", + "gear_crit_damage_resist": "375", "gear_healing_boost": "376", "gear_max_health": "379", + "gear_nether_resist": "377", "gear_life_resist": "378", "gear_overpower": "388", + "gear_overpower_resist": "389", + } + ratings := map[string]any{} + for col, k := range ratingKeys { + ratings[col] = ivI(iv, k, -1) + } + + // spells: union of Spells + ActiveSpells, is_active = in ActiveSpells. + spells := toIntList(raw["Spells"]) + active := toIntList(raw["ActiveSpells"]) + activeSet := map[int]bool{} + for _, id := range active { + activeSet[id] = true + } + seen := map[int]bool{} + var spellRows []map[string]any + for _, id := range append(append([]int{}, spells...), active...) { + if seen[id] { + continue + } + seen[id] = true + spellRows = append(spellRows, map[string]any{"spell_id": id, "is_active": activeSet[id]}) + } + + return map[string]any{ + "items": items, + "combat": nullify(combat, sentinelCombat), + "requirements": nullify(requirements, sentinelReq), + "enhancements": nullifyKeep(enhancements, sentinelEnh), // ALWAYS inserts a row + "ratings": nullify(ratings, sentinelRating), + "spells": spellRows, + } +} + +func topElseIvI(raw, iv map[string]any, topKey, ivKey string, def int) int { + if v, ok := raw[topKey]; ok { + if f, ok := v.(float64); ok { + return int(f) + } + } + return ivI(iv, ivKey, def) +} + +func rawValue(raw map[string]any, key string) any { + if v, ok := raw[key]; ok { + if f, ok := v.(float64); ok { + return int64(f) + } + return v + } + if v, ok := raw[strings.ToLower(key)]; ok { // Id -> id fallback + if f, ok := v.(float64); ok { + return int64(f) + } + return v + } + return nil +} +func rawValueStr(raw map[string]any, key string) any { + if s, ok := raw[key].(string); ok && s != "" { + return s + } + return nil +} + +func nilNeg(v int) any { + if v == -1 { + return nil + } + return v +} + +// per-table sentinel predicates: true => value should become NULL. +func sentinelCombat(v any) bool { return isNeg1(v) || isNeg1f(v) } +func sentinelReq(v any) bool { return isNeg1(v) || v == nil || v == "" } +func sentinelEnh(v any) bool { return isNeg1(v) || isNeg1f(v) || v == nil || v == "" } +func sentinelRating(v any) bool { return isNeg1(v) || isNeg1f(v) || v == nil } + +func isNeg1(v any) bool { i, ok := v.(int); return ok && i == -1 } +func isNeg1f(v any) bool { f, ok := v.(float64); return ok && f == -1.0 } + +// nullify replaces sentinel values with nil. Returns nil for the whole table if +// every value is sentinel (the per-table "skip insert" guard) — EXCEPT +// enhancements, which always inserts; we keep its map even if all-null. +func nullify(m map[string]any, isSentinel func(any) bool) map[string]any { + any_ := false + out := make(map[string]any, len(m)) + for k, v := range m { + if isSentinel(v) { + out[k] = nil + } else { + out[k] = v + any_ = true + } + } + if !any_ { + return nil // combat/req/ratings: skip the insert when all-sentinel + } + return out +} + +// nullifyKeep is like nullify but ALWAYS returns the map (for item_enhancements, +// which inserts a row even when every value is NULL). +func nullifyKeep(m map[string]any, isSentinel func(any) bool) map[string]any { + out := make(map[string]any, len(m)) + for k, v := range m { + if isSentinel(v) { + out[k] = nil + } else { + out[k] = v + } + } + return out +} + +// mergeBaseValues reverses active-spell buffs into base_* columns (compute_base_values). +type spellEffect struct { + key int + change, bonus float64 +} + +var intEffects = map[int]spellEffect{ + 1616: {218103842, 20, 0}, 2096: {218103842, 22, 0}, 5183: {218103842, 24, 0}, 4395: {218103842, 24, 0}, 3688: {218103842, 300, 0}, + 2598: {218103842, 2, 2}, 2586: {218103842, 4, 4}, 4661: {218103842, 7, 7}, 6089: {218103842, 10, 10}, + 1486: {28, 200, 0}, 2108: {28, 220, 0}, 4407: {28, 240, 0}, + 2604: {28, 20, 20}, 2592: {28, 40, 40}, 4667: {28, 60, 60}, 6095: {28, 80, 80}, +} +var doubleEffects = map[int]spellEffect{ + 3258: {152, 0.06, 0}, 3259: {152, 0.07, 0}, 5182: {152, 0.08, 0}, 4414: {152, 0.08, 0}, 3735: {152, 0.15, 0}, + 3251: {152, 0.01, 0.01}, 3250: {152, 0.03, 0.03}, 4670: {152, 0.05, 0.05}, 6098: {152, 0.07, 0.07}, + 1592: {167772172, 0.15, 0}, 2106: {167772172, 0.17, 0}, 4405: {167772172, 0.20, 0}, + 2603: {167772172, 0.03, 0.03}, 2591: {167772172, 0.05, 0.05}, 4666: {167772172, 0.07, 0.07}, 6094: {167772172, 0.09, 0.09}, + 1605: {29, 0.15, 0}, 2101: {29, 0.17, 0}, 4400: {29, 0.20, 0}, 3699: {29, 0.25, 0}, + 2600: {29, 0.03, 0.03}, 3985: {29, 0.04, 0.04}, 2588: {29, 0.05, 0.05}, 4663: {29, 0.07, 0.07}, 6091: {29, 0.09, 0.09}, + 1480: {144, 1.60, 0}, 2117: {144, 1.70, 0}, 4418: {144, 1.80, 0}, + 3201: {144, 1.05, 1.05}, 3199: {144, 1.10, 1.10}, 3202: {144, 1.15, 1.15}, 3200: {144, 1.20, 1.20}, 6086: {144, 1.25, 1.25}, 6087: {144, 1.30, 1.30}, +} + +func (s *Server) mergeBaseValues(raw, combat map[string]any) { + spells := toIntList(raw["Spells"]) + active := toIntList(raw["ActiveSpells"]) + for _, p := range []struct { + prop string + key int + }{{"max_damage", 218103842}, {"armor_level", 28}} { + val, ok := combat[p.prop].(int) + if !ok || val == -1 { + continue + } + for _, sid := range active { + if e, ok := intEffects[sid]; ok && e.key == p.key { + val -= int(e.change) + } + } + for _, sid := range spells { + if e, ok := intEffects[sid]; ok && e.key == p.key && e.bonus != 0 { + val += int(e.bonus) + } + } + combat["base_"+p.prop] = val + } + for _, p := range []struct { + prop string + key int + }{{"attack_bonus", 167772172}, {"melee_defense_bonus", 29}, {"elemental_damage_vs_monsters", 152}, {"mana_conversion_bonus", 144}} { + val, ok := combat[p.prop].(float64) + if !ok || val == -1.0 { + continue + } + for _, sid := range active { + if e, ok := doubleEffects[sid]; ok && e.key == p.key { + val -= e.change + } + } + for _, sid := range spells { + if e, ok := doubleEffects[sid]; ok && e.key == p.key && e.bonus != 0 { + val += e.bonus + } + } + combat["base_"+p.prop] = math.Round(val*10000) / 10000 + } +} + +// POST /debug/process — returns the normalized columns for a raw item JSON body +// (loopback validation against production's stored rows; never writes). +func (s *Server) handleDebugProcess(w http.ResponseWriter, r *http.Request) { + body, _ := io.ReadAll(io.LimitReader(r.Body, 8<<20)) + var raw map[string]any + if json.Unmarshal(body, &raw) != nil { + writeJSON(w, http.StatusBadRequest, map[string]any{"detail": "invalid JSON"}) + return + } + writeJSON(w, http.StatusOK, s.processItem(raw)) +} diff --git a/go-services/inventory-go/schema.go b/go-services/inventory-go/schema.go new file mode 100644 index 00000000..43950bbb --- /dev/null +++ b/go-services/inventory-go/schema.go @@ -0,0 +1,127 @@ +package main + +import ( + "context" + "log/slog" + + "github.com/jackc/pgx/v5/pgxpool" +) + +// initSchema creates the normalized inventory schema on an ingest-owned database +// (a faithful replica of inventory-service/database.py). Run only when this +// instance owns its DB (READ_ONLY=false) — never against production. Idempotent; +// logs and continues per statement. +func initSchema(ctx context.Context, pool *pgxpool.Pool, log *slog.Logger) { + stmts := []string{ + `CREATE TABLE IF NOT EXISTS items ( + id SERIAL PRIMARY KEY, + character_name VARCHAR(50) NOT NULL, + item_id BIGINT NOT NULL, + timestamp TIMESTAMP NOT NULL, + name VARCHAR(200) NOT NULL, + icon INTEGER NOT NULL, + object_class INTEGER NOT NULL, + value INTEGER DEFAULT 0, + burden INTEGER DEFAULT 0, + current_wielded_location INTEGER DEFAULT 0, + container_id BIGINT DEFAULT 0, + slot INTEGER DEFAULT -1, + bonded INTEGER DEFAULT 0, + attuned INTEGER DEFAULT 0, + "unique" BOOLEAN DEFAULT false, + stack_size INTEGER DEFAULT 1, + max_stack_size INTEGER DEFAULT 1, + items_capacity INTEGER, + containers_capacity INTEGER, + structure INTEGER, + max_structure INTEGER, + rare_id INTEGER, + lifespan INTEGER, + remaining_lifespan INTEGER, + has_id_data BOOLEAN DEFAULT false, + last_id_time BIGINT DEFAULT 0, + CONSTRAINT uq_char_item UNIQUE (character_name, item_id) + )`, + `CREATE INDEX IF NOT EXISTS ix_items_character_name ON items (character_name)`, + `CREATE INDEX IF NOT EXISTS ix_items_name ON items (name)`, + `CREATE INDEX IF NOT EXISTS ix_items_object_class ON items (object_class)`, + `CREATE INDEX IF NOT EXISTS ix_items_current_wielded_location ON items (current_wielded_location)`, + + `CREATE TABLE IF NOT EXISTS item_combat_stats ( + item_id INTEGER PRIMARY KEY REFERENCES items(id), + max_damage INTEGER, damage INTEGER, damage_type INTEGER, damage_bonus DOUBLE PRECISION, + elemental_damage_bonus INTEGER, elemental_damage_vs_monsters DOUBLE PRECISION, variance DOUBLE PRECISION, + cleaving INTEGER, crit_damage_rating INTEGER, damage_over_time INTEGER, + attack_bonus DOUBLE PRECISION, weapon_time INTEGER, weapon_skill INTEGER, + armor_level INTEGER, shield_value INTEGER, melee_defense_bonus DOUBLE PRECISION, + missile_defense_bonus DOUBLE PRECISION, magic_defense_bonus DOUBLE PRECISION, + resist_magic INTEGER, crit_resist_rating INTEGER, crit_damage_resist_rating INTEGER, + dot_resist_rating INTEGER, life_resist_rating INTEGER, nether_resist_rating INTEGER, + heal_over_time INTEGER, healing_resist_rating INTEGER, mana_conversion_bonus DOUBLE PRECISION, + pk_damage_rating INTEGER, pk_damage_resist_rating INTEGER, gear_pk_damage_rating INTEGER, + gear_pk_damage_resist_rating INTEGER, + base_armor_level INTEGER, base_max_damage INTEGER, base_attack_bonus DOUBLE PRECISION, + base_melee_defense_bonus DOUBLE PRECISION, base_elemental_damage_vs_monsters DOUBLE PRECISION, + base_mana_conversion_bonus DOUBLE PRECISION + )`, + `CREATE INDEX IF NOT EXISTS ix_combat_armor ON item_combat_stats (armor_level)`, + + `CREATE TABLE IF NOT EXISTS item_requirements ( + item_id INTEGER PRIMARY KEY REFERENCES items(id), + wield_level INTEGER, wield_requirement INTEGER, skill_level INTEGER, + lore_requirement INTEGER, equip_skill VARCHAR(50), mastery VARCHAR(50) + )`, + `CREATE INDEX IF NOT EXISTS ix_req_level ON item_requirements (wield_level)`, + + `CREATE TABLE IF NOT EXISTS item_enhancements ( + item_id INTEGER PRIMARY KEY REFERENCES items(id), + material VARCHAR(50), imbue VARCHAR(50), tinks INTEGER, workmanship DOUBLE PRECISION, + salvage_workmanship DOUBLE PRECISION, num_times_tinkered INTEGER DEFAULT 0, + free_tinkers_bitfield INTEGER, num_items_in_material INTEGER, + imbue_attempts INTEGER DEFAULT 0, imbue_successes INTEGER DEFAULT 0, + imbued_effect2 INTEGER, imbued_effect3 INTEGER, imbued_effect4 INTEGER, imbued_effect5 INTEGER, + imbue_stacking_bits INTEGER, item_set VARCHAR(100), equipment_set_extra INTEGER, + aetheria_bitfield INTEGER, heritage_specific_armor INTEGER, shared_cooldown INTEGER + )`, + `CREATE INDEX IF NOT EXISTS ix_enh_material_set ON item_enhancements (material, item_set)`, + + `CREATE TABLE IF NOT EXISTS item_ratings ( + item_id INTEGER PRIMARY KEY REFERENCES items(id), + damage_rating INTEGER, damage_resist_rating INTEGER, crit_rating INTEGER, + crit_resist_rating INTEGER, crit_damage_rating INTEGER, crit_damage_resist_rating INTEGER, + heal_boost_rating INTEGER, vitality_rating INTEGER, healing_rating INTEGER, + mana_conversion_rating INTEGER, weakness_rating INTEGER, nether_over_time INTEGER, + healing_resist_rating INTEGER, nether_resist_rating INTEGER, dot_resist_rating INTEGER, + life_resist_rating INTEGER, sneak_attack_rating INTEGER, recklessness_rating INTEGER, + deception_rating INTEGER, pk_damage_rating INTEGER, pk_damage_resist_rating INTEGER, + gear_pk_damage_rating INTEGER, gear_pk_damage_resist_rating INTEGER, + gear_damage INTEGER, gear_damage_resist INTEGER, gear_crit INTEGER, gear_crit_resist INTEGER, + gear_crit_damage INTEGER, gear_crit_damage_resist INTEGER, gear_healing_boost INTEGER, + gear_max_health INTEGER, gear_nether_resist INTEGER, gear_life_resist INTEGER, + gear_overpower INTEGER, gear_overpower_resist INTEGER, total_rating INTEGER + )`, + + `CREATE TABLE IF NOT EXISTS item_spells ( + item_id INTEGER REFERENCES items(id), + spell_id INTEGER, + is_active BOOLEAN DEFAULT false, + PRIMARY KEY (item_id, spell_id) + )`, + + `CREATE TABLE IF NOT EXISTS item_raw_data ( + item_id INTEGER PRIMARY KEY REFERENCES items(id), + int_values JSONB, double_values JSONB, string_values JSONB, bool_values JSONB, + original_json JSONB + )`, + } + ok, failed := 0, 0 + for _, s := range stmts { + if _, err := pool.Exec(ctx, s); err != nil { + failed++ + log.Warn("schema statement failed (continuing)", "err", err) + continue + } + ok++ + } + log.Info("inventory schema init complete", "ok", ok, "failed", failed) +} diff --git a/go-services/inventory-go/search.go b/go-services/inventory-go/search.go new file mode 100644 index 00000000..a5422e67 --- /dev/null +++ b/go-services/inventory-go/search.go @@ -0,0 +1,677 @@ +package main + +import ( + "context" + "fmt" + "net/http" + "net/url" + "strconv" + "strings" + "time" +) + +// /search/items — port of inventory-service main.py:2892. This slice implements +// the search QUERY (the CTE + all SQL filters + sort + pagination + count) and +// returns each row's direct DB columns plus the computed booleans. The deep +// per-row translation (material_name, spells, slot_name, ...) from +// extract_item_properties is layered on in a later slice; the filter/count logic +// — "which items match" — is validated here against the Python service. + +// cteSelect is the items_with_slots CTE body (everything up to FROM/JOINs). The +// rating columns are extracted from the item_raw_data int_values JSONB exactly +// as Python does (paired ids via GREATEST, singletons via COALESCE). +const cteSelect = ` +SELECT DISTINCT + i.id AS db_item_id, i.character_name, i.name, i.icon, i.object_class, i.value, i.burden, + i.current_wielded_location, i.bonded, i.attuned, i."unique", i.stack_size, i.max_stack_size, + i.structure, i.max_structure, i.rare_id, i.timestamp AS last_updated, + COALESCE(cs.max_damage, -1) AS max_damage, + COALESCE(cs.armor_level, -1) AS armor_level, + COALESCE(cs.attack_bonus, -1.0) AS attack_bonus, + COALESCE(cs.melee_defense_bonus, -1.0) AS melee_defense_bonus, + COALESCE(cs.weapon_time, -1) AS weapon_time, + COALESCE(cs.base_armor_level, cs.armor_level, -1) AS base_armor_level, + COALESCE(cs.base_max_damage, cs.max_damage, -1) AS base_max_damage, + GREATEST(COALESCE((rd.int_values->>'314')::int, -1), COALESCE((rd.int_values->>'374')::int, -1)) AS crit_damage_rating, + GREATEST(COALESCE((rd.int_values->>'307')::int, -1), COALESCE((rd.int_values->>'370')::int, -1)) AS damage_rating, + GREATEST(COALESCE((rd.int_values->>'323')::int, -1), COALESCE((rd.int_values->>'376')::int, -1)) AS heal_boost_rating, + COALESCE((rd.int_values->>'379')::int, -1) AS vitality_rating, + GREATEST(COALESCE((rd.int_values->>'308')::int, -1), COALESCE((rd.int_values->>'371')::int, -1)) AS damage_resist_rating, + COALESCE((rd.int_values->>'315')::int, -1) AS crit_resist_rating, + GREATEST(COALESCE((rd.int_values->>'316')::int, -1), COALESCE((rd.int_values->>'375')::int, -1)) AS crit_damage_resist_rating, + COALESCE((rd.int_values->>'317')::int, -1) AS healing_resist_rating, + COALESCE((rd.int_values->>'331')::int, -1) AS nether_resist_rating, + COALESCE((rd.int_values->>'342')::int, -1) AS healing_rating, + COALESCE((rd.int_values->>'350')::int, -1) AS dot_resist_rating, + COALESCE((rd.int_values->>'351')::int, -1) AS life_resist_rating, + COALESCE((rd.int_values->>'356')::int, -1) AS sneak_attack_rating, + COALESCE((rd.int_values->>'357')::int, -1) AS recklessness_rating, + COALESCE((rd.int_values->>'358')::int, -1) AS deception_rating, + COALESCE((rd.int_values->>'381')::int, -1) AS pk_damage_rating, + COALESCE((rd.int_values->>'382')::int, -1) AS pk_damage_resist_rating, + COALESCE((rd.int_values->>'383')::int, -1) AS gear_pk_damage_rating, + COALESCE((rd.int_values->>'384')::int, -1) AS gear_pk_damage_resist_rating, + COALESCE(req.wield_level, -1) AS wield_level, + COALESCE(enh.material, '') AS material, + COALESCE(enh.workmanship, -1.0) AS workmanship, + COALESCE(enh.imbue, '') AS imbue, + COALESCE(enh.tinks, -1) AS tinks, + COALESCE(enh.item_set, '') AS item_set, + COALESCE((rd.int_values->>'218103821')::int, 0) AS coverage_mask, + COALESCE((rd.int_values->>'218103822')::int, 0) AS equippable_slots, + CASE + WHEN rd.original_json IS NOT NULL + AND rd.original_json->'IntValues'->>'218103822' IS NOT NULL + AND (rd.original_json->'IntValues'->>'218103822')::int > 0 + THEN + CASE (rd.original_json->'IntValues'->>'218103822')::int + WHEN 1 THEN 'Head' WHEN 2 THEN 'Neck' WHEN 4 THEN 'Shirt' + WHEN 16 THEN 'Chest' WHEN 32 THEN 'Hands' WHEN 256 THEN 'Feet' + WHEN 512 THEN 'Chest' WHEN 1024 THEN 'Abdomen' WHEN 2048 THEN 'Upper Arms' + WHEN 4096 THEN 'Lower Arms' WHEN 8192 THEN 'Upper Legs' WHEN 16384 THEN 'Lower Legs' + WHEN 33554432 THEN 'Shield' + WHEN 15 THEN 'Chest, Abdomen, Upper Arms, Lower Arms' + WHEN 30 THEN 'Shirt' + WHEN 14336 THEN 'Chest, Abdomen, Upper Arms, Lower Arms' + WHEN 25600 THEN 'Abdomen, Upper Legs, Lower Legs' + ELSE CONCAT_WS(', ', + CASE WHEN (rd.original_json->'IntValues'->>'218103822')::int & 1 = 1 THEN 'Head' END, + CASE WHEN (rd.original_json->'IntValues'->>'218103822')::int & 512 = 512 THEN 'Chest' END, + CASE WHEN (rd.original_json->'IntValues'->>'218103822')::int & 1024 = 1024 THEN 'Abdomen' END, + CASE WHEN (rd.original_json->'IntValues'->>'218103822')::int & 2048 = 2048 THEN 'Upper Arms' END, + CASE WHEN (rd.original_json->'IntValues'->>'218103822')::int & 4096 = 4096 THEN 'Lower Arms' END, + CASE WHEN (rd.original_json->'IntValues'->>'218103822')::int & 32 = 32 THEN 'Hands' END, + CASE WHEN (rd.original_json->'IntValues'->>'218103822')::int & 8192 = 8192 THEN 'Upper Legs' END, + CASE WHEN (rd.original_json->'IntValues'->>'218103822')::int & 16384 = 16384 THEN 'Lower Legs' END, + CASE WHEN (rd.original_json->'IntValues'->>'218103822')::int & 256 = 256 THEN 'Feet' END) + END + WHEN i.object_class = 4 THEN + CASE + WHEN i.current_wielded_location = 32768 THEN 'Neck' + WHEN i.current_wielded_location = 262144 THEN 'Left Ring' + WHEN i.current_wielded_location = 524288 THEN 'Right Ring' + WHEN i.current_wielded_location = 786432 THEN 'Left Ring, Right Ring' + WHEN i.current_wielded_location = 131072 THEN 'Left Wrist' + WHEN i.current_wielded_location = 1048576 THEN 'Right Wrist' + WHEN i.current_wielded_location = 1179648 THEN 'Left Wrist, Right Wrist' + WHEN i.name ILIKE '%amulet%' OR i.name ILIKE '%necklace%' OR i.name ILIKE '%gorget%' THEN 'Neck' + WHEN i.name ILIKE '%ring%' AND i.name NOT ILIKE '%keyring%' AND i.name NOT ILIKE '%signet%' THEN 'Left Ring, Right Ring' + WHEN i.name ILIKE '%bracelet%' THEN 'Left Wrist, Right Wrist' + WHEN i.name ILIKE '%trinket%' THEN 'Trinket' + ELSE 'Jewelry' + END + WHEN i.object_class = 6 THEN 'Melee Weapon' + WHEN i.object_class = 7 THEN 'Missile Weapon' + WHEN i.object_class = 8 THEN 'Held' + WHEN i.current_wielded_location = 67108864 THEN 'Two-Handed' + WHEN i.name ILIKE '%cloak%' THEN 'Cloak' + ELSE '-' + END AS computed_slot_name, + COALESCE((SELECT STRING_AGG(CAST(sp_inner.spell_id AS VARCHAR), ',' ORDER BY sp_inner.spell_id) + FROM item_spells sp_inner WHERE sp_inner.item_id = i.id), '') AS computed_spell_names, + -- Ordered passive Spells from the raw item (matches extract_item_properties: + -- spell_names = [translate_spell(id) for id in original_json["Spells"]], in + -- array order, with duplicates preserved). Internal; stripped after enrich. + (SELECT STRING_AGG(elem, ',' ORDER BY ord) + FROM jsonb_array_elements_text( + CASE WHEN jsonb_typeof(rd.original_json->'Spells') = 'array' + THEN rd.original_json->'Spells' ELSE '[]'::jsonb END) + WITH ORDINALITY AS t(elem, ord)) AS spell_ids_ordered +FROM items i +LEFT JOIN item_combat_stats cs ON i.id = cs.item_id +LEFT JOIN item_requirements req ON i.id = req.item_id +LEFT JOIN item_enhancements enh ON i.id = enh.item_id +LEFT JOIN item_ratings rt ON i.id = rt.item_id +LEFT JOIN item_raw_data rd ON i.id = rd.item_id` + +var sortMapping = map[string]string{ + "name": "name", "character_name": "character_name", "value": "value", + "damage": "max_damage", "armor": "armor_level", "armor_level": "armor_level", + "workmanship": "workmanship", "level": "wield_level", "damage_rating": "damage_rating", + "crit_damage_rating": "crit_damage_rating", "heal_boost_rating": "heal_boost_rating", + "vitality_rating": "vitality_rating", "damage_resist_rating": "damage_resist_rating", + "crit_damage_resist_rating": "crit_damage_resist_rating", "item_set": "item_set", + "coverage": "coverage_mask", "item_type_name": "object_class", + "last_updated": "last_updated", "spell_names": "computed_spell_names", +} + +// argBuilder accumulates positional ($N) query args. +type argBuilder struct{ args []any } + +func (b *argBuilder) add(v any) string { + b.args = append(b.args, v) + return "$" + strconv.Itoa(len(b.args)) +} + +func (s *Server) handleSearchItems(w http.ResponseWriter, r *http.Request) { + res, err := s.runSearch(r.Context(), r.URL.Query()) + if err != nil { + s.dbErr(w, "search/items", err) + return + } + writeJSON(w, http.StatusOK, res) +} + +// runSearch executes /search/items and returns the response object (items + +// pagination, or an {error,...} object for invalid params). Shared by the HTTP +// handler and the suitbuilder solver's load_items, so both see identical rows. +func (s *Server) runSearch(ctx context.Context, q url.Values) (map[string]any, error) { + ab := &argBuilder{} + var conds []string + + // --- character (mutually exclusive cascade) --- + if c := q.Get("character"); c != "" { + conds = append(conds, "character_name = "+ab.add(c)) + } else if cs := q.Get("characters"); cs != "" { + names := splitNonEmpty(cs) + if len(names) == 0 { + return map[string]any{"error": "Empty characters list provided", "items": []any{}, "total_count": 0}, nil + } + ph := make([]string, len(names)) + for i, n := range names { + ph[i] = ab.add(n) + } + conds = append(conds, "character_name IN ("+strings.Join(ph, ", ")+")") + } else if !qBool(q, "include_all_characters") { + return map[string]any{"error": "Must specify character, characters, or set include_all_characters=true", "items": []any{}, "total_count": 0}, nil + } + + // --- text --- + if t := q.Get("text"); t != "" { + p := ab.add("%" + t + "%") + conds = append(conds, "(CONCAT(COALESCE(material,''),' ',name) ILIKE "+p+" OR name ILIKE "+p+" OR COALESCE(material,'') ILIKE "+p+")") + } + + // --- category (mutually exclusive) --- + switch { + case qBool(q, "armor_only"): + conds = append(conds, "(object_class = 2 AND COALESCE(armor_level,0) > 0)") + case qBool(q, "jewelry_only"): + conds = append(conds, "object_class = 4") + case qBool(q, "weapon_only"): + conds = append(conds, weaponTypeClause(q.Get("weapon_type"))) + case qBool(q, "clothing_only"): + conds = append(conds, "(object_class = 3 AND name NOT ILIKE '%cloak%' AND name NOT ILIKE '%robe%' AND name NOT ILIKE '%pallium%' AND name NOT ILIKE '%armet%' AND (name ILIKE '%shirt%' OR name ILIKE '%pants%' OR name ILIKE '%breeches%' OR name ILIKE '%baggy%' OR name ILIKE '%tunic%'))") + } + + // --- equipment status / slot --- + switch q.Get("equipment_status") { + case "equipped": + conds = append(conds, "current_wielded_location > 0") + case "unequipped": + conds = append(conds, "current_wielded_location = 0") + } + if v, ok := qInt(q, "equipment_slot"); ok { + conds = append(conds, "current_wielded_location = "+ab.add(v)) + } + + // --- combat + all rating filters (column >= :param) --- + geFilters := []struct{ param, col string }{ + {"min_damage", "max_damage"}, {"min_armor", "armor_level"}, + {"min_crit_damage_rating", "crit_damage_rating"}, {"min_damage_rating", "damage_rating"}, + {"min_heal_boost_rating", "heal_boost_rating"}, {"min_vitality_rating", "vitality_rating"}, + {"min_damage_resist_rating", "damage_resist_rating"}, {"min_crit_resist_rating", "crit_resist_rating"}, + {"min_crit_damage_resist_rating", "crit_damage_resist_rating"}, {"min_healing_resist_rating", "healing_resist_rating"}, + {"min_nether_resist_rating", "nether_resist_rating"}, {"min_healing_rating", "healing_rating"}, + {"min_dot_resist_rating", "dot_resist_rating"}, {"min_life_resist_rating", "life_resist_rating"}, + {"min_sneak_attack_rating", "sneak_attack_rating"}, {"min_recklessness_rating", "recklessness_rating"}, + {"min_deception_rating", "deception_rating"}, {"min_pk_damage_rating", "pk_damage_rating"}, + {"min_pk_damage_resist_rating", "pk_damage_resist_rating"}, {"min_gear_pk_damage_rating", "gear_pk_damage_rating"}, + {"min_gear_pk_damage_resist_rating", "gear_pk_damage_resist_rating"}, {"min_tinks", "tinks"}, + {"min_value", "value"}, {"min_workmanship", "workmanship"}, + } + for _, f := range geFilters { + if v := q.Get(f.param); v != "" { + if n, err := strconv.ParseFloat(v, 64); err == nil { + conds = append(conds, f.col+" >= "+ab.add(n)) + } + } + } + leFilters := []struct{ param, col string }{ + {"max_damage", "max_damage"}, {"max_armor", "armor_level"}, + {"max_value", "value"}, {"max_burden", "burden"}, + } + for _, f := range leFilters { + if v, ok := qInt(q, f.param); ok { + conds = append(conds, f.col+" <= "+ab.add(v)) + } + } + if v := q.Get("min_attack_bonus"); v != "" { + if n, err := strconv.ParseFloat(v, 64); err == nil { + conds = append(conds, "attack_bonus >= "+ab.add(n)) + } + } + + // --- requirements (wield level) --- + if v, ok := qInt(q, "max_level"); ok { + conds = append(conds, "(wield_level <= "+ab.add(v)+" OR wield_level IS NULL)") + } + if v, ok := qInt(q, "min_level"); ok { + conds = append(conds, "wield_level >= "+ab.add(v)) + } + + // --- enhancements --- + if m := q.Get("material"); m != "" { + conds = append(conds, "material ILIKE "+ab.add("%"+m+"%")) + } + if v := q.Get("has_imbue"); v != "" { + if qBool(q, "has_imbue") { + conds = append(conds, "(imbue IS NOT NULL AND imbue != '')") + } else { + conds = append(conds, "(imbue IS NULL OR imbue = '')") + } + } + + // --- item state --- + if v := q.Get("bonded"); v != "" { + conds = append(conds, ternary(qBool(q, "bonded"), "bonded > 0", "bonded = 0")) + } + if v := q.Get("attuned"); v != "" { + conds = append(conds, ternary(qBool(q, "attuned"), "attuned > 0", "attuned = 0")) + } + if v := q.Get("unique"); v != "" { + conds = append(conds, `"unique" = `+ab.add(qBool(q, "unique"))) + } + if v := q.Get("is_rare"); v != "" { + conds = append(conds, ternary(qBool(q, "is_rare"), "rare_id IS NOT NULL AND rare_id > 0", "(rare_id IS NULL OR rare_id <= 0)")) + } + if v, ok := qInt(q, "min_condition"); ok { + conds = append(conds, "((structure * 100.0 / NULLIF(max_structure, 0)) >= "+ab.add(v)+" OR max_structure IS NULL)") + } + + // --- item_set / item_sets (translate id->name, bug-for-bug) --- + if v := q.Get("item_set"); v != "" { + conds = append(conds, "item_set = "+ab.add(s.translateSetID(v))) + } else if v := q.Get("item_sets"); v != "" { + ids := splitNonEmpty(v) + if len(ids) != 1 { + conds = append(conds, "1 = 0") // 0 or >1 set ids => impossible + } else { + conds = append(conds, "item_set = "+ab.add(s.translateSetID(ids[0]))) + } + } + + // --- slot_names (OR of per-slot approaches over computed_slot_name) --- + if v := q.Get("slot_names"); v != "" { + var slotClauses []string + for _, name := range splitNonEmpty(v) { + slotClauses = append(slotClauses, slotNameClause(name, ab)) + } + if len(slotClauses) > 0 { + conds = append(conds, "("+strings.Join(slotClauses, " OR ")+")") + } + } + + where := "" + if len(conds) > 0 { + where = " WHERE " + strings.Join(conds, " AND ") + } + + // --- sort --- + sortField, ok := sortMapping[q.Get("sort_by")] + if !ok { + sortField = "name" + } + dir, nulls := "ASC", "NULLS LAST" + if strings.EqualFold(q.Get("sort_dir"), "desc") { + dir, nulls = "DESC", "NULLS FIRST" + } + orderBy := fmt.Sprintf(" ORDER BY %s %s %s, character_name, db_item_id", sortField, dir, nulls) + + // --- pagination --- + page := clampInt(qIntDefault(q, "page", 1), 1, 1<<30) + limit := clampInt(qIntDefault(q, "limit", 200), 1, 50000) + offset := (page - 1) * limit + + ctx, cancel := context.WithTimeout(ctx, 30*time.Second) + defer cancel() + + // Underwear filters (shirt_only/pants_only/underwear_only) are injected into + // the CTE body itself (they filter on raw i./rd. columns), mirroring Python's + // cte_where_clause insertion. Mutually exclusive, shirt > pants > underwear. + cteBody := cteSelect + if cw := underwearCTEWhere(q); cw != "" { + cteBody += "\n" + cw + } + cte := "WITH items_with_slots AS (" + cteBody + ")\n" + mainSQL := cte + "SELECT * FROM items_with_slots" + where + orderBy + + " LIMIT " + ab.add(limit) + " OFFSET " + ab.add(offset) + rows, err := queryRowsAsMaps(ctx, s.pool, mainSQL, ab.args...) + if err != nil { + return nil, err + } + + // count uses the SAME CTE (incl. the underwear injection) + conditions, so + // total_count is always consistent with the returned items. Python builds a + // SEPARATE count CTE (main.py:3747) that omits the underwear injection and + // uses a simpler computed_slot_name, so its total_count is inconsistent with + // its own items for underwear/slot_names filters (e.g. shirt_only reports the + // whole table). We deliberately do NOT replicate that bug. Normal browse + // filters apply to both CTEs identically, so those counts match Python. + // LIMIT/OFFSET args are unused here. + countSQL := cte + "SELECT COUNT(DISTINCT db_item_id) FROM items_with_slots" + where + var totalCount int64 + if err := s.pool.QueryRow(ctx, countSQL, ab.args[:len(ab.args)-2]...).Scan(&totalCount); err != nil { + return nil, err + } + + items := s.enrichRows(rows) + + return map[string]any{ + "items": items, + "total_count": totalCount, + "page": page, + "limit": limit, + "has_next": int64(page*limit) < totalCount, + "has_previous": page > 1, + }, nil +} + +// enrichRows applies the direct-column transforms (computed booleans, condition, +// timestamp), the material-name prefix, and the item_set name, then strips +// internal columns. Deeper enrichment (spells, slot_name, weapon damage/mana, +// rating fallbacks) is a later slice. +func (s *Server) enrichRows(rows []map[string]any) []map[string]any { + out := make([]map[string]any, 0, len(rows)) + for _, row := range rows { + row["is_equipped"] = toInt64(row["current_wielded_location"]) > 0 + row["is_bonded"] = toInt64(row["bonded"]) > 0 + row["is_attuned"] = toInt64(row["attuned"]) > 0 + row["is_rare"] = toInt64(row["rare_id"]) > 0 + st, mx := row["structure"], row["max_structure"] + if st != nil && mx != nil && toFloat(mx) != 0 { + row["condition_percent"] = roundTo(toFloat(st)*100/toFloat(mx), 1) + } else { + row["condition_percent"] = nil + } + if t, ok := row["last_updated"].(time.Time); ok { + row["last_updated"] = pyISO(t) + } + + // object_class_name — gem(11) context uses the ORIGINAL item name, so + // compute before the material prefix below (translate_object_class). + if oc := int(toInt64(row["object_class"])); oc != 0 { + row["object_class_name"] = s.translateObjectClass(oc, toStr(row["name"])) + } + + // material_name + material prefix on name (material is already a + // translated string in the DB; enrich_db_item:2371-2602). + if mat := toStr(row["material"]); mat != "" { + row["material_name"] = mat + if name := toStr(row["name"]); name != "" && + !strings.HasPrefix(strings.ToLower(name), strings.ToLower(mat)) { + row["original_name"] = name + row["name"] = mat + " " + name + } + } + // item_set_name (enrich_db_item:2551-2562). + if iset := strings.TrimSpace(toStr(row["item_set"])); iset != "" { + if n, ok := s.attributeSets[iset]; ok { + row["item_set_name"] = n + } else { + row["item_set_name"] = "Set " + iset + } + } + + // spells / spell_names from the ordered passive Spells array + // (enrich_db_item:3942-3951; only set when the item has spells). + if raw := toStr(row["spell_ids_ordered"]); raw != "" { + parts := strings.Split(raw, ",") + spells := make([]map[string]any, 0, len(parts)) + names := make([]string, 0, len(parts)) + for _, p := range parts { + id, err := strconv.Atoi(strings.TrimSpace(p)) + if err != nil { + continue + } + sp := s.translateSpell(id) + spells = append(spells, sp) + if n, _ := sp["name"].(string); n != "" { + names = append(names, n) + } + } + if len(spells) > 0 { + row["spells"] = spells + row["spell_names"] = names + } + } + delete(row, "spell_ids_ordered") + + // slot_name — sophisticated equipment-slot translation (main.py:3977-4033). + // Load-bearing for the suitbuilder: jewelry has an empty computed_slot_name, + // so load_items falls back to this to bucket rings/neck/wrists/trinket. + eq := int(toInt64(row["equippable_slots"])) + hasMat := toStr(row["material"]) != "" + row["slot_name"] = s.computeSlotName(eq, int(toInt64(row["coverage_mask"])), hasMat) + delete(row, "equippable_slots") + + // Gear-total display ratings (main.py:4035-4072): damage_rating, + // crit_damage_rating, heal_boost_rating only. The CTE already does + // GREATEST(individual, gear-key 370/374/376), so the gear-positive rescue + // branch is dead — the net rule is simply -1 -> null. The other three + // solver ratings (damage_resist/crit_damage_resist/vitality) stay -1. + for _, f := range []string{"damage_rating", "crit_damage_rating", "heal_boost_rating"} { + if toInt64(row[f]) == -1 { + row[f] = nil + } + } + + delete(row, "db_item_id") + out = append(out, row) + } + return out +} + +// translateObjectClass mirrors translate_object_class: ObjectClass enum lookup, +// with the context-aware Gem(11) classification by item name. The aetheria-by- +// IntValues path (for gem-class items not named crystal/gem/mana stone) is not +// reproduced here (it needs original_json) — a documented rare edge. +func (s *Server) translateObjectClass(oc int, name string) string { + base, ok := s.objectClasses[oc] + if !ok { + return fmt.Sprintf("Unknown_ObjectClass_%d", oc) + } + if base == "Gem" && oc == 11 { + n := strings.ToLower(name) + switch { + case strings.Contains(n, "mana stone"): + return "Mana Stone" + case strings.Contains(n, "crystal"): + return "Crystal" + case strings.Contains(n, "gem"): + return "Gem" + case strings.Contains(n, "aetheria"): + return "Aetheria" + } + return "Gem" + } + return base +} + +// translateSetID mirrors translate_equipment_set_id (AttributeSetInfo lookup, +// ID-string fallback). +func (s *Server) translateSetID(setID string) string { + if name, ok := s.attributeSets[setID]; ok { + return name + } + return setID +} + +// underwearCTEWhere returns the WHERE clause injected into the search CTE for +// the shirt_only / pants_only / underwear_only filters (main.py:3220-3251). +// Coverage bits on key 218103821: UnderwearUpperLegs=2, UnderwearLowerLegs=4, +// UnderwearChest=8, UnderwearAbdomen=16. +func underwearCTEWhere(q map[string][]string) string { + switch { + case qBool(q, "shirt_only"): + return `WHERE i.object_class = 3 + AND ((rd.int_values->>'218103821')::int & 8) > 0 + AND NOT ((rd.int_values->>'218103821')::int & 6) = 6 + AND i.name NOT ILIKE '%robe%' + AND i.name NOT ILIKE '%cloak%' + AND i.name NOT ILIKE '%pallium%' + AND i.name NOT ILIKE '%armet%' + AND i.name NOT ILIKE '%pants%' + AND i.name NOT ILIKE '%breeches%'` + case qBool(q, "pants_only"): + return `WHERE i.object_class = 3 + AND ((rd.int_values->>'218103821')::int & 2) = 2 + AND i.name NOT ILIKE '%robe%' + AND i.name NOT ILIKE '%cloak%' + AND i.name NOT ILIKE '%pallium%' + AND i.name NOT ILIKE '%armet%'` + case qBool(q, "underwear_only"): + return `WHERE i.object_class = 3 + AND ((rd.int_values->>'218103821')::int & 30) > 0 + AND i.name NOT ILIKE '%robe%' + AND i.name NOT ILIKE '%cloak%' + AND i.name NOT ILIKE '%pallium%' + AND i.name NOT ILIKE '%armet%'` + } + return "" +} + +func weaponTypeClause(wt string) string { + exists := func(skill int) string { + return fmt.Sprintf("(object_class = 1 AND EXISTS (SELECT 1 FROM item_raw_data wrd WHERE wrd.item_id = db_item_id AND (wrd.int_values->>'218103840')::int = %d))", skill) + } + switch strings.ToLower(wt) { + case "heavy": + return exists(44) + case "light": + return exists(45) + case "finesse": + return exists(46) + case "two_handed": + return exists(41) + case "bow": + return "(object_class = 9 AND name ILIKE '%bow%' AND name NOT ILIKE '%crossbow%')" + case "crossbow": + return "(object_class = 9 AND name ILIKE '%crossbow%')" + case "thrown": + return "(object_class = 9 AND (name ILIKE '%atlatl%' OR name ILIKE '%throwing%' OR name ILIKE '%javelin%' OR name ILIKE '%shuriken%' OR name ILIKE '%dart%' OR name ILIKE '%slingshot%'))" + case "caster": + return "object_class = 31" + default: + return "object_class IN (1, 9, 31)" + } +} + +func slotNameClause(name string, ab *argBuilder) string { + switch strings.ToLower(name) { + case "ring": + return "((computed_slot_name ILIKE '%Ring%') OR (object_class = 4 AND name ILIKE '%ring%' AND name NOT ILIKE '%keyring%' AND name NOT ILIKE '%signet%'))" + case "bracelet", "wrist": + return "((computed_slot_name ILIKE '%Wrist%') OR (object_class = 4 AND name ILIKE '%bracelet%'))" + case "neck": + return "((computed_slot_name ILIKE " + ab.add("%neck%") + ") OR (object_class = 4 AND (name ILIKE '%amulet%' OR name ILIKE '%necklace%' OR name ILIKE '%gorget%')))" + case "trinket": + // Approach 5 (jewelry fallback) MUST exclude %bracelet% — without it the + // Trinket fetch sweeps in bracelets, which then duplicate the Wrist buckets + // (also fetched via slot_names=Bracelet) and the DFS re-emits suits. + return "((computed_slot_name ILIKE " + ab.add("%trinket%") + ") OR (current_wielded_location = 67108864) OR (object_class = 4 AND (name ILIKE '%trinket%' OR name ILIKE '%compass%' OR name ILIKE '%goggles%')) OR (object_class = 11 AND name ILIKE '%trinket%') OR (object_class = 4 AND name NOT ILIKE '%ring%' AND name NOT ILIKE '%bracelet%' AND name NOT ILIKE '%amulet%' AND name NOT ILIKE '%necklace%' AND name NOT ILIKE '%gorget%'))" + case "cloak": + return "((computed_slot_name ILIKE " + ab.add("%cloak%") + ") OR (name ILIKE '%cloak%') OR (computed_slot_name = 'Cloak'))" + default: + return "(computed_slot_name ILIKE " + ab.add("%"+name+"%") + ")" + } +} + +func splitNonEmpty(s string) []string { + var out []string + for _, p := range strings.Split(s, ",") { + if p = strings.TrimSpace(p); p != "" { + out = append(out, p) + } + } + return out +} + +func qBool(q map[string][]string, key string) bool { + v := "" + if vs, ok := q[key]; ok && len(vs) > 0 { + v = vs[0] + } + switch strings.ToLower(v) { + case "1", "true", "yes", "on": + return true + } + return false +} + +func qInt(q map[string][]string, key string) (int, bool) { + if vs, ok := q[key]; ok && len(vs) > 0 && vs[0] != "" { + if n, err := strconv.Atoi(vs[0]); err == nil { + return n, true + } + } + return 0, false +} + +func qIntDefault(q map[string][]string, key string, def int) int { + if n, ok := qInt(q, key); ok { + return n + } + return def +} + +func ternary(c bool, a, b string) string { + if c { + return a + } + return b +} + +func clampInt(v, lo, hi int) int { + if v < lo { + return lo + } + if v > hi { + return hi + } + return v +} + +func toInt64(v any) int64 { + switch x := v.(type) { + case int64: + return x + case int32: + return int64(x) + case int: + return int64(x) + case float64: + return int64(x) + } + return 0 +} + +func toFloat(v any) float64 { + switch x := v.(type) { + case float64: + return x + case float32: + return float64(x) + case int64: + return float64(x) + case int32: + return float64(x) + case int: + return float64(x) + } + return 0 +} + +func roundTo(v float64, places int) float64 { + p := 1.0 + for i := 0; i < places; i++ { + p *= 10 + } + r := v * p + if r < 0 { + r -= 0.5 + } else { + r += 0.5 + } + return float64(int64(r)) / p +} diff --git a/go-services/inventory-go/slotname.go b/go-services/inventory-go/slotname.go new file mode 100644 index 00000000..449eb8ad --- /dev/null +++ b/go-services/inventory-go/slotname.go @@ -0,0 +1,183 @@ +package main + +import ( + "fmt" + "math/bits" + "strings" +) + +// Port of main.py's sophisticated equipment-slot translation, used to emit the +// `slot_name` field. This is load-bearing for the suitbuilder: jewelry items get +// an empty computed_slot_name (their EquipMask isn't an armor-coverage value, so +// the SQL CONCAT_WS yields ''), and load_items falls back to slot_name +// (`computed_slot_name or slot_name`) to bucket them as Left Ring / Neck / etc. + +// equipMaskEntry is one EquipMask enum row, kept in ascending-mask order so the +// bit-flag decode joins parts deterministically (Left before Right). +type equipMaskEntry struct { + Mask int + Name string +} + +// equipFriendly maps technical EquipMask names to display names +// (translate_equipment_slot's name_mapping, identical in both branches). +var equipFriendly = map[string]string{ + "HeadWear": "Head", "ChestWear": "Chest", "ChestArmor": "Chest", + "AbdomenWear": "Abdomen", "AbdomenArmor": "Abdomen", + "UpperArmWear": "Upper Arms", "UpperArmArmor": "Upper Arms", + "LowerArmWear": "Lower Arms", "LowerArmArmor": "Lower Arms", + "HandWear": "Hands", "UpperLegWear": "Upper Legs", "UpperLegArmor": "Upper Legs", + "LowerLegWear": "Lower Legs", "LowerLegArmor": "Lower Legs", "FootWear": "Feet", + "NeckWear": "Neck", "WristWearLeft": "Left Wrist", "WristWearRight": "Right Wrist", + "FingerWearLeft": "Left Ring", "FingerWearRight": "Right Ring", + "MeleeWeapon": "Melee Weapon", "Shield": "Shield", "MissileWeapon": "Missile Weapon", + "MissileAmmo": "Ammo", "Held": "Held", "TwoHanded": "Two-Handed", + "TrinketOne": "Trinket", "Cloak": "Cloak", "Robe": "Robe", +} + +var commonSlots = map[int]string{ + 30: "Shirt", + 786432: "Left Ring, Right Ring", + 262144: "Left Ring", + 524288: "Right Ring", +} + +func friendlySlot(name string) string { + if f, ok := equipFriendly[name]; ok { + return f + } + return name +} + +func isBodyArmorEquipMask(v int) bool { return v&0x00007F21 != 0 } +func isBodyArmorCoverageMask(v int) bool { return v&0x0001FF00 != 0 } +func totalBitsSet(v int) int { return bits.OnesCount(uint(uint32(v))) } + +// getCoverageReductionOptions mirrors main.py:658. +func getCoverageReductionOptions(coverage int) []int { + const ( + oUpperArms = 4096 + oLowerArms = 8192 + oUpperLegs = 256 + oLowerLegs = 512 + oChest = 1024 + oAbdomen = 2048 + head = 16384 + hands = 32768 + feet = 65536 + ) + if totalBitsSet(coverage) <= 1 || !isBodyArmorCoverageMask(coverage) { + return []int{coverage} + } + switch coverage { + case oUpperArms | oLowerArms: + return []int{oUpperArms, oLowerArms} + case oUpperLegs | oLowerLegs: + return []int{oUpperLegs, oLowerLegs} + case oLowerLegs | feet: + return []int{feet} + case oChest | oAbdomen: + return []int{oChest} + case oChest | oAbdomen | oUpperArms: + return []int{oChest} + case oChest | oUpperArms | oLowerArms: + return []int{oChest} + case oChest | oUpperArms: + return []int{oChest} + case oAbdomen | oUpperLegs | oLowerLegs: + return []int{oAbdomen, oUpperLegs, oLowerLegs} + case oChest | oAbdomen | oUpperArms | oLowerArms: + return []int{oChest} + case oAbdomen | oUpperLegs: + return []int{oAbdomen} + } + return []int{coverage} +} + +// coverageToEquipMask mirrors main.py:717. +func coverageToEquipMask(coverage int) int { + m := map[int]int{ + 16384: 1, 1024: 512, 4096: 2048, 8192: 4096, 32768: 32, + 2048: 1024, 256: 8192, 512: 16384, 65536: 256, + } + if v, ok := m[coverage]; ok { + return v + } + return coverage +} + +// getSophisticatedSlotOptions mirrors main.py:734. +func getSophisticatedSlotOptions(equippableSlots, coverageValue int, hasMaterial bool) []int { + const lowerLegWear, footWear = 128, 256 + if equippableSlots == (lowerLegWear | footWear) { + return []int{footWear} + } + if isBodyArmorEquipMask(equippableSlots) && totalBitsSet(equippableSlots) > 1 { + if !hasMaterial { + return []int{equippableSlots} + } + var slotOpts []int + for _, o := range getCoverageReductionOptions(coverageValue) { + slotOpts = append(slotOpts, coverageToEquipMask(o)) + } + if len(slotOpts) > 0 { + return slotOpts + } + return []int{equippableSlots} + } + return []int{equippableSlots} +} + +// translateEquipmentSlot mirrors main.py:807. +func (s *Server) translateEquipmentSlot(loc int) string { + if loc == 0 { + return "Inventory" + } + if name, ok := s.equipMaskMap[loc]; ok { + return friendlySlot(name) + } + if cs, ok := commonSlots[loc]; ok { + return cs + } + var parts []string + for _, e := range s.equipMaskOrdered { + if e.Mask > 0 && loc&e.Mask == e.Mask { + parts = append(parts, friendlySlot(e.Name)) + } + } + if len(parts) > 0 { + return strings.Join(parts, ", ") + } + if loc >= 268435456 { + switch loc { + case 268435456: + return "Aetheria Blue" + case 536870912: + return "Aetheria Yellow" + case 1073741824: + return "Aetheria Red" + default: + return fmt.Sprintf("Special Slot (%d)", loc) + } + } + return "-" +} + +// computeSlotName mirrors the slot_name block in search_items (main.py:3977-4033). +func (s *Server) computeSlotName(equippableSlots, coverageValue int, hasMaterial bool) string { + if equippableSlots <= 0 { + return "-" + } + opts := getSophisticatedSlotOptions(equippableSlots, coverageValue, hasMaterial) + var names []string + for _, o := range opts { + n := s.translateEquipmentSlot(o) + if n != "" && !containsString(names, n) { + names = append(names, n) + } + } + if len(names) > 0 { + return strings.Join(names, ", ") + } + return "-" +} diff --git a/go-services/inventory-go/store.go b/go-services/inventory-go/store.go new file mode 100644 index 00000000..8ebb1b9e --- /dev/null +++ b/go-services/inventory-go/store.go @@ -0,0 +1,83 @@ +package main + +import ( + "context" + "encoding/json" + "fmt" + "log/slog" + "net/http" + "time" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgxpool" +) + +// newPool creates a pgx pool. When readOnly (the default in parallel mode), every +// connection is forced into read-only transactions so the Go service can never +// mutate the production inventory_db it shares with the Python service. +func newPool(ctx context.Context, dsn string, readOnly bool) (*pgxpool.Pool, error) { + cfg, err := pgxpool.ParseConfig(dsn) + if err != nil { + return nil, fmt.Errorf("parse DATABASE_URL: %w", err) + } + cfg.MaxConns = 10 + cfg.MaxConnIdleTime = 5 * time.Minute + if readOnly { + cfg.AfterConnect = func(ctx context.Context, conn *pgx.Conn) error { + _, err := conn.Exec(ctx, "SET default_transaction_read_only = on") + return err + } + } + return pgxpool.NewWithConfig(ctx, cfg) +} + +func queryRowsAsMaps(ctx context.Context, pool *pgxpool.Pool, sql string, args ...any) ([]map[string]any, error) { + rows, err := pool.Query(ctx, sql, args...) + if err != nil { + return nil, err + } + out, err := pgx.CollectRows(rows, pgx.RowToMap) + if err != nil { + return nil, err + } + if out == nil { + out = []map[string]any{} + } + return out, nil +} + +// pyISO mirrors Python datetime.isoformat() for a UTC value (matches FastAPI's +// jsonable_encoder). Note the inventory-service stores naive datetimes (no tz), +// so isoformat has no offset — we format without one. +func pyISO(t time.Time) string { + t = t.UTC() + if t.Nanosecond() == 0 { + return t.Format("2006-01-02T15:04:05") + } + return t.Format("2006-01-02T15:04:05") + fmt.Sprintf(".%06d", t.Nanosecond()/1000) +} + +func formatTimes(rows []map[string]any, keys ...string) { + for _, m := range rows { + for _, k := range keys { + if t, ok := m[k].(time.Time); ok { + m[k] = pyISO(t) + } + } + } +} + +func toStr(v any) string { + if s, ok := v.(string); ok { + return s + } + return "" +} + +func writeJSON(w http.ResponseWriter, status int, v any) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(status) + if err := json.NewEncoder(w).Encode(v); err != nil { + slog.Error("json encode failed", "err", err) + } +} diff --git a/go-services/inventory-go/suit_cd.go b/go-services/inventory-go/suit_cd.go new file mode 100644 index 00000000..d0d4f843 --- /dev/null +++ b/go-services/inventory-go/suit_cd.go @@ -0,0 +1,74 @@ +package main + +import "strings" + +// CD-tier filtering for the suitbuilder. The allowed_crit_damage constraint +// restricts which crit-damage tiers are permitted on ARMOR pieces; jewelry and +// clothing are never affected. "Prefer the highest allowed tier" is NOT done +// here — it falls out of the existing scoring (CritDamage2 > CritDamage1) and +// the CD-descending armor sort once disallowed tiers are removed. + +// critTier normalizes a raw crit_damage_rating into a tier in {0,1,2}. Rare +// high-crit gear (rating >= 2, including 3+) collapses to tier 2 so it counts +// as "CD2" rather than being silently excluded. +func critTier(rating int) int { + switch { + case rating <= 0: + return 0 + case rating == 1: + return 1 + default: + return 2 + } +} + +// isArmorSlot reports whether a slot name denotes an armor coverage slot, +// including comma-joined multi-coverage slots like "Chest, Abdomen". +func isArmorSlot(slot string) bool { + if armorSlotSet[slot] { + return true + } + if strings.Contains(slot, ", ") { + for _, p := range strings.Split(slot, ", ") { + if armorSlotSet[strings.TrimSpace(p)] { + return true + } + } + } + return false +} + +// allowedCritSet normalizes the constraint's allowed crit-damage tiers into a +// set, or returns nil when the filter is INACTIVE: no values, or all three +// tiers {0,1,2} present (== default). A nil result means "no filter" and keeps +// the default search path byte-identical to the unfiltered solver. +func allowedCritSet(vals []int) map[int]bool { + if len(vals) == 0 { + return nil + } + set := map[int]bool{} + for _, v := range vals { + set[critTier(v)] = true + } + if set[0] && set[1] && set[2] { + return nil + } + return set +} + +// filterArmorByCD drops armor items whose crit-damage tier is not in allowed. +// Non-armor items (jewelry, clothing, unknown) always pass through. When +// allowed is nil the input is returned unchanged. +func filterArmorByCD(items []*SuitItem, allowed map[int]bool) []*SuitItem { + if allowed == nil { + return items + } + out := make([]*SuitItem, 0, len(items)) + for _, it := range items { + if isArmorSlot(it.Slot) && !allowed[critTier(it.Ratings["crit_damage_rating"])] { + continue + } + out = append(out, it) + } + return out +} diff --git a/go-services/inventory-go/suit_cd_test.go b/go-services/inventory-go/suit_cd_test.go new file mode 100644 index 00000000..a366b218 --- /dev/null +++ b/go-services/inventory-go/suit_cd_test.go @@ -0,0 +1,82 @@ +package main + +import "testing" + +func TestCritTier(t *testing.T) { + cases := []struct { + rating, want int + }{{-1, 0}, {0, 0}, {1, 1}, {2, 2}, {3, 2}, {5, 2}} + for _, c := range cases { + if got := critTier(c.rating); got != c.want { + t.Errorf("critTier(%d) = %d, want %d", c.rating, got, c.want) + } + } +} + +func TestAllowedCritSet(t *testing.T) { + for _, vals := range [][]int{nil, {}, {0, 1, 2}, {0, 1, 3}} { + if allowedCritSet(vals) != nil { + t.Errorf("allowedCritSet(%v) should be nil (inactive)", vals) + } + } + if s := allowedCritSet([]int{1}); s == nil || !s[1] || s[0] || s[2] { + t.Errorf("allowedCritSet({1}) = %v, want only tier 1", s) + } + if s := allowedCritSet([]int{0, 1}); s == nil || !s[0] || !s[1] || s[2] { + t.Errorf("allowedCritSet({0,1}) = %v, want tiers 0,1", s) + } + if s := allowedCritSet([]int{3}); s == nil || !s[2] || s[0] || s[1] { + t.Errorf("allowedCritSet({3}) = %v, want only tier 2 (normalized)", s) + } +} + +func TestIsArmorSlot(t *testing.T) { + for _, s := range []string{"Chest", "Head", "Feet", "Chest, Abdomen", "Upper Legs, Lower Legs"} { + if !isArmorSlot(s) { + t.Errorf("isArmorSlot(%q) = false, want true", s) + } + } + for _, s := range []string{"Neck", "Left Ring", "Left Wrist", "Trinket", "Shirt", "Pants", "Unknown", ""} { + if isArmorSlot(s) { + t.Errorf("isArmorSlot(%q) = true, want false", s) + } + } +} + +func cdItem(slot string, cd int) *SuitItem { + return &SuitItem{Slot: slot, Ratings: map[string]int{"crit_damage_rating": cd}} +} + +func TestFilterArmorByCD(t *testing.T) { + items := []*SuitItem{ + cdItem("Chest", 0), cdItem("Head", 1), cdItem("Feet", 2), + cdItem("Chest, Abdomen", 2), // multi-coverage armor, CD2 + cdItem("Neck", 0), // jewelry — never filtered + cdItem("Shirt", 0), // clothing — never filtered + } + + if got := filterArmorByCD(items, nil); len(got) != len(items) { + t.Errorf("nil filter dropped items: got %d, want %d", len(got), len(items)) + } + + got := filterArmorByCD(items, map[int]bool{1: true}) + keep := map[string]bool{"Head": true, "Neck": true, "Shirt": true} + if len(got) != 3 { + t.Fatalf("allowed{1}: got %d items, want 3", len(got)) + } + for _, it := range got { + if !keep[it.Slot] { + t.Errorf("allowed{1}: unexpected slot %q survived", it.Slot) + } + } + + got = filterArmorByCD(items, map[int]bool{0: true, 1: true}) + if len(got) != 4 { // Chest(0), Head(1), Neck, Shirt + t.Errorf("allowed{0,1}: got %d items, want 4", len(got)) + } + for _, it := range got { + if isArmorSlot(it.Slot) && it.Ratings["crit_damage_rating"] >= 2 { + t.Errorf("allowed{0,1}: CD2 armor %q should have been dropped", it.Slot) + } + } +} diff --git a/go-services/inventory-go/suit_http.go b/go-services/inventory-go/suit_http.go new file mode 100644 index 00000000..d071040a --- /dev/null +++ b/go-services/inventory-go/suit_http.go @@ -0,0 +1,92 @@ +package main + +import ( + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "sync" + "time" +) + +// Suitbuilder endpoints — port of suitbuilder.py's router (mounted at +// /suitbuilder in the Python service). The live UI hits /inv/suitbuilder/* on +// the tracker, which proxies here; we expose the same contract for parallel +// validation. + +// POST /suitbuilder/search — streams SSE events (event: \ndata: \n\n). +func (s *Server) handleSuitSearch(w http.ResponseWriter, r *http.Request) { + body, _ := io.ReadAll(io.LimitReader(r.Body, 1<<20)) + // Pydantic defaults applied before decode; json.Unmarshal only overwrites + // fields present in the body. + c := SearchConstraints{IncludeEquipped: true, IncludeInventory: true, MaxResults: 50, SearchTimeout: 300} + if err := json.Unmarshal(body, &c); err != nil { + writeJSON(w, http.StatusUnprocessableEntity, map[string]any{"detail": "invalid SearchConstraints"}) + return + } + + flusher, ok := w.(http.Flusher) + if !ok { + writeJSON(w, http.StatusInternalServerError, map[string]any{"detail": "streaming unsupported"}) + return + } + h := w.Header() + h.Set("Content-Type", "text/event-stream") + h.Set("Cache-Control", "no-cache") + h.Set("Connection", "keep-alive") + h.Set("Access-Control-Allow-Origin", "*") + h.Set("Access-Control-Allow-Headers", "Cache-Control") + w.WriteHeader(http.StatusOK) + + var mu sync.Mutex + emit := func(event string, data map[string]any) { + b, err := json.Marshal(data) + if err != nil { + b, _ = json.Marshal(map[string]any{"message": "Serialization error: " + err.Error()}) + event = "error" + } + mu.Lock() + fmt.Fprintf(w, "event: %s\n", event) + fmt.Fprintf(w, "data: %s\n\n", b) + flusher.Flush() + mu.Unlock() + } + cancelled := func() bool { + select { + case <-r.Context().Done(): + return true + default: + return false + } + } + + sv := newSolver(s, c, emit, cancelled) + sv.Search(r.Context()) +} + +// GET /suitbuilder/characters (suitbuilder.py:2085). +func (s *Server) handleSuitCharacters(w http.ResponseWriter, r *http.Request) { + ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second) + defer cancel() + rows, err := queryRowsAsMaps(ctx, s.pool, `SELECT DISTINCT character_name FROM items ORDER BY character_name`) + if err != nil { + s.dbErr(w, "suitbuilder/characters", err) + return + } + chars := make([]any, 0, len(rows)) + for _, row := range rows { + chars = append(chars, row["character_name"]) + } + writeJSON(w, http.StatusOK, map[string]any{"characters": chars}) +} + +// GET /suitbuilder/sets (suitbuilder.py:2195) — the hardcoded set list. +func (s *Server) handleSuitSets(w http.ResponseWriter, r *http.Request) { + order := []int{14, 16, 13, 21, 40, 41, 46, 47, 48, 15, 19, 20, 22, 24, 26, 29} + sets := make([]map[string]any, 0, len(order)) + for _, id := range order { + sets = append(sets, map[string]any{"id": id, "name": setNames[id]}) + } + writeJSON(w, http.StatusOK, map[string]any{"sets": sets}) +} diff --git a/go-services/inventory-go/suit_model.go b/go-services/inventory-go/suit_model.go new file mode 100644 index 00000000..d42b1960 --- /dev/null +++ b/go-services/inventory-go/suit_model.go @@ -0,0 +1,592 @@ +package main + +import ( + "fmt" + "hash/fnv" + "math/bits" + "sort" + "strings" +) + +// Port of inventory-service/suitbuilder.py data model. This is the LIVE solver +// (mounted at /suitbuilder/search; main.py's /optimize/suits is legacy/unused). +// Every sort carries (character_name, name) tiebreakers so results are +// deterministic and reproducible, exactly as the Python source documents. + +// --- Equipment set name<->id maps (suitbuilder.py SET_NAMES / _convert_set_name_to_id) --- + +var setNames = map[int]string{ + 14: "Adept's", 16: "Defender's", 13: "Soldier's", 21: "Wise", + 40: "Heroic Protector", 41: "Heroic Destroyer", 46: "Relic Alduressa", + 47: "Ancient Relic", 48: "Noble Relic", 15: "Archer's", 19: "Hearty", + 20: "Dexterous", 22: "Swift", 24: "Reinforced", 26: "Flame Proof", + 29: "Lightning Proof", +} + +// nameToSetID is the reverse map used by load_items to turn the search's +// item_set field into a numeric set id (note the " Set" suffix, verbatim). +var nameToSetID = map[string]int{ + "Adept's Set": 14, "Defender's Set": 16, "Soldier's Set": 13, "Wise Set": 21, + "Heroic Protector Set": 40, "Heroic Destroyer Set": 41, "Relic Alduressa Set": 46, + "Ancient Relic Set": 47, "Noble Relic Set": 48, "Archer's Set": 15, + "Hearty Set": 19, "Dexterous Set": 20, "Swift Set": 22, "Reinforced Set": 24, + "Flame Proof Set": 26, "Lightning Proof Set": 29, +} + +// getSetName mirrors suitbuilder.get_set_name (None/0 -> ""). +func getSetName(setID int) string { + if setID == 0 { + return "" + } + if n, ok := setNames[setID]; ok { + return n + } + return fmt.Sprintf("Set %d", setID) +} + +func convertSetNameToID(setName string) int { return nameToSetID[setName] } + +// --- CoverageMask (suitbuilder.py:81) --- + +const ( + covUnderwearUpperLegs = 0x00000002 + covUnderwearLowerLegs = 0x00000004 + covUnderwearChest = 0x00000008 + covUnderwearAbdomen = 0x00000010 + covUnderwearUpperArms = 0x00000020 + covUnderwearLowerArms = 0x00000040 + covOuterUpperLegs = 0x00000100 + covOuterLowerLegs = 0x00000200 + covOuterChest = 0x00000400 + covOuterAbdomen = 0x00000800 + covOuterUpperArms = 0x00001000 + covOuterLowerArms = 0x00002000 + covHead = 0x00004000 + covHands = 0x00008000 + covFeet = 0x00010000 + + // Aliases matching slot names (suitbuilder.py:110-115). + covChest = covOuterChest + covAbdomen = covOuterAbdomen + covUpperArms = covOuterUpperArms + covLowerArms = covOuterLowerArms + covUpperLegs = covOuterUpperLegs + covLowerLegs = covOuterLowerLegs + + magRobePattern = 0x00013F00 +) + +// coverageReductionOptions mirrors CoverageMask.reduction_options(). +func coverageReductionOptions(v int) []int { + if bits.OnesCount(uint(v)) <= 1 { + return nil + } + if coverageIsRobe(v) { + return nil + } + switch v { + case covUpperArms | covLowerArms: + return []int{covUpperArms, covLowerArms} + case covUpperLegs | covLowerLegs: + return []int{covUpperLegs, covLowerLegs} + case covLowerLegs | covFeet: + return []int{covFeet} + case covChest | covAbdomen: + return []int{covChest} + case covChest | covAbdomen | covUpperArms: + return []int{covChest} + case covChest | covUpperArms | covLowerArms: + return []int{covChest} + case covChest | covUpperArms: + return []int{covChest} + case covAbdomen | covUpperLegs | covLowerLegs: + return []int{covAbdomen, covUpperLegs, covLowerLegs} + case covChest | covAbdomen | covUpperArms | covLowerArms: + return []int{covChest} + case covAbdomen | covUpperLegs: + return []int{covAbdomen} + } + return nil +} + +// coverageIsRobe mirrors CoverageMask.is_robe() (exact pattern == component +// pattern == 0x13F00; otherwise the 6+ coverage-areas fallback). +func coverageIsRobe(v int) bool { + if v == magRobePattern { + return true + } + return bits.OnesCount(uint(v)) >= 6 +} + +// coverageToSlotName mirrors CoverageMask.to_slot_name() (single coverage only). +func coverageToSlotName(v int) string { + switch v { + case covHead: + return "Head" + case covChest: + return "Chest" + case covUpperArms: + return "Upper Arms" + case covLowerArms: + return "Lower Arms" + case covHands: + return "Hands" + case covAbdomen: + return "Abdomen" + case covUpperLegs: + return "Upper Legs" + case covLowerLegs: + return "Lower Legs" + case covFeet: + return "Feet" + } + return "" +} + +// --- SuitItem (suitbuilder.py:221) --- + +type SuitItem struct { + ID string // unique per (character,name); used for uniqueness checks + Name string + CharacterName string + Slot string + Coverage int // 0 == None + HasCoverage bool + SetID int // 0 == None + ArmorLevel int + Ratings map[string]int + SpellBitmap uint64 + SpellNames []string + IsLocked bool + Material string +} + +func (it *SuitItem) ratingsSum() int { + s := 0 + for _, v := range it.Ratings { + s += v + } + return s +} + +func (it *SuitItem) ratingsSumExcept(skip string) int { + s := 0 + for k, v := range it.Ratings { + if k != skip { + s += v + } + } + return s +} + +func (it *SuitItem) clone(slot string, name string, coverage int, hasCov bool) *SuitItem { + r := make(map[string]int, len(it.Ratings)) + for k, v := range it.Ratings { + r[k] = v + } + sn := make([]string, len(it.SpellNames)) + copy(sn, it.SpellNames) + return &SuitItem{ + ID: it.ID, Name: name, CharacterName: it.CharacterName, Slot: slot, + Coverage: coverage, HasCoverage: hasCov, SetID: it.SetID, ArmorLevel: it.ArmorLevel, + Ratings: r, SpellBitmap: it.SpellBitmap, SpellNames: sn, IsLocked: it.IsLocked, + Material: it.Material, + } +} + +// --- ItemBucket (suitbuilder.py:247) --- + +type ItemBucket struct { + Slot string + Items []*SuitItem + IsArmor bool +} + +var clothingSortSlots = map[string]bool{"Shirt": true, "Pants": true} + +// sortItems mirrors ItemBucket.sort_items() (reverse=True over the key tuple, +// stable so equal keys keep prior order). +func (b *ItemBucket) sortItems() { + items := b.Items + if _, isClothing := clothingSortSlots[b.Slot]; isClothing { + sort.SliceStable(items, func(i, j int) bool { + return descTuple( + cmpInt(items[i].Ratings["damage_rating"], items[j].Ratings["damage_rating"]), + cmpInt(len(items[i].SpellNames), len(items[j].SpellNames)), + cmpInt(items[i].ratingsSumExcept("damage_rating"), items[j].ratingsSumExcept("damage_rating")), + cmpStr(items[i].CharacterName, items[j].CharacterName), + cmpStr(items[i].Name, items[j].Name), + ) + }) + } else if b.IsArmor { + sort.SliceStable(items, func(i, j int) bool { + return descTuple( + cmpInt(items[i].ArmorLevel, items[j].ArmorLevel), + cmpInt(items[i].Ratings["crit_damage_rating"], items[j].Ratings["crit_damage_rating"]), + cmpInt(len(items[i].SpellNames), len(items[j].SpellNames)), + cmpInt(items[i].ratingsSum(), items[j].ratingsSum()), + cmpStr(items[i].CharacterName, items[j].CharacterName), + cmpStr(items[i].Name, items[j].Name), + ) + }) + } else { + sort.SliceStable(items, func(i, j int) bool { + return descTuple( + cmpInt(len(items[i].SpellNames), len(items[j].SpellNames)), + cmpInt(items[i].ratingsSum(), items[j].ratingsSum()), + cmpStr(items[i].CharacterName, items[j].CharacterName), + cmpStr(items[i].Name, items[j].Name), + ) + }) + } +} + +// descTuple returns true if the left tuple sorts before the right under Python's +// reverse=True (i.e. the larger tuple comes first). cmp* return -1/0/1. +func descTuple(cmps ...int) bool { + for _, c := range cmps { + if c != 0 { + return c > 0 // larger first + } + } + return false +} + +func cmpInt(a, b int) int { + switch { + case a < b: + return -1 + case a > b: + return 1 + } + return 0 +} + +func cmpStr(a, b string) int { return strings.Compare(a, b) } + +// --- SpellBitmapIndex (suitbuilder.py:299) --- + +type SpellBitmapIndex struct { + spellToBit map[string]uint64 + order []struct { + bit uint64 + name string + } + nextBit uint +} + +func newSpellBitmapIndex() *SpellBitmapIndex { + return &SpellBitmapIndex{spellToBit: map[string]uint64{}} +} + +func (s *SpellBitmapIndex) registerSpell(name string) uint64 { + if b, ok := s.spellToBit[name]; ok { + return b + } + var bit uint64 + if s.nextBit < 64 { + bit = uint64(1) << s.nextBit + } // >=64: bit stays 0 (only non-required spells ever reach here; required + // spells are registered first, so their low bits are always exact). + s.spellToBit[name] = bit + s.order = append(s.order, struct { + bit uint64 + name string + }{bit, name}) + s.nextBit++ + return bit +} + +func (s *SpellBitmapIndex) getBitmap(spells []string) uint64 { + var m uint64 + for _, sp := range spells { + m |= s.registerSpell(sp) + } + return m +} + +func (s *SpellBitmapIndex) getSpellNames(bitmap uint64) []string { + var out []string + for _, e := range s.order { + if e.bit != 0 && bitmap&e.bit != 0 { + out = append(out, e.name) + } + } + return out +} + +// --- SuitState (suitbuilder.py:342) --- + +type SuitState struct { + Items map[string]*SuitItem + SpellBitmap uint64 + SetCounts map[int]int + TotalArmor int + TotalRatings map[string]int + Occupied map[string]bool +} + +func newSuitState() *SuitState { + return &SuitState{ + Items: map[string]*SuitItem{}, SetCounts: map[int]int{}, + TotalRatings: map[string]int{}, Occupied: map[string]bool{}, + } +} + +func (st *SuitState) push(it *SuitItem) { + st.Items[it.Slot] = it + st.Occupied[it.Slot] = true + st.SpellBitmap |= it.SpellBitmap + if it.SetID != 0 { + st.SetCounts[it.SetID]++ + } + st.TotalArmor += it.ArmorLevel + for k, v := range it.Ratings { + st.TotalRatings[k] += v + } +} + +func (st *SuitState) pop(slot string) { + it, ok := st.Items[slot] + if !ok { + return + } + delete(st.Items, slot) + delete(st.Occupied, slot) + // Rebuild spell bitmap (overlaps prevent simple subtraction). + st.SpellBitmap = 0 + for _, r := range st.Items { + st.SpellBitmap |= r.SpellBitmap + } + if it.SetID != 0 { + st.SetCounts[it.SetID]-- + if st.SetCounts[it.SetID] == 0 { + delete(st.SetCounts, it.SetID) + } + } + st.TotalArmor -= it.ArmorLevel + for k, v := range it.Ratings { + if _, present := st.TotalRatings[k]; present { + st.TotalRatings[k] -= v + if st.TotalRatings[k] <= 0 { + delete(st.TotalRatings, k) + } + } + } +} + +// --- ScoringWeights / SearchConstraints (suitbuilder.py:409,426) --- + +type ScoringWeights struct { + ArmorSetComplete int + MissingSetPenalty int + CritDamage1 int + CritDamage2 int + DamageRating1 int + DamageRating2 int + DamageRating3 int +} + +func defaultScoringWeights() ScoringWeights { + return ScoringWeights{ + ArmorSetComplete: 1000, MissingSetPenalty: -200, + CritDamage1: 10, CritDamage2: 20, + DamageRating1: 10, DamageRating2: 20, DamageRating3: 30, + } +} + +type LockedSlotInfo struct { + SetID int `json:"set_id"` + Spells []string `json:"spells"` +} + +type SearchConstraints struct { + Characters []string `json:"characters"` + PrimarySet int `json:"primary_set"` + SecondarySet int `json:"secondary_set"` + RequiredSpells []string `json:"required_spells"` + LockedSlots map[string]LockedSlotInfo `json:"locked_slots"` + IncludeEquipped bool `json:"include_equipped"` + IncludeInventory bool `json:"include_inventory"` + MinArmor *int `json:"min_armor"` + MaxArmor *int `json:"max_armor"` + AllowedCritDamage []int `json:"allowed_crit_damage"` + MinDamageRating *int `json:"min_damage_rating"` + MaxDamageRating *int `json:"max_damage_rating"` + ScoringWeights *ScoringWeights `json:"scoring_weights"` + MaxResults int `json:"max_results"` + SearchTimeout int `json:"search_timeout"` +} + +// --- CompletedSuit (suitbuilder.py:446) --- + +type CompletedSuit struct { + Items map[string]*SuitItem + Score int + TotalArmor int + TotalRatings map[string]int + SetCounts map[int]int + FulfilledSpells []string + MissingSpells []string +} + +func fnvInt(s string) int { + h := fnv.New32a() + _, _ = h.Write([]byte(s)) + return int(h.Sum32()) +} + +// toDict mirrors CompletedSuit.to_dict(). The opaque id fields are derived +// deterministically (Python uses salted hash(); we use FNV) — never compared in +// validation. +func (c *CompletedSuit) toDict() map[string]any { + transferByChar := map[string][]string{} + totalItems := 0 + // Slots iterated in Python dict order; use a sorted-slot order for stable + // transfer instructions (instructions are display-only). + for _, it := range c.Items { + transferByChar[it.CharacterName] = append(transferByChar[it.CharacterName], it.Name) + totalItems++ + } + chars := make([]string, 0, len(transferByChar)) + for ch := range transferByChar { + chars = append(chars, ch) + } + sort.Strings(chars) + instructions := []string{} + step := 1 + for _, ch := range chars { + for _, name := range transferByChar[ch] { + instructions = append(instructions, fmt.Sprintf("%d. Transfer %s from %s to new character", step, name, ch)) + step++ + } + } + instructions = append(instructions, fmt.Sprintf("%d. Equip all transferred items on new character", step)) + + slotKeys := make([]string, 0, len(c.Items)) + for slot := range c.Items { + slotKeys = append(slotKeys, slot) + } + sort.Strings(slotKeys) + itemsOut := map[string]any{} + for _, slot := range slotKeys { + it := c.Items[slot] + var setIDOut any + if it.SetID != 0 { + setIDOut = it.SetID + } + itemsOut[slot] = map[string]any{ + "id": fnvInt(it.ID), + "name": it.Name, + "source_character": it.CharacterName, + "armor_level": it.ArmorLevel, + "ratings": it.Ratings, + "spells": it.SpellNames, + "set_id": setIDOut, + "set_name": getSetName(it.SetID), + } + } + + return map[string]any{ + "id": fnvInt(strings.Join(slotKeys, "|")), + "score": c.Score, + "items": itemsOut, + "stats": map[string]any{ + "total_armor": c.TotalArmor, + "total_crit_damage": c.TotalRatings["crit_damage_rating"], + "total_damage_rating": c.TotalRatings["damage_rating"], + "primary_set_count": 0, + "secondary_set_count": 0, + "spell_coverage": len(c.FulfilledSpells), + }, + "missing": c.MissingSpells, + "notes": []any{}, + "transfer_summary": map[string]any{ + "total_items": totalItems, + "from_characters": transferByChar, + }, + "instructions": instructions, + } +} + +// --- ItemPreFilter (suitbuilder.py:519) --- + +func removeSurpassedItems(items []*SuitItem) []*SuitItem { + out := make([]*SuitItem, 0, len(items)) + for _, it := range items { + surpassed := false + for _, cmp := range items { + if cmp == it { + continue + } + if isSurpassedBy(it, cmp) { + surpassed = true + break + } + } + if !surpassed { + out = append(out, it) + } + } + return out +} + +func isSurpassedBy(item, cmp *SuitItem) bool { + if item.Slot != cmp.Slot { + return false + } + if item.SetID != cmp.SetID { + return false + } + if !spellsSurpassOrEqual(cmp.SpellNames, item.SpellNames) { + return false + } + betterInSomething := false + for _, key := range []string{"crit_damage_rating", "damage_rating"} { + ir := item.Ratings[key] + cr := cmp.Ratings[key] + if cr > ir { + betterInSomething = true + } else if ir > cr { + return false + } + } + if item.ArmorLevel > 0 && cmp.ArmorLevel > 0 { + if cmp.ArmorLevel > item.ArmorLevel { + betterInSomething = true + } else if item.ArmorLevel > cmp.ArmorLevel { + return false + } + } + return betterInSomething +} + +func spellsSurpassOrEqual(spells1, spells2 []string) bool { + for _, s2 := range spells2 { + found := false + for _, s1 := range spells1 { + if s1 == s2 || spellSurpasses(s1, s2) { + found = true + break + } + } + if !found { + return false + } + } + return true +} + +func spellSurpasses(s1, s2 string) bool { + if strings.Contains(s1, "Legendary") && (strings.Contains(s2, "Epic") || strings.Contains(s2, "Major")) { + b1 := strings.ReplaceAll(s1, "Legendary ", "") + b2 := strings.ReplaceAll(strings.ReplaceAll(s2, "Epic ", ""), "Major ", "") + return b1 == b2 + } + if strings.Contains(s1, "Epic") && strings.Contains(s2, "Major") { + b1 := strings.ReplaceAll(s1, "Epic ", "") + b2 := strings.ReplaceAll(s2, "Major ", "") + return b1 == b2 + } + return false +} diff --git a/go-services/inventory-go/suit_solver.go b/go-services/inventory-go/suit_solver.go new file mode 100644 index 00000000..9023b000 --- /dev/null +++ b/go-services/inventory-go/suit_solver.go @@ -0,0 +1,870 @@ +package main + +import ( + "context" + "net/url" + "sort" + "strconv" + "strings" + "time" +) + +// Solver is the Go port of suitbuilder.py ConstraintSatisfactionSolver (the live +// /suitbuilder/search DFS). It streams events via emit; cancellation is checked +// through cancelled (the request context). + +type Solver struct { + s *Server + c SearchConstraints + spellIndex *SpellBitmapIndex + bestSuits []*CompletedSuit + evaluated int + weights ScoringWeights + + neededSpellBitmap uint64 + bestSuitItemCount int + highestArmorCount int + armorBucketsItems int + allowedCD map[int]bool // nil == no CD filter (default / all tiers) + + lockedSetCounts map[int]int + lockedSpells map[string]bool + effPrimary int + effSecondary int + + start time.Time + emit func(event string, data map[string]any) + cancelled func() bool + stopped bool +} + +func newSolver(s *Server, c SearchConstraints, emit func(string, map[string]any), cancelled func() bool) *Solver { + w := defaultScoringWeights() + if c.ScoringWeights != nil { + w = *c.ScoringWeights + } + if c.MaxResults == 0 { + c.MaxResults = 50 + } + sv := &Solver{ + s: s, c: c, spellIndex: newSpellBitmapIndex(), weights: w, + lockedSetCounts: map[int]int{}, lockedSpells: map[string]bool{}, + effPrimary: 5, effSecondary: 4, start: time.Now(), + emit: emit, cancelled: cancelled, + } + // Required spells register first, so they always get the low bits. + sv.neededSpellBitmap = sv.spellIndex.getBitmap(c.RequiredSpells) + sv.allowedCD = allowedCritSet(c.AllowedCritDamage) + return sv +} + +var armorSlotSet = map[string]bool{ + "Head": true, "Chest": true, "Upper Arms": true, "Lower Arms": true, + "Hands": true, "Abdomen": true, "Upper Legs": true, "Lower Legs": true, "Feet": true, +} +var jewelrySlotSet = map[string]bool{ + "Neck": true, "Left Ring": true, "Right Ring": true, + "Left Wrist": true, "Right Wrist": true, "Trinket": true, +} + +func (sv *Solver) elapsed() float64 { return time.Since(sv.start).Seconds() } + +// Search drives the 5-phase pipeline, emitting events as it goes. +func (sv *Solver) Search(ctx context.Context) { + sv.emit("phase", map[string]any{"phase": "loading", "message": "Loading items from database...", "phase_number": 1, "total_phases": 5}) + + items, err := sv.loadItems(ctx) + if err != nil { + sv.emit("error", map[string]any{"message": err.Error()}) + return + } + sv.emit("phase", map[string]any{"phase": "loaded", "message": "Loaded items", "items_count": len(items), "phase_number": 1, "total_phases": 5}) + sv.emit("log", map[string]any{"level": "info", "message": "Loaded items from characters", "timestamp": sv.elapsed()}) + if len(items) == 0 { + sv.emit("error", map[string]any{"message": "No items found for specified characters"}) + return + } + + sv.emit("phase", map[string]any{"phase": "buckets", "message": "Creating equipment buckets...", "phase_number": 2, "total_phases": 5}) + buckets := sv.createBuckets(items) + summary := map[string]any{} + for _, b := range buckets { + summary[b.Slot] = len(b.Items) + } + sv.emit("phase", map[string]any{"phase": "buckets_done", "message": "Created buckets", "bucket_count": len(buckets), "bucket_summary": summary, "phase_number": 2, "total_phases": 5}) + + sv.emit("phase", map[string]any{"phase": "reducing", "message": "Applying armor reduction rules...", "phase_number": 3, "total_phases": 5}) + buckets = sv.applyReductionOptions(buckets) + + sv.emit("phase", map[string]any{"phase": "sorting", "message": "Optimizing search order...", "phase_number": 4, "total_phases": 5}) + buckets = sv.sortBuckets(buckets) + + // Locked slots: drop those buckets, accumulate locked set/spell contributions. + if len(sv.c.LockedSlots) > 0 { + locked := map[string]bool{} + for slot := range sv.c.LockedSlots { + locked[slot] = true + } + kept := buckets[:0] + for _, b := range buckets { + if !locked[b.Slot] { + kept = append(kept, b) + } + } + buckets = kept + for _, info := range sv.c.LockedSlots { + if info.SetID != 0 { + sv.lockedSetCounts[info.SetID]++ + } + for _, sp := range info.Spells { + sv.lockedSpells[sp] = true + } + } + } + sv.effPrimary, sv.effSecondary = 5, 4 + if sv.c.PrimarySet != 0 { + sv.effPrimary = max0(5 - sv.lockedSetCounts[sv.c.PrimarySet]) + } + if sv.c.SecondarySet != 0 { + sv.effSecondary = max0(4 - sv.lockedSetCounts[sv.c.SecondarySet]) + } + + sv.emit("phase", map[string]any{"phase": "searching", "message": "Searching for optimal suits...", "total_buckets": len(buckets), "phase_number": 5, "total_phases": 5}) + sv.emit("log", map[string]any{"level": "info", "message": "Starting search", "timestamp": sv.elapsed()}) + + sv.recursiveSearch(buckets, 0, newSuitState()) + + sv.emit("complete", map[string]any{"suits_found": len(sv.bestSuits), "duration": round1(sv.elapsed())}) +} + +// loadItems mirrors suitbuilder.load_items: fetch via the in-process search with +// the exact same filter param sets, convert to SuitItem, register spell bitmaps, +// pre-filter, and sort into armor+jewelry+clothing order. +func (sv *Solver) loadItems(ctx context.Context) ([]*SuitItem, error) { + s := sv.s + primaryName, secondaryName := "", "" + if sv.c.PrimarySet != 0 { + primaryName = s.translateSetID(strconv.Itoa(sv.c.PrimarySet)) + } + if sv.c.SecondarySet != 0 { + secondaryName = s.translateSetID(strconv.Itoa(sv.c.SecondarySet)) + } + equipmentStatus := "" + if sv.c.IncludeEquipped && sv.c.IncludeInventory { + equipmentStatus = "" + } else if sv.c.IncludeEquipped { + equipmentStatus = "equipped" + } else if sv.c.IncludeInventory { + equipmentStatus = "unequipped" + } + + var apiItems []map[string]any + fetch := func(extra map[string]string) error { + q := url.Values{} + if len(sv.c.Characters) > 0 { + q.Set("characters", strings.Join(sv.c.Characters, ",")) + } else { + q.Set("include_all_characters", "true") + } + if equipmentStatus != "" { + q.Set("equipment_status", equipmentStatus) + } + q.Set("limit", "1000") + for k, v := range extra { + q.Set(k, v) + } + res, err := s.runSearch(ctx, q) + if err != nil { + return err + } + if items, ok := res["items"].([]map[string]any); ok { + apiItems = append(apiItems, items...) + } + return nil + } + + if primaryName != "" { + if err := fetch(map[string]string{"item_set": primaryName}); err != nil { + return nil, err + } + } + if secondaryName != "" { + if err := fetch(map[string]string{"item_set": secondaryName}); err != nil { + return nil, err + } + } + // Clothing: DR3 shirts/pants only. + _ = fetch(map[string]string{"shirt_only": "true", "min_damage_rating": "3"}) + _ = fetch(map[string]string{"pants_only": "true", "min_damage_rating": "3"}) + // Jewelry: one fetch per type via slot_names. + for _, slot := range []string{"Ring", "Bracelet", "Neck", "Trinket"} { + _ = fetch(map[string]string{"jewelry_only": "true", "slot_names": slot}) + } + + items := make([]*SuitItem, 0, len(apiItems)) + for _, api := range apiItems { + name := toStr(api["name"]) + char := toStr(api["character_name"]) + coverageVal := int(toInt64(api["coverage_mask"])) + slot := toStr(api["computed_slot_name"]) + if slot == "" { + slot = toStr(api["slot_name"]) + } + if slot == "" { + slot = "Unknown" + } + if int(toInt64(api["object_class"])) == 3 { + switch coverageVal { + case 104: + slot = "Shirt" + case 19, 22: + slot = "Pants" + } + } + rg := func(k string) int { + v := api[k] + if v == nil { + return 0 + } + return int(toInt64(v)) + } + var spellNames []string + if sn, ok := api["spell_names"].([]string); ok { + spellNames = sn + } + it := &SuitItem{ + ID: char + "_" + name, + Name: name, + CharacterName: char, + Slot: slot, + Coverage: coverageVal, + HasCoverage: coverageVal != 0, + SetID: convertSetNameToID(toStr(api["item_set"])), + ArmorLevel: int(toInt64(api["armor_level"])), + Ratings: map[string]int{ + "crit_damage_rating": rg("crit_damage_rating"), + "damage_rating": rg("damage_rating"), + "damage_resist_rating": rg("damage_resist_rating"), + "crit_damage_resist_rating": rg("crit_damage_resist_rating"), + "heal_boost_rating": rg("heal_boost_rating"), + "vitality_rating": rg("vitality_rating"), + }, + SpellNames: spellNames, + Material: toStr(api["material_name"]), + } + items = append(items, it) + } + + for _, it := range items { + if len(it.SpellNames) > 0 { + it.SpellBitmap = sv.spellIndex.getBitmap(it.SpellNames) + } + } + + // Drop armor whose CD tier is disallowed BEFORE domination, so a CD2 piece + // can't surpass-and-remove an allowed CD1 piece we'd then exclude. + items = filterArmorByCD(items, sv.allowedCD) + + filtered := removeSurpassedItems(items) + + jewelryFallback := map[string]bool{"Ring": true, "Bracelet": true, "Jewelry": true, "Necklace": true, "Amulet": true} + matches := func(slot string, set, fallback map[string]bool) bool { + if set[slot] { + return true + } + if strings.Contains(slot, ", ") { + for _, p := range strings.Split(slot, ", ") { + if set[strings.TrimSpace(p)] { + return true + } + } + } + if fallback != nil && fallback[slot] { + return true + } + return false + } + var armor, jewelry, clothing []*SuitItem + for _, it := range filtered { + if matches(it.Slot, armorSlotSet, nil) { + armor = append(armor, it) + } + if matches(it.Slot, jewelrySlotSet, jewelryFallback) { + jewelry = append(jewelry, it) + } + if matches(it.Slot, clothingSortSlots, nil) { + clothing = append(clothing, it) + } + } + sortBySpellThenName := func(list []*SuitItem) { + sort.SliceStable(list, func(i, j int) bool { + return descTuple( + cmpInt(len(list[i].SpellNames), len(list[j].SpellNames)), + cmpStr(list[i].CharacterName, list[j].CharacterName), + cmpStr(list[i].Name, list[j].Name), + ) + }) + } + sortBySpellThenName(armor) + sortBySpellThenName(jewelry) + sort.SliceStable(clothing, func(i, j int) bool { + return descTuple( + cmpInt(clothing[i].Ratings["damage_rating"], clothing[j].Ratings["damage_rating"]), + cmpStr(clothing[i].CharacterName, clothing[j].CharacterName), + cmpStr(clothing[i].Name, clothing[j].Name), + ) + }) + + out := make([]*SuitItem, 0, len(armor)+len(jewelry)+len(clothing)) + out = append(out, armor...) + out = append(out, jewelry...) + out = append(out, clothing...) + return out, nil +} + +var allSlots = []string{ + "Head", "Chest", "Upper Arms", "Lower Arms", "Hands", + "Abdomen", "Upper Legs", "Lower Legs", "Feet", + "Neck", "Left Ring", "Right Ring", "Left Wrist", "Right Wrist", "Trinket", + "Shirt", "Pants", +} + +func (sv *Solver) createBuckets(items []*SuitItem) []*ItemBucket { + slotItems := map[string][]*SuitItem{} + inSlots := map[string]bool{} + for _, slot := range allSlots { + slotItems[slot] = nil + inSlots[slot] = true + } + genericJewelry := map[string][]string{ + "Ring": {"Left Ring", "Right Ring"}, + "Bracelet": {"Left Wrist", "Right Wrist"}, + "Jewelry": {"Neck", "Left Ring", "Right Ring", "Left Wrist", "Right Wrist", "Trinket"}, + "Necklace": {"Neck"}, + "Amulet": {"Neck"}, + } + for _, it := range items { + if inSlots[it.Slot] { + slotItems[it.Slot] = append(slotItems[it.Slot], it) + } else if strings.Contains(it.Slot, ", ") { + for _, p := range strings.Split(it.Slot, ", ") { + p = strings.TrimSpace(p) + if inSlots[p] { + slotItems[p] = append(slotItems[p], it.clone(p, it.Name, it.Coverage, it.HasCoverage)) + } + } + } else if targets, ok := genericJewelry[it.Slot]; ok { + for _, t := range targets { + slotItems[t] = append(slotItems[t], it.clone(t, it.Name, it.Coverage, it.HasCoverage)) + } + } else { + lower := strings.ToLower(it.Slot) + for _, known := range allSlots { + if strings.Contains(lower, strings.ToLower(known)) { + slotItems[known] = append(slotItems[known], it.clone(known, it.Name, it.Coverage, it.HasCoverage)) + } + } + } + } + + buckets := make([]*ItemBucket, 0, len(allSlots)) + for _, slot := range allSlots { + b := &ItemBucket{Slot: slot, Items: slotItems[slot], IsArmor: armorSlotSet[slot]} + b.sortItems() + buckets = append(buckets, b) + } + // armor first, then item count ascending (overridden by sortBuckets, but the + // stable item order set here feeds the later stable re-sorts). + sort.SliceStable(buckets, func(i, j int) bool { + ai, aj := boolToInt(!buckets[i].IsArmor), boolToInt(!buckets[j].IsArmor) + if ai != aj { + return ai < aj + } + return len(buckets[i].Items) < len(buckets[j].Items) + }) + sv.armorBucketsItems = 0 + for _, b := range buckets { + if b.IsArmor && len(b.Items) > 0 { + sv.armorBucketsItems++ + } + } + return buckets +} + +func (sv *Solver) applyReductionOptions(buckets []*ItemBucket) []*ItemBucket { + var newBuckets []*ItemBucket + findBucket := func(slot string) *ItemBucket { + for _, b := range newBuckets { + if b.Slot == slot { + return b + } + } + return nil + } + for _, bucket := range buckets { + if !bucket.IsArmor { + newBuckets = append(newBuckets, bucket) + continue + } + var original, reducible []*SuitItem + for _, it := range bucket.Items { + if it.HasCoverage && it.Material != "" && len(coverageReductionOptions(it.Coverage)) > 0 { + reducible = append(reducible, it) + } else { + original = append(original, it) + } + } + if len(original) > 0 || len(reducible) == 0 { + nb := &ItemBucket{Slot: bucket.Slot, Items: original, IsArmor: bucket.IsArmor} + nb.sortItems() + newBuckets = append(newBuckets, nb) + } + for _, it := range reducible { + for _, rc := range coverageReductionOptions(it.Coverage) { + reducedSlot := coverageToSlotName(rc) + if reducedSlot == "" { + continue + } + reduced := it.clone(reducedSlot, it.Name+" (tailored to "+reducedSlot+")", rc, true) + target := findBucket(reducedSlot) + if target == nil { + target = &ItemBucket{Slot: reducedSlot, IsArmor: true} + newBuckets = append(newBuckets, target) + } + target.Items = append(target.Items, reduced) + } + } + } + for _, b := range newBuckets { + b.sortItems() + } + return newBuckets +} + +var coreArmorPriority = []string{"Chest", "Head", "Hands", "Feet", "Upper Arms", "Lower Arms", "Abdomen", "Upper Legs", "Lower Legs"} +var jewelryPriority = []string{"Neck", "Left Ring", "Right Ring", "Left Wrist", "Right Wrist", "Trinket"} +var clothingPriority = []string{"Shirt", "Pants"} + +func (sv *Solver) sortBuckets(buckets []*ItemBucket) []*ItemBucket { + for _, bucket := range buckets { + items := bucket.Items + sort.SliceStable(items, func(i, j int) bool { + pi, pj := sv.setPriority(items[i].SetID), sv.setPriority(items[j].SetID) + if pi != pj { + return pi < pj + } + if c := cmpInt(items[i].Ratings["crit_damage_rating"], items[j].Ratings["crit_damage_rating"]); c != 0 { + return c > 0 + } + if c := cmpInt(items[i].Ratings["damage_rating"], items[j].Ratings["damage_rating"]); c != 0 { + return c > 0 + } + return items[i].ArmorLevel > items[j].ArmorLevel + }) + } + sort.SliceStable(buckets, func(i, j int) bool { + gi, ii := bucketPriority(buckets[i].Slot) + gj, ij := bucketPriority(buckets[j].Slot) + if gi != gj { + return gi < gj + } + if ii != ij { + return ii < ij + } + return len(buckets[i].Items) < len(buckets[j].Items) + }) + return buckets +} + +func (sv *Solver) setPriority(setID int) int { + if setID != 0 && setID == sv.c.PrimarySet { + return 0 + } + if setID != 0 && setID == sv.c.SecondarySet { + return 1 + } + return 2 +} + +func bucketPriority(slot string) (int, int) { + for i, s := range coreArmorPriority { + if s == slot { + return 0, i + } + } + for i, s := range jewelryPriority { + if s == slot { + return 1, i + } + } + for i, s := range clothingPriority { + if s == slot { + return 2, i + } + } + return 3, 0 +} + +func (sv *Solver) recursiveSearch(buckets []*ItemBucket, idx int, state *SuitState) { + if sv.stopped { + return + } + if sv.cancelled != nil && sv.cancelled() { + sv.stopped = true + return + } + + if sv.highestArmorCount > 0 { + currentCount := len(state.Items) + remaining := sv.armorBucketsItems - minInt(idx, sv.armorBucketsItems) + minRequired := sv.highestArmorCount - remaining + if currentCount+1 < minRequired { + return + } + } + remainingBuckets := len(buckets) - idx + maxPossible := len(state.Items) + remainingBuckets + if sv.bestSuitItemCount > 0 && maxPossible < sv.bestSuitItemCount { + return + } + + if idx >= len(buckets) { + suit := sv.finalizeSuit(state) + if suit != nil && sv.isBetterThanExisting(suit) { + sv.bestSuits = append(sv.bestSuits, suit) + if len(suit.Items) > sv.bestSuitItemCount { + sv.bestSuitItemCount = len(suit.Items) + } + armorPieces := 0 + for slot := range suit.Items { + if armorSlotSet[slot] { + armorPieces++ + } + } + if armorPieces > sv.highestArmorCount { + sv.highestArmorCount = armorPieces + } + sort.SliceStable(sv.bestSuits, func(i, j int) bool { return sv.bestSuits[i].Score > sv.bestSuits[j].Score }) + if len(sv.bestSuits) > sv.c.MaxResults { + sv.bestSuits = sv.bestSuits[:sv.c.MaxResults] + } + sv.emit("suit", sv.suitData(suit)) + sv.emit("log", map[string]any{"level": "success", "message": "Found suit", "timestamp": sv.elapsed()}) + } + return + } + + sv.evaluated++ + if sv.evaluated%100 == 0 { + if sv.cancelled != nil && sv.cancelled() { + sv.stopped = true + return + } + bestScore := 0 + if len(sv.bestSuits) > 0 { + bestScore = sv.bestSuits[0].Score + } + var curBucket any + if idx < len(buckets) { + curBucket = buckets[idx].Slot + } + el := sv.elapsed() + rate := 0.0 + if el > 0 { + rate = round1(float64(sv.evaluated) / el) + } + sv.emit("progress", map[string]any{ + "evaluated": sv.evaluated, "found": len(sv.bestSuits), "current_depth": idx, + "total_buckets": len(buckets), "current_items": len(state.Items), "elapsed": el, + "rate": rate, "current_bucket": curBucket, "best_score": bestScore, + }) + if sv.evaluated%500 == 0 { + sv.emit("log", map[string]any{"level": "info", "message": "Evaluating combinations", "timestamp": el}) + } + } + + bucket := buckets[idx] + accepted := 0 + for _, it := range bucket.Items { + if sv.canAddItem(it, state) { + accepted++ + state.push(it) + sv.recursiveSearch(buckets, idx+1, state) + state.pop(it.Slot) + if sv.stopped { + return + } + } + } + if accepted == 0 { + sv.recursiveSearch(buckets, idx+1, state) + } +} + +func (sv *Solver) canAddItem(it *SuitItem, state *SuitState) bool { + if state.Occupied[it.Slot] { + return false + } + for _, ex := range state.Items { + if ex.ID == it.ID { + return false + } + } + if it.SetID != 0 { + current := state.SetCounts[it.SetID] + if it.SetID == sv.c.PrimarySet { + if current >= sv.effPrimary { + return false + } + } else if it.SetID == sv.c.SecondarySet { + if current >= sv.effSecondary { + return false + } + } else { + if jewelrySlotSet[it.Slot] { + if !sv.jewelryContributesRequiredSpell(it, state) { + return false + } + } else { + return false + } + } + } else { + if it.Slot == "Shirt" || it.Slot == "Pants" { + // clothing allowed without set id + } else if jewelrySlotSet[it.Slot] { + if !sv.jewelryContributesRequiredSpell(it, state) { + return false + } + } else { + return false + } + } + if len(sv.c.RequiredSpells) > 0 && len(it.SpellNames) > 0 { + if !sv.canGetBeneficialSpellFrom(it, state) { + return false + } + } + return true +} + +func (sv *Solver) canGetBeneficialSpellFrom(it *SuitItem, state *SuitState) bool { + if len(it.SpellNames) == 0 { + return true + } + if len(sv.c.RequiredSpells) == 0 { + return true + } + newBeneficial := it.SpellBitmap & sv.neededSpellBitmap & ^state.SpellBitmap + return newBeneficial != 0 +} + +func (sv *Solver) jewelryContributesRequiredSpell(it *SuitItem, state *SuitState) bool { + if len(sv.c.RequiredSpells) == 0 { + return false + } + if len(it.SpellNames) == 0 { + return false + } + for _, sp := range it.SpellNames { + bit := sv.spellIndex.getBitmap([]string{sp}) + if bit&sv.neededSpellBitmap != 0 && state.SpellBitmap&bit == 0 { + return true + } + } + return false +} + +func (sv *Solver) finalizeSuit(state *SuitState) *CompletedSuit { + if len(state.Items) == 0 { + return nil + } + score := sv.calculateScore(state) + var fulfilled, missing []string + if len(sv.c.RequiredSpells) > 0 { + fulfilled = sv.spellIndex.getSpellNames(state.SpellBitmap & sv.neededSpellBitmap) + missing = sv.spellIndex.getSpellNames(sv.neededSpellBitmap & ^state.SpellBitmap) + if len(sv.lockedSpells) > 0 { + for sp := range sv.lockedSpells { + missing = removeString(missing, sp) + if !containsString(fulfilled, sp) { + fulfilled = append(fulfilled, sp) + } + } + } + } + items := make(map[string]*SuitItem, len(state.Items)) + for k, v := range state.Items { + items[k] = v + } + ratings := map[string]int{} + for k, v := range state.TotalRatings { + ratings[k] = v + } + setCounts := map[int]int{} + for k, v := range state.SetCounts { + setCounts[k] = v + } + return &CompletedSuit{ + Items: items, Score: score, TotalArmor: state.TotalArmor, + TotalRatings: ratings, SetCounts: setCounts, + FulfilledSpells: fulfilled, MissingSpells: missing, + } +} + +func (sv *Solver) calculateScore(state *SuitState) int { + score := 0 + w := sv.weights + foundPrimary, foundSecondary := 0, 0 + if sv.c.PrimarySet != 0 { + foundPrimary = state.SetCounts[sv.c.PrimarySet] + } + if sv.c.SecondarySet != 0 { + foundSecondary = state.SetCounts[sv.c.SecondarySet] + } + if foundPrimary >= sv.effPrimary { + score += w.ArmorSetComplete + if foundPrimary > sv.effPrimary { + score -= (foundPrimary - sv.effPrimary) * 500 + } + } else if sv.c.PrimarySet != 0 && foundPrimary > 0 { + score += (sv.effPrimary - foundPrimary) * w.MissingSetPenalty + } + if foundSecondary >= sv.effSecondary { + score += w.ArmorSetComplete + if foundSecondary > sv.effSecondary { + score -= (foundSecondary - sv.effSecondary) * 500 + } + } else if sv.c.SecondarySet != 0 && foundSecondary > 0 { + score += (sv.effSecondary - foundSecondary) * w.MissingSetPenalty + } + for _, it := range state.Items { + switch it.Ratings["crit_damage_rating"] { + case 1: + score += w.CritDamage1 + case 2: + score += w.CritDamage2 + } + } + for _, it := range state.Items { + if it.Slot == "Shirt" || it.Slot == "Pants" { + switch it.Ratings["damage_rating"] { + case 1: + score += w.DamageRating1 + case 2: + score += w.DamageRating2 + case 3: + score += w.DamageRating3 + } + } + } + if len(sv.c.RequiredSpells) > 0 { + score += popcount(state.SpellBitmap&sv.neededSpellBitmap) * 100 + } + score += len(state.Items) * 5 + // Python uses floor division (//); total_armor can be negative because + // non-armor items carry armor_level = -1. Go's / truncates toward zero, so a + // slightly-negative total would be +1 too high. + score += floorDiv(state.TotalArmor, 100) + if score < 0 { + return 0 + } + return score +} + +func (sv *Solver) isBetterThanExisting(suit *CompletedSuit) bool { + if len(sv.bestSuits) < sv.c.MaxResults { + return true + } + lowest := sv.bestSuits[len(sv.bestSuits)-1] + if len(suit.Items) > len(lowest.Items) { + return true + } + return suit.Score > lowest.Score +} + +// suitData builds the streamed suit payload (CompletedSuit.to_dict plus the +// constraint-derived stats overrides from recursive_search). +func (sv *Solver) suitData(suit *CompletedSuit) map[string]any { + d := suit.toDict() + stats := d["stats"].(map[string]any) + primaryCount, secondaryCount := 0, 0 + if sv.c.PrimarySet != 0 { + primaryCount = suit.SetCounts[sv.c.PrimarySet] + sv.lockedSetCounts[sv.c.PrimarySet] + } + if sv.c.SecondarySet != 0 { + secondaryCount = suit.SetCounts[sv.c.SecondarySet] + sv.lockedSetCounts[sv.c.SecondarySet] + } + var primaryName, secondaryName any + if sv.c.PrimarySet != 0 { + primaryName = sv.s.translateSetID(strconv.Itoa(sv.c.PrimarySet)) + } + if sv.c.SecondarySet != 0 { + secondaryName = sv.s.translateSetID(strconv.Itoa(sv.c.SecondarySet)) + } + stats["primary_set_count"] = primaryCount + stats["secondary_set_count"] = secondaryCount + stats["primary_set"] = primaryName + stats["secondary_set"] = secondaryName + stats["locked_slots"] = len(sv.c.LockedSlots) + stats["primary_locked"] = sv.lockedSetCounts[sv.c.PrimarySet] + stats["secondary_locked"] = sv.lockedSetCounts[sv.c.SecondarySet] + return d +} + +// --- small helpers --- + +func max0(v int) int { + if v < 0 { + return 0 + } + return v +} +func minInt(a, b int) int { + if a < b { + return a + } + return b +} + +// floorDiv matches Python's // (floor toward -inf), unlike Go's / (toward zero). +func floorDiv(a, b int) int { + q := a / b + if a%b != 0 && (a < 0) != (b < 0) { + q-- + } + return q +} +func boolToInt(b bool) int { + if b { + return 1 + } + return 0 +} +func popcount(v uint64) int { + c := 0 + for v != 0 { + v &= v - 1 + c++ + } + return c +} +func round1(v float64) float64 { + return float64(int64(v*10+0.5)) / 10 +} +func containsString(list []string, s string) bool { + for _, x := range list { + if x == s { + return true + } + } + return false +} +func removeString(list []string, s string) []string { + for i, x := range list { + if x == s { + return append(list[:i], list[i+1:]...) + } + } + return list +} diff --git a/go-services/nginx/go-location.conf b/go-services/nginx/go-location.conf new file mode 100644 index 00000000..eec52819 --- /dev/null +++ b/go-services/nginx/go-location.conf @@ -0,0 +1,41 @@ +# Parallel-run nginx wiring for the Go tracker (dereth-tracker-go, 127.0.0.1:8770). +# +# Deploying needs root (the agent cannot sudo). Apply on the host: +# +# 1) Add the upstream to the http{} block of /etc/nginx/nginx.conf, next to the +# existing `tracker` and `grafana` upstreams (around line 55): +# +# upstream tracker_go { server 127.0.0.1:8770; } +# +# 2) Insert the `location /go/` block below into the server{} block of +# /etc/nginx/sites-enabled/overlord (anywhere in server{}; nginx matches the +# longer /go/ prefix before /, so order doesn't matter). Mirror it into the +# repo copy nginx/overlord.conf too — but note the live file has DRIFTED from +# the repo copy, so reconcile by hand rather than cp-overwriting. +# +# 3) sudo nginx -t && sudo nginx -s reload +# +# After reload: +# https://overlord.snakedesert.se/go/health -> 200 (public) +# https://overlord.snakedesert.se/go/api-version -> 200 (logged-in) / 401 (no cookie) +# https://overlord.snakedesert.se/go/live -> matches /live (same login cookie) +# +# The Go service is auth-gated identically to Python (session cookie + internal +# trust), and X-Forwarded-For below is REQUIRED — without it the Go service would +# treat all internet traffic as internal-trust and skip auth (security invariant). + +location /go/ { + proxy_pass http://tracker_go/; # trailing slash strips the /go/ prefix + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; # REQUIRED — security invariant + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + proxy_cache_bypass $http_upgrade; + # Go will serve long-lived browser WebSockets in a later phase; match the + # /websocket/ and / blocks so idle sockets aren't cut at nginx's default 60s. + proxy_read_timeout 1d; + proxy_send_timeout 1d; +} diff --git a/go-services/tracker-go/Dockerfile b/go-services/tracker-go/Dockerfile new file mode 100644 index 00000000..a9032a63 --- /dev/null +++ b/go-services/tracker-go/Dockerfile @@ -0,0 +1,21 @@ +# Multi-stage build: compile a static Go binary, ship it on distroless. +# No host Go toolchain required — everything happens inside the build stage. +FROM golang:1.25-bookworm AS build +WORKDIR /src + +# No local Go toolchain is available to maintain go.sum, so resolve and lock +# dependencies inside the build (network is available here). `go mod tidy` +# reads the imports from the source and writes go.mod/go.sum, then we build. +COPY . . +RUN go mod tidy +RUN go test ./... +ARG BUILD_VERSION=dev +RUN CGO_ENABLED=0 GOOS=linux go build \ + -trimpath \ + -ldflags "-s -w -X main.buildVersion=${BUILD_VERSION}" \ + -o /out/tracker-go . + +FROM gcr.io/distroless/static-debian12:nonroot +COPY --from=build /out/tracker-go /tracker-go +EXPOSE 8770 +ENTRYPOINT ["/tracker-go"] diff --git a/go-services/tracker-go/aclog.go b/go-services/tracker-go/aclog.go new file mode 100644 index 00000000..608d86a7 --- /dev/null +++ b/go-services/tracker-go/aclog.go @@ -0,0 +1,145 @@ +package main + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "log/slog" + "net/http" + "strconv" + "strings" + "sync" + "time" + + "github.com/jackc/pgx/v5/pgxpool" +) + +// trimFloat formats a vitae value without a trailing ".0" for whole numbers. +func trimFloat(f float64) string { + if f == float64(int64(f)) { + return strconv.FormatInt(int64(f), 10) + } + return strconv.FormatFloat(f, 'f', -1, 64) +} + +// aclogPoster posts death + idle alerts to the #aclog Discord webhook, porting +// main.py's _send_discord_aclog / death detection / _idle_detection_loop. nil +// when DISCORD_ACLOG_WEBHOOK is unset (or in shadow mode). +type aclogPoster struct { + webhook string + client *http.Client + log *slog.Logger + + mu sync.Mutex + deathAlerted map[string]time.Time // char -> last death alert (max 1 / 5min) + idleSince map[string]time.Time // char -> first detected idle + idleAlerted map[string]bool // char -> already alerted this idle period +} + +func newACLogPoster(webhook string, log *slog.Logger) *aclogPoster { + return &aclogPoster{ + webhook: webhook, + client: &http.Client{Timeout: 5 * time.Second}, + log: log, + deathAlerted: map[string]time.Time{}, + idleSince: map[string]time.Time{}, + idleAlerted: map[string]bool{}, + } +} + +func (a *aclogPoster) post(message string) { + if a == nil || a.webhook == "" { + return + } + body, _ := json.Marshal(map[string]any{"content": message}) + resp, err := a.client.Post(a.webhook, "application/json", bytes.NewReader(body)) + if err != nil { + a.log.Debug("discord webhook failed", "err", err) + return + } + drain(resp) +} + +// maybeDeath fires a death alert when vitae crosses 0 -> >0, capped at 1 per +// 5 minutes per character (main.py:3419). +func (a *aclogPoster) maybeDeath(name string, vitae float64) { + if a == nil || a.webhook == "" { + return + } + a.mu.Lock() + last, ok := a.deathAlerted[name] + if ok && time.Since(last) <= 5*time.Minute { + a.mu.Unlock() + return + } + a.deathAlerted[name] = time.Now() + a.mu.Unlock() + go a.post(fmt.Sprintf("☠️ **%s** died! (vitae: %s%%)", name, trimFloat(vitae))) +} + +// runIdleLoop polls online players every 60s and alerts on idle (main.py:2694). +func (a *aclogPoster) runIdleLoop(ctx context.Context, pool *pgxpool.Pool) { + if a == nil || a.webhook == "" { + return + } + select { + case <-time.After(30 * time.Second): // let telemetry arrive first + case <-ctx.Done(): + return + } + t := time.NewTicker(60 * time.Second) + defer t.Stop() + for { + a.checkIdleOnce(ctx, pool) + select { + case <-t.C: + case <-ctx.Done(): + return + } + } +} + +func (a *aclogPoster) checkIdleOnce(ctx context.Context, pool *pgxpool.Pool) { + qctx, cancel := context.WithTimeout(ctx, 10*time.Second) + defer cancel() + rows, err := pool.Query(qctx, ` + SELECT DISTINCT ON (character_name) character_name, COALESCE(vt_state,''), COALESCE(kills_per_hour, 0) + FROM telemetry_events + WHERE COALESCE(received_at, timestamp) > now() - interval '30 seconds' + ORDER BY character_name, timestamp DESC`) + if err != nil { + a.log.Debug("idle query failed", "err", err) + return + } + defer rows.Close() + now := time.Now() + a.mu.Lock() + defer a.mu.Unlock() + for rows.Next() { + var name, vtState string + var kph float64 + if rows.Scan(&name, &vtState, &kph) != nil { + continue + } + s := strings.ToLower(vtState) + kphi := int(kph) + isIdle := s == "default" || s == "idle" || s == "" || ((s == "combat" || s == "hunt") && kphi == 0) + if isIdle { + if _, seen := a.idleSince[name]; !seen { + a.idleSince[name] = now + } else if !a.idleAlerted[name] && now.Sub(a.idleSince[name]) >= 5*time.Minute { + a.idleAlerted[name] = true + idleMins := int(now.Sub(a.idleSince[name]).Minutes()) + stateText := vtState + if stateText == "" { + stateText = "idle" + } + go a.post(fmt.Sprintf("⚠️ **%s** appears idle for %dmin (state: %s, KPH: %d)", name, idleMins, stateText, kphi)) + } + } else { + delete(a.idleAlerted, name) + delete(a.idleSince, name) + } + } +} diff --git a/go-services/tracker-go/auth.go b/go-services/tracker-go/auth.go new file mode 100644 index 00000000..78c483ad --- /dev/null +++ b/go-services/tracker-go/auth.go @@ -0,0 +1,236 @@ +package main + +import ( + "bytes" + "compress/zlib" + "context" + "crypto/hmac" + "crypto/sha1" + "encoding/base64" + "encoding/json" + "io" + "net" + "net/http" + "strings" + "time" +) + +type userCtxKey struct{} + +func withUser(ctx context.Context, u *sessionUser) context.Context { + return context.WithValue(ctx, userCtxKey{}, u) +} + +// currentUser returns the authenticated user for the request, or nil (e.g. +// internal-trust loopback requests carry no user identity). +func currentUser(r *http.Request) *sessionUser { + u, _ := r.Context().Value(userCtxKey{}).(*sessionUser) + return u +} + +// requireAdmin writes 403 and returns false unless the request is an admin +// (main.py _require_admin). +func requireAdmin(w http.ResponseWriter, r *http.Request) bool { + if u := currentUser(r); u != nil && u.IsAdmin { + return true + } + writeJSON(w, http.StatusForbidden, map[string]any{"detail": "Admin access required"}) + return false +} + +// Session-cookie verification compatible with the Python service's +// itsdangerous URLSafeTimedSerializer(SECRET_KEY) (itsdangerous 2.2): +// - HMAC-SHA1 signature +// - django-concat key derivation: sha1(salt + b"signer" + secret_key) +// - salt "itsdangerous", separator ".", Unix-epoch timestamp +// - payload = urlsafe-base64(no pad) of compact JSON {"u":username,"a":is_admin}, +// optionally zlib-compressed with a leading "." marker +// Reusing the same SECRET_KEY means a login on the Python service authenticates +// on the Go service during the parallel run. + +const sessionMaxAge = 30 * 24 * 3600 // SESSION_MAX_AGE seconds (30 days) + +type sessionUser struct { + Username string + IsAdmin bool +} + +func deriveSignerKey(secretKey string) []byte { + h := sha1.New() + h.Write([]byte("itsdangerous")) // salt + h.Write([]byte("signer")) + h.Write([]byte(secretKey)) + return h.Sum(nil) +} + +// verifySessionCookie returns the user encoded in a valid, unexpired token, or +// nil. Constant-time signature comparison; never partially trusts a bad token. +func verifySessionCookie(secretKey, token string) *sessionUser { + if secretKey == "" || token == "" { + return nil + } + // signature is everything after the final separator. + i := strings.LastIndexByte(token, '.') + if i <= 0 { + return nil + } + signed := token[:i] // payload + "." + timestamp + sig, err := base64.RawURLEncoding.DecodeString(token[i+1:]) + if err != nil { + return nil + } + mac := hmac.New(sha1.New, deriveSignerKey(secretKey)) + mac.Write([]byte(signed)) + if !hmac.Equal(sig, mac.Sum(nil)) { + return nil + } + + // timestamp is after the second-to-last separator; payload precedes it + // (the payload may itself start with "." when zlib-compressed). + j := strings.LastIndexByte(signed, '.') + if j < 0 { + return nil + } + tsBytes, err := base64.RawURLEncoding.DecodeString(signed[j+1:]) + if err != nil { + return nil + } + var ts int64 + for _, b := range tsBytes { + ts = ts<<8 | int64(b) + } + if time.Now().Unix()-ts > sessionMaxAge { + return nil // expired + } + + payload, err := decodeItsdangerousPayload(signed[:j]) + if err != nil { + return nil + } + var data struct { + U string `json:"u"` + A bool `json:"a"` + } + if json.Unmarshal(payload, &data) != nil { + return nil + } + return &sessionUser{Username: data.U, IsAdmin: data.A} +} + +// issueSessionCookie produces an itsdangerous URLSafeTimedSerializer token +// compatible with the Python service (so Go-issued cookies verify on Python and +// vice-versa). Inverse of verifySessionCookie. +func issueSessionCookie(secretKey string, u sessionUser) string { + payload, _ := json.Marshal(struct { + U string `json:"u"` + A bool `json:"a"` + }{u.Username, u.IsAdmin}) + payloadPart := encodeItsdangerousPayload(payload) + tsPart := base64.RawURLEncoding.EncodeToString(int64ToBytes(time.Now().Unix())) + signed := payloadPart + "." + tsPart + mac := hmac.New(sha1.New, deriveSignerKey(secretKey)) + mac.Write([]byte(signed)) + return signed + "." + base64.RawURLEncoding.EncodeToString(mac.Sum(nil)) +} + +// encodeItsdangerousPayload mirrors URLSafeSerializerBase.dump_payload: zlib- +// compress only when it actually saves more than one byte (it won't for our tiny +// payload), then urlsafe-base64 (no pad), with a "." marker if compressed. +func encodeItsdangerousPayload(jsonBytes []byte) string { + var buf bytes.Buffer + zw := zlib.NewWriter(&buf) + _, _ = zw.Write(jsonBytes) + _ = zw.Close() + if compressed := buf.Bytes(); len(compressed) < len(jsonBytes)-1 { + return "." + base64.RawURLEncoding.EncodeToString(compressed) + } + return base64.RawURLEncoding.EncodeToString(jsonBytes) +} + +// int64ToBytes encodes a non-negative int as minimal big-endian bytes, matching +// itsdangerous int_to_bytes (verifySessionCookie reads it back the same way). +func int64ToBytes(n int64) []byte { + if n == 0 { + return []byte{0} + } + var b []byte + for n > 0 { + b = append([]byte{byte(n & 0xff)}, b...) + n >>= 8 + } + return b +} + +func decodeItsdangerousPayload(p string) ([]byte, error) { + compressed := strings.HasPrefix(p, ".") + if compressed { + p = p[1:] + } + raw, err := base64.RawURLEncoding.DecodeString(p) + if err != nil { + return nil, err + } + if !compressed { + return raw, nil + } + zr, err := zlib.NewReader(bytes.NewReader(raw)) + if err != nil { + return nil, err + } + defer zr.Close() + return io.ReadAll(zr) +} + +// authMiddleware replicates main.py's AuthMiddleware: public paths pass through; +// private-source + no X-Forwarded-For is internal-trust (skip auth); otherwise a +// valid session cookie is required. +func (s *Server) authMiddleware(next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if isPublicPath(r.URL.Path) { + next.ServeHTTP(w, r) + return + } + // Internal trust: only when the peer is private AND nginx did not add + // X-Forwarded-For (nginx sets XFF on all proxied internet traffic). + if r.Header.Get("X-Forwarded-For") == "" && isPrivateAddr(clientIP(r)) { + next.ServeHTTP(w, r) + return + } + if c, err := r.Cookie("session"); err == nil { + if u := verifySessionCookie(s.secretKey, c.Value); u != nil { + next.ServeHTTP(w, r.WithContext(withUser(r.Context(), u))) + return + } + } + if strings.Contains(r.Header.Get("Accept"), "text/html") { + http.Redirect(w, r, "/login", http.StatusFound) + return + } + writeJSON(w, http.StatusUnauthorized, map[string]any{"detail": "Not authenticated"}) + }) +} + +func isPublicPath(p string) bool { + switch p { + case "/login", "/logout", "/login.html", "/login-style.css", "/health": + return true + } + // WS endpoints authenticate inside their own handlers. + return strings.HasPrefix(p, "/icons/") || strings.HasPrefix(p, "/ws/") +} + +func clientIP(r *http.Request) string { + host, _, err := net.SplitHostPort(r.RemoteAddr) + if err != nil { + return r.RemoteAddr + } + return host +} + +func isPrivateAddr(ip string) bool { + a := net.ParseIP(ip) + if a == nil { + return false + } + return a.IsLoopback() || a.IsPrivate() +} diff --git a/go-services/tracker-go/charstats.go b/go-services/tracker-go/charstats.go new file mode 100644 index 00000000..65e10c59 --- /dev/null +++ b/go-services/tracker-go/charstats.go @@ -0,0 +1,114 @@ +package main + +import ( + "net/http" + "sort" +) + +// GET /character-stats/{name} — latest full stats. Phase 1 reads the DB +// (character_stats is authoritative); the live_character_stats overlay is an +// ingest-only freshness layer we don't have yet. (main.py:4137) +func (s *Server) handleCharacterStats(w http.ResponseWriter, r *http.Request) { + name := r.PathValue("name") + // Live overlay first (ingest mode), like Python's live_character_stats check. + if s.ingestor != nil { + if v, ok := s.ingestor.getCharacterStats(name); ok { + writeJSON(w, http.StatusOK, v) + return + } + } + ctx, cancel := reqCtx(r) + defer cancel() + + row, err := queryRowAsMap(ctx, s.pool, `SELECT * FROM character_stats WHERE character_name = $1`, name) + if err != nil { + s.dbErr(w, "character-stats", err) + return + } + if row == nil { + writeJSON(w, http.StatusNotFound, map[string]any{"error": "No stats available for this character"}) + return + } + // Merge stats_data JSONB up to the top level, matching the frontend contract. + sd := asJSONMap(row["stats_data"]) + delete(row, "stats_data") + formatTimes([]map[string]any{row}, "timestamp") + for k, v := range sd { + row[k] = v + } + writeJSON(w, http.StatusOK, row) +} + +// GET /combat-stats/{character_name} — lifetime combat blob. Phase 1: DB only, +// so session is always null. (main.py:1819) +func (s *Server) handleCombatStatsOne(w http.ResponseWriter, r *http.Request) { + cn := r.PathValue("character_name") + if s.ingestor != nil { + if live, ok := s.ingestor.getCombatStats(cn); ok { + writeJSON(w, http.StatusOK, map[string]any{ + "character_name": cn, "session": live["session"], "lifetime": live["lifetime"], + }) + return + } + } + ctx, cancel := reqCtx(r) + defer cancel() + + row, err := queryRowAsMap(ctx, s.pool, `SELECT stats_data FROM combat_stats WHERE character_name = $1`, cn) + if err != nil { + s.dbErr(w, "combat-stats/one", err) + return + } + if row == nil { + writeJSON(w, http.StatusOK, map[string]any{"character_name": cn, "session": nil, "lifetime": nil}) + return + } + writeJSON(w, http.StatusOK, map[string]any{ + "character_name": cn, + "session": nil, + "lifetime": decodeJSONValue(row["stats_data"]), + }) +} + +// GET /combat-stats — all characters' lifetime combat blobs. Phase 1: DB only. (main.py:1850) +func (s *Server) handleCombatStatsAll(w http.ResponseWriter, r *http.Request) { + ctx, cancel := reqCtx(r) + defer cancel() + + results := make([]map[string]any, 0) + seen := map[string]bool{} + if s.ingestor != nil { // live overlay first, like Python + for char, live := range s.ingestor.allCombatStats() { + seen[char] = true + results = append(results, map[string]any{ + "character_name": char, "session": live["session"], "lifetime": live["lifetime"], + }) + } + } + rows, err := queryRowsAsMaps(ctx, s.pool, `SELECT character_name, stats_data FROM combat_stats`) + if err != nil { + s.dbErr(w, "combat-stats/all", err) + return + } + for _, row := range rows { + if seen[toStr(row["character_name"])] { + continue + } + results = append(results, map[string]any{ + "character_name": row["character_name"], + "session": nil, + "lifetime": decodeJSONValue(row["stats_data"]), + }) + } + sort.Slice(results, func(i, j int) bool { + return toStr(results[i]["character_name"]) < toStr(results[j]["character_name"]) + }) + writeJSON(w, http.StatusOK, map[string]any{"stats": results}) +} + +func toStr(v any) string { + if s, ok := v.(string); ok { + return s + } + return "" +} diff --git a/go-services/tracker-go/combat.go b/go-services/tracker-go/combat.go new file mode 100644 index 00000000..243b173f --- /dev/null +++ b/go-services/tracker-go/combat.go @@ -0,0 +1,242 @@ +package main + +import ( + "context" + "encoding/json" + "io" + "os" + "time" +) + +// runCombatMergeCLI folds a JSON array of cumulative session snapshots (stdin) +// through combatSessionDelta + combatMergeIntoLifetime and prints the resulting +// lifetime, mirroring exactly what the combat_stats handler accumulates. Used to +// diff the Go accumulator against the Python functions on identical input. +func runCombatMergeCLI() { + raw, _ := io.ReadAll(os.Stdin) + var snapshots []map[string]any + if err := json.Unmarshal(raw, &snapshots); err != nil { + os.Stderr.WriteString("combat-merge: invalid JSON: " + err.Error() + "\n") + os.Exit(1) + } + lifetime := map[string]any{} + var last map[string]any + for _, s := range snapshots { + var delta map[string]any + if last != nil { + delta = combatSessionDelta(s, last) + } else { + delta = s + } + lifetime = combatMergeIntoLifetime(lifetime, delta) + last = s + } + out, _ := json.Marshal(lifetime) + os.Stdout.Write(out) +} + +// Combat stats accumulation — a faithful port of main.py's +// _combat_session_delta / _combat_merge_into_lifetime (incl. the documented +// quirk that offense/defense use the latest snapshot rather than a true delta). +// JSON numbers decode to float64; Go marshals whole floats without a decimal, +// so the stored JSONB matches Python's integer output. + +func num(v any) float64 { + switch x := v.(type) { + case float64: + return x + case int: + return float64(x) + case int64: + return float64(x) + } + return 0 +} + +func asMap(v any) map[string]any { + if m, ok := v.(map[string]any); ok { + return m + } + return map[string]any{} +} + +func combatSessionDelta(newS, oldS map[string]any) map[string]any { + delta := map[string]any{ + "total_damage_given": num(newS["total_damage_given"]) - num(oldS["total_damage_given"]), + "total_damage_received": num(newS["total_damage_received"]) - num(oldS["total_damage_received"]), + "total_kills": num(newS["total_kills"]) - num(oldS["total_kills"]), + "total_aetheria_surges": num(newS["total_aetheria_surges"]) - num(oldS["total_aetheria_surges"]), + "total_cloak_surges": num(newS["total_cloak_surges"]) - num(oldS["total_cloak_surges"]), + "monsters": map[string]any{}, + } + newMonsters := asMap(newS["monsters"]) + oldMonsters := asMap(oldS["monsters"]) + dMonsters := delta["monsters"].(map[string]any) + for name, nmv := range newMonsters { + nm := asMap(nmv) + om := asMap(oldMonsters[name]) + dm := map[string]any{ + "name": name, + "kill_count": num(nm["kill_count"]) - num(om["kill_count"]), + "damage_given": num(nm["damage_given"]) - num(om["damage_given"]), + "damage_received": num(nm["damage_received"]) - num(om["damage_received"]), + "aetheria_surges": num(nm["aetheria_surges"]) - num(om["aetheria_surges"]), + "cloak_surges": num(nm["cloak_surges"]) - num(om["cloak_surges"]), + "offense": asMap(nm["offense"]), // latest snapshot, per main.py + "defense": asMap(nm["defense"]), + } + if num(dm["kill_count"]) > 0 || num(dm["damage_given"]) > 0 || num(dm["damage_received"]) > 0 { + dMonsters[name] = dm + } + } + return delta +} + +func combatMergeIntoLifetime(lifetime, delta map[string]any) map[string]any { + if len(lifetime) == 0 { + lifetime = map[string]any{ + "total_damage_given": float64(0), "total_damage_received": float64(0), + "total_kills": float64(0), "total_aetheria_surges": float64(0), + "total_cloak_surges": float64(0), "monsters": map[string]any{}, + } + } + lifetime["total_damage_given"] = num(lifetime["total_damage_given"]) + num(delta["total_damage_given"]) + lifetime["total_damage_received"] = num(lifetime["total_damage_received"]) + num(delta["total_damage_received"]) + lifetime["total_kills"] = num(lifetime["total_kills"]) + num(delta["total_kills"]) + lifetime["total_aetheria_surges"] = num(lifetime["total_aetheria_surges"]) + num(delta["total_aetheria_surges"]) + lifetime["total_cloak_surges"] = num(lifetime["total_cloak_surges"]) + num(delta["total_cloak_surges"]) + + ltMonsters := asMap(lifetime["monsters"]) + lifetime["monsters"] = ltMonsters + for name, dmv := range asMap(delta["monsters"]) { + dm := asMap(dmv) + lmv, ok := ltMonsters[name] + if !ok { + lmv = map[string]any{ + "name": name, "kill_count": float64(0), "damage_given": float64(0), + "damage_received": float64(0), "aetheria_surges": float64(0), + "cloak_surges": float64(0), "offense": map[string]any{}, "defense": map[string]any{}, + } + ltMonsters[name] = lmv + } + lm := asMap(lmv) + lm["kill_count"] = num(lm["kill_count"]) + num(dm["kill_count"]) + lm["damage_given"] = num(lm["damage_given"]) + num(dm["damage_given"]) + lm["damage_received"] = num(lm["damage_received"]) + num(dm["damage_received"]) + lm["aetheria_surges"] = num(lm["aetheria_surges"]) + num(dm["aetheria_surges"]) + lm["cloak_surges"] = num(lm["cloak_surges"]) + num(dm["cloak_surges"]) + for _, side := range []string{"offense", "defense"} { + deltaSide := asMap(dm[side]) + if len(deltaSide) == 0 { + continue + } + ltSide := asMap(lm[side]) + lm[side] = ltSide + for atkType, byElV := range deltaSide { + ltByEl := asMap(ltSide[atkType]) + ltSide[atkType] = ltByEl + for el, statsV := range asMap(byElV) { + stats := asMap(statsV) + ltS := asMap(ltByEl[el]) + if len(ltS) == 0 { + ltS = map[string]any{ + "total_attacks": float64(0), "failed_attacks": float64(0), "crits": float64(0), + "total_normal_damage": float64(0), "max_normal_damage": float64(0), + "total_crit_damage": float64(0), "max_crit_damage": float64(0), + } + } + ltByEl[el] = ltS + ltS["total_attacks"] = num(ltS["total_attacks"]) + num(stats["total_attacks"]) + ltS["failed_attacks"] = num(ltS["failed_attacks"]) + num(stats["failed_attacks"]) + ltS["crits"] = num(ltS["crits"]) + num(stats["crits"]) + ltS["total_normal_damage"] = num(ltS["total_normal_damage"]) + num(stats["total_normal_damage"]) + ltS["max_normal_damage"] = maxF(num(ltS["max_normal_damage"]), num(stats["max_normal_damage"])) + ltS["total_crit_damage"] = num(ltS["total_crit_damage"]) + num(stats["total_crit_damage"]) + ltS["max_crit_damage"] = maxF(num(ltS["max_crit_damage"]), num(stats["max_crit_damage"])) + } + } + } + } + return lifetime +} + +func maxF(a, b float64) float64 { + if a > b { + return a + } + return b +} + +// handleCombatStats mirrors main.py:3305: compute the session delta vs the last +// snapshot, merge into the (DB-backed) lifetime, persist lifetime + the session +// snapshot (delete-then-insert), and update the live overlay. +func (i *Ingestor) handleCombatStats(ctx context.Context, data map[string]any) { + char := toStr(data["character_name"]) + if char == "" { + return + } + sessionV, hasSession := data["session"] + sessionID := toStr(data["session_id"]) + + if hasSession && sessionV != nil { + sessionData := asMap(sessionV) + prevKey := char + ":" + sessionID + + i.mu.Lock() + prev, hadPrev := i.combatLastSession[prevKey] + i.combatLastSession[prevKey] = sessionData + i.mu.Unlock() + + var delta map[string]any + if hadPrev { + delta = combatSessionDelta(sessionData, prev) + } else { + delta = sessionData + } + + // Load lifetime from cache, else DB (else empty). + i.mu.Lock() + lifetime, cached := i.combatLifetimeCache[char] + i.mu.Unlock() + if !cached { + lifetime = map[string]any{} + if row, err := queryRowAsMap(ctx, i.pool, + `SELECT stats_data FROM combat_stats WHERE character_name=$1`, char); err == nil && row != nil { + if m := asJSONMap(row["stats_data"]); m != nil { + lifetime = m + } + } + } + lifetime = combatMergeIntoLifetime(lifetime, delta) + i.mu.Lock() + i.combatLifetimeCache[char] = lifetime + i.mu.Unlock() + + now := time.Now().UTC() + ltJSON, _ := json.Marshal(lifetime) + // delete-then-insert lifetime (no ON CONFLICT, matching Python) + if _, err := i.pool.Exec(ctx, `DELETE FROM combat_stats WHERE character_name=$1`, char); err == nil { + if _, err := i.pool.Exec(ctx, + `INSERT INTO combat_stats (character_name,timestamp,stats_data) VALUES ($1,$2,$3)`, + char, now, ltJSON); err != nil { + i.log.Error("combat_stats insert failed", "err", err, "char", char) + } + } + if sessionID != "" { + sdJSON, _ := json.Marshal(sessionData) + if _, err := i.pool.Exec(ctx, + `DELETE FROM combat_stats_sessions WHERE character_name=$1 AND session_id=$2`, char, sessionID); err == nil { + if _, err := i.pool.Exec(ctx, + `INSERT INTO combat_stats_sessions (character_name,session_id,timestamp,stats_data) VALUES ($1,$2,$3,$4)`, + char, sessionID, now, sdJSON); err != nil { + i.log.Error("combat_stats_sessions insert failed", "err", err, "char", char) + } + } + } + data["lifetime"] = lifetime + } + + i.mu.Lock() + i.liveCombatStats[char] = data + i.mu.Unlock() +} diff --git a/go-services/tracker-go/combat_test.go b/go-services/tracker-go/combat_test.go new file mode 100644 index 00000000..25bc15cd --- /dev/null +++ b/go-services/tracker-go/combat_test.go @@ -0,0 +1,54 @@ +package main + +import "testing" + +// Golden values cross-checked against the Python _combat_session_delta / +// _combat_merge_into_lifetime on identical input (see compare run). Folds two +// cumulative snapshots; the first is treated as the whole delta. +func TestCombatMerge(t *testing.T) { + snap1 := map[string]any{ + "total_damage_given": 100.0, "total_kills": 2.0, + "monsters": map[string]any{ + "Drudge": map[string]any{ + "name": "Drudge", "kill_count": 2.0, "damage_given": 100.0, + "offense": map[string]any{"melee": map[string]any{"slashing": map[string]any{ + "total_attacks": 10.0, "max_normal_damage": 15.0, + }}}, + }, + }, + } + snap2 := map[string]any{ + "total_damage_given": 250.0, "total_kills": 5.0, + "monsters": map[string]any{ + "Drudge": map[string]any{ + "name": "Drudge", "kill_count": 4.0, "damage_given": 200.0, + "offense": map[string]any{"melee": map[string]any{"slashing": map[string]any{ + "total_attacks": 20.0, "max_normal_damage": 18.0, + }}}, + }, + }, + } + + lifetime := map[string]any{} + lifetime = combatMergeIntoLifetime(lifetime, snap1) // first = whole delta + lifetime = combatMergeIntoLifetime(lifetime, combatSessionDelta(snap2, snap1)) + + if got := num(lifetime["total_kills"]); got != 5 { + t.Errorf("total_kills = %v, want 5", got) + } + if got := num(lifetime["total_damage_given"]); got != 250 { + t.Errorf("total_damage_given = %v, want 250", got) + } + drudge := asMap(asMap(lifetime["monsters"])["Drudge"]) + if got := num(drudge["kill_count"]); got != 4 { + t.Errorf("Drudge.kill_count = %v, want 4", got) + } + slashing := asMap(asMap(asMap(drudge["offense"])["melee"])["slashing"]) + // offense uses the latest snapshot additively (the documented quirk): 10 + 20. + if got := num(slashing["total_attacks"]); got != 30 { + t.Errorf("offense slashing total_attacks = %v, want 30 (latest-additive quirk)", got) + } + if got := num(slashing["max_normal_damage"]); got != 18 { + t.Errorf("offense slashing max_normal_damage = %v, want 18 (max)", got) + } +} diff --git a/go-services/tracker-go/go.mod b/go-services/tracker-go/go.mod new file mode 100644 index 00000000..38254972 --- /dev/null +++ b/go-services/tracker-go/go.mod @@ -0,0 +1,5 @@ +module git.snakedesert.se/SawatoMosswartsEnjoyersClub/MosswartOverlord/go-services/tracker-go + +go 1.25 + +require github.com/jackc/pgx/v5 v5.10.0 diff --git a/go-services/tracker-go/ingest.go b/go-services/tracker-go/ingest.go new file mode 100644 index 00000000..1b8c0ca2 --- /dev/null +++ b/go-services/tracker-go/ingest.go @@ -0,0 +1,519 @@ +package main + +import ( + "context" + "encoding/json" + "log/slog" + "strings" + "sync" + "time" + + "github.com/jackc/pgx/v5/pgxpool" +) + +// Ingestor implements the plugin event handlers (the /ws/position logic), +// faithfully mirroring main.py's write semantics. It owns the in-memory live +// state and writes to a read-write pool (its own DB in shadow/cutover mode). +// +// It is fed either by the real /ws/position server (cutover) or by the shadow +// consumer replaying Python's /ws/live broadcast firehose. broadcast is invoked +// after each handled event (nil = no browser fan-out, e.g. shadow mode). +type Ingestor struct { + pool *pgxpool.Pool + log *slog.Logger + broadcast func(map[string]any) + + mu sync.RWMutex + liveSnapshots map[string]map[string]any + liveVitals map[string]map[string]any + liveCharacterStats map[string]map[string]any + liveEquipmentCantrip map[string]map[string]any + liveNearbyObjects map[string]map[string]any + liveCombatStats map[string]map[string]any + dungeonMapCache map[string]map[string]any + questStatus map[string]map[string]string + lastKills map[string]int // "session_id|character_name" -> kills + combatLastSession map[string]map[string]any // "char:session_id" -> last cumulative session + combatLifetimeCache map[string]map[string]any // character_name -> accumulated lifetime + vitalSubscribers map[string]bool + vitalPeerState map[string]map[string]any + + plugins *pluginRegistry // for share_* fan-out + plugin_connected status + + invFwd *invForwarder // inventory forwarding (cutover only; nil in shadow/read) + aclog *aclogPoster // death/idle Discord alerts (cutover only; nil otherwise) +} + +func newIngestor(pool *pgxpool.Pool, log *slog.Logger, broadcast func(map[string]any), plugins *pluginRegistry) *Ingestor { + return &Ingestor{ + pool: pool, + log: log, + broadcast: broadcast, + plugins: plugins, + liveSnapshots: map[string]map[string]any{}, + liveVitals: map[string]map[string]any{}, + liveCharacterStats: map[string]map[string]any{}, + liveEquipmentCantrip: map[string]map[string]any{}, + liveNearbyObjects: map[string]map[string]any{}, + liveCombatStats: map[string]map[string]any{}, + dungeonMapCache: map[string]map[string]any{}, + questStatus: map[string]map[string]string{}, + lastKills: map[string]int{}, + combatLastSession: map[string]map[string]any{}, + combatLifetimeCache: map[string]map[string]any{}, + vitalSubscribers: map[string]bool{}, + vitalPeerState: map[string]map[string]any{}, + } +} + +// dispatch routes a parsed message to the right handler. Over /ws/position the +// discriminator is the "type" field; over the /ws/live broadcast, telemetry has +// NO type (it's the raw snapshot), so we also match it by shape. +func (i *Ingestor) dispatch(ctx context.Context, data map[string]any) { + t := toStr(data["type"]) + switch { + case t == "telemetry" || (t == "" && hasTelemetryShape(data)): + i.handleTelemetry(ctx, data) + // Python broadcasts telemetry as a TYPELESS snapshot (snap.dict()); the + // browser intentionally ignores typeless messages (useLiveData drops + // `if (!msg.type) return`) and takes player data from the 5s /live poll + // instead. Broadcasting it WITH a type makes the UI overwrite the + // /live-derived telemetry (which has total_kills/total_rares/session_rares) + // with the raw plugin payload (which lacks them), flapping those counters + // 0<->value. Strip the type to match. + if i.broadcast != nil { + i.broadcast(stripType(data)) + } + return + case t == "rare": + i.handleRare(ctx, data) + case t == "portal": + i.handlePortal(ctx, data) + case t == "character_stats": + i.handleCharacterStats(ctx, data) + case t == "spawn": + i.handleSpawn(ctx, data) + case t == "vitals": + i.handleVitals(data) + case t == "quest": + i.handleQuest(data) + case t == "equipment_cantrip_state": + i.handleEquipmentCantrip(data) + case t == "nearby_objects": + i.handleNearbyObjects(data) + case t == "dungeon_map": + i.handleDungeonMap(data) + case t == "combat_stats": + i.handleCombatStats(ctx, data) + case t == "full_inventory": + // Forward the full snapshot to the inventory service; not browser-broadcast. + if i.invFwd != nil { + i.invFwd.forwardFullInventory(data) + } + return + case t == "inventory_delta": + // Fire-and-forget forward; the forwarder broadcasts the enriched delta. + if i.invFwd != nil { + i.invFwd.handleInventoryDelta(data) + } + return + case t == "share_subscribe": + i.handleShareSubscribe(data) + case t == "share_unsubscribe": + i.handleShareUnsubscribe(data) + return // unsubscribe broadcasts its own share_peer_removed; don't re-broadcast + case strings.HasPrefix(t, "share_"): + i.handleShareUpdate(t, data) + case t == "register": + // no DB / no broadcast; plugin_conns belongs to the /ws/position server + case t == "chat": + // broadcast-only + } + if i.broadcast != nil { + i.broadcast(data) + } +} + +// stripType returns a shallow copy of the message without its "type" key, so the +// browser treats it as a typeless snapshot (and ignores it, deferring to /live). +func stripType(data map[string]any) map[string]any { + cp := make(map[string]any, len(data)) + for k, v := range data { + if k != "type" { + cp[k] = v + } + } + return cp +} + +func hasTelemetryShape(d map[string]any) bool { + _, a := d["session_id"] + _, b := d["ew"] + _, c := d["kills"] + return a && b && c +} + +// --- telemetry: INSERT telemetry_events + kill-delta into char_stats (main.py:3124) --- + +const insTelemetry = `INSERT INTO telemetry_events +(character_name,char_tag,session_id,timestamp,ew,ns,z,kills,kills_per_hour,onlinetime, + deaths,total_deaths,rares_found,prismatic_taper_count,vt_state,mem_mb,cpu_pct,mem_handles,latency_ms,received_at) +VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,0,$13,$14,$15,$16,$17,$18,$19)` + +const upsertCharKills = `INSERT INTO char_stats (character_name,total_kills) VALUES ($1,$2) +ON CONFLICT (character_name) DO UPDATE SET total_kills = char_stats.total_kills + $2` + +func (i *Ingestor) handleTelemetry(ctx context.Context, data map[string]any) { + name := toStr(data["character_name"]) + sessionID := toStr(data["session_id"]) + if name == "" || sessionID == "" { + return + } + kills := toInt(data["kills"]) + received := time.Now().UTC() + + key := sessionID + "|" + name + i.mu.RLock() + last, ok := i.lastKills[key] + i.mu.RUnlock() + if !ok { + if row, err := queryRowAsMap(ctx, i.pool, + `SELECT kills FROM telemetry_events WHERE character_name=$1 AND session_id=$2 ORDER BY timestamp DESC LIMIT 1`, + name, sessionID); err == nil && row != nil { + last = toInt(row["kills"]) + } + } + delta := kills - last + + tx, err := i.pool.Begin(ctx) + if err != nil { + i.log.Error("telemetry tx begin failed", "err", err) + return + } + defer tx.Rollback(ctx) + if _, err := tx.Exec(ctx, insTelemetry, + name, nstr(data["char_tag"]), sessionID, parseTSAny(data["timestamp"]), + toFloat(data["ew"]), toFloat(data["ns"]), toFloat(data["z"]), kills, + nfloat(data["kills_per_hour"]), nstr(data["onlinetime"]), toInt(data["deaths"]), + nint(data["total_deaths"]), toInt(data["prismatic_taper_count"]), nstr(data["vt_state"]), + nfloat(data["mem_mb"]), nfloat(data["cpu_pct"]), nint(data["mem_handles"]), + nfloat(data["latency_ms"]), received, + ); err != nil { + i.log.Error("telemetry insert failed", "err", err, "char", name) + return + } + if delta > 0 { + if _, err := tx.Exec(ctx, upsertCharKills, name, delta); err != nil { + i.log.Error("char_stats upsert failed", "err", err, "char", name) + return + } + } + if err := tx.Commit(ctx); err != nil { + i.log.Error("telemetry commit failed", "err", err, "char", name) + return + } + + i.mu.Lock() + i.lastKills[key] = kills + i.liveSnapshots[name] = data + i.mu.Unlock() +} + +// --- rare: rare_stats + rare_stats_sessions + rare_events (main.py:3234) --- + +const upsertRareStats = `INSERT INTO rare_stats (character_name,total_rares) VALUES ($1,1) +ON CONFLICT (character_name) DO UPDATE SET total_rares = rare_stats.total_rares + 1` +const upsertRareSession = `INSERT INTO rare_stats_sessions (character_name,session_id,session_rares) VALUES ($1,$2,1) +ON CONFLICT (character_name,session_id) DO UPDATE SET session_rares = rare_stats_sessions.session_rares + 1` +const insRareEvent = `INSERT INTO rare_events (character_name,name,timestamp,ew,ns,z) VALUES ($1,$2,$3,$4,$5,$6)` + +func (i *Ingestor) handleRare(ctx context.Context, data map[string]any) { + name := toStr(data["character_name"]) + if strings.TrimSpace(name) == "" { + return + } + if _, err := i.pool.Exec(ctx, upsertRareStats, name); err != nil { + i.log.Error("rare_stats upsert failed", "err", err, "char", name) + return + } + // Session id: live snapshot first, else latest telemetry row. + i.mu.RLock() + sessionID := toStr(i.liveSnapshots[name]["session_id"]) + i.mu.RUnlock() + if sessionID == "" { + if row, err := queryRowAsMap(ctx, i.pool, + `SELECT session_id FROM telemetry_events WHERE character_name=$1 ORDER BY timestamp DESC LIMIT 1`, name); err == nil && row != nil { + sessionID = toStr(row["session_id"]) + } + } + if sessionID != "" { + if _, err := i.pool.Exec(ctx, upsertRareSession, name, sessionID); err != nil { + i.log.Error("rare_stats_sessions upsert failed", "err", err, "char", name) + } + } + if _, err := i.pool.Exec(ctx, insRareEvent, + name, toStr(data["name"]), parseTSAny(data["timestamp"]), + toFloat(data["ew"]), toFloat(data["ns"]), toFloatOr(data["z"], 0), + ); err != nil { + i.log.Error("rare_events insert failed", "err", err, "char", name) + } +} + +// --- portal: upsert on rounded coords (main.py:3567) --- + +const upsertPortal = `INSERT INTO portals (portal_name,ns,ew,z,discovered_at,discovered_by) +VALUES ($1,$2,$3,$4,$5,$6) +ON CONFLICT (ROUND(ns::numeric,1), ROUND(ew::numeric,1)) DO UPDATE SET + discovered_at = EXCLUDED.discovered_at, + discovered_by = EXCLUDED.discovered_by, + portal_name = EXCLUDED.portal_name` + +func (i *Ingestor) handlePortal(ctx context.Context, data map[string]any) { + name := toStr(data["character_name"]) + portalName := toStr(data["portal_name"]) + ts := data["timestamp"] + if name == "" || portalName == "" || data["ns"] == nil || data["ew"] == nil || data["z"] == nil || ts == nil { + return + } + if _, err := i.pool.Exec(ctx, upsertPortal, + portalName, toFloat(data["ns"]), toFloat(data["ew"]), toFloat(data["z"]), + parseTSAny(ts), name, + ); err != nil { + i.log.Error("portal upsert failed", "err", err, "char", name) + } +} + +// --- character_stats: build stats_data subset + upsert (main.py:3443) --- + +const upsertCharacterStats = `INSERT INTO character_stats +(character_name,timestamp,level,total_xp,unassigned_xp,luminance_earned,luminance_total,deaths,stats_data) +VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9) +ON CONFLICT (character_name) DO UPDATE SET + timestamp=EXCLUDED.timestamp, level=EXCLUDED.level, total_xp=EXCLUDED.total_xp, + unassigned_xp=EXCLUDED.unassigned_xp, luminance_earned=EXCLUDED.luminance_earned, + luminance_total=EXCLUDED.luminance_total, deaths=EXCLUDED.deaths, stats_data=EXCLUDED.stats_data` + +var statsDataKeys = []string{ + "attributes", "vitals", "skills", "allegiance", "active_item_enchantments", + "race", "gender", "birth", "current_title", "skill_credits", "burden", + "burden_units", "encumbrance_capacity", "properties", "titles", +} + +func (i *Ingestor) handleCharacterStats(ctx context.Context, data map[string]any) { + name := toStr(data["character_name"]) + if name == "" { + return + } + statsData := map[string]any{} + for _, k := range statsDataKeys { + if v, ok := data[k]; ok && v != nil { + statsData[k] = v + } + } + sdJSON, _ := json.Marshal(statsData) + if _, err := i.pool.Exec(ctx, upsertCharacterStats, + name, parseTSAny(data["timestamp"]), nint(data["level"]), nint(data["total_xp"]), + nint(data["unassigned_xp"]), nint(data["luminance_earned"]), nint(data["luminance_total"]), + nint(data["deaths"]), sdJSON, + ); err != nil { + i.log.Error("character_stats upsert failed", "err", err, "char", name) + return + } + i.mu.Lock() + i.liveCharacterStats[name] = data + i.mu.Unlock() +} + +// --- spawn: INSERT spawn_events (main.py:3110). Not broadcast, so only the real +// /ws/position path feeds this; covered by ingest_test.go. --- + +const insSpawn = `INSERT INTO spawn_events (character_name,mob,timestamp,ew,ns,z) VALUES ($1,$2,$3,$4,$5,$6)` + +func (i *Ingestor) handleSpawn(ctx context.Context, data map[string]any) { + name := toStr(data["character_name"]) + mob := toStr(data["mob"]) + if name == "" || mob == "" { + return + } + if _, err := i.pool.Exec(ctx, insSpawn, + name, mob, parseTSAny(data["timestamp"]), + toFloat(data["ew"]), toFloat(data["ns"]), toFloatOr(data["z"], 0), + ); err != nil { + i.log.Error("spawn insert failed", "err", err, "char", name) + } +} + +// --- memory-only handlers --- + +func (i *Ingestor) handleVitals(data map[string]any) { + name := toStr(data["character_name"]) + if name == "" { + return + } + // Death detection (main.py:3419): vitae crossing 0 -> >0. Only in cutover + // (i.aclog != nil); in shadow mode it stays off to avoid duplicating the + // production alert. + if i.aclog != nil { + i.mu.RLock() + prev := i.liveVitals[name] + i.mu.RUnlock() + var prevVitae float64 + if prev != nil { + prevVitae = toFloat(prev["vitae"]) + } + if newVitae := toFloat(data["vitae"]); prevVitae == 0 && newVitae > 0 { + i.aclog.maybeDeath(name, newVitae) + } + } + i.mu.Lock() + i.liveVitals[name] = data + i.mu.Unlock() +} + +var allowedQuests = map[string]bool{ + "Stipend Collection Timer": true, + "Blank Augmentation Gem Pickup Timer": true, + "Insatiable Eater Jaw": true, +} + +func (i *Ingestor) handleQuest(data map[string]any) { + name := toStr(data["character_name"]) + quest := toStr(data["quest_name"]) + countdown, ok := data["countdown"] + if name == "" || quest == "" || !ok || countdown == nil || !allowedQuests[quest] { + return + } + i.mu.Lock() + if i.questStatus[name] == nil { + i.questStatus[name] = map[string]string{} + } + i.questStatus[name][quest] = toStr(countdown) + i.mu.Unlock() +} + +func (i *Ingestor) handleEquipmentCantrip(data map[string]any) { + name := toStr(data["character_name"]) + if name == "" { + return + } + i.mu.Lock() + i.liveEquipmentCantrip[name] = data + i.mu.Unlock() +} + +// clearEquipmentCantrip drops a character's cantrip overlay on plugin register +// (main.py:3106). +func (i *Ingestor) clearEquipmentCantrip(name string) { + i.mu.Lock() + delete(i.liveEquipmentCantrip, name) + i.mu.Unlock() +} + +func (i *Ingestor) handleNearbyObjects(data map[string]any) { + name := toStr(data["character_name"]) + if name == "" { + return + } + i.mu.Lock() + i.liveNearbyObjects[name] = data + i.mu.Unlock() +} + +func (i *Ingestor) handleDungeonMap(data map[string]any) { + lb := toStr(data["landblock"]) + if lb == "" { + return + } + i.mu.Lock() + i.dungeonMapCache[lb] = data + i.mu.Unlock() +} + +// --- read-side overlay accessors (used by the HTTP handlers when an ingestor +// is present, mirroring Python's "live cache first, DB fallback") --- + +func (i *Ingestor) snapshot(m map[string]map[string]any, name string) (map[string]any, bool) { + i.mu.RLock() + defer i.mu.RUnlock() + v, ok := m[name] + return v, ok +} + +func (i *Ingestor) getCharacterStats(name string) (map[string]any, bool) { + return i.snapshot(i.liveCharacterStats, name) +} +func (i *Ingestor) getEquipmentCantrip(name string) (map[string]any, bool) { + return i.snapshot(i.liveEquipmentCantrip, name) +} +func (i *Ingestor) getCombatStats(name string) (map[string]any, bool) { + return i.snapshot(i.liveCombatStats, name) +} +func (i *Ingestor) allCombatStats() map[string]map[string]any { + i.mu.RLock() + defer i.mu.RUnlock() + out := make(map[string]map[string]any, len(i.liveCombatStats)) + for k, v := range i.liveCombatStats { + out[k] = v + } + return out +} +func (i *Ingestor) questData() (map[string]map[string]string, int) { + i.mu.RLock() + defer i.mu.RUnlock() + out := make(map[string]map[string]string, len(i.questStatus)) + for c, qs := range i.questStatus { + cp := make(map[string]string, len(qs)) + for k, v := range qs { + cp[k] = v + } + out[c] = cp + } + return out, len(i.questStatus) +} + +// --- small value helpers (JSON numbers decode as float64) --- + +func nstr(v any) any { + if s, ok := v.(string); ok { + return s + } + return nil +} +// nint/nfloat return a typed number or nil (for nullable columns), coercing +// string-encoded numbers the plugin sends (see coerceNum). +func nint(v any) any { + if f, ok := coerceNum(v); ok { + return int64(f) + } + return nil +} +func nfloat(v any) any { + if f, ok := coerceNum(v); ok { + return f + } + return nil +} +func toFloatOr(v any, def float64) float64 { + if f, ok := coerceNum(v); ok { + return f + } + return def +} + +func parseTSAny(v any) time.Time { + s, ok := v.(string) + if !ok { + return time.Now().UTC() + } + s = strings.Replace(s, "Z", "+00:00", 1) + for _, l := range []string{ + time.RFC3339Nano, time.RFC3339, + "2006-01-02T15:04:05.999999-07:00", "2006-01-02T15:04:05-07:00", + "2006-01-02T15:04:05.999999", "2006-01-02T15:04:05", + } { + if t, err := time.Parse(l, s); err == nil { + return t + } + } + return time.Now().UTC() +} diff --git a/go-services/tracker-go/inventory_forward.go b/go-services/tracker-go/inventory_forward.go new file mode 100644 index 00000000..a5c6e70b --- /dev/null +++ b/go-services/tracker-go/inventory_forward.go @@ -0,0 +1,144 @@ +package main + +import ( + "bytes" + "encoding/json" + "fmt" + "io" + "log/slog" + "net/http" + "net/url" + "strings" + "sync" + "time" +) + +// invForwarder forwards plugin inventory events to the inventory service, +// porting main.py's _forward_to_inventory_service / _handle_inventory_delta. +// Only active in cutover (write) mode; nil in shadow/read-only mode, where the +// plugin firehose never carries inventory anyway. +// +// full_inventory -> POST {url}/process-inventory (full replace) +// inventory_delta add/update -> POST {url}/inventory/{char}/item +// inventory_delta remove -> DELETE {url}/inventory/{char}/item/{item_id} +// +// Deltas are fire-and-forget (never block the /ws/position read loop), serialized +// per-character (so a char's rapid deltas don't race the inventory DELETE+INSERT), +// and globally capped at 8 concurrent forwards. +type invForwarder struct { + url string + client *http.Client + sem chan struct{} + mu sync.Mutex + locks map[string]*sync.Mutex + log *slog.Logger + broadcast func(map[string]any) +} + +func newInvForwarder(rawURL string, log *slog.Logger, broadcast func(map[string]any)) *invForwarder { + return &invForwarder{ + url: strings.TrimRight(rawURL, "/"), + client: &http.Client{Timeout: 30 * time.Second}, + sem: make(chan struct{}, 8), + locks: map[string]*sync.Mutex{}, + log: log, + broadcast: broadcast, + } +} + +func (f *invForwarder) charLock(name string) *sync.Mutex { + f.mu.Lock() + defer f.mu.Unlock() + l := f.locks[name] + if l == nil { + l = &sync.Mutex{} + f.locks[name] = l + } + return l +} + +// forwardFullInventory POSTs a full inventory snapshot (full replace). Runs +// inline on the /ws/position handler — main.py awaits _store_inventory too. +func (f *invForwarder) forwardFullInventory(data map[string]any) { + char := toStr(data["character_name"]) + body, _ := json.Marshal(map[string]any{ + "character_name": char, + "timestamp": data["timestamp"], + "items": data["items"], + }) + resp, err := f.client.Post(f.url+"/process-inventory", "application/json", bytes.NewReader(body)) + if err != nil { + f.log.Error("full_inventory forward failed", "err", err, "char", char) + return + } + defer drain(resp) + if resp.StatusCode >= 400 { + f.log.Warn("inventory service error (full_inventory)", "status", resp.StatusCode, "char", char) + } +} + +// handleInventoryDelta forwards a single add/update/remove. Fire-and-forget. +func (f *invForwarder) handleInventoryDelta(data map[string]any) { + go func() { + char := toStr(data["character_name"]) + lock := f.charLock(char) + lock.Lock() + defer lock.Unlock() + f.sem <- struct{}{} + defer func() { <-f.sem }() + + out := data + switch toStr(data["action"]) { + case "remove": + if itemID := data["item_id"]; itemID != nil { + req, _ := http.NewRequest(http.MethodDelete, + fmt.Sprintf("%s/inventory/%s/item/%v", f.url, url.PathEscape(char), itemID), nil) + if resp, err := f.client.Do(req); err != nil { + f.log.Warn("inventory delta remove failed", "err", err, "char", char) + } else { + if resp.StatusCode >= 400 { + f.log.Warn("inventory service error (delta remove)", "status", resp.StatusCode, "char", char) + } + drain(resp) + } + } + case "add", "update": + if item := data["item"]; item != nil { + b, _ := json.Marshal(item) + resp, err := f.client.Post(fmt.Sprintf("%s/inventory/%s/item", f.url, url.PathEscape(char)), + "application/json", bytes.NewReader(b)) + if err != nil { + f.log.Warn("inventory delta add/update failed", "err", err, "char", char) + } else { + if resp.StatusCode < 400 { + // Re-broadcast the enriched item the service returns. + var r map[string]any + if json.NewDecoder(resp.Body).Decode(&r) == nil { + if enriched, ok := r["item"]; ok && enriched != nil { + out = map[string]any{ + "type": "inventory_delta", + "action": toStr(data["action"]), + "character_name": char, + "item": enriched, + } + } + } + } else { + f.log.Warn("inventory service error (delta add/update)", "status", resp.StatusCode, "char", char) + } + drain(resp) + } + } + } + if f.broadcast != nil { + f.broadcast(out) + } + }() +} + +func drain(resp *http.Response) { + if resp != nil && resp.Body != nil { + _, _ = io.Copy(io.Discard, resp.Body) + _ = resp.Body.Close() + } +} diff --git a/go-services/tracker-go/live.go b/go-services/tracker-go/live.go new file mode 100644 index 00000000..2e571602 --- /dev/null +++ b/go-services/tracker-go/live.go @@ -0,0 +1,150 @@ +package main + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "sync" + "time" +) + +// Timing constants mirror main.py. +const ( + activeWindow = 30 * time.Second // ACTIVE_WINDOW — the real "online" test + chunkLookback = 10 * time.Minute // coarse bound, only so TimescaleDB can prune chunks + trailsWindow = 600 * time.Second // /trails lookback (hardcoded; the `seconds` param is ignored) + cacheInterval = 5 * time.Second // _refresh_cache_loop cadence +) + +// liveSQL mirrors main.py:837 exactly. $1 = chunk_cutoff (now-10min), $2 = cutoff (now-30s). +// Online-ness is decided on COALESCE(received_at, timestamp) — server receive-time — because +// game clients' clocks drift up to ~90s and would otherwise flap the player count. +const liveSQL = ` +SELECT sub.*, + COALESCE(rs.total_rares, 0) AS total_rares, + COALESCE(rss.session_rares, 0) AS session_rares, + COALESCE(cs.total_kills, 0) AS total_kills +FROM ( + SELECT DISTINCT ON (character_name) * + FROM telemetry_events + WHERE timestamp > $1 + AND COALESCE(received_at, timestamp) > $2 + ORDER BY character_name, timestamp DESC +) sub +LEFT JOIN rare_stats rs ON sub.character_name = rs.character_name +LEFT JOIN rare_stats_sessions rss ON sub.character_name = rss.character_name + AND sub.session_id = rss.session_id +LEFT JOIN char_stats cs ON sub.character_name = cs.character_name` + +// trailsSQL mirrors main.py:874 — last 600s of position points, ordered for the map. +const trailsSQL = ` +SELECT timestamp, character_name, ew, ns, z +FROM telemetry_events +WHERE timestamp >= $1 +ORDER BY character_name, timestamp` + +// liveCache holds the pre-marshaled JSON bodies for /live and /trails, swapped +// atomically every cacheInterval by the refresh loop. +type liveCache struct { + mu sync.RWMutex + liveJSON []byte + trailsJSON []byte +} + +func newLiveCache() *liveCache { + return &liveCache{ + liveJSON: []byte(`{"players":[]}`), + trailsJSON: []byte(`{"trails":[]}`), + } +} + +func (c *liveCache) getLive() []byte { + c.mu.RLock() + defer c.mu.RUnlock() + return c.liveJSON +} + +func (c *liveCache) getTrails() []byte { + c.mu.RLock() + defer c.mu.RUnlock() + return c.trailsJSON +} + +func (c *liveCache) set(live, trails []byte) { + c.mu.Lock() + defer c.mu.Unlock() + c.liveJSON = live + c.trailsJSON = trails +} + +// refresh recomputes both caches from the DB. Both queries use the SAME `now` +// so the online window and trails window are consistent within a tick. +func (s *Server) refreshLiveCache(ctx context.Context) error { + qctx, cancel := context.WithTimeout(ctx, 15*time.Second) + defer cancel() + + now := time.Now().UTC() + + players, err := queryRowsAsMaps(qctx, s.pool, liveSQL, now.Add(-chunkLookback), now.Add(-activeWindow)) + if err != nil { + return fmt.Errorf("live query: %w", err) + } + formatTimes(players, "timestamp", "received_at") + liveJSON, err := json.Marshal(map[string]any{"players": players}) + if err != nil { + return fmt.Errorf("marshal live: %w", err) + } + + trails, err := queryRowsAsMaps(qctx, s.pool, trailsSQL, now.Add(-trailsWindow)) + if err != nil { + return fmt.Errorf("trails query: %w", err) + } + formatTimes(trails, "timestamp") + trailsJSON, err := json.Marshal(map[string]any{"trails": trails}) + if err != nil { + return fmt.Errorf("marshal trails: %w", err) + } + + s.cache.set(liveJSON, trailsJSON) + return nil +} + +// runCacheLoop refreshes the cache every cacheInterval until ctx is cancelled. +// It refreshes immediately on entry (refresh-then-sleep) so the cache is warm +// shortly after startup. pgxpool handles reconnection transparently, so we just +// log failures and keep serving the last good snapshot. +func (s *Server) runCacheLoop(ctx context.Context) { + failures := 0 + for { + if err := s.refreshLiveCache(ctx); err != nil { + failures++ + s.log.Error("live cache refresh failed", "err", err, "consecutive", failures) + } else { + if failures > 0 { + s.log.Info("live cache refresh recovered", "after_failures", failures) + } + failures = 0 + } + select { + case <-ctx.Done(): + return + case <-time.After(cacheInterval): + } + } +} + +func (s *Server) handleLive(w http.ResponseWriter, r *http.Request) { + writeRawJSON(w, s.cache.getLive()) +} + +func (s *Server) handleTrails(w http.ResponseWriter, r *http.Request) { + // `seconds` query param is accepted but ignored, matching main.py:2001. + writeRawJSON(w, s.cache.getTrails()) +} + +func writeRawJSON(w http.ResponseWriter, body []byte) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(http.StatusOK) + _, _ = w.Write(body) +} diff --git a/go-services/tracker-go/main.go b/go-services/tracker-go/main.go new file mode 100644 index 00000000..b5cf5a44 --- /dev/null +++ b/go-services/tracker-go/main.go @@ -0,0 +1,337 @@ +// Command tracker-go is a Go reimplementation of the MosswartOverlord +// "dereth-tracker" backend, deployed in parallel with the live Python service +// for side-by-side comparison (strangler-fig migration). +// +// Phase 1: read-side parity. Connects READ-ONLY to the existing dereth +// TimescaleDB and reimplements the HTTP read API, starting with the /live and +// /trails caches (the 5s _refresh_cache_loop). It never touches anything the +// Python service writes. +// +// Routes are declared WITHOUT the nginx-stripped "/go/" prefix, mirroring the +// Python service's "no /api/ prefix" convention. nginx's `location /go/` strips +// the prefix before proxying to this service on 127.0.0.1:8770. +package main + +import ( + "context" + "encoding/json" + "errors" + "log/slog" + "net/http" + "net/http/httputil" + "os" + "os/signal" + "syscall" + "time" + + "github.com/jackc/pgx/v5/pgxpool" +) + +// buildVersion is injected at build time via -ldflags "-X main.buildVersion=...". +// Mirrors the Python service's APP_VERSION / "/api-version" stamp. +var buildVersion = "dev" + +// Server holds the shared dependencies for HTTP handlers. +type Server struct { + pool *pgxpool.Pool + cache *liveCache + totals *totalsCache + invProxy *httputil.ReverseProxy + staticDir string + secretKey string + sharedSecret string + sharedSecretLegacy string + ingestor *Ingestor // non-nil only in ingest/shadow mode + hub *Hub // browser /ws/live fan-out + plugins *pluginRegistry + loginLimiter *loginLimiter + log *slog.Logger +} + +func main() { + // `tracker-go combat-merge` reads a JSON array of cumulative session + // snapshots from stdin and prints the folded lifetime — a deterministic hook + // for cross-language parity testing against the Python combat functions. + if len(os.Args) > 1 && os.Args[1] == "combat-merge" { + runCombatMergeCLI() + return + } + // `tracker-go issue-cookie ` prints a + // session token — a hook to cross-check itsdangerous cookie interop with the + // Python service. + if len(os.Args) > 1 && os.Args[1] == "issue-cookie" { + runIssueCookieCLI() + return + } + + logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo})) + slog.SetDefault(logger) + + cfg := loadConfig() + logger.Info("starting tracker-go", "version", buildVersion, "addr", cfg.Addr) + + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) + defer stop() + + srv := &Server{ + cache: newLiveCache(), + totals: newTotalsCache(), + loginLimiter: newLoginLimiter(), + staticDir: cfg.StaticDir, + secretKey: cfg.SecretKey, + sharedSecret: cfg.SharedSecret, + sharedSecretLegacy: cfg.SharedSecretLegacy, + hub: newHub(), + plugins: newPluginRegistry(logger), + log: logger, + } + if cfg.SecretKey == "" { + // Fail closed like the Python service: with no key, no external cookie + // can verify, so only internal-trust (loopback/compose) requests pass. + logger.Warn("SECRET_KEY unset — external (nginx-proxied) requests will all be rejected") + } + + // Inventory-service reverse proxy (independent of the DB). + if err := srv.initInvProxy(cfg.InventoryURL); err != nil { + logger.Error("inventory proxy init failed", "err", err, "target", cfg.InventoryURL) + os.Exit(1) + } + + // Connect to the dereth DB (read-only). If DATABASE_URL is unset we still + // serve health/version (Phase-0 mode) so the container is observable. + if cfg.DatabaseURL == "" { + logger.Warn("DATABASE_URL unset — running without DB; DB-backed endpoints will be empty") + } else { + connectCtx, cancel := context.WithTimeout(ctx, 15*time.Second) + pool, err := newPool(connectCtx, cfg.DatabaseURL, cfg.ReadOnly) + cancel() + if err != nil { + logger.Error("db pool init failed", "err", err) + os.Exit(1) + } + defer pool.Close() + srv.pool = pool + + // Write mode (shadow OR cutover) owns the ingest path; read-only mode + // (parallel read API) skips all of this. + if !cfg.ReadOnly { + // Schema init only when we own a fresh DB. In cutover (reusing the + // production DB) SKIP_SCHEMA_INIT keeps us from running ANY DDL. + if !cfg.SkipSchemaInit { + schemaCtx, cancel := context.WithTimeout(ctx, 60*time.Second) + initSchema(schemaCtx, pool, logger) + cancel() + } + + srv.ingestor = newIngestor(pool, logger, srv.hub.broadcast, srv.plugins) + + if cfg.IngestWS != "" { + // Shadow: replay the Python /ws/live firehose. Inventory forwarding + // + Discord alerts stay OFF (would double production writes/alerts; + // inventory isn't in the firehose anyway). + go srv.runShadowConsumer(ctx, cfg.IngestWS) + logger.Info("shadow ingest enabled", "source", cfg.IngestWS) + } else { + // Cutover: the real plugin connects to /ws/position. Forward + // inventory to the inventory service and post death/idle alerts. + srv.ingestor.invFwd = newInvForwarder(cfg.InventoryURL, logger, srv.hub.broadcast) + if cfg.DiscordACLog != "" { + srv.ingestor.aclog = newACLogPoster(cfg.DiscordACLog, logger) + go srv.ingestor.aclog.runIdleLoop(ctx, pool) + } + logger.Info("cutover ingest enabled", "inventory_url", cfg.InventoryURL, "aclog", cfg.DiscordACLog != "") + } + } else if cfg.IngestWS != "" { + logger.Error("SHADOW_INGEST_WS set but READ_ONLY=true; refusing to ingest into the production DB") + os.Exit(1) + } + + go srv.runCacheLoop(ctx) + go srv.runTotalsLoop(ctx) + logger.Info("db connected; cache loops started", + "read_only", cfg.ReadOnly, "live_interval", cacheInterval.String(), "totals_interval", totalsInterval.String()) + } + + mux := http.NewServeMux() + srv.registerRoutes(mux) + + httpSrv := &http.Server{ + Addr: cfg.Addr, + Handler: withRequestLogging(srv.authMiddleware(mux)), + ReadHeaderTimeout: 10 * time.Second, + } + + go func() { + if err := httpSrv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) { + logger.Error("http server failed", "err", err) + os.Exit(1) + } + }() + logger.Info("listening", "addr", cfg.Addr) + + <-ctx.Done() + logger.Info("shutdown signal received, draining") + shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + if err := httpSrv.Shutdown(shutdownCtx); err != nil { + logger.Error("graceful shutdown failed", "err", err) + } + logger.Info("stopped") +} + +// config holds runtime configuration sourced from environment variables, +// matching the Python service's env var names where they overlap. +type config struct { + Addr string // listen address, e.g. ":8770" + DatabaseURL string // dereth TimescaleDB DSN + ReadOnly bool // true = read-side parity (force read-only txns); false = ingest/shadow (owns its DB) + InventoryURL string // inventory-service base URL + StaticDir string // directory for static assets / openissues.json + SecretKey string // session-cookie signing key (must match the Python service) + SharedSecret string // plugin /ws/position auth + SharedSecretLegacy string // plugin auth rotation fallback + IngestWS string // optional: a /ws/live URL to shadow-ingest from (Python tracker) + SkipSchemaInit bool // cutover: trust the existing prod schema, run no DDL + DiscordACLog string // #aclog webhook for death/idle alerts (cutover only) +} + +func loadConfig() config { + return config{ + Addr: ":" + envOr("PORT", "8770"), + DatabaseURL: os.Getenv("DATABASE_URL"), + ReadOnly: envOr("READ_ONLY", "true") != "false", + InventoryURL: envOr("INVENTORY_SERVICE_URL", "http://inventory-service:8000"), + StaticDir: envOr("STATIC_DIR", "static"), + SecretKey: os.Getenv("SECRET_KEY"), + SharedSecret: os.Getenv("SHARED_SECRET"), + SharedSecretLegacy: os.Getenv("SHARED_SECRET_LEGACY"), + IngestWS: os.Getenv("SHADOW_INGEST_WS"), + SkipSchemaInit: envOr("SKIP_SCHEMA_INIT", "false") == "true", + DiscordACLog: os.Getenv("DISCORD_ACLOG_WEBHOOK"), + } +} + +func envOr(key, def string) string { + if v := os.Getenv(key); v != "" { + return v + } + return def +} + +func (s *Server) registerRoutes(mux *http.ServeMux) { + mux.HandleFunc("GET /health", s.handleHealth) + // Mirrors Python's GET /api-version (hyphenated so nginx never strips it). + mux.HandleFunc("GET /api-version", s.handleVersion) + + // Phase 1 read-side: the 5s caches. + mux.HandleFunc("GET /live", s.handleLive) + mux.HandleFunc("GET /live/", s.handleLive) + mux.HandleFunc("GET /trails", s.handleTrails) + mux.HandleFunc("GET /trails/", s.handleTrails) + + // Totals (5-minute caches). + mux.HandleFunc("GET /total-rares", s.handleTotalRares) + mux.HandleFunc("GET /total-rares/", s.handleTotalRares) + mux.HandleFunc("GET /total-kills", s.handleTotalKills) + mux.HandleFunc("GET /total-kills/", s.handleTotalKills) + + // Per-character & aggregate DB reads. + mux.HandleFunc("GET /stats/{character_name}", s.handleStats) + mux.HandleFunc("GET /portals", s.handlePortals) + mux.HandleFunc("GET /spawns/heatmap", s.handleSpawnHeatmap) + mux.HandleFunc("GET /server-health", s.handleServerHealth) + mux.HandleFunc("GET /character-stats/{name}", s.handleCharacterStats) + mux.HandleFunc("GET /combat-stats", s.handleCombatStatsAll) + mux.HandleFunc("GET /combat-stats/{character_name}", s.handleCombatStatsOne) + mux.HandleFunc("GET /inventories", s.handleInventories) + mux.HandleFunc("GET /inventory/{character_name}/search", s.handleInventorySearch) + + // Ingest-only state (empty/default in Phase 1). + mux.HandleFunc("GET /quest-status", s.handleQuestStatus) + mux.HandleFunc("GET /vital-sharing/peers", s.handleVitalSharingPeers) + mux.HandleFunc("GET /equipment-cantrip-state/{name}", s.handleEquipmentCantrip) + mux.HandleFunc("GET /issues", s.handleIssues) + mux.HandleFunc("GET /me", s.handleMe) + + // WebSocket servers (cutover-ready): browser fan-out + plugin ingest. + mux.HandleFunc("GET /ws/live", s.handleWSLive) + mux.HandleFunc("GET /ws/position", s.handleWSPosition) + + // Inventory-service reverse proxies. + s.registerProxyRoutes(mux) + + // Website layer: login/logout + icons + static frontend (cutover). + mux.HandleFunc("GET /login", s.handleLoginGet) + mux.HandleFunc("POST /login", s.handleLoginPost) + mux.HandleFunc("GET /logout", s.handleLogout) + mux.HandleFunc("GET /icons/{filename}", s.handleIcon) + + // Admin user management. + mux.HandleFunc("GET /admin/users", s.handleAdminPage) + mux.HandleFunc("GET /api-admin/users", s.handleListUsers) + mux.HandleFunc("POST /api-admin/users", s.handleCreateUser) + mux.HandleFunc("PATCH /api-admin/users/{user_id}", s.handleUpdateUser) + mux.HandleFunc("DELETE /api-admin/users/{user_id}", s.handleDeleteUser) + + // Issue board write side (GET /issues is registered above). + mux.HandleFunc("POST /issues", s.handleAddIssue) + mux.HandleFunc("PATCH /issues/{issue_id}", s.handleUpdateIssue) + mux.HandleFunc("POST /issues/{issue_id}/comments", s.handleAddComment) + mux.HandleFunc("DELETE /issues/{issue_id}", s.handleDeleteIssue) + // Catch-all: serve the static frontend (SPA). Registered last; every + // specific route above is more specific, so this only handles the rest. + mux.HandleFunc("GET /", s.handleStatic) +} + +func (s *Server) handleHealth(w http.ResponseWriter, r *http.Request) { + writeJSON(w, http.StatusOK, map[string]any{ + "status": "ok", + "service": "tracker-go", + "version": buildVersion, + "db": s.pool != nil, + }) +} + +func (s *Server) handleVersion(w http.ResponseWriter, r *http.Request) { + writeJSON(w, http.StatusOK, map[string]any{"version": buildVersion}) +} + +func writeJSON(w http.ResponseWriter, status int, v any) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(status) + if err := json.NewEncoder(w).Encode(v); err != nil { + slog.Error("json encode failed", "err", err) + } +} + +// withRequestLogging is a thin access-log middleware. +func withRequestLogging(next http.Handler) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + start := time.Now() + sr := &statusRecorder{ResponseWriter: w, status: http.StatusOK} + next.ServeHTTP(sr, r) + slog.Info("http", + "method", r.Method, + "path", r.URL.Path, + "status", sr.status, + "dur_ms", time.Since(start).Milliseconds(), + ) + }) +} + +type statusRecorder struct { + http.ResponseWriter + status int +} + +func (s *statusRecorder) WriteHeader(code int) { + s.status = code + s.ResponseWriter.WriteHeader(code) +} + +// Unwrap lets http.ResponseController (used by coder/websocket to hijack the +// connection for /ws upgrades) reach the underlying ResponseWriter through this +// logging wrapper. Without it, WebSocket handshakes fail. +func (s *statusRecorder) Unwrap() http.ResponseWriter { + return s.ResponseWriter +} diff --git a/go-services/tracker-go/memstate.go b/go-services/tracker-go/memstate.go new file mode 100644 index 00000000..27ba80d8 --- /dev/null +++ b/go-services/tracker-go/memstate.go @@ -0,0 +1,99 @@ +package main + +import ( + "encoding/json" + "net/http" + "os" + "path/filepath" + "sort" +) + +// These endpoints are backed by ingest-only in-memory state in the Python +// service (populated from /ws/position events). Phase 1 has no ingest, so they +// return the same empty/default shapes the Python service emits when no data is +// present — preserving the API contract for the frontend. + +// GET /quest-status (main.py:1940) +func (s *Server) handleQuestStatus(w http.ResponseWriter, r *http.Request) { + questData := map[string]any{} + playerCount := 0 + if s.ingestor != nil { + qd, n := s.ingestor.questData() + playerCount = n + for c, qs := range qd { + m := map[string]any{} + for k, v := range qs { + m[k] = v + } + questData[c] = m + } + } + writeJSON(w, http.StatusOK, map[string]any{ + "quest_data": questData, + "tracked_quests": []string{ + "Stipend Collection Timer", + "Blank Augmentation Gem Pickup Timer", + "Insatiable Eater Jaw", + }, + "player_count": playerCount, + }) +} + +// GET /vital-sharing/peers (main.py:1800) +func (s *Server) handleVitalSharingPeers(w http.ResponseWriter, r *http.Request) { + if s.ingestor == nil { + writeJSON(w, http.StatusOK, map[string]any{"peers": []any{}, "subscriber_count": 0}) + return + } + peers, subCount := s.ingestor.vitalSharingPeers() + sort.Slice(peers, func(i, j int) bool { + return toStr(peers[i]["character_name"]) < toStr(peers[j]["character_name"]) + }) + writeJSON(w, http.StatusOK, map[string]any{"peers": peers, "subscriber_count": subCount}) +} + +// GET /equipment-cantrip-state/{name} (main.py:4167) +func (s *Server) handleEquipmentCantrip(w http.ResponseWriter, r *http.Request) { + name := r.PathValue("name") + if s.ingestor != nil { + if v, ok := s.ingestor.getEquipmentCantrip(name); ok { + writeJSON(w, http.StatusOK, v) + return + } + } + writeJSON(w, http.StatusOK, map[string]any{ + "type": "equipment_cantrip_state", + "character_name": name, + "items": []any{}, + }) +} + +// GET /issues — flat-file issue board. (main.py:1709) +func (s *Server) handleIssues(w http.ResponseWriter, r *http.Request) { + issues := s.loadIssues() + writeJSON(w, http.StatusOK, map[string]any{"issues": issues}) +} + +func (s *Server) loadIssues() []any { + empty := []any{} + b, err := os.ReadFile(filepath.Join(s.staticDir, "openissues.json")) + if err != nil { + return empty + } + var v []any + if json.Unmarshal(b, &v) != nil { + return empty + } + return v +} + +// GET /me — current user from the session (main.py:1455). Internal-trust +// loopback requests carry no user identity, so they get 401 too. +func (s *Server) handleMe(w http.ResponseWriter, r *http.Request) { + u := currentUser(r) + if u == nil { + writeJSON(w, http.StatusUnauthorized, map[string]any{"detail": "Not authenticated"}) + return + } + writeJSON(w, http.StatusOK, map[string]any{"username": u.Username, "is_admin": u.IsAdmin}) +} diff --git a/go-services/tracker-go/proxy.go b/go-services/tracker-go/proxy.go new file mode 100644 index 00000000..f0995edb --- /dev/null +++ b/go-services/tracker-go/proxy.go @@ -0,0 +1,74 @@ +package main + +import ( + "net/http" + "net/http/httputil" + "net/url" +) + +// initInvProxy builds a streaming reverse proxy to the inventory-service. +// FlushInterval=-1 flushes writes immediately so SSE endpoints (the suitbuilder +// search stream) work. Connection errors map to 503, mirroring the Python +// service's "Inventory service unavailable". +func (s *Server) initInvProxy(target string) error { + u, err := url.Parse(target) + if err != nil { + return err + } + rp := httputil.NewSingleHostReverseProxy(u) + rp.FlushInterval = -1 + rp.ErrorHandler = func(w http.ResponseWriter, r *http.Request, err error) { + s.log.Error("inventory proxy error", "err", err, "path", r.URL.Path) + writeJSON(w, http.StatusServiceUnavailable, map[string]any{"detail": "Inventory service unavailable"}) + } + s.invProxy = rp + return nil +} + +// proxyInv returns a handler that rewrites the request path (via rewrite) and +// forwards it to the inventory-service, preserving method, query, headers, and +// body. The original /inv/* prefix etc. is mapped to the upstream path. +func (s *Server) proxyInv(rewrite func(r *http.Request) string) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + if s.invProxy == nil { + writeJSON(w, http.StatusServiceUnavailable, map[string]any{"detail": "Inventory service unavailable"}) + return + } + r.URL.Path = rewrite(r) + r.URL.RawPath = "" // force re-encode from the (decoded) Path + s.invProxy.ServeHTTP(w, r) + } +} + +func (s *Server) registerProxyRoutes(mux *http.ServeMux) { + mux.HandleFunc("GET /inventory/{character_name}", s.proxyInv(func(r *http.Request) string { + return "/inventory/" + r.PathValue("character_name") + })) + mux.HandleFunc("GET /inventory-characters", s.proxyInv(func(r *http.Request) string { + return "/characters/list" + })) + mux.HandleFunc("GET /search/items", s.proxyInv(func(r *http.Request) string { + return "/search/items" + })) + mux.HandleFunc("GET /search/equipped/{character_name}", s.proxyInv(func(r *http.Request) string { + return "/search/equipped/" + r.PathValue("character_name") + })) + mux.HandleFunc("GET /search/upgrades/{character_name}/{slot}", s.proxyInv(func(r *http.Request) string { + return "/search/upgrades/" + r.PathValue("character_name") + "/" + r.PathValue("slot") + })) + mux.HandleFunc("GET /sets/list", s.proxyInv(func(r *http.Request) string { + return "/sets/list" + })) + + // /inv/test is a static liveness probe in the Python service. + mux.HandleFunc("GET /inv/test", func(w http.ResponseWriter, r *http.Request) { + writeJSON(w, http.StatusOK, map[string]any{"message": "Inventory proxy route is working"}) + }) + // Generic catch-all proxy: /inv/{path...} -> {SVC}/{path}. Covers GET and + // POST (incl. the SSE suitbuilder search). Registered for both methods. + invAll := s.proxyInv(func(r *http.Request) string { + return "/" + r.PathValue("path") + }) + mux.HandleFunc("GET /inv/{path...}", invAll) + mux.HandleFunc("POST /inv/{path...}", invAll) +} diff --git a/go-services/tracker-go/reads.go b/go-services/tracker-go/reads.go new file mode 100644 index 00000000..0de945f1 --- /dev/null +++ b/go-services/tracker-go/reads.go @@ -0,0 +1,367 @@ +package main + +import ( + "context" + "fmt" + "net/http" + "strconv" + "strings" + "time" +) + +// coerceNum converts a JSON value to a float64, parsing string-encoded numbers. +// The plugin sends several telemetry fields as strings (kills_per_hour, deaths, +// total_deaths, prismatic_taper_count via .ToString()); Python's pydantic +// coerced them, so Go must too or it writes null/0 (causing the live counters +// to flap 0<->value between the WS broadcast and the DB-derived /live poll). +func coerceNum(v any) (float64, bool) { + switch x := v.(type) { + case float64: + return x, true + case float32: + return float64(x), true + case int: + return float64(x), true + case int32: + return float64(x), true + case int64: + return float64(x), true + case string: + s := strings.TrimSpace(x) + if s == "" { + return 0, false + } + f, err := strconv.ParseFloat(s, 64) + return f, err == nil + } + return 0, false +} + +// reqCtx returns a child of the request context with a query timeout. +func reqCtx(r *http.Request) (context.Context, context.CancelFunc) { + return context.WithTimeout(r.Context(), 15*time.Second) +} + +func (s *Server) dbErr(w http.ResponseWriter, where string, err error) { + s.log.Error("db query failed", "where", where, "err", err) + writeJSON(w, http.StatusInternalServerError, map[string]any{"detail": "Internal server error"}) +} + +// GET /stats/{character_name} — latest telemetry snapshot + lifetime totals. (main.py:3927) +func (s *Server) handleStats(w http.ResponseWriter, r *http.Request) { + cn := r.PathValue("character_name") + ctx, cancel := reqCtx(r) + defer cancel() + + const sql = ` +WITH latest AS ( + SELECT * FROM telemetry_events + WHERE character_name = $1 + ORDER BY timestamp DESC LIMIT 1 +) +SELECT l.*, + COALESCE(cs.total_kills, 0) AS total_kills, + COALESCE(rs.total_rares, 0) AS total_rares +FROM latest l +LEFT JOIN char_stats cs ON l.character_name = cs.character_name +LEFT JOIN rare_stats rs ON l.character_name = rs.character_name` + + row, err := queryRowAsMap(ctx, s.pool, sql, cn) + if err != nil { + s.dbErr(w, "stats", err) + return + } + if row == nil { + writeJSON(w, http.StatusNotFound, map[string]any{"detail": "Character not found"}) + return + } + totalKills := row["total_kills"] + totalRares := row["total_rares"] + delete(row, "total_kills") + delete(row, "total_rares") + formatTimes([]map[string]any{row}, "timestamp", "received_at") + writeJSON(w, http.StatusOK, map[string]any{ + "character_name": cn, + "latest_snapshot": row, + "total_kills": totalKills, + "total_rares": totalRares, + }) +} + +// GET /portals — all active portals (cleanup job handles 1h expiry). (main.py:1959) +func (s *Server) handlePortals(w http.ResponseWriter, r *http.Request) { + ctx, cancel := reqCtx(r) + defer cancel() + + rows, err := queryRowsAsMaps(ctx, s.pool, + `SELECT portal_name, ns, ew, z, discovered_at, discovered_by FROM portals ORDER BY discovered_at DESC`) + if err != nil { + s.dbErr(w, "portals", err) + return + } + portals := make([]map[string]any, 0, len(rows)) + for _, row := range rows { + da := "" + if t, ok := row["discovered_at"].(time.Time); ok { + da = pyISO(t) + } + portals = append(portals, map[string]any{ + "portal_name": row["portal_name"], + "coordinates": map[string]any{"ns": row["ns"], "ew": row["ew"], "z": row["z"]}, + "discovered_at": da, + "discovered_by": row["discovered_by"], + }) + } + writeJSON(w, http.StatusOK, map[string]any{"portals": portals, "portal_count": len(portals)}) +} + +// GET /spawns/heatmap?hours=&limit= — aggregated spawn density. (main.py:2037) +func (s *Server) handleSpawnHeatmap(w http.ResponseWriter, r *http.Request) { + hours := clampInt(queryInt(r, "hours", 24), 1, 168) + limit := clampInt(queryInt(r, "limit", 10000), 100, 50000) + ctx, cancel := reqCtx(r) + defer cancel() + + cutoff := time.Now().UTC().Add(-time.Duration(hours) * time.Hour) + rows, err := queryRowsAsMaps(ctx, s.pool, + `SELECT ew, ns, COUNT(*) AS spawn_count FROM spawn_events + WHERE timestamp >= $1 GROUP BY ew, ns ORDER BY spawn_count DESC LIMIT $2`, + cutoff, limit) + if err != nil { + s.dbErr(w, "spawns/heatmap", err) + return + } + points := make([]map[string]any, 0, len(rows)) + for _, row := range rows { + points = append(points, map[string]any{ + "ew": toFloat(row["ew"]), + "ns": toFloat(row["ns"]), + "intensity": toInt(row["spawn_count"]), + }) + } + writeJSON(w, http.StatusOK, map[string]any{ + "spawn_points": points, + "total_points": len(points), + "timestamp": pyISO(time.Now().UTC()), + "hours_window": hours, + }) +} + +// GET /server-health — current Coldeve status + computed uptime. (main.py:1881) +func (s *Server) handleServerHealth(w http.ResponseWriter, r *http.Request) { + ctx, cancel := reqCtx(r) + defer cancel() + + row, err := queryRowAsMap(ctx, s.pool, `SELECT * FROM server_status WHERE server_name = $1`, "Coldeve") + if err != nil { + s.dbErr(w, "server-health", err) + return + } + + status := "unknown" + var latency, playerCount, lastRestart, lastCheck any + var uptimeSeconds int64 + if row != nil { + if v, ok := row["current_status"].(string); ok && v != "" { + status = v + } + latency = row["last_latency_ms"] + playerCount = row["last_player_count"] + uptimeSeconds = toInt64(row["total_uptime_seconds"]) + if t, ok := row["last_restart"].(time.Time); ok { + lastRestart = pyISO(t) + } + if t, ok := row["last_check"].(time.Time); ok { + lastCheck = pyISO(t) + } + } + days := uptimeSeconds / 86400 + hours := (uptimeSeconds % 86400) / 3600 + minutes := (uptimeSeconds % 3600) / 60 + uptime := fmt.Sprintf("%dh %dm", hours, minutes) + if days > 0 { + uptime = fmt.Sprintf("%dd %dh %dm", days, hours, minutes) + } + writeJSON(w, http.StatusOK, map[string]any{ + "server_name": "Coldeve", + "status": status, + "latency_ms": latency, + "player_count": playerCount, + "uptime": uptime, + "uptime_seconds": uptimeSeconds, + "last_restart": lastRestart, + "last_check": lastCheck, + }) +} + +// GET /inventories — characters with stored inventories. (main.py:2212) +func (s *Server) handleInventories(w http.ResponseWriter, r *http.Request) { + ctx, cancel := reqCtx(r) + defer cancel() + + rows, err := queryRowsAsMaps(ctx, s.pool, + `SELECT character_name, COUNT(*) AS item_count, MAX(timestamp) AS last_updated + FROM character_inventories GROUP BY character_name ORDER BY last_updated DESC`) + if err != nil { + s.dbErr(w, "inventories", err) + return + } + formatTimes(rows, "last_updated") + chars := make([]map[string]any, 0, len(rows)) + for _, row := range rows { + chars = append(chars, map[string]any{ + "character_name": row["character_name"], + "item_count": row["item_count"], + "last_updated": row["last_updated"], + }) + } + writeJSON(w, http.StatusOK, map[string]any{"characters": chars, "total_characters": len(chars)}) +} + +// GET /inventory/{character_name}/search — filtered local inventory rows. (main.py:2135) +func (s *Server) handleInventorySearch(w http.ResponseWriter, r *http.Request) { + cn := r.PathValue("character_name") + name := optStr(r, "name") + objectClass := optInt(r, "object_class") + minValue := optInt(r, "min_value") + maxValue := optInt(r, "max_value") + minBurden := optInt(r, "min_burden") + maxBurden := optInt(r, "max_burden") + + conds := []string{"character_name = $1"} + args := []any{cn} + add := func(tmpl string, val any) { + args = append(args, val) + conds = append(conds, fmt.Sprintf(tmpl, len(args))) + } + if name != nil && *name != "" { + add("name ILIKE $%d", "%"+*name+"%") + } + if objectClass != nil { + add("object_class = $%d", *objectClass) + } + if minValue != nil { + add("value >= $%d", *minValue) + } + if maxValue != nil { + add("value <= $%d", *maxValue) + } + if minBurden != nil { + add("burden >= $%d", *minBurden) + } + if maxBurden != nil { + add("burden <= $%d", *maxBurden) + } + + sql := `SELECT name, icon, object_class, value, burden, has_id_data, item_data, timestamp + FROM character_inventories WHERE ` + join(conds, " AND ") + ` ORDER BY value DESC, name` + + ctx, cancel := reqCtx(r) + defer cancel() + rows, err := queryRowsAsMaps(ctx, s.pool, sql, args...) + if err != nil { + s.dbErr(w, "inventory-search", err) + return + } + formatTimes(rows, "timestamp") + for _, row := range rows { + if v, ok := row["item_data"]; ok { + row["item_data"] = decodeJSONValue(v) + } + } + writeJSON(w, http.StatusOK, map[string]any{ + "character_name": cn, + "item_count": len(rows), + "search_criteria": map[string]any{ + "name": derefStr(name), + "object_class": derefInt(objectClass), + "min_value": derefInt(minValue), + "max_value": derefInt(maxValue), + "min_burden": derefInt(minBurden), + "max_burden": derefInt(maxBurden), + }, + "items": rows, + }) +} + +// ---- small param/number helpers ---- + +func queryInt(r *http.Request, key string, def int) int { + if v := r.URL.Query().Get(key); v != "" { + if n, err := strconv.Atoi(v); err == nil { + return n + } + } + return def +} + +func optInt(r *http.Request, key string) *int { + v := r.URL.Query().Get(key) + if v == "" { + return nil + } + n, err := strconv.Atoi(v) + if err != nil { + return nil + } + return &n +} + +func optStr(r *http.Request, key string) *string { + vs := r.URL.Query() + if !vs.Has(key) { + return nil + } + v := vs.Get(key) + return &v +} + +func derefStr(p *string) any { + if p == nil { + return nil + } + return *p +} + +func derefInt(p *int) any { + if p == nil { + return nil + } + return *p +} + +func clampInt(v, lo, hi int) int { + if v < lo { + return lo + } + if v > hi { + return hi + } + return v +} + +func join(parts []string, sep string) string { + out := "" + for i, p := range parts { + if i > 0 { + out += sep + } + out += p + } + return out +} + +func toFloat(v any) float64 { + f, _ := coerceNum(v) + return f +} + +func toInt(v any) int { + f, _ := coerceNum(v) + return int(f) +} + +func toInt64(v any) int64 { + f, _ := coerceNum(v) + return int64(f) +} diff --git a/go-services/tracker-go/schema.go b/go-services/tracker-go/schema.go new file mode 100644 index 00000000..df3326dc --- /dev/null +++ b/go-services/tracker-go/schema.go @@ -0,0 +1,197 @@ +package main + +import ( + "context" + "log/slog" + "strings" + + "github.com/jackc/pgx/v5/pgxpool" +) + +// initSchema creates the dereth schema on an ingest-owned database, faithfully +// replicating db_async.init_db_async (idempotent DDL). It runs ONLY for an +// instance that owns its DB (read-write shadow/ingest mode) — never against the +// production dereth DB. Like the Python init, it logs and continues per +// statement so an optional step (e.g. a timescale policy) can't abort the rest. +// +// One deliberate divergence from db_async.py: the portal unique index uses +// ROUND(..,1), matching main.py's ON CONFLICT target, so portal upserts resolve +// on a fresh DB (db_async.py creates ROUND(..,2) — the known production drift). +func initSchema(ctx context.Context, pool *pgxpool.Pool, log *slog.Logger) { + stmts := []string{ + `CREATE EXTENSION IF NOT EXISTS timescaledb`, + + `CREATE TABLE IF NOT EXISTS telemetry_events ( + character_name VARCHAR NOT NULL, + char_tag VARCHAR, + session_id VARCHAR NOT NULL, + timestamp TIMESTAMPTZ NOT NULL, + ew DOUBLE PRECISION NOT NULL, + ns DOUBLE PRECISION NOT NULL, + z DOUBLE PRECISION NOT NULL, + kills INTEGER NOT NULL, + kills_per_hour DOUBLE PRECISION, + onlinetime VARCHAR, + deaths INTEGER NOT NULL, + total_deaths INTEGER, + rares_found INTEGER NOT NULL, + prismatic_taper_count INTEGER NOT NULL, + vt_state VARCHAR, + mem_mb DOUBLE PRECISION, + cpu_pct DOUBLE PRECISION, + mem_handles INTEGER, + latency_ms DOUBLE PRECISION, + received_at TIMESTAMPTZ + )`, + `SELECT create_hypertable('telemetry_events','timestamp', if_not_exists => true, migrate_data => true, create_default_indexes => false)`, + `CREATE INDEX IF NOT EXISTS ix_telemetry_events_char_ts ON telemetry_events (character_name, timestamp)`, + `CREATE INDEX IF NOT EXISTS ix_telemetry_events_character_name ON telemetry_events (character_name)`, + `CREATE INDEX IF NOT EXISTS ix_telemetry_events_session_id ON telemetry_events (session_id)`, + `CREATE INDEX IF NOT EXISTS ix_telemetry_events_timestamp ON telemetry_events (timestamp)`, + `SELECT add_retention_policy('telemetry_events', INTERVAL '7 days', if_not_exists => TRUE)`, + // Compression must be enabled on the hypertable before a policy can be added. + `ALTER TABLE telemetry_events SET (timescaledb.compress, timescaledb.compress_segmentby = 'character_name')`, + `SELECT add_compression_policy('telemetry_events', INTERVAL '1 day', if_not_exists => TRUE)`, + + `CREATE TABLE IF NOT EXISTS char_stats ( + character_name VARCHAR PRIMARY KEY, + total_kills INTEGER NOT NULL DEFAULT 0 + )`, + `CREATE TABLE IF NOT EXISTS rare_stats ( + character_name VARCHAR PRIMARY KEY, + total_rares INTEGER NOT NULL DEFAULT 0 + )`, + `CREATE TABLE IF NOT EXISTS rare_stats_sessions ( + character_name VARCHAR NOT NULL, + session_id VARCHAR NOT NULL, + session_rares INTEGER NOT NULL DEFAULT 0, + PRIMARY KEY (character_name, session_id) + )`, + `CREATE TABLE IF NOT EXISTS combat_stats ( + character_name VARCHAR PRIMARY KEY, + timestamp TIMESTAMPTZ NOT NULL, + stats_data JSONB NOT NULL + )`, + `CREATE TABLE IF NOT EXISTS combat_stats_sessions ( + id SERIAL PRIMARY KEY, + character_name VARCHAR NOT NULL, + session_id VARCHAR NOT NULL, + timestamp TIMESTAMPTZ NOT NULL, + stats_data JSONB NOT NULL + )`, + `CREATE INDEX IF NOT EXISTS ix_combat_stats_sessions_character_name ON combat_stats_sessions (character_name)`, + `CREATE INDEX IF NOT EXISTS ix_combat_stats_sessions_session_id ON combat_stats_sessions (session_id)`, + `CREATE INDEX IF NOT EXISTS ix_combat_stats_sessions_timestamp ON combat_stats_sessions (timestamp)`, + + // No sole-id PRIMARY KEY: TimescaleDB requires the partition column + // (timestamp) in every unique index, so a bare id PK blocks hypertable + // conversion. id stays an auto-increment column for an append-only log. + `CREATE TABLE IF NOT EXISTS spawn_events ( + id BIGSERIAL, + character_name VARCHAR NOT NULL, + mob VARCHAR NOT NULL, + timestamp TIMESTAMPTZ NOT NULL, + ew DOUBLE PRECISION NOT NULL, + ns DOUBLE PRECISION NOT NULL, + z DOUBLE PRECISION NOT NULL + )`, + `SELECT create_hypertable('spawn_events','timestamp', if_not_exists => TRUE, migrate_data => FALSE, chunk_time_interval => INTERVAL '1 day')`, + `CREATE INDEX IF NOT EXISTS ix_spawn_events_timestamp ON spawn_events (timestamp)`, + `SELECT add_retention_policy('spawn_events', INTERVAL '7 days', if_not_exists => TRUE)`, + + `CREATE TABLE IF NOT EXISTS rare_events ( + id SERIAL PRIMARY KEY, + character_name VARCHAR NOT NULL, + name VARCHAR NOT NULL, + timestamp TIMESTAMPTZ NOT NULL, + ew DOUBLE PRECISION NOT NULL, + ns DOUBLE PRECISION NOT NULL, + z DOUBLE PRECISION NOT NULL + )`, + `CREATE INDEX IF NOT EXISTS ix_rare_events_timestamp ON rare_events (timestamp)`, + + `CREATE TABLE IF NOT EXISTS character_inventories ( + id SERIAL PRIMARY KEY, + character_name VARCHAR NOT NULL, + item_id BIGINT NOT NULL, + timestamp TIMESTAMPTZ NOT NULL, + name VARCHAR, + icon INTEGER, + object_class INTEGER, + value INTEGER, + burden INTEGER, + has_id_data BOOLEAN, + item_data JSONB NOT NULL, + CONSTRAINT uq_char_item UNIQUE (character_name, item_id) + )`, + `CREATE INDEX IF NOT EXISTS ix_character_inventories_character_name ON character_inventories (character_name)`, + `CREATE INDEX IF NOT EXISTS ix_character_inventories_object_class ON character_inventories (object_class)`, + `CREATE INDEX IF NOT EXISTS ix_character_inventories_value ON character_inventories (value)`, + + `CREATE TABLE IF NOT EXISTS portals ( + id SERIAL PRIMARY KEY, + portal_name VARCHAR NOT NULL, + ns DOUBLE PRECISION NOT NULL, + ew DOUBLE PRECISION NOT NULL, + z DOUBLE PRECISION NOT NULL, + discovered_at TIMESTAMPTZ NOT NULL, + discovered_by VARCHAR NOT NULL + )`, + `CREATE INDEX IF NOT EXISTS ix_portals_discovered_at ON portals (discovered_at)`, + `CREATE UNIQUE INDEX IF NOT EXISTS unique_portal_coords ON portals (ROUND(ns::numeric, 1), ROUND(ew::numeric, 1))`, + `CREATE INDEX IF NOT EXISTS idx_portals_coords ON portals (ns, ew)`, + + `CREATE TABLE IF NOT EXISTS server_status ( + server_name VARCHAR PRIMARY KEY, + current_status VARCHAR(10) NOT NULL, + last_seen_up TIMESTAMPTZ, + last_restart TIMESTAMPTZ, + total_uptime_seconds BIGINT DEFAULT 0, + last_check TIMESTAMPTZ, + last_latency_ms DOUBLE PRECISION, + last_player_count INTEGER + )`, + + `CREATE TABLE IF NOT EXISTS character_stats ( + character_name VARCHAR(255) PRIMARY KEY, + timestamp TIMESTAMPTZ NOT NULL DEFAULT NOW(), + level INTEGER, + total_xp BIGINT, + unassigned_xp BIGINT, + luminance_earned BIGINT, + luminance_total BIGINT, + deaths INTEGER, + stats_data JSONB NOT NULL + )`, + + `CREATE TABLE IF NOT EXISTS users ( + id SERIAL PRIMARY KEY, + username VARCHAR NOT NULL UNIQUE, + password_hash VARCHAR NOT NULL, + is_admin BOOLEAN NOT NULL DEFAULT false, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW() + )`, + } + + ok, failed := 0, 0 + for _, s := range stmts { + if _, err := pool.Exec(ctx, s); err != nil { + failed++ + log.Warn("schema statement failed (continuing)", "stmt", firstLine(s), "err", err) + continue + } + ok++ + } + log.Info("schema init complete", "ok", ok, "failed", failed) +} + +func firstLine(s string) string { + s = strings.TrimSpace(s) + if i := strings.IndexByte(s, '\n'); i >= 0 { + return strings.TrimSpace(s[:i]) + } + if len(s) > 80 { + return s[:80] + } + return s +} diff --git a/go-services/tracker-go/shadow.go b/go-services/tracker-go/shadow.go new file mode 100644 index 00000000..619869b6 --- /dev/null +++ b/go-services/tracker-go/shadow.go @@ -0,0 +1,106 @@ +package main + +import ( + "context" + "encoding/json" + "time" + + "github.com/coder/websocket" +) + +// runShadowConsumer connects to the Python tracker's /ws/live, receives the full +// broadcast firehose (no subscribe = all types), and replays each message +// through the ingest handlers into THIS instance's own DB. This validates the +// Go ingest path against real traffic without touching production or stealing +// plugin connections. Reconnects with exponential backoff. +// +// Note: telemetry broadcasts carry no "type" field (dispatch matches by shape); +// spawn and full_inventory are NOT broadcast, so they don't arrive here (covered +// by unit tests / the future /ws/position path). +func (s *Server) runShadowConsumer(ctx context.Context, wsURL string) { + backoff := time.Second + const maxBackoff = 60 * time.Second + for ctx.Err() == nil { + err := s.shadowConnect(ctx, wsURL) + if ctx.Err() != nil { + return + } + s.log.Warn("shadow consumer disconnected; reconnecting", "err", err, "backoff", backoff.String()) + select { + case <-ctx.Done(): + return + case <-time.After(backoff): + } + backoff *= 2 + if backoff > maxBackoff { + backoff = maxBackoff + } + } +} + +func (s *Server) shadowConnect(ctx context.Context, wsURL string) error { + c, _, err := websocket.Dial(ctx, wsURL, nil) + if err != nil { + return err + } + defer c.CloseNow() + c.SetReadLimit(32 << 20) // nearby_objects / dungeon_map payloads can be large + + connCtx, cancel := context.WithCancel(ctx) + defer cancel() + // No outbound keepalive ping: the firehose is constant, so the connection is + // never idle, and the read-deadline watchdog below handles dead connections. + + // Decouple socket read from ALL processing, including JSON parsing: the read + // loop only copies raw frames onto a queue, so it drains the socket as fast + // as the network delivers. If parsing or DB-bound dispatch ran inline, the + // read would stall, the upstream /ws/live broadcast send would error, and + // Python would evict us (Read then blocks forever). A single worker + // unmarshals + dispatches in order, preserving per-char kill-delta / combat + // accumulation. + queue := make(chan []byte, 16384) + done := make(chan struct{}) + go func() { + defer close(done) + for raw := range queue { + var m map[string]any + if json.Unmarshal(raw, &m) != nil { + continue + } + s.ingestor.dispatch(connCtx, m) + } + }() + + s.log.Info("shadow consumer connected; replaying /ws/live into ingest", "url", wsURL) + var n, dropped int + loopErr := s.shadowReadLoop(ctx, c, queue, &n, &dropped) + close(queue) + <-done + return loopErr +} + +func (s *Server) shadowReadLoop(ctx context.Context, c *websocket.Conn, queue chan []byte, n, dropped *int) error { + for { + // Read deadline acts as a liveness watchdog: the firehose is constant, so + // a multi-second silence means the upstream evicted us without closing — + // time out quickly and let runShadowConsumer reconnect (high duty cycle). + rctx, rcancel := context.WithTimeout(ctx, 12*time.Second) + _, raw, err := c.Read(rctx) + rcancel() + if err != nil { + return err + } + select { + case queue <- raw: + default: + *dropped++ + if *dropped%1000 == 1 { + s.log.Warn("shadow queue full; dropping messages", "dropped", *dropped) + } + } + *n++ + if *n%5000 == 0 { + s.log.Info("shadow consumer progress", "messages", *n, "queued", len(queue), "dropped", *dropped) + } + } +} diff --git a/go-services/tracker-go/share.go b/go-services/tracker-go/share.go new file mode 100644 index 00000000..63090ea0 --- /dev/null +++ b/go-services/tracker-go/share.go @@ -0,0 +1,111 @@ +package main + +// Cross-machine vital sharing (share_*), a faithful port of main.py:3658-3703 + +// _update_vital_sharing_peer_state / _broadcast_share_to_plugin_clients. +// Memory-only: subscriber set + last-known peer snapshot, fanned out to other +// opted-in plugin clients and to browsers. In shadow mode there are no plugin +// connections, so the fan-out is a no-op; the peer state still drives +// /vital-sharing/peers. + +func (i *Ingestor) handleShareSubscribe(data map[string]any) { + char := toStr(data["character_name"]) + if char == "" { + return + } + i.mu.Lock() + i.vitalSubscribers[char] = true + entry := i.vitalPeerEntry(char) + if tags, ok := data["tags"].([]any); ok { + entry["tags"] = tags + } + entry["connected"] = true + i.mu.Unlock() +} + +func (i *Ingestor) handleShareUnsubscribe(data map[string]any) { + char := toStr(data["character_name"]) + if char == "" { + return + } + i.mu.Lock() + delete(i.vitalSubscribers, char) + delete(i.vitalPeerState, char) + i.mu.Unlock() + if i.broadcast != nil { + i.broadcast(map[string]any{"type": "share_peer_removed", "character_name": char}) + } +} + +func (i *Ingestor) handleShareUpdate(msgType string, data map[string]any) { + origin := toStr(data["character_name"]) + i.mu.Lock() + i.updateVitalPeerState(msgType, data) + // Snapshot subscribers for the fan-out. + subs := make(map[string]bool, len(i.vitalSubscribers)) + for k := range i.vitalSubscribers { + subs[k] = true + } + i.mu.Unlock() + // Fan out to other opted-in plugin clients (no-op when no plugins connected). + if i.plugins != nil && len(subs) > 0 { + i.plugins.fanoutShare(data, origin, subs) + } +} + +// vitalPeerEntry returns (creating if needed) the peer snapshot for char. Caller +// holds i.mu. +func (i *Ingestor) vitalPeerEntry(char string) map[string]any { + entry, ok := i.vitalPeerState[char] + if !ok { + entry = map[string]any{ + "character_name": char, "tags": []any{}, "vitals": nil, + "position": nil, "items": nil, "connected": true, "last_update": nil, + } + i.vitalPeerState[char] = entry + } + return entry +} + +// updateVitalPeerState mirrors _update_vital_sharing_peer_state. Caller holds i.mu. +func (i *Ingestor) updateVitalPeerState(msgType string, data map[string]any) { + char := toStr(data["character_name"]) + if char == "" { + return + } + entry := i.vitalPeerEntry(char) + entry["last_update"] = data["timestamp"] + if tags, ok := data["tags"].([]any); ok { + entry["tags"] = tags + } + switch msgType { + case "share_vital_update": + entry["vitals"] = map[string]any{ + "current_health": data["current_health"], "max_health": data["max_health"], + "current_stamina": data["current_stamina"], "max_stamina": data["max_stamina"], + "current_mana": data["current_mana"], "max_mana": data["max_mana"], + } + case "share_position_update": + entry["position"] = map[string]any{ + "ew": data["ew"], "ns": data["ns"], "z": data["z"], "heading": data["heading"], + } + case "share_item_update": + entry["items"] = data["items"] + } +} + +// vitalSharingPeers returns the peer list for /vital-sharing/peers (main.py:1800). +func (i *Ingestor) vitalSharingPeers() ([]map[string]any, int) { + i.mu.RLock() + defer i.mu.RUnlock() + peers := make([]map[string]any, 0, len(i.vitalPeerState)) + for char, entry := range i.vitalPeerState { + p := make(map[string]any, len(entry)+2) + for k, v := range entry { + p[k] = v + } + p["subscribed"] = i.vitalSubscribers[char] + p["plugin_connected"] = i.plugins != nil && i.plugins.isConnected(char) + peers = append(peers, p) + } + return peers, len(i.vitalSubscribers) +} diff --git a/go-services/tracker-go/store.go b/go-services/tracker-go/store.go new file mode 100644 index 00000000..6979622b --- /dev/null +++ b/go-services/tracker-go/store.go @@ -0,0 +1,145 @@ +package main + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "time" + + "github.com/jackc/pgx/v5" + "github.com/jackc/pgx/v5/pgxpool" +) + +// newPool creates a pgx pool against a dereth TimescaleDB. +// +// When readOnly is true (the default — read-side parity against the live +// production dereth DB), every pooled connection is forced into read-only +// transaction mode as defense-in-depth, so even a buggy write cannot mutate the +// data the Python service owns. When false (ingest/shadow mode against this +// instance's OWN database), writes are permitted. +func newPool(ctx context.Context, dsn string, readOnly bool) (*pgxpool.Pool, error) { + cfg, err := pgxpool.ParseConfig(dsn) + if err != nil { + return nil, fmt.Errorf("parse DATABASE_URL: %w", err) + } + cfg.MaxConns = 10 + cfg.MaxConnIdleTime = 5 * time.Minute + if readOnly { + cfg.AfterConnect = func(ctx context.Context, conn *pgx.Conn) error { + if _, err := conn.Exec(ctx, "SET default_transaction_read_only = on"); err != nil { + return fmt.Errorf("set read-only: %w", err) + } + return nil + } + } + pool, err := pgxpool.NewWithConfig(ctx, cfg) + if err != nil { + return nil, fmt.Errorf("create pool: %w", err) + } + return pool, nil +} + +// queryRowsAsMaps runs a query and returns each row as a column-name->value map, +// mirroring how the Python service builds response dicts directly from rows. +// A nil result is coerced to an empty (non-nil) slice so JSON encodes "[]". +func queryRowsAsMaps(ctx context.Context, pool *pgxpool.Pool, sql string, args ...any) ([]map[string]any, error) { + rows, err := pool.Query(ctx, sql, args...) + if err != nil { + return nil, err + } + out, err := pgx.CollectRows(rows, pgx.RowToMap) + if err != nil { + return nil, err + } + if out == nil { + out = []map[string]any{} + } + return out, nil +} + +// queryRowAsMap runs a query expected to return at most one row. It returns +// (nil, nil) when there are no rows, so callers can map that to a 404. +func queryRowAsMap(ctx context.Context, pool *pgxpool.Pool, sql string, args ...any) (map[string]any, error) { + rows, err := pool.Query(ctx, sql, args...) + if err != nil { + return nil, err + } + m, err := pgx.CollectExactlyOneRow(rows, pgx.RowToMap) + if err != nil { + if errors.Is(err, pgx.ErrNoRows) { + return nil, nil + } + return nil, err + } + return m, nil +} + +// asJSONMap coerces a value that may be JSON bytes, a JSON string, or an +// already-decoded map into a map[string]any. Used for JSONB columns where pgx's +// decoding can vary. Returns nil if the value can't be interpreted as an object. +func asJSONMap(v any) map[string]any { + switch x := v.(type) { + case nil: + return nil + case map[string]any: + return x + case []byte: + var m map[string]any + if json.Unmarshal(x, &m) == nil { + return m + } + case string: + var m map[string]any + if json.Unmarshal([]byte(x), &m) == nil { + return m + } + } + return nil +} + +// decodeJSONValue coerces a JSON/JSONB column value into its natural Go value +// (map, slice, scalar). Bytes/strings are unmarshaled; anything else is +// returned unchanged. +func decodeJSONValue(v any) any { + switch x := v.(type) { + case []byte: + var out any + if json.Unmarshal(x, &out) == nil { + return out + } + case string: + var out any + if json.Unmarshal([]byte(x), &out) == nil { + return out + } + } + return v +} + +// pyISO formats a timestamp the way Python's datetime.isoformat() does for a +// UTC tz-aware value, so output matches FastAPI's jsonable_encoder: +// - no fractional part when microseconds are zero +// - otherwise exactly 6 fractional digits +// - "+00:00" offset (not "Z") +// Postgres timestamptz has microsecond resolution, so ns is always a multiple +// of 1000. +func pyISO(t time.Time) string { + t = t.UTC() + if t.Nanosecond() == 0 { + return t.Format("2006-01-02T15:04:05+00:00") + } + return t.Format("2006-01-02T15:04:05") + fmt.Sprintf(".%06d+00:00", t.Nanosecond()/1000) +} + +// formatTimes rewrites the named time.Time columns in-place to pyISO strings. +// Missing or NULL (nil) values are left untouched, so they encode as JSON null. +func formatTimes(rows []map[string]any, keys ...string) { + for _, m := range rows { + for _, k := range keys { + if t, ok := m[k].(time.Time); ok { + m[k] = pyISO(t) + } + } + } +} diff --git a/go-services/tracker-go/totals.go b/go-services/tracker-go/totals.go new file mode 100644 index 00000000..34caa6e7 --- /dev/null +++ b/go-services/tracker-go/totals.go @@ -0,0 +1,80 @@ +package main + +import ( + "context" + "encoding/json" + "net/http" + "sync" + "time" +) + +const totalsInterval = 300 * time.Second // _refresh_total_rares_cache cadence + +// totalsCache holds the pre-marshaled bodies for /total-rares and /total-kills, +// refreshed every totalsInterval, mirroring main.py:924. +type totalsCache struct { + mu sync.RWMutex + raresJSON []byte + killsJSON []byte +} + +func newTotalsCache() *totalsCache { + return &totalsCache{ + raresJSON: []byte(`{"all_time":0,"today":0,"last_updated":null}`), + killsJSON: []byte(`{"total":0,"last_updated":null}`), + } +} + +func (c *totalsCache) getRares() []byte { c.mu.RLock(); defer c.mu.RUnlock(); return c.raresJSON } +func (c *totalsCache) getKills() []byte { c.mu.RLock(); defer c.mu.RUnlock(); return c.killsJSON } + +func (c *totalsCache) set(rares, kills []byte) { + c.mu.Lock() + defer c.mu.Unlock() + c.raresJSON = rares + c.killsJSON = kills +} + +func (s *Server) refreshTotals(ctx context.Context) error { + qctx, cancel := context.WithTimeout(ctx, 15*time.Second) + defer cancel() + + var allTime, today, totalKills int64 + // Each query degrades to 0 on error, mirroring the Python try/except blocks. + _ = s.pool.QueryRow(qctx, "SELECT COALESCE(SUM(total_rares), 0) FROM rare_stats").Scan(&allTime) + _ = s.pool.QueryRow(qctx, "SELECT COUNT(*) FROM rare_events WHERE timestamp >= CURRENT_DATE").Scan(&today) + _ = s.pool.QueryRow(qctx, "SELECT COALESCE(SUM(total_kills), 0) FROM char_stats").Scan(&totalKills) + + lastUpdated := pyISO(time.Now().UTC()) + raresJSON, err := json.Marshal(map[string]any{"all_time": allTime, "today": today, "last_updated": lastUpdated}) + if err != nil { + return err + } + killsJSON, err := json.Marshal(map[string]any{"total": totalKills, "last_updated": lastUpdated}) + if err != nil { + return err + } + s.totals.set(raresJSON, killsJSON) + return nil +} + +func (s *Server) runTotalsLoop(ctx context.Context) { + for { + if err := s.refreshTotals(ctx); err != nil { + s.log.Error("totals cache refresh failed", "err", err) + } + select { + case <-ctx.Done(): + return + case <-time.After(totalsInterval): + } + } +} + +func (s *Server) handleTotalRares(w http.ResponseWriter, r *http.Request) { + writeRawJSON(w, s.totals.getRares()) +} + +func (s *Server) handleTotalKills(w http.ResponseWriter, r *http.Request) { + writeRawJSON(w, s.totals.getKills()) +} diff --git a/go-services/tracker-go/website.go b/go-services/tracker-go/website.go new file mode 100644 index 00000000..f3f9e2cd --- /dev/null +++ b/go-services/tracker-go/website.go @@ -0,0 +1,164 @@ +package main + +import ( + "context" + "encoding/json" + "net/http" + "os" + "path" + "path/filepath" + "strings" + "sync" + "time" + + "golang.org/x/crypto/bcrypt" +) + +// Website-serving layer: static frontend + login/logout, porting main.py so the +// unchanged frontend loads on the Go tracker. Cookie issuing/verifying is in +// auth.go; this file is the handlers + the static file server. + +// A fixed bcrypt hash used to keep the no-such-user path constant-time, matching +// Python's _DUMMY_HASH. (Hash of an arbitrary constant; never matches input.) +var dummyBcryptHash = []byte("$2a$12$C6UzMDM.H6dfI/f/IKcEeO3Jj6Q1jK7Z1qkq9b2yY6m4eW7N0pZ2K") + +type loginLimiter struct { + mu sync.Mutex + last map[string]time.Time +} + +func newLoginLimiter() *loginLimiter { return &loginLimiter{last: map[string]time.Time{}} } + +// allow returns false if this IP attempted within the 5s cooldown (main.py). +func (l *loginLimiter) allow(ip string) bool { + l.mu.Lock() + defer l.mu.Unlock() + now := time.Now() + if t, ok := l.last[ip]; ok && now.Sub(t) < 5*time.Second { + return false + } + l.last[ip] = now + return true +} + +// GET /login — serve the login page (main.py:login_page). +func (s *Server) handleLoginGet(w http.ResponseWriter, r *http.Request) { + s.serveStaticFile(w, r, "login.html") +} + +// POST /login — authenticate and set the session cookie (main.py:login). +func (s *Server) handleLoginPost(w http.ResponseWriter, r *http.Request) { + ip := clientIP(r) + if xff := r.Header.Get("X-Forwarded-For"); xff != "" { + ip = strings.TrimSpace(strings.Split(xff, ",")[0]) + } + if !s.loginLimiter.allow(ip) { + writeJSON(w, http.StatusTooManyRequests, map[string]any{"detail": "Too many login attempts. Try again in a few seconds."}) + return + } + var body struct { + Username string `json:"username"` + Password string `json:"password"` + } + if json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<20)).Decode(&body) != nil { + writeJSON(w, http.StatusBadRequest, map[string]any{"detail": "Invalid request body"}) + return + } + username := strings.ToLower(strings.TrimSpace(body.Username)) + if username == "" || body.Password == "" { + writeJSON(w, http.StatusBadRequest, map[string]any{"detail": "Username and password required"}) + return + } + + ctx, cancel := context.WithTimeout(r.Context(), 10*time.Second) + defer cancel() + var dbUser, hash string + var isAdmin bool + err := s.pool.QueryRow(ctx, + "SELECT username, password_hash, is_admin FROM users WHERE LOWER(username) = $1", username, + ).Scan(&dbUser, &hash, &isAdmin) + // Constant-time: always run bcrypt, even when the user doesn't exist. + pwOK := false + if err == nil { + pwOK = bcrypt.CompareHashAndPassword([]byte(hash), []byte(body.Password)) == nil + } else { + _ = bcrypt.CompareHashAndPassword(dummyBcryptHash, []byte(body.Password)) + } + if !pwOK { + writeJSON(w, http.StatusUnauthorized, map[string]any{"detail": "Invalid username or password"}) + return + } + + token := issueSessionCookie(s.secretKey, sessionUser{Username: dbUser, IsAdmin: isAdmin}) + http.SetCookie(w, &http.Cookie{ + Name: "session", Value: token, Path: "/", MaxAge: sessionMaxAge, + HttpOnly: true, Secure: true, SameSite: http.SameSiteLaxMode, + }) + writeJSON(w, http.StatusOK, map[string]any{"ok": true, "username": dbUser, "is_admin": isAdmin}) +} + +// GET /logout — clear the cookie and redirect to /login (main.py:logout). +func (s *Server) handleLogout(w http.ResponseWriter, r *http.Request) { + http.SetCookie(w, &http.Cookie{Name: "session", Value: "", Path: "/", MaxAge: -1}) + http.Redirect(w, r, "/login", http.StatusFound) +} + +// GET /icons/{filename} — serve an icon file (main.py:serve_icon). +func (s *Server) handleIcon(w http.ResponseWriter, r *http.Request) { + name := r.PathValue("filename") + if name == "" || strings.ContainsAny(name, "/\\") || strings.Contains(name, "..") { + http.NotFound(w, r) + return + } + s.serveStaticFile(w, r, filepath.Join("icons", name)) +} + +// handleStatic is the catch-all GET handler: serves files from staticDir, falls +// back to index.html for SPA routes (React client-side routing). Registered last +// so the specific API routes take precedence. +func (s *Server) handleStatic(w http.ResponseWriter, r *http.Request) { + upath := path.Clean("/" + r.URL.Path) + full := filepath.Join(s.staticDir, filepath.FromSlash(upath)) + // Guard against path traversal escaping staticDir. + if rel, err := filepath.Rel(s.staticDir, full); err != nil || strings.HasPrefix(rel, "..") { + http.NotFound(w, r) + return + } + if info, err := os.Stat(full); err == nil { + if info.IsDir() { + if idx := filepath.Join(full, "index.html"); fileExists(idx) { + http.ServeFile(w, r, idx) + return + } + } else { + http.ServeFile(w, r, full) + return + } + } + // SPA fallback — serve the app shell for unknown (client-routed) paths. + http.ServeFile(w, r, filepath.Join(s.staticDir, "index.html")) +} + +func (s *Server) serveStaticFile(w http.ResponseWriter, r *http.Request, rel string) { + full := filepath.Join(s.staticDir, filepath.FromSlash(rel)) + if !fileExists(full) { + http.Error(w, "Not found", http.StatusNotFound) + return + } + http.ServeFile(w, r, full) +} + +func fileExists(p string) bool { + info, err := os.Stat(p) + return err == nil && !info.IsDir() +} + +// runIssueCookieCLI prints a session token for cross-checking itsdangerous +// cookie interop with the Python service. +func runIssueCookieCLI() { + if len(os.Args) < 5 { + os.Stderr.WriteString("usage: tracker-go issue-cookie \n") + os.Exit(2) + } + os.Stdout.WriteString(issueSessionCookie(os.Args[4], sessionUser{Username: os.Args[2], IsAdmin: os.Args[3] == "true"})) +} diff --git a/go-services/tracker-go/website_admin.go b/go-services/tracker-go/website_admin.go new file mode 100644 index 00000000..bf2a1dd4 --- /dev/null +++ b/go-services/tracker-go/website_admin.go @@ -0,0 +1,151 @@ +package main + +import ( + "context" + "encoding/json" + "errors" + "net/http" + "strconv" + "strings" + "time" + + "github.com/jackc/pgx/v5" + "golang.org/x/crypto/bcrypt" +) + +// Admin user management — port of main.py's /admin + /api-admin/users routes. +// All require an admin session (requireAdmin). Writes only succeed in write +// (cutover) mode; on the read-only parallel instance the txn is rejected. + +// GET /admin/users — serve the admin page (admin only). +func (s *Server) handleAdminPage(w http.ResponseWriter, r *http.Request) { + if !requireAdmin(w, r) { + return + } + s.serveStaticFile(w, r, "admin.html") +} + +// GET /api-admin/users — list users (admin only). +func (s *Server) handleListUsers(w http.ResponseWriter, r *http.Request) { + if !requireAdmin(w, r) { + return + } + ctx, cancel := context.WithTimeout(r.Context(), 10*time.Second) + defer cancel() + rows, err := s.pool.Query(ctx, "SELECT id, username, is_admin, created_at FROM users ORDER BY id") + if err != nil { + writeJSON(w, http.StatusInternalServerError, map[string]any{"detail": "db error"}) + return + } + defer rows.Close() + users := []map[string]any{} + for rows.Next() { + var id int + var username string + var isAdmin bool + var createdAt time.Time + if rows.Scan(&id, &username, &isAdmin, &createdAt) != nil { + continue + } + users = append(users, map[string]any{ + "id": id, "username": username, "is_admin": isAdmin, + "created_at": createdAt.UTC().Format("2006-01-02T15:04:05.999999"), + }) + } + writeJSON(w, http.StatusOK, map[string]any{"users": users}) +} + +// POST /api-admin/users — create a user (admin only). +func (s *Server) handleCreateUser(w http.ResponseWriter, r *http.Request) { + if !requireAdmin(w, r) { + return + } + var body struct { + Username string `json:"username"` + Password string `json:"password"` + IsAdmin bool `json:"is_admin"` + } + _ = json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<20)).Decode(&body) + username := strings.TrimSpace(body.Username) + if username == "" || body.Password == "" { + writeJSON(w, http.StatusBadRequest, map[string]any{"detail": "Username and password required"}) + return + } + if len(body.Password) < 4 { + writeJSON(w, http.StatusBadRequest, map[string]any{"detail": "Password must be at least 4 characters"}) + return + } + ctx, cancel := context.WithTimeout(r.Context(), 10*time.Second) + defer cancel() + var existing int + if s.pool.QueryRow(ctx, "SELECT id FROM users WHERE LOWER(username) = $1", strings.ToLower(username)).Scan(&existing) == nil { + writeJSON(w, http.StatusConflict, map[string]any{"detail": "Username already exists"}) + return + } + hash, _ := bcrypt.GenerateFromPassword([]byte(body.Password), 12) + if _, err := s.pool.Exec(ctx, "INSERT INTO users (username, password_hash, is_admin) VALUES ($1,$2,$3)", username, string(hash), body.IsAdmin); err != nil { + writeJSON(w, http.StatusInternalServerError, map[string]any{"detail": "Failed to create user"}) + return + } + writeJSON(w, http.StatusOK, map[string]any{"ok": true, "username": username}) +} + +// PATCH /api-admin/users/{user_id} — password reset / admin toggle (admin only). +func (s *Server) handleUpdateUser(w http.ResponseWriter, r *http.Request) { + if !requireAdmin(w, r) { + return + } + id, _ := strconv.Atoi(r.PathValue("user_id")) + var body map[string]any + _ = json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<20)).Decode(&body) + ctx, cancel := context.WithTimeout(r.Context(), 10*time.Second) + defer cancel() + var exists int + if errors.Is(s.pool.QueryRow(ctx, "SELECT id FROM users WHERE id = $1", id).Scan(&exists), pgx.ErrNoRows) { + writeJSON(w, http.StatusNotFound, map[string]any{"detail": "User not found"}) + return + } + if pw, ok := body["password"].(string); ok { + if len(pw) < 4 { + writeJSON(w, http.StatusBadRequest, map[string]any{"detail": "Password must be at least 4 characters"}) + return + } + hash, _ := bcrypt.GenerateFromPassword([]byte(pw), 12) + if _, err := s.pool.Exec(ctx, "UPDATE users SET password_hash = $1 WHERE id = $2", string(hash), id); err != nil { + writeJSON(w, http.StatusInternalServerError, map[string]any{"detail": "update failed"}) + return + } + } + if a, ok := body["is_admin"]; ok { + isAdmin, _ := a.(bool) + if _, err := s.pool.Exec(ctx, "UPDATE users SET is_admin = $1 WHERE id = $2", isAdmin, id); err != nil { + writeJSON(w, http.StatusInternalServerError, map[string]any{"detail": "update failed"}) + return + } + } + writeJSON(w, http.StatusOK, map[string]any{"ok": true}) +} + +// DELETE /api-admin/users/{user_id} — delete a user (admin only, not yourself). +func (s *Server) handleDeleteUser(w http.ResponseWriter, r *http.Request) { + if !requireAdmin(w, r) { + return + } + id, _ := strconv.Atoi(r.PathValue("user_id")) + ctx, cancel := context.WithTimeout(r.Context(), 10*time.Second) + defer cancel() + var username string + if errors.Is(s.pool.QueryRow(ctx, "SELECT username FROM users WHERE id = $1", id).Scan(&username), pgx.ErrNoRows) { + writeJSON(w, http.StatusNotFound, map[string]any{"detail": "User not found"}) + return + } + if cur := currentUser(r); cur != nil && strings.EqualFold(username, cur.Username) { + writeJSON(w, http.StatusBadRequest, map[string]any{"detail": "Cannot delete yourself"}) + return + } + if _, err := s.pool.Exec(ctx, "DELETE FROM users WHERE id = $1", id); err != nil { + writeJSON(w, http.StatusInternalServerError, map[string]any{"detail": "delete failed"}) + return + } + writeJSON(w, http.StatusOK, map[string]any{"ok": true}) +} diff --git a/go-services/tracker-go/website_issues.go b/go-services/tracker-go/website_issues.go new file mode 100644 index 00000000..810ddaec --- /dev/null +++ b/go-services/tracker-go/website_issues.go @@ -0,0 +1,192 @@ +package main + +import ( + "crypto/rand" + "encoding/hex" + "encoding/json" + "net/http" + "os" + "path/filepath" + "strings" + "sync" + "time" +) + +// Issue board write side — port of main.py's POST/PATCH/DELETE /issues. Issues +// live in static/openissues.json (the same flat file the read side uses); writes +// are serialized by issuesMu. Needs the file mounted read-write in cutover. + +var issuesMu sync.Mutex + +func (s *Server) issuesPath() string { return filepath.Join(s.staticDir, "openissues.json") } + +func (s *Server) loadIssuesRW() []map[string]any { + b, err := os.ReadFile(s.issuesPath()) + if err != nil { + return []map[string]any{} + } + var v []map[string]any + if json.Unmarshal(b, &v) != nil { + return []map[string]any{} + } + return v +} + +func (s *Server) saveIssues(issues []map[string]any) error { + b, _ := json.MarshalIndent(issues, "", " ") + return os.WriteFile(s.issuesPath(), b, 0o644) +} + +func issueAuthor(r *http.Request) string { + if u := currentUser(r); u != nil { + return u.Username + } + return "Anonymous" +} + +func nowISO() string { return time.Now().UTC().Format("2006-01-02T15:04:05.999999") } + +func randHex8() string { + b := make([]byte, 4) + _, _ = rand.Read(b) + return hex.EncodeToString(b) +} + +// pyHTMLEscape matches Python's html.escape(s, quote=True). +func pyHTMLEscape(s string) string { + s = strings.ReplaceAll(s, "&", "&") + s = strings.ReplaceAll(s, "<", "<") + s = strings.ReplaceAll(s, ">", ">") + s = strings.ReplaceAll(s, "\"", """) + s = strings.ReplaceAll(s, "'", "'") + return s +} + +// POST /issues +func (s *Server) handleAddIssue(w http.ResponseWriter, r *http.Request) { + var body map[string]any + _ = json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<20)).Decode(&body) + title := pyHTMLEscape(strings.TrimSpace(toStr(body["title"]))) + if title == "" { + writeJSON(w, http.StatusBadRequest, map[string]any{"detail": "Title is required"}) + return + } + category := strings.TrimSpace(toStr(body["category"])) + if category == "" { + category = "other" + } + newIssue := map[string]any{ + "id": randHex8(), + "title": title, + "description": pyHTMLEscape(strings.TrimSpace(toStr(body["description"]))), + "category": pyHTMLEscape(category), + "author": issueAuthor(r), + "created": nowISO(), + "resolved": false, + "comments": []any{}, + } + issuesMu.Lock() + defer issuesMu.Unlock() + issues := append([]map[string]any{newIssue}, s.loadIssuesRW()...) + if err := s.saveIssues(issues); err != nil { + writeJSON(w, http.StatusInternalServerError, map[string]any{"detail": "save failed"}) + return + } + writeJSON(w, http.StatusOK, newIssue) +} + +// PATCH /issues/{issue_id} +func (s *Server) handleUpdateIssue(w http.ResponseWriter, r *http.Request) { + id := r.PathValue("issue_id") + var update map[string]any + _ = json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<20)).Decode(&update) + issuesMu.Lock() + defer issuesMu.Unlock() + issues := s.loadIssuesRW() + var found map[string]any + for _, i := range issues { + if toStr(i["id"]) == id { + if v, ok := update["resolved"]; ok { + b, _ := v.(bool) + i["resolved"] = b + } + if v, ok := update["title"]; ok { + t := pyHTMLEscape(strings.TrimSpace(toStr(v))) + if t == "" { + writeJSON(w, http.StatusBadRequest, map[string]any{"detail": "Title cannot be empty"}) + return + } + i["title"] = t + } + if v, ok := update["description"]; ok { + i["description"] = pyHTMLEscape(strings.TrimSpace(toStr(v))) + } + if v, ok := update["category"]; ok { + i["category"] = pyHTMLEscape(toStr(v)) + } + found = i + break + } + } + if found == nil { + writeJSON(w, http.StatusNotFound, map[string]any{"detail": "Issue not found"}) + return + } + if err := s.saveIssues(issues); err != nil { + writeJSON(w, http.StatusInternalServerError, map[string]any{"detail": "save failed"}) + return + } + writeJSON(w, http.StatusOK, found) +} + +// POST /issues/{issue_id}/comments +func (s *Server) handleAddComment(w http.ResponseWriter, r *http.Request) { + id := r.PathValue("issue_id") + var body map[string]any + _ = json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<20)).Decode(&body) + issuesMu.Lock() + defer issuesMu.Unlock() + issues := s.loadIssuesRW() + var found map[string]any + for _, i := range issues { + if toStr(i["id"]) == id { + found = i + break + } + } + if found == nil { + writeJSON(w, http.StatusNotFound, map[string]any{"detail": "Issue not found"}) + return + } + text := pyHTMLEscape(strings.TrimSpace(toStr(body["text"]))) + if text == "" { + writeJSON(w, http.StatusBadRequest, map[string]any{"detail": "Comment text is required"}) + return + } + comment := map[string]any{"id": randHex8(), "author": issueAuthor(r), "text": text, "created": nowISO()} + comments, _ := found["comments"].([]any) + found["comments"] = append(comments, comment) + if err := s.saveIssues(issues); err != nil { + writeJSON(w, http.StatusInternalServerError, map[string]any{"detail": "save failed"}) + return + } + writeJSON(w, http.StatusOK, comment) +} + +// DELETE /issues/{issue_id} +func (s *Server) handleDeleteIssue(w http.ResponseWriter, r *http.Request) { + id := r.PathValue("issue_id") + issuesMu.Lock() + defer issuesMu.Unlock() + kept := []map[string]any{} + for _, i := range s.loadIssuesRW() { + if toStr(i["id"]) != id { + kept = append(kept, i) + } + } + if err := s.saveIssues(kept); err != nil { + writeJSON(w, http.StatusInternalServerError, map[string]any{"detail": "save failed"}) + return + } + writeJSON(w, http.StatusOK, map[string]any{"status": "ok"}) +} diff --git a/go-services/tracker-go/wslive.go b/go-services/tracker-go/wslive.go new file mode 100644 index 00000000..0fe1eb27 --- /dev/null +++ b/go-services/tracker-go/wslive.go @@ -0,0 +1,185 @@ +package main + +import ( + "context" + "encoding/json" + "net/http" + "sync" + "time" + + "github.com/coder/websocket" +) + +// Hub is the browser broadcast fan-out for /ws/live, mirroring main.py's +// browser_conns + _do_broadcast: each client has an optional message-type +// filter (nil = all); a message is delivered when the filter is nil or contains +// the message's "type". Telemetry broadcasts carry no type, so only unfiltered +// clients receive them (matching Python — which is why the React map polls /live +// over HTTP rather than relying on the WS for positions). +type Hub struct { + mu sync.RWMutex + clients map[*browserClient]bool +} + +type browserClient struct { + filter map[string]bool // nil = all types + send chan []byte +} + +func newHub() *Hub { return &Hub{clients: map[*browserClient]bool{}} } + +func (h *Hub) add(c *browserClient) { + h.mu.Lock() + h.clients[c] = true + h.mu.Unlock() +} + +func (h *Hub) remove(c *browserClient) { + h.mu.Lock() + if h.clients[c] { + delete(h.clients, c) + close(c.send) + } + h.mu.Unlock() +} + +func (h *Hub) count() int { + h.mu.RLock() + defer h.mu.RUnlock() + return len(h.clients) +} + +// broadcast serializes once and delivers to matching clients. A slow client +// (full send buffer) is skipped for this message rather than blocking the +// ingest path, matching the spirit of Python's per-send timeout + eviction. +func (h *Hub) broadcast(data map[string]any) { + h.mu.RLock() + empty := len(h.clients) == 0 + h.mu.RUnlock() + if empty { + return // no browsers: skip the marshal entirely + } + msg, err := json.Marshal(data) + if err != nil { + return + } + msgType, _ := data["type"].(string) + h.mu.RLock() + for c := range h.clients { + if c.filter != nil && (msgType == "" || !c.filter[msgType]) { + continue + } + select { + case c.send <- msg: + default: + } + } + h.mu.RUnlock() +} + +func (s *Server) handleWSLive(w http.ResponseWriter, r *http.Request) { + // Auth: internal-trust (private peer + no XFF) OR a valid session cookie. + if !(r.Header.Get("X-Forwarded-For") == "" && isPrivateAddr(clientIP(r))) { + c, err := r.Cookie("session") + if err != nil || verifySessionCookie(s.secretKey, c.Value) == nil { + http.Error(w, "Not authenticated", http.StatusUnauthorized) + return + } + } + conn, err := websocket.Accept(w, r, &websocket.AcceptOptions{InsecureSkipVerify: true}) + if err != nil { + return + } + defer conn.CloseNow() + conn.SetReadLimit(8 << 20) + + client := &browserClient{send: make(chan []byte, 256)} + s.hub.add(client) + defer s.hub.remove(client) + + ctx := r.Context() + // Writer goroutine: the only writer for this conn (serializes writes). + go func() { + for msg := range client.send { + wctx, cancel := context.WithTimeout(ctx, 5*time.Second) + err := conn.Write(wctx, websocket.MessageText, msg) + cancel() + if err != nil { + conn.CloseNow() + return + } + } + }() + + for { + _, raw, err := conn.Read(ctx) + if err != nil { + return + } + var m map[string]any + if json.Unmarshal(raw, &m) != nil { + continue + } + s.handleBrowserMessage(client, m) + } +} + +// handleBrowserMessage handles subscribe / request_dungeon_map / command +// envelopes from a browser client (main.py:3846). +func (s *Server) handleBrowserMessage(c *browserClient, m map[string]any) { + switch toStr(m["type"]) { + case "subscribe": + types := toStringSlice(m["message_types"]) + if len(types) == 0 { + c.filter = nil // all + return + } + f := make(map[string]bool, len(types)) + for _, t := range types { + f[t] = true + } + c.filter = f + return + case "request_dungeon_map": + lb := toStr(m["landblock"]) + if lb != "" && s.ingestor != nil { + if dm, ok := s.ingestor.snapshot(s.ingestor.dungeonMapCache, lb); ok { + if b, err := json.Marshal(dm); err == nil { + select { + case c.send <- b: + default: + } + } + } + } + return + } + // Command envelopes: new {player_name, command} or legacy {type:command, character_name, text}. + if pn, ok := m["player_name"].(string); ok { + if cmd, ok := m["command"].(string); ok { + s.plugins.send(pn, map[string]any{"player_name": pn, "command": cmd}) + return + } + } + if toStr(m["type"]) == "command" { + pn := toStr(m["character_name"]) + text := toStr(m["text"]) + if pn != "" { + s.plugins.send(pn, map[string]any{"player_name": pn, "command": text}) + } + } +} + +func toStringSlice(v any) []string { + arr, ok := v.([]any) + if !ok { + return nil + } + out := make([]string, 0, len(arr)) + for _, e := range arr { + if s, ok := e.(string); ok { + out = append(out, s) + } + } + return out +} diff --git a/go-services/tracker-go/wsposition.go b/go-services/tracker-go/wsposition.go new file mode 100644 index 00000000..84892558 --- /dev/null +++ b/go-services/tracker-go/wsposition.go @@ -0,0 +1,156 @@ +package main + +import ( + "context" + "crypto/hmac" + "encoding/json" + "log/slog" + "net/http" + "sync" + "time" + + "github.com/coder/websocket" +) + +// pluginRegistry maps character_name -> plugin connection for backend->plugin +// command routing (main.py plugin_conns). +type pluginRegistry struct { + mu sync.RWMutex + conns map[string]*websocket.Conn + log *slog.Logger +} + +func newPluginRegistry(log *slog.Logger) *pluginRegistry { + return &pluginRegistry{conns: map[string]*websocket.Conn{}, log: log} +} + +func (p *pluginRegistry) register(name string, c *websocket.Conn) { + p.mu.Lock() + p.conns[name] = c + p.mu.Unlock() +} + +// removeConn drops every name bound to this connection (on disconnect). +func (p *pluginRegistry) removeConn(c *websocket.Conn) { + p.mu.Lock() + for n, cc := range p.conns { + if cc == c { + delete(p.conns, n) + } + } + p.mu.Unlock() +} + +func (p *pluginRegistry) isConnected(name string) bool { + p.mu.RLock() + defer p.mu.RUnlock() + _, ok := p.conns[name] + return ok +} + +// send routes an opaque {player_name, command} envelope to a plugin; evicts the +// connection on write failure (main.py command-forward semantics). +func (p *pluginRegistry) send(name string, payload map[string]any) { + p.mu.RLock() + c := p.conns[name] + p.mu.RUnlock() + if c == nil { + return + } + b, _ := json.Marshal(payload) + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + if err := c.Write(ctx, websocket.MessageText, b); err != nil { + p.mu.Lock() + if p.conns[name] == c { + delete(p.conns, name) + } + p.mu.Unlock() + } +} + +// fanoutShare forwards a share_* message to other opted-in plugin clients +// (every connected name that is subscribed and isn't the origin). Send failures +// are logged-and-ignored, not evicted (main.py:2829). +func (p *pluginRegistry) fanoutShare(data map[string]any, origin string, subs map[string]bool) { + p.mu.RLock() + type target struct { + name string + c *websocket.Conn + } + var targets []target + for n, c := range p.conns { + if n != origin && subs[n] { + targets = append(targets, target{n, c}) + } + } + p.mu.RUnlock() + if len(targets) == 0 { + return + } + b, _ := json.Marshal(data) + for _, t := range targets { + ctx, cancel := context.WithTimeout(context.Background(), time.Second) + _ = t.c.Write(ctx, websocket.MessageText, b) + cancel() + } +} + +// pluginAuthOK constant-time-compares the supplied secret to SHARED_SECRET (and +// the optional rotation fallback). Fails closed when unset or left at the +// placeholder, matching main.py. +func (s *Server) pluginAuthOK(key string) bool { + ok := s.sharedSecret != "" && s.sharedSecret != "your_shared_secret" && + hmac.Equal([]byte(key), []byte(s.sharedSecret)) + if !ok && s.sharedSecretLegacy != "" { + ok = hmac.Equal([]byte(key), []byte(s.sharedSecretLegacy)) + } + return ok +} + +func (s *Server) handleWSPosition(w http.ResponseWriter, r *http.Request) { + if s.ingestor == nil { + http.Error(w, "ingest disabled on this instance", http.StatusServiceUnavailable) + return + } + key := r.URL.Query().Get("secret") + if key == "" { + key = r.Header.Get("X-Plugin-Secret") + } + if !s.pluginAuthOK(key) { + http.Error(w, "unauthorized", http.StatusUnauthorized) + return + } + conn, err := websocket.Accept(w, r, &websocket.AcceptOptions{InsecureSkipVerify: true}) + if err != nil { + return + } + defer conn.CloseNow() + defer s.plugins.removeConn(conn) + conn.SetReadLimit(32 << 20) + + ctx := r.Context() + for { + _, raw, err := conn.Read(ctx) + if err != nil { + return + } + var m map[string]any + if json.Unmarshal(raw, &m) != nil { + continue + } + if toStr(m["type"]) == "register" { + name := toStr(m["character_name"]) + if name == "" { + name = toStr(m["player_name"]) + } + if name != "" { + s.plugins.register(name, conn) + s.ingestor.clearEquipmentCantrip(name) + s.log.Info("plugin registered", "character", name) + } + continue + } + s.ingestor.dispatch(ctx, m) + } +} diff --git a/static/suitbuilder.css b/static/suitbuilder.css index ef6bb4d2..db10085b 100644 --- a/static/suitbuilder.css +++ b/static/suitbuilder.css @@ -1529,4 +1529,26 @@ body { color: #95a5a6; font-size: 10px; margin-left: auto; -} \ No newline at end of file +} + +.cd-toggle { + display: inline-flex; + align-items: center; + gap: 4px; + margin-right: 10px; + font-weight: normal; + cursor: pointer; +} +.cd-toggle input { margin: 0; } + +.select-all-btn { + margin-left: 8px; + padding: 2px 8px; + font-size: 11px; + font-weight: normal; + cursor: pointer; + border: 1px solid #ccc; + border-radius: 3px; + background: #f0f0f0; +} +.select-all-btn:hover { background: #e0e0e0; } \ No newline at end of file diff --git a/static/suitbuilder.html b/static/suitbuilder.html index 5ede36cc..52412f2e 100644 --- a/static/suitbuilder.html +++ b/static/suitbuilder.html @@ -51,10 +51,10 @@
- - - - - + + + +
@@ -245,7 +245,7 @@
-

Legendary Wards

+

Legendary Wards

diff --git a/static/suitbuilder.js b/static/suitbuilder.js index f523134b..72558a08 100644 --- a/static/suitbuilder.js +++ b/static/suitbuilder.js @@ -152,13 +152,32 @@ function setupEventListeners() { // Main action buttons document.getElementById('searchSuits').addEventListener('click', performSuitSearch); document.getElementById('clearAll').addEventListener('click', clearAllConstraints); - + document.getElementById('wardsSelectAll').addEventListener('click', toggleAllWards); + // Slot control buttons document.getElementById('lockSelectedSlots').addEventListener('click', lockSelectedSlots); document.getElementById('clearAllLocks').addEventListener('click', clearAllLocks); document.getElementById('resetSlotView').addEventListener('click', resetSlotView); } +// Legendary Ward checkboxes (toggled together by the "Select All" button). +const WARD_IDS = [ + 'protection_flame', 'protection_frost', 'protection_acid', 'protection_storm', + 'protection_slashing', 'protection_piercing', 'protection_bludgeoning', 'protection_armor' +]; + +/** + * Toggle all Legendary Ward checkboxes. If every ward is already checked, + * clears them; otherwise selects all. The button label tracks the state. + */ +function toggleAllWards() { + const boxes = WARD_IDS.map(id => document.getElementById(id)).filter(Boolean); + const allChecked = boxes.every(cb => cb.checked); + boxes.forEach(cb => { cb.checked = !allChecked; }); + const btn = document.getElementById('wardsSelectAll'); + if (btn) btn.textContent = allChecked ? 'Select All' : 'Clear All'; +} + /** * Setup slot interaction functionality */ @@ -307,8 +326,11 @@ function gatherConstraints() { characters: selectedCharacters, min_armor: document.getElementById('minArmor').value || null, max_armor: document.getElementById('maxArmor').value || null, - min_crit_damage: document.getElementById('minCritDmg').value || null, - max_crit_damage: document.getElementById('maxCritDmg').value || null, + allowed_crit_damage: [ + document.getElementById('allowCD0').checked ? 0 : null, + document.getElementById('allowCD1').checked ? 1 : null, + document.getElementById('allowCD2').checked ? 2 : null, + ].filter(v => v !== null), min_damage_rating: document.getElementById('minDmgRating').value || null, max_damage_rating: document.getElementById('maxDmgRating').value || null, @@ -357,7 +379,7 @@ function validateConstraints(constraints) { if (!constraints.primary_set && !constraints.secondary_set && constraints.legendary_cantrips.length === 0 && constraints.protection_spells.length === 0 && - !constraints.min_armor && !constraints.min_crit_damage && !constraints.min_damage_rating) { + !constraints.min_armor && !constraints.min_damage_rating) { alert('Please specify at least one constraint (equipment sets, cantrips, legendary wards, or rating minimums).'); return false; } @@ -383,8 +405,7 @@ async function streamOptimalSuits(constraints) { include_inventory: constraints.include_inventory, min_armor: constraints.min_armor ? parseInt(constraints.min_armor) : null, max_armor: constraints.max_armor ? parseInt(constraints.max_armor) : null, - min_crit_damage: constraints.min_crit_damage ? parseInt(constraints.min_crit_damage) : null, - max_crit_damage: constraints.max_crit_damage ? parseInt(constraints.max_crit_damage) : null, + allowed_crit_damage: constraints.allowed_crit_damage, min_damage_rating: constraints.min_damage_rating ? parseInt(constraints.min_damage_rating) : null, max_damage_rating: constraints.max_damage_rating ? parseInt(constraints.max_damage_rating) : null, max_results: 10,