MosswartOverlord/README.md

# Mosswart Overlord (Dereth Tracker)

Real-time telemetry, inventory, and analytics platform for Asheron's Call —
driven by a firehose of WebSocket events from the companion
[MosswartMassacre](https://github.com/SawatoMosswartsEnjoyersClub/MosswartMassacre)
DECAL plugin running on 60+ characters.

**The production backend is written in Go** (`go-services/`). It replaced the
original Python/FastAPI implementation via a strangler-fig migration: the Go
services ran in parallel against live traffic until every endpoint was proven
byte-identical, then production was cut over. The Python implementation is
preserved on the `python-legacy` branch.

---

## Architecture

```
 MosswartMassacre plugin ──wss──> nginx ──> Go tracker (tracker-go) ──> dereth  (TimescaleDB)
   (60+ game clients)                 │                │
                                      │                ├──HTTP──> Go inventory (inventory-go) ──> inventory_db
 Browsers ──https──────────────────> nginx            │
                                      │                └──/ws/live──> Discord rare bot (relays rares + chat)
                                      └──> Grafana (/grafana/)        death/idle alerts → Discord webhook
```

| Component | Path | Runs as | Notes |
|---|---|---|---|
| **Tracker** (ingest + website + read API + WS) | `go-services/tracker-go/` | Docker `dereth-tracker-go`, 127.0.0.1:8770 | serves the React frontend, login/admin, the plugin `/ws/position`, browser `/ws/live`, and the full read API; writes the `dereth` DB |
| **Inventory** (search + suitbuilder + ingestion) | `go-services/inventory-go/` | Docker `inventory-go`, 127.0.0.1:8772 | normalized item search, the suitbuilder solver (SSE), inventory ingestion; writes `inventory_db` |
| Telemetry DB | TimescaleDB | Docker `dereth-db`, 5432 | hypertables `telemetry_events`, `spawn_events` |
| Inventory DB | postgres:14 | Docker `inventory-db`, 5433 | 7-table normalized item schema |
| React frontend | `frontend/` → `static/` | served by `tracker-go` | unchanged by the migration — same paths, same API |
| Classic v1 / legacy pages | `static/classic/`, `static/*.html` | served by `tracker-go` | `/classic`, `/suitbuilder.html`, `/inventory.html` |
| Grafana | compose `dereth-grafana` | 127.0.0.1:3000 | anonymous Viewer auth, proxied at `/grafana/` |
| Discord rare bot | `discord-rare-monitor/` (Python) | Docker, reads Go `/ws/live` | posts rares + relays allegiance chat |
| Overlord Agent (assistant) | `agent/` | host-side systemd `overlord-agent`, 127.0.0.1:8767 | shells out to `claude -p`; outside Docker by design |

**Stack:** Go 1.25 (stdlib `net/http` with 1.22 method+path routing, `pgx/v5`,
`coder/websocket`, `bwmarrin/discordgo`, `golang.org/x/crypto/bcrypt`), distroless
multi-stage images. React 19 + Vite + TypeScript. PostgreSQL/TimescaleDB. nginx
reverse proxy (host-side). Unlike the old single-worker Python service, the Go
tracker uses `GOMAXPROCS` = all available cores, so traffic bursts parallelize
instead of bottlenecking on one core.

---

## Build & run

Everything builds and runs in Docker — **no host Go toolchain needed** (the
multi-stage images compile from source). The production stack is the base compose
(databases, Grafana, Discord bot) plus two override files for the Go services and
the cutover wiring.

```bash
# --- build the Go service images ---
export BUILD_VERSION="$(date -u +%Y.%-m.%-d.%H%M)-$(git rev-parse --short HEAD)"
docker compose -f docker-compose.yml -f go-services/docker-compose.go.yml \
  build dereth-tracker-go inventory-go

# --- production: Go services in write mode, serving the site + ingest ---
docker compose -f docker-compose.yml \
  -f go-services/docker-compose.go.yml \
  -f go-services/docker-compose.cutover.yml \
  up -d --no-deps dereth-tracker-go inventory-go
```

- `docker-compose.go.yml` defines the Go services (plus the isolated shadow DBs used during the parallel run).
- `docker-compose.cutover.yml` flips the Go services to **write mode** against the production DBs (`READ_ONLY=false`, `SKIP_SCHEMA_INIT=true` so they run no DDL and trust the existing schema) and points the Discord bot at the Go `/ws/live`. Drop this file to return the Go services to read-only parallel mode.
- `BUILD_VERSION` is shown in the frontend sidebar (CalVer: `YYYY.M.D.HHMM-gitshorthash`).
- Required env (server `.env`, **never committed**): `SHARED_SECRET`, `SECRET_KEY`, `POSTGRES_PASSWORD`, `INVENTORY_DB_PASSWORD`, `DISCORD_ACLOG_WEBHOOK`, `DISCORD_RARE_BOT_TOKEN`, the Discord channel IDs, and Grafana admin. See `.env.example`.

### Frontend (unchanged by the migration)

The React app and the legacy static pages call the same absolute paths
(`/api/...`, `/inv/...`, `/live`, …) — the Go tracker answers them, so the
frontend ships as-is.

```bash
cd frontend && npm run dev          # local dev, port 5173, /api → :8770
bash deploy-frontend.sh             # complete build + copy into static/ (runs npm run build itself)
```

The tracker serves `static/` directly (bind-mounted), so static/JS/CSS changes
need no restart. ⚠️ `npm run build` writes to `static/_build/`; only
`deploy-frontend.sh` copies it into the served `static/`.

### nginx

The live config is host-side at `/etc/nginx/sites-enabled/overlord` (source copy
in `nginx/overlord.conf`); the `tracker_go` upstream is in
`/etc/nginx/conf.d/tracker_go.conf` (`server 127.0.0.1:8770;`). Production routes
`/`, `/api/`, `/websocket/` to the Go tracker. Every location that proxies to the
tracker **must** set `X-Forwarded-For` — it drives the internal-trust auth rule.

### Overlord Agent

Unchanged by the migration — it's a host-side Python systemd service. Code change:
`git pull && sudo systemctl restart overlord-agent`. Its env lives separately at
`/etc/overlord/agent.env`. See `agent/` and `CLAUDE.md`.

---

## WebSocket contract

- **`/ws/position`** — plugin → backend. Telemetry, vitals, inventory, portal, rare, combat, quest, chat, share_*, … Authenticated by the `X-Plugin-Secret` header against `SHARED_SECRET` (constant-time; fails closed when unset). The tracker forwards inventory to `inventory-go`, accumulates kill/combat stats, and re-broadcasts to browsers.
- **`/ws/live`** — browser ↔ backend. Session-cookie (or internal-trust) authenticated. Accepts `subscribe`, `request_dungeon_map`, and `{player_name, command}` envelopes routed to the matching plugin socket. **Telemetry is broadcast typeless** so the browser ignores it and takes player data from the 5 s `/live` poll (matching the original design — broadcasting it typed flaps the per-player counters).
- **Internal-trust rule:** a request skips cookie auth only when its source is private/loopback **and** carries no `X-Forwarded-For`. nginx sets XFF on all internet traffic, so only host-side / compose-network callers qualify.

### Payload note

Payloads are snake_case JSON; keep field names and shapes stable across plugin +
backend. The plugin sends several numeric telemetry fields as **strings**
(`kills_per_hour`, `deaths`, `total_deaths`, `prismatic_taper_count`) — the backend
coerces them (`coerceNum` in `tracker-go/reads.go`).

## Auth & users

Session cookies are signed with `SECRET_KEY` via an itsdangerous-compatible
`URLSafeTimedSerializer` (HMAC-SHA1, 30-day expiry) — cookies interoperate with
the legacy Python service. Login at `/login` (bcrypt against the `users` table),
admin user CRUD at `/api-admin/users`, current user at `/me`.

## Databases

Two separate Postgres databases, both schema-from-code:

- **`dereth`** (TimescaleDB, `dereth-db`): hypertables `telemetry_events` + `spawn_events`, plus `char_stats`, `combat_stats(_sessions)`, `rare_*`, `portals`, `character_stats`, `users`. Persisted event types: telemetry, spawn, rare, portal, character_stats, combat_stats. Everything else (vitals, quest, cantrips, nearby_objects, dungeon_map, share_*) is memory-only.
- **`inventory_db`** (postgres:14, `inventory-db`): 7 normalized tables (`items` + combat/requirements/enhancements/ratings/spells/raw_data).

In cutover mode the Go services reuse these production databases directly; the
shadow DBs in `docker-compose.go.yml` exist only for isolated parallel-run
validation. **Backups:** `pg_dump -Fc` of both DBs; TimescaleDB restore needs
`timescaledb_pre_restore()` / `post_restore()` around `pg_restore`.

## Route conventions

- nginx strips `/api/` before proxying, so backend routes do **not** start with `/api/`.
- Hyphenated routes (`/api-version`, `/api-admin/...`) deliberately bypass the strip (they fall through nginx's `location /`).
- The static SPA is the catch-all (`GET /`), registered after the API routes, with `index.html` fallback for client-side routing.
- `/inv/*` reverse-proxies to the inventory service; `/api/agent/*` is proxied by nginx (not the tracker) to the host-side agent.

## Operational notes

- Discord: the rare bot posts rares + relays allegiance chat; **death/idle alerts come from the tracker itself** via `DISCORD_ACLOG_WEBHOOK`.
- Issue board persists to the flat file `static/openissues.json` (web-served, mounted read-write).
- Logs: `docker logs dereth-tracker-go`, `docker logs inventory-go`. Read-only psql: `docker exec dereth-db psql -U postgres -d dereth`, `docker exec inventory-db psql -U inventory_user -d inventory_db`.
- **This repo is PUBLIC** on git.snakedesert.se — never commit secrets. `.env` is gitignored; `.env.example` is the template.

## Branches

- **`master`** — the Go production backend (this).
- **`python-legacy`** — the original Python/FastAPI implementation, preserved for reference and rollback.

See [`CLAUDE.md`](CLAUDE.md) for contributor/agent guidance and deeper internals.