docs: rewrite README to reflect current architecture
Full rewrite covering: - React v2 frontend at /, classic v1 preserved at /classic - WebSocket message-type subscription mechanism (bot filter fix) - Death + idle alerts via Discord webhook with 5-min grace period - spawn_events now a TimescaleDB hypertable with 7-day retention - server_health_checks removed (write-only bloat) - PostgreSQL memory tuning (shared_buffers 8GB on 32GB host) - Uvicorn runs without --reload in production - deploy-frontend.sh requirement for React builds - Combat stats (Mag-Tools style), vital sharing, all WS event types - Cross-machine vital sharing via WebSocket relay - Deploy flows (quick / frontend / full rebuild) - BUILD_VERSION CalVer stamp format Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Erik
parent
f7f04d6a84
commit
3c634adbdc
1 changed files with 254 additions and 347 deletions
585
README.md
585
README.md
|
|
@ -1,412 +1,319 @@
|
||||||
# Dereth Tracker
|
# Mosswart Overlord (Dereth Tracker)
|
||||||
|
|
||||||
Dereth Tracker is a real-time telemetry service for the world of Dereth. It collects player data, stores it in a PostgreSQL (TimescaleDB) database for efficient time-series storage, provides a live map interface, and includes a comprehensive inventory management system for tracking and searching character equipment.
|
Real-time telemetry, inventory, and analytics platform for Asheron's Call.
|
||||||
|
FastAPI backend + React frontend + PostgreSQL (TimescaleDB) + Discord integrations,
|
||||||
|
all driven by WebSocket events from the companion [MosswartMassacre](https://github.com/SawatoMosswartsEnjoyersClub/MosswartMassacre) DECAL plugin.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Table of Contents
|
## Table of Contents
|
||||||
- [Overview](#overview)
|
- [Overview](#overview)
|
||||||
|
- [Architecture](#architecture)
|
||||||
- [Features](#features)
|
- [Features](#features)
|
||||||
- [Requirements](#requirements)
|
- [Requirements](#requirements)
|
||||||
- [Installation](#installation)
|
- [Installation](#installation)
|
||||||
- [Configuration](#configuration)
|
- [Configuration](#configuration)
|
||||||
- [Usage](#usage)
|
- [Deploying Changes](#deploying-changes)
|
||||||
- [API Reference](#api-reference)
|
- [WebSocket Contract](#websocket-contract)
|
||||||
|
- [HTTP API Reference](#http-api-reference)
|
||||||
- [Frontend](#frontend)
|
- [Frontend](#frontend)
|
||||||
- [Database Schema](#database-schema)
|
- [Database Schema](#database-schema)
|
||||||
|
- [Operations & Health](#operations--health)
|
||||||
- [Contributing](#contributing)
|
- [Contributing](#contributing)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
This project provides:
|
Mosswart Overlord is the backend that consumes a firehose of telemetry, vitals, inventory, combat, and chat events from 60+ characters running the `MosswartMassacre` plugin. It stores selected data in TimescaleDB, runs analytics (combat stats, idle/death detection), and broadcasts live updates to connected browser clients.
|
||||||
- A FastAPI backend with endpoints for receiving and querying telemetry data.
|
|
||||||
- PostgreSQL/TimescaleDB-based storage for time-series telemetry and per-character stats.
|
The frontend is a React + Vite app served at `/` with a live map, draggable windows (inventory, chat, combat, radar, etc.), and a server uptime sidebar. The previous vanilla JS frontend is preserved at `/classic`.
|
||||||
- A live, interactive map using static HTML, CSS, and JavaScript.
|
|
||||||
- A comprehensive inventory management system with search capabilities.
|
## Architecture
|
||||||
- Real-time inventory updates via WebSocket when characters log in/out.
|
|
||||||
- A sample data generator script (`generate_data.py`) for simulating telemetry snapshots.
|
```
|
||||||
|
┌─────────────────────────┐
|
||||||
|
│ MosswartMassacre (C#) │ ← plugin per game client
|
||||||
|
└────────────┬────────────┘
|
||||||
|
│ WebSocket /ws/position (authenticated)
|
||||||
|
▼
|
||||||
|
┌────────────────────────────────────────────────────────┐
|
||||||
|
│ dereth-tracker (FastAPI) │
|
||||||
|
│ • main.py — WS routing, analytics, broadcasts │
|
||||||
|
│ • idle/death detection → Discord webhook │
|
||||||
|
│ • combat stats delta/lifetime accumulation │
|
||||||
|
│ • vital sharing relay (cross-machine) │
|
||||||
|
└──┬──────────────────┬────────────────────┬────────────┘
|
||||||
|
│ │ │
|
||||||
|
│ WS /ws/live │ HTTP │ HTTP
|
||||||
|
▼ ▼ ▼
|
||||||
|
┌──────────┐ ┌──────────────────┐ ┌──────────────────┐
|
||||||
|
│ Browsers │ │ inventory-svc │ │ Discord bot │
|
||||||
|
│ (React) │ │ (FastAPI) │ │ (rare monitor) │
|
||||||
|
└──────────┘ └────────┬─────────┘ └──────────────────┘
|
||||||
|
▼
|
||||||
|
┌──────────────┐
|
||||||
|
│ inventory-db │
|
||||||
|
└──────────────┘
|
||||||
|
┌──────────────┐
|
||||||
|
│ dereth-db │ ← TimescaleDB (telemetry, spawns, rares, portals)
|
||||||
|
└──────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
All services run via Docker Compose.
|
||||||
|
|
||||||
## Features
|
## Features
|
||||||
|
|
||||||
- **WebSocket /ws/position**: Stream telemetry snapshots and inventory updates (protected by a shared secret).
|
### Live Data
|
||||||
- **GET /live**: Fetch active players seen in the last 30 seconds.
|
- **Live Map** — real-time player positions, dots, trails, portals, heatmap
|
||||||
- **GET /history**: Retrieve historical telemetry data with optional time filtering.
|
- **WebSocket firehose** (`/ws/live`) — broadcasts every incoming event to browsers
|
||||||
- **GET /debug**: Health check endpoint.
|
- **Per-client subscriptions** — clients can send `{"type":"subscribe","message_types":[...]}` to receive only specific event types (the Discord rare monitor bot uses this to filter the 82GB/day firehose down to just `rare` and `chat`)
|
||||||
- **Live Map**: Interactive map interface with panning, zooming, and sorting.
|
|
||||||
- **Inventory Management**:
|
### Inventory
|
||||||
- Real-time inventory updates via WebSocket on character login/logout
|
- Full inventory snapshot on login + incremental `inventory_delta` updates (add/update/remove)
|
||||||
- Advanced search across all character inventories
|
- Per-character live refresh in the browser (debounced 2s)
|
||||||
- Filter by character, equipment type, material, stats, and more
|
- Advanced search with filters: material, set, armor level, spells, tinks, workmanship, etc.
|
||||||
- Sort by any column with live results
|
- **Suitbuilder** at `/suitbuilder.html` — constraint-based armor optimization across multiple mule inventories with primary/secondary set support, cantrip overlap detection, and real-time SSE streaming
|
||||||
- Track item properties including spells, armor level, damage ratings
|
|
||||||
- **Suitbuilder**:
|
### Combat Stats (Mag-Tools style)
|
||||||
- Equipment optimization across multiple character inventories
|
- Plugin parses combat chat into session deltas
|
||||||
- Constraint-based search for optimal armor combinations
|
- Backend accumulates lifetime totals from per-session snapshots
|
||||||
- Support for primary and secondary armor sets
|
- Offense/defense broken out per damage element
|
||||||
- Real-time streaming results during long-running searches
|
- Browser combat window shows monster-by-monster damage
|
||||||
- **Portal Tracking**:
|
|
||||||
- Automatic discovery and tracking of in-game portals
|
### Cross-Machine Vital Sharing
|
||||||
- 1-hour retention for discovered portals
|
- WebSocket relay replaces UtilityBelt's localhost-only `VTankFellowHeals`
|
||||||
- Coordinate-based uniqueness (rounded to 0.1 precision)
|
- Plugin broadcasts its own vitals and consumes peer vitals
|
||||||
- Real-time portal updates on the map interface
|
- In-game `DxHud` overlay shows peer health/stamina/mana bars with direction arrows
|
||||||
- **Discord Rare Monitor Bot**: Monitors rare discoveries and posts filtered notifications to Discord channels
|
|
||||||
- **Sample Data Generator**: `generate_data.py` sends telemetry snapshots over WebSocket for testing.
|
### Discord Integration
|
||||||
|
- **Rare Monitor Bot** — posts rares (split by common/great) to configured channels
|
||||||
|
- **Death Alerts** — webhook to `#alerts` when a character's vitae goes from 0 → >0 (rate-limited to one per character per 5 min)
|
||||||
|
- **Idle Alerts** — webhook after 5 minutes of continuous idle state (caught portals, stuck nav, etc.). The grace period prevents false positives on brief idle blips.
|
||||||
|
- **Vortex Warning** — bot watches for "whirlwind of vortexes" chat and posts a warning embed
|
||||||
|
|
||||||
|
### Portals
|
||||||
|
- Automatic discovery + 1-hour retention
|
||||||
|
- Coordinate-deduplicated (rounded to 0.1 precision)
|
||||||
|
|
||||||
|
### Stats
|
||||||
|
- Per-character lifetime kills, deaths, rares, taper counts
|
||||||
|
- Grafana dashboards (2x2 iframe grid in the stats window)
|
||||||
|
|
||||||
|
### Health & Monitoring
|
||||||
|
- Server uptime + latency + player count from TreeStats.net (checked every 30s)
|
||||||
|
- Only current state is kept — no historical `server_health_checks` table (removed April 2026 as write-only bloat)
|
||||||
|
|
||||||
## Requirements
|
## Requirements
|
||||||
|
|
||||||
- Python 3.9 or newer (only if running without Docker)
|
|
||||||
- pip (only if running without Docker)
|
|
||||||
- Docker & Docker Compose (recommended)
|
- Docker & Docker Compose (recommended)
|
||||||
|
- OR: Python 3.11+, Node.js 20+, and a PostgreSQL 14+ with TimescaleDB
|
||||||
Python packages (if using local virtualenv):
|
|
||||||
|
|
||||||
- fastapi
|
|
||||||
- uvicorn
|
|
||||||
- pydantic
|
|
||||||
- databases
|
|
||||||
- asyncpg
|
|
||||||
- sqlalchemy
|
|
||||||
- websockets # required for sample data generator
|
|
||||||
|
|
||||||
## Installation
|
## Installation
|
||||||
|
|
||||||
1. Clone the repository:
|
```bash
|
||||||
```bash
|
git clone git@git.snakedesert.se:SawatoMosswartsEnjoyersClub/MosswartOverlord.git
|
||||||
git clone https://github.com/yourusername/dereth-tracker.git
|
cd MosswartOverlord
|
||||||
cd dereth-tracker
|
cp .env.example .env # fill in secrets (see Configuration below)
|
||||||
```
|
docker compose up -d
|
||||||
2. Create and activate a virtual environment:
|
```
|
||||||
```bash
|
|
||||||
python3 -m venv venv
|
### Frontend development loop
|
||||||
source venv/bin/activate # Windows: venv\Scripts\activate
|
|
||||||
```
|
```bash
|
||||||
3. Install dependencies:
|
cd frontend
|
||||||
```bash
|
npm install
|
||||||
pip install fastapi uvicorn pydantic websockets
|
npm run dev # local Vite server
|
||||||
```
|
# ...edit files, hot reload...
|
||||||
|
cd ..
|
||||||
|
bash deploy-frontend.sh # builds + copies to static/ for production serving
|
||||||
|
```
|
||||||
|
|
||||||
|
⚠️ **`npm run build` writes to `static/_build/` but the web server serves from `static/`.** You must run `deploy-frontend.sh` to copy `_build/ → static/`. Otherwise the browser keeps loading the previous bundle.
|
||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
- Configure the plugin shared secret via the `SHARED_SECRET` environment variable (default in code: `"your_shared_secret"`).
|
All secrets go in `.env`:
|
||||||
- The database connection is controlled by the `DATABASE_URL` environment variable (e.g. `postgresql://postgres:password@db:5432/dereth`).
|
|
||||||
By default, when using Docker Compose, a TimescaleDB container is provisioned for you.
|
|
||||||
- If you need to tune Timescale or Postgres settings (retention, checkpoint, etc.), set the corresponding `DB_*` environment variables as documented in `docker-compose.yml`.
|
|
||||||
|
|
||||||
## Usage
|
| Variable | Purpose |
|
||||||
|
|---|---|
|
||||||
|
| `POSTGRES_PASSWORD` | Telemetry DB password |
|
||||||
|
| `INVENTORY_DB_PASSWORD` | Inventory DB password |
|
||||||
|
| `SHARED_SECRET` | Plugin auth for `/ws/position` |
|
||||||
|
| `SECRET_KEY` | Session cookie signing |
|
||||||
|
| `DISCORD_RARE_BOT_TOKEN` | Bot token for rare monitor |
|
||||||
|
| `DISCORD_ACLOG_WEBHOOK` | Webhook URL for death/idle alerts |
|
||||||
|
| `GF_SECURITY_ADMIN_PASSWORD` | Grafana admin |
|
||||||
|
| `COMMON_RARE_CHANNEL_ID` | Discord channel ID for common rares |
|
||||||
|
| `GREAT_RARE_CHANNEL_ID` | Discord channel ID for great rares |
|
||||||
|
| `ACLOG_CHANNEL_ID` | Discord channel ID for the rare bot's status/vortex messages |
|
||||||
|
| `MONITOR_CHARACTER` | Which character's chat the bot monitors |
|
||||||
|
|
||||||
### Using Docker (Recommended)
|
## Deploying Changes
|
||||||
|
|
||||||
1. Build and start all services:
|
Live backend host: `overlord.snakedesert.se` (SSH user `erik`, key-based auth).
|
||||||
```bash
|
|
||||||
docker compose up -d
|
|
||||||
```
|
|
||||||
|
|
||||||
2. Rebuild container after code changes:
|
### Quick deploy — Python / static file changes
|
||||||
```bash
|
|
||||||
docker compose build --no-cache dereth-tracker
|
|
||||||
docker compose up -d dereth-tracker
|
|
||||||
```
|
|
||||||
|
|
||||||
3. View logs:
|
|
||||||
```bash
|
|
||||||
docker logs mosswartoverlord-dereth-tracker-1
|
|
||||||
docker logs dereth-db
|
|
||||||
```
|
|
||||||
|
|
||||||
### Without Docker
|
|
||||||
|
|
||||||
Start the server using Uvicorn:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uvicorn main:app --reload --host 0.0.0.0 --port 8000
|
ssh erik@overlord.snakedesert.se \
|
||||||
|
"cd /home/erik/MosswartOverlord && git pull --ff-only origin master"
|
||||||
|
# Python changes require a restart:
|
||||||
|
ssh erik@overlord.snakedesert.se "docker compose restart dereth-tracker"
|
||||||
|
# Static files (JS/CSS/HTML) are served from the bind-mounted static/ — no restart.
|
||||||
```
|
```
|
||||||
|
|
||||||
# Grafana Dashboard UI
|
⚠️ Uvicorn runs **without** `--reload` in production. Do not add it back — without the `watchfiles` package it falls back to a polling reloader that busy-loops at 100% CPU and eats a whole core.
|
||||||
```nginx
|
|
||||||
location /grafana/ {
|
|
||||||
# Optional: require basic auth on the Grafana UI
|
|
||||||
auth_basic "Restricted";
|
|
||||||
auth_basic_user_file /etc/nginx/.htpasswd;
|
|
||||||
|
|
||||||
proxy_pass http://127.0.0.1:3000/;
|
### React frontend deploy
|
||||||
proxy_http_version 1.1;
|
|
||||||
proxy_set_header Host $host;
|
```bash
|
||||||
proxy_set_header X-Real-IP $remote_addr;
|
cd frontend && npm run build && cd ..
|
||||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
bash deploy-frontend.sh
|
||||||
proxy_set_header X-Forwarded-Proto $scheme;
|
git add static/ && git commit -m "deploy frontend" && git push
|
||||||
# Inject Grafana service account token for anonymous panel embeds
|
ssh erik@overlord.snakedesert.se "cd /home/erik/MosswartOverlord && git pull"
|
||||||
proxy_set_header Authorization "Bearer <YOUR_SERVICE_ACCOUNT_TOKEN>";
|
# No container restart needed.
|
||||||
# WebSocket support (for live panels)
|
|
||||||
proxy_set_header Upgrade $http_upgrade;
|
|
||||||
proxy_set_header Connection "upgrade";
|
|
||||||
proxy_cache_bypass $http_upgrade;
|
|
||||||
}
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## NGINX Proxy Configuration
|
### Full rebuild — Dockerfile / pip package / version stamp changes
|
||||||
|
|
||||||
If you cannot reassign the existing `/live` and `/trails` routes, you can namespace this service under `/api` (or any other prefix) and configure NGINX accordingly. Be sure to forward WebSocket upgrade headers so that `/ws/live` and `/ws/position` continue to work. Example:
|
```bash
|
||||||
```nginx
|
ssh erik@overlord.snakedesert.se "cd /home/erik/MosswartOverlord && \
|
||||||
location /api/ {
|
git pull --ff-only origin master && \
|
||||||
proxy_pass http://127.0.0.1:8765/;
|
export BUILD_VERSION=\"\$(date -u +%Y.%-m.%-d.%H%M)-\$(git rev-parse --short HEAD)\" && \
|
||||||
proxy_http_version 1.1;
|
docker compose build --no-cache --build-arg BUILD_VERSION=\$BUILD_VERSION dereth-tracker && \
|
||||||
proxy_set_header Host $host;
|
docker compose up -d dereth-tracker"
|
||||||
proxy_set_header X-Real-IP $remote_addr;
|
|
||||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
|
||||||
proxy_set_header X-Forwarded-Proto $scheme;
|
|
||||||
# WebSocket support
|
|
||||||
proxy_set_header Upgrade $http_upgrade;
|
|
||||||
proxy_set_header Connection "upgrade";
|
|
||||||
proxy_cache_bypass $http_upgrade;
|
|
||||||
}
|
|
||||||
```
|
|
||||||
Then the browser client (static/script.js) will fetch `/api/live/` and `/api/trails/` to reach this new server.
|
|
||||||
|
|
||||||
- Live Map: `http://localhost:8000/` (or `http://<your-domain>/api/` if behind a prefix)
|
|
||||||
- Grafana UI: `http://localhost:3000/grafana/` (or `http://<your-domain>/grafana/` if proxied under that path)
|
|
||||||
|
|
||||||
### Frontend Configuration
|
|
||||||
|
|
||||||
- In `static/script.js`, the constant `API_BASE` controls where live/trails data and WebSocket `/ws/live` are fetched. By default:
|
|
||||||
```js
|
|
||||||
const API_BASE = '/api';
|
|
||||||
```
|
|
||||||
Update `API_BASE` if you mount the service under a different path or serve it at root.
|
|
||||||
|
|
||||||
### Debugging WebSockets
|
|
||||||
|
|
||||||
- Server logs now print every incoming WebSocket frame in `main.py`:
|
|
||||||
- `[WS-PLUGIN RX] <client>: <raw-payload>` for plugin messages on `/ws/position`
|
|
||||||
- `[WS-LIVE RX] <client>: <parsed-json>` for browser messages on `/ws/live`
|
|
||||||
- Use these logs to verify messages and troubleshoot handshake failures.
|
|
||||||
|
|
||||||
### Styling Adjustments
|
|
||||||
|
|
||||||
- Chat input bar is fixed at the bottom of the chat window (`.chat-form { position:absolute; bottom:0; }`).
|
|
||||||
- Input text and placeholder are white for readability (`.chat-input, .chat-input::placeholder { color:#fff; }`).
|
|
||||||
- Incoming chat messages forced white via `.chat-messages div { color:#fff !important; }`.
|
|
||||||
|
|
||||||
## API Reference
|
|
||||||
|
|
||||||
### WebSocket /ws/position
|
|
||||||
Stream telemetry snapshots over a WebSocket connection. Provide your shared secret either as a query parameter or WebSocket header:
|
|
||||||
|
|
||||||
```
|
|
||||||
ws://<host>:<port>/ws/position?secret=<shared_secret>
|
|
||||||
```
|
|
||||||
or
|
|
||||||
```
|
|
||||||
X-Plugin-Secret: <shared_secret>
|
|
||||||
```
|
```
|
||||||
|
|
||||||
After connecting, send JSON messages matching the `TelemetrySnapshot` schema. For example:
|
`BUILD_VERSION` is displayed in the sidebar of the live frontend. Format is CalVer: `YYYY.M.D.HHMM-gitshorthash`.
|
||||||
|
|
||||||
```json
|
## WebSocket Contract
|
||||||
{
|
|
||||||
"type": "telemetry",
|
|
||||||
"character_name": "Dunking Rares",
|
|
||||||
"char_tag": "moss",
|
|
||||||
"session_id": "dunk-20250422-xyz",
|
|
||||||
"timestamp": "2025-04-22T13:45:00Z",
|
|
||||||
"ew": 123.4,
|
|
||||||
"ns": 567.8,
|
|
||||||
"z": 10.2,
|
|
||||||
"kills": 42,
|
|
||||||
"deaths": 1,
|
|
||||||
"prismatic_taper_count": 17,
|
|
||||||
"vt_state": "Combat",
|
|
||||||
"kills_per_hour": "N/A",
|
|
||||||
"onlinetime": "00:05:00"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Each message above is sent as its own JSON object over the WebSocket (one frame per event). When you want to report a rare spawn, send a standalone `rare` event instead of embedding rare counts in telemetry. For example:
|
### `/ws/position` (plugin → backend)
|
||||||
|
|
||||||
```json
|
Authenticated via `?secret=<SHARED_SECRET>` or `X-Plugin-Secret` header. Accepts JSON frames with a `type` discriminator:
|
||||||
{
|
|
||||||
"type": "rare",
|
|
||||||
"timestamp": "2025-04-22T13:48:00Z",
|
|
||||||
"character_name": "MyCharacter",
|
|
||||||
"name": "Golden Gryphon",
|
|
||||||
"ew": 150.5,
|
|
||||||
"ns": 350.7,
|
|
||||||
"z": 5.0,
|
|
||||||
"additional_info": "first sighting of the day"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Chat messages
|
| `type` | Purpose |
|
||||||
You can also send chat envelopes over the same WebSocket to display messages in the browser. Fields:
|
|---|---|
|
||||||
- `type`: must be "chat"
|
| `telemetry` | Position, kills, session metrics (every 2s per character) |
|
||||||
- `character_name`: target player name
|
| `vitals` | Health/stamina/mana/vitae percentages |
|
||||||
- `text`: message content
|
| `character_stats` | Full attributes/skills/allegiance (every 10 min) |
|
||||||
- `color` (optional): CSS color string (e.g. "#ff8800"); if sent as an integer (0xRRGGBB), it will be converted to hex.
|
| `inventory` / `full_inventory` | Complete inventory dump on login |
|
||||||
|
| `inventory_delta` | Incremental add/update/remove of a single item |
|
||||||
|
| `equipment_cantrip_state` | Equipped spell effects |
|
||||||
|
| `portal` | Discovered portal with coordinates |
|
||||||
|
| `spawn` | Monster spawn observation |
|
||||||
|
| `chat` | In-game chat line (any channel) |
|
||||||
|
| `quest` | Quest timer / progress |
|
||||||
|
| `rare` | Rare item find notification |
|
||||||
|
| `nearby_objects` | On-demand radar data (nearby entities) |
|
||||||
|
| `combat_stats` | Session combat snapshot (Mag-Tools parser output) |
|
||||||
|
| `share_*` | Cross-machine vital/debuff sharing envelopes |
|
||||||
|
| `dungeon_map` | Dungeon floor tile data for radar overlay |
|
||||||
|
|
||||||
Example chat payload:
|
See `EVENT_FORMATS.json` for exact per-type schemas.
|
||||||
```json
|
|
||||||
{
|
|
||||||
"type": "chat",
|
|
||||||
"character_name": "MyCharacter",
|
|
||||||
"text": "Hello world!",
|
|
||||||
"color": "#88f"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Event Payload Formats
|
### `/ws/live` (browser → backend)
|
||||||
|
|
||||||
For a complete reference of JSON payloads accepted by the backend (over `/ws/position`), see the file `EVENT_FORMATS.json` in the project root. It contains example schemas for:
|
Session-cookie authenticated (except for internal Docker network clients, which are trusted by IP). Clients can:
|
||||||
- **Telemetry events** (`type`: "telemetry")
|
|
||||||
- **Spawn events** (`type`: "spawn")
|
|
||||||
- **Chat events** (`type`: "chat")
|
|
||||||
- **Rare events** (`type`: "rare")
|
|
||||||
- **Inventory events** (`type`: "inventory")
|
|
||||||
|
|
||||||
Notes on payload changes:
|
- Send `{"type":"subscribe","message_types":["rare","chat"]}` to filter which events they receive. Without subscribing, all types are forwarded (browser default).
|
||||||
- Spawn events no longer require the `z` coordinate; if omitted, the server defaults it to 0.0.
|
- Send `{"player_name":"Larsson","command":"/radar start"}` to route a command to that character's plugin client.
|
||||||
Coordinates (`ew`, `ns`, `z`) may be sent as JSON numbers or strings; the backend will coerce them to floats.
|
- Send `{"type":"request_dungeon_map","landblock":"..."}` to pull cached dungeon tile data.
|
||||||
- Telemetry events have removed the `latency_ms` field; please omit it from your payloads.
|
|
||||||
- Inventory events are sent automatically on character login/logout containing complete inventory data.
|
|
||||||
|
|
||||||
Each entry shows all required and optional fields, their types, and example values.
|
Backend pushes the same firehose (subject to subscription filter) to every browser client.
|
||||||
|
|
||||||
### GET /live
|
## HTTP API Reference
|
||||||
Returns active players seen within the last 30 seconds:
|
|
||||||
|
|
||||||
```json
|
See `EVENT_FORMATS.json` for event schemas. Major HTTP endpoints:
|
||||||
{
|
|
||||||
"players": [ { ... } ]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### GET /history
|
- `GET /live` — active players seen in the last 30s
|
||||||
Retrieve historical snapshots with optional `from` and `to` ISO8601 timestamps:
|
- `GET /history?from=…&to=…` — historical telemetry snapshots
|
||||||
|
- `GET /trails` — recent player trails for the map
|
||||||
```
|
- `GET /spawns/heatmap?hours=N` — aggregated spawn density
|
||||||
GET /history?from=2025-04-22T12:00:00Z&to=2025-04-22T13:00:00Z
|
- `GET /portals` — discovered portals within retention window
|
||||||
```
|
- `GET /inventory/{character}` — current inventory (proxied to inventory-service)
|
||||||
|
- `GET /character-stats/{character}` — full character attributes/skills
|
||||||
Response:
|
- `GET /combat-stats/{character}` — session + lifetime combat stats
|
||||||
|
- `GET /vital-sharing/peers` — currently-registered vital sharing peers
|
||||||
```json
|
- `GET /api-version` — build version stamp
|
||||||
{
|
- `GET /server-health` — current Coldeve server status + player count
|
||||||
"data": [ { ... } ]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Frontend
|
## Frontend
|
||||||
|
|
||||||
- **Live Map**: `static/index.html` – Real-time player positions on a map.
|
### React v2 (primary, at `/`)
|
||||||
- **Inventory Search**: `static/inventory.html` – Search and browse character inventories with advanced filtering.
|
- Map-first layout with draggable/resizable windows
|
||||||
|
- Code-split bundles: one chunk per window type, lazy-loaded on open
|
||||||
|
- Window types: Chat, Stats, Inventory, Character, Radar, CombatStats, CombatPicker, Issues, VitalSharing, QuestStatus, PlayerDashboard
|
||||||
|
- Per-character inventory version counter — an open inventory window refreshes 2s after its own character's last `inventory_delta`, ignoring unrelated traffic
|
||||||
|
- Direct DOM pan/zoom on the map (no React state per frame)
|
||||||
|
- Service worker caches a small whitelist of static assets
|
||||||
|
- Version badge in the sidebar confirms which build is loaded
|
||||||
|
|
||||||
|
### Classic v1 (preserved at `/classic`)
|
||||||
|
The original vanilla JS frontend with element-pooling optimization is kept for fallback and reference.
|
||||||
|
|
||||||
## Database Schema
|
## Database Schema
|
||||||
|
|
||||||
This service uses PostgreSQL with the TimescaleDB extension to store telemetry time-series data,
|
### Telemetry DB (`dereth`, TimescaleDB)
|
||||||
aggregate character statistics, and a separate inventory database for equipment management.
|
|
||||||
|
|
||||||
### Telemetry Database Tables:
|
| Table | Type | Retention | Purpose |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `telemetry_events` | hypertable | 30 days | Position/stats snapshots |
|
||||||
|
| `spawn_events` | hypertable | 7 days | Monster spawn observations (heatmap source) |
|
||||||
|
| `rare_events` | regular | forever | Rare find history |
|
||||||
|
| `portals` | regular | 1 hour | Discovered portals, dedup by rounded coords |
|
||||||
|
| `char_stats` | regular | forever | Per-character lifetime kill total |
|
||||||
|
| `rare_stats` | regular | forever | Per-character lifetime rare total |
|
||||||
|
| `rare_stats_sessions` | regular | forever | Per-session rare count |
|
||||||
|
| `combat_stats` | regular | forever | Lifetime combat accumulator |
|
||||||
|
| `combat_stats_sessions` | regular | forever | Per-session combat snapshots |
|
||||||
|
| `character_stats` | regular | forever | Latest full stats JSON per character |
|
||||||
|
| `server_status` | regular | forever | Current Coldeve server state (single row) |
|
||||||
|
|
||||||
- **telemetry_events** (hypertable):
|
### Inventory DB (`inventory_db`, PostgreSQL)
|
||||||
- `id` (PK, serial)
|
|
||||||
- `character_name` (text, indexed)
|
|
||||||
- `char_tag` (text, nullable)
|
|
||||||
- `session_id` (text, indexed)
|
|
||||||
- `timestamp` (timestamptz, indexed)
|
|
||||||
- `ew`, `ns`, `z` (float)
|
|
||||||
- `kills`, `deaths`, `rares_found`, `prismatic_taper_count` (integer)
|
|
||||||
- `kills_per_hour` (float)
|
|
||||||
- `onlinetime`, `vt_state` (text)
|
|
||||||
- Optional metrics: `mem_mb`, `cpu_pct`, `mem_handles`, `latency_ms` (float)
|
|
||||||
|
|
||||||
- **char_stats**:
|
Normalized schema: `items`, `item_combat_stats`, `item_requirements`, `item_enhancements`, `item_ratings`, `item_spells`, `item_raw_data`.
|
||||||
- `character_name` (text, PK)
|
|
||||||
- `total_kills` (integer)
|
|
||||||
|
|
||||||
- **rare_stats**:
|
`items.container_id` stores the in-game ID of the container holding the item (0 = character body). The frontend groups items into packs by this ID.
|
||||||
- `character_name` (text, PK)
|
|
||||||
- `total_rares` (integer)
|
|
||||||
|
|
||||||
- **rare_stats_sessions**:
|
## Operations & Health
|
||||||
- `character_name`, `session_id` (composite PK)
|
|
||||||
- `session_rares` (integer)
|
|
||||||
|
|
||||||
- **spawn_events**:
|
### PostgreSQL tuning
|
||||||
- `id` (PK, serial)
|
`dereth-db` runs with explicit memory overrides in `docker-compose.yml`:
|
||||||
- `character_name` (text)
|
- `shared_buffers=8GB` (was 96GB via auto-tune on a 32GB host — caused thrashing)
|
||||||
- `mob` (text)
|
- `effective_cache_size=16GB`
|
||||||
- `timestamp` (timestamptz)
|
- `work_mem=16MB`, `maintenance_work_mem=1GB`
|
||||||
- `ew`, `ns`, `z` (float)
|
- `max_wal_size=4GB`
|
||||||
|
|
||||||
- **rare_events**:
|
### Retention policies
|
||||||
- `id` (PK, serial)
|
- `telemetry_events`: 30-day drop, daily
|
||||||
- `character_name` (text)
|
- `spawn_events`: 7-day drop, daily
|
||||||
- `name` (text)
|
- `portals`: 1-hour cleanup (background task in `main.py`)
|
||||||
- `timestamp` (timestamptz)
|
- `server_health_checks`: **removed** — was write-only, 850K rows of nothing
|
||||||
- `ew`, `ns`, `z` (float)
|
|
||||||
|
|
||||||
- **portals**:
|
### Log levels
|
||||||
- `id` (PK, serial)
|
Both `dereth-tracker` and `inventory-service` run at `LOG_LEVEL=INFO`. Do not set to `DEBUG` in production — it dumps full inventory_delta payloads for every item update (hundreds of KB/sec).
|
||||||
- `portal_name` (text)
|
|
||||||
- `ns`, `ew`, `z` (float coordinates)
|
|
||||||
- `discovered_at` (timestamptz, indexed)
|
|
||||||
- `discovered_by` (text)
|
|
||||||
- Unique constraint: `ROUND(ns::numeric, 1), ROUND(ew::numeric, 1)`
|
|
||||||
|
|
||||||
### Inventory Database Tables:
|
### Host (Proxmox VM)
|
||||||
|
- 6 vCPU, 32 GiB RAM (of which ~30 GiB is normally free under current load)
|
||||||
|
- Live host: `overlord.snakedesert.se`
|
||||||
|
- Reverse proxy: Nginx on the host terminates TLS and strips the `/api/` prefix before forwarding to port 8765
|
||||||
|
|
||||||
- **items**:
|
### Debug commands
|
||||||
- `id` (PK, serial)
|
```bash
|
||||||
- `character_name` (text, indexed)
|
docker ps
|
||||||
- `item_id` (bigint)
|
docker logs mosswartoverlord-dereth-tracker-1 --tail 100
|
||||||
- `name` (text)
|
docker logs mosswartoverlord-inventory-service-1 --tail 100
|
||||||
- `object_class` (integer)
|
docker logs mosswartoverlord-discord-rare-monitor-1 --tail 100
|
||||||
- `icon`, `value`, `burden` (integer)
|
docker exec dereth-db psql -U postgres -d dereth
|
||||||
- `current_wielded_location`, `bonded`, `attuned`, `unique` (various)
|
```
|
||||||
- `timestamp` (timestamptz)
|
|
||||||
|
|
||||||
- **item_combat_stats**:
|
|
||||||
- `item_id` (FK to items.id)
|
|
||||||
- `armor_level`, `max_damage` (integer)
|
|
||||||
- `damage_bonus`, `attack_bonus` (float)
|
|
||||||
- Various defense bonuses
|
|
||||||
|
|
||||||
- **item_enhancements**:
|
|
||||||
- `item_id` (FK to items.id)
|
|
||||||
- `material` (varchar)
|
|
||||||
- `item_set` (varchar)
|
|
||||||
- `tinks`, `workmanship` (integer/float)
|
|
||||||
|
|
||||||
- **item_spells**:
|
|
||||||
- `item_id` (FK to items.id)
|
|
||||||
- `spell_id` (integer)
|
|
||||||
- `spell_name` (text)
|
|
||||||
- `is_legendary`, `is_epic` (boolean)
|
|
||||||
|
|
||||||
- **item_raw_data**:
|
|
||||||
- `item_id` (FK to items.id)
|
|
||||||
- `int_values`, `double_values`, `string_values`, `bool_values` (JSONB)
|
|
||||||
- `original_json` (JSONB)
|
|
||||||
|
|
||||||
## Contributing
|
## Contributing
|
||||||
|
|
||||||
Contributions are welcome! Feel free to open issues or submit pull requests.
|
Contributions welcome. Please:
|
||||||
|
- Keep cross-repo protocol changes additive (new optional fields > renames/removes)
|
||||||
|
- Update both this README and `CLAUDE.md` when workflows change
|
||||||
|
- Test end-to-end: plugin → backend → browser for any new event type
|
||||||
|
|
||||||
## Roadmap & TODO
|
For detailed architecture notes and ongoing investigations, see `CLAUDE.md` and `docs/plans/`.
|
||||||
For detailed tasks, migration steps, and future enhancements, see [TODO.md](TODO.md).
|
|
||||||
|
|
||||||
### Local Development Database
|
|
||||||
This service uses PostgreSQL with the TimescaleDB extension. You can configure local development using the provided Docker Compose setup or connect to an external instance:
|
|
||||||
|
|
||||||
1. PostgreSQL/TimescaleDB via Docker Compose (recommended):
|
|
||||||
- Pros:
|
|
||||||
- Reproducible, isolated environment out-of-the-box
|
|
||||||
- No need to install Postgres locally
|
|
||||||
- Aligns development with production setups
|
|
||||||
- Cons:
|
|
||||||
- Additional resource usage (memory, CPU)
|
|
||||||
- Slightly more complex Docker configuration
|
|
||||||
|
|
||||||
2. External PostgreSQL instance:
|
|
||||||
- Pros:
|
|
||||||
- Leverages existing infrastructure
|
|
||||||
- No Docker overhead
|
|
||||||
- Cons:
|
|
||||||
- Requires manual setup and Timescale extension
|
|
||||||
- Less portable for new contributors
|
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue