SawatoMosswartsEnjoyersClub/MosswartOverlord
3
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: rewrite README to reflect current architecture

Browse source 
Full rewrite covering:
- React v2 frontend at /, classic v1 preserved at /classic
- WebSocket message-type subscription mechanism (bot filter fix)
- Death + idle alerts via Discord webhook with 5-min grace period
- spawn_events now a TimescaleDB hypertable with 7-day retention
- server_health_checks removed (write-only bloat)
- PostgreSQL memory tuning (shared_buffers 8GB on 32GB host)
- Uvicorn runs without --reload in production
- deploy-frontend.sh requirement for React builds
- Combat stats (Mag-Tools style), vital sharing, all WS event types
- Cross-machine vital sharing via WebSocket relay
- Deploy flows (quick / frontend / full rebuild)
- BUILD_VERSION CalVer stamp format

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Erik 2026-04-24 14:55:30 +02:00
parent f7f04d6a84
commit 3c634adbdc
1 changed files with 254 additions and 347 deletions
Download patch file Download diff file Expand all files Collapse all files

601
README.md
View file

@ -1,412 +1,319 @@
# Dereth Tracker
# Mosswart Overlord (Dereth Tracker)
Dereth Tracker is a real-time telemetry service for the world of Dereth. It collects player data, stores it in a PostgreSQL (TimescaleDB) database for efficient time-series storage, provides a live map interface, and includes a comprehensive inventory management system for tracking and searching character equipment.
Real-time telemetry, inventory, and analytics platform for Asheron's Call.
FastAPI backend + React frontend + PostgreSQL (TimescaleDB) + Discord integrations,
all driven by WebSocket events from the companion [MosswartMassacre](https://github.com/SawatoMosswartsEnjoyersClub/MosswartMassacre) DECAL plugin.
---
## Table of Contents
- [Overview](#overview)
- [Architecture](#architecture)
- [Features](#features)
- [Requirements](#requirements)
- [Installation](#installation)
- [Configuration](#configuration)
- [Usage](#usage)
- [API Reference](#api-reference)
- [Deploying Changes](#deploying-changes)
- [WebSocket Contract](#websocket-contract)
- [HTTP API Reference](#http-api-reference)
- [Frontend](#frontend)
- [Database Schema](#database-schema)
- [Operations & Health](#operations--health)
- [Contributing](#contributing)
---
## Overview
This project provides:
- A FastAPI backend with endpoints for receiving and querying telemetry data.
- PostgreSQL/TimescaleDB-based storage for time-series telemetry and per-character stats.
- A live, interactive map using static HTML, CSS, and JavaScript.
- A comprehensive inventory management system with search capabilities.
- Real-time inventory updates via WebSocket when characters log in/out.
- A sample data generator script (`generate_data.py`) for simulating telemetry snapshots.
Mosswart Overlord is the backend that consumes a firehose of telemetry, vitals, inventory, combat, and chat events from 60+ characters running the `MosswartMassacre` plugin. It stores selected data in TimescaleDB, runs analytics (combat stats, idle/death detection), and broadcasts live updates to connected browser clients.
The frontend is a React + Vite app served at `/` with a live map, draggable windows (inventory, chat, combat, radar, etc.), and a server uptime sidebar. The previous vanilla JS frontend is preserved at `/classic`.
## Architecture
```
┌─────────────────────────┐
│ MosswartMassacre (C#) │ ← plugin per game client
└────────────┬────────────┘
│ WebSocket /ws/position (authenticated)
▼
┌────────────────────────────────────────────────────────┐
│ dereth-tracker (FastAPI) │
│ • main.py — WS routing, analytics, broadcasts │
│ • idle/death detection → Discord webhook │
│ • combat stats delta/lifetime accumulation │
│ • vital sharing relay (cross-machine) │
└──┬──────────────────┬────────────────────┬────────────┘
│ │ │
│ WS /ws/live │ HTTP │ HTTP
▼ ▼ ▼
┌──────────┐ ┌──────────────────┐ ┌──────────────────┐
│ Browsers │ │ inventory-svc │ │ Discord bot │
│ (React) │ │ (FastAPI) │ │ (rare monitor) │
└──────────┘ └────────┬─────────┘ └──────────────────┘
▼
┌──────────────┐
│ inventory-db │
└──────────────┘
┌──────────────┐
│ dereth-db │ ← TimescaleDB (telemetry, spawns, rares, portals)
└──────────────┘
```
All services run via Docker Compose.
## Features
- **WebSocket /ws/position**: Stream telemetry snapshots and inventory updates (protected by a shared secret).
- **GET /live**: Fetch active players seen in the last 30 seconds.
- **GET /history**: Retrieve historical telemetry data with optional time filtering.
- **GET /debug**: Health check endpoint.
- **Live Map**: Interactive map interface with panning, zooming, and sorting.
- **Inventory Management**:
- Real-time inventory updates via WebSocket on character login/logout
- Advanced search across all character inventories
- Filter by character, equipment type, material, stats, and more
- Sort by any column with live results
- Track item properties including spells, armor level, damage ratings
- **Suitbuilder**:
- Equipment optimization across multiple character inventories
- Constraint-based search for optimal armor combinations
- Support for primary and secondary armor sets
- Real-time streaming results during long-running searches
- **Portal Tracking**:
- Automatic discovery and tracking of in-game portals
- 1-hour retention for discovered portals
- Coordinate-based uniqueness (rounded to 0.1 precision)
- Real-time portal updates on the map interface
- **Discord Rare Monitor Bot**: Monitors rare discoveries and posts filtered notifications to Discord channels
- **Sample Data Generator**: `generate_data.py` sends telemetry snapshots over WebSocket for testing.
### Live Data
- **Live Map** — real-time player positions, dots, trails, portals, heatmap
- **WebSocket firehose** (`/ws/live`) — broadcasts every incoming event to browsers
- **Per-client subscriptions** — clients can send `{"type":"subscribe","message_types":[...]}` to receive only specific event types (the Discord rare monitor bot uses this to filter the 82GB/day firehose down to just `rare` and `chat`)
### Inventory
- Full inventory snapshot on login + incremental `inventory_delta` updates (add/update/remove)
- Per-character live refresh in the browser (debounced 2s)
- Advanced search with filters: material, set, armor level, spells, tinks, workmanship, etc.
- **Suitbuilder** at `/suitbuilder.html` — constraint-based armor optimization across multiple mule inventories with primary/secondary set support, cantrip overlap detection, and real-time SSE streaming
### Combat Stats (Mag-Tools style)
- Plugin parses combat chat into session deltas
- Backend accumulates lifetime totals from per-session snapshots
- Offense/defense broken out per damage element
- Browser combat window shows monster-by-monster damage
### Cross-Machine Vital Sharing
- WebSocket relay replaces UtilityBelt's localhost-only `VTankFellowHeals`
- Plugin broadcasts its own vitals and consumes peer vitals
- In-game `DxHud` overlay shows peer health/stamina/mana bars with direction arrows
### Discord Integration
- **Rare Monitor Bot** — posts rares (split by common/great) to configured channels
- **Death Alerts** — webhook to `#alerts` when a character's vitae goes from 0 → >0 (rate-limited to one per character per 5 min)
- **Idle Alerts** — webhook after 5 minutes of continuous idle state (caught portals, stuck nav, etc.). The grace period prevents false positives on brief idle blips.
- **Vortex Warning** — bot watches for "whirlwind of vortexes" chat and posts a warning embed
### Portals
- Automatic discovery + 1-hour retention
- Coordinate-deduplicated (rounded to 0.1 precision)
### Stats
- Per-character lifetime kills, deaths, rares, taper counts
- Grafana dashboards (2x2 iframe grid in the stats window)
### Health & Monitoring
- Server uptime + latency + player count from TreeStats.net (checked every 30s)
- Only current state is kept — no historical `server_health_checks` table (removed April 2026 as write-only bloat)
## Requirements
- Python 3.9 or newer (only if running without Docker)
- pip (only if running without Docker)
- Docker & Docker Compose (recommended)
Python packages (if using local virtualenv):
- fastapi
- uvicorn
- pydantic
- databases
- asyncpg
- sqlalchemy
- websockets # required for sample data generator
- OR: Python 3.11+, Node.js 20+, and a PostgreSQL 14+ with TimescaleDB
## Installation
1. Clone the repository:
```bash
git clone https://github.com/yourusername/dereth-tracker.git
cd dereth-tracker
```
2. Create and activate a virtual environment:
```bash
python3 -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
```
3. Install dependencies:
```bash
pip install fastapi uvicorn pydantic websockets
```
```bash
git clone git@git.snakedesert.se:SawatoMosswartsEnjoyersClub/MosswartOverlord.git
cd MosswartOverlord
cp .env.example .env # fill in secrets (see Configuration below)
docker compose up -d
```
### Frontend development loop
```bash
cd frontend
npm install
npm run dev # local Vite server
# ...edit files, hot reload...
cd ..
bash deploy-frontend.sh # builds + copies to static/ for production serving
```
⚠️ **`npm run build` writes to `static/_build/` but the web server serves from `static/`.** You must run `deploy-frontend.sh` to copy `_build/ → static/`. Otherwise the browser keeps loading the previous bundle.
## Configuration
- Configure the plugin shared secret via the `SHARED_SECRET` environment variable (default in code: `"your_shared_secret"`).
- The database connection is controlled by the `DATABASE_URL` environment variable (e.g. `postgresql://postgres:password@db:5432/dereth`).
By default, when using Docker Compose, a TimescaleDB container is provisioned for you.
- If you need to tune Timescale or Postgres settings (retention, checkpoint, etc.), set the corresponding `DB_*` environment variables as documented in `docker-compose.yml`.
All secrets go in `.env`:
## Usage
| Variable | Purpose |
|---|---|
| `POSTGRES_PASSWORD` | Telemetry DB password |
| `INVENTORY_DB_PASSWORD` | Inventory DB password |
| `SHARED_SECRET` | Plugin auth for `/ws/position` |
| `SECRET_KEY` | Session cookie signing |
| `DISCORD_RARE_BOT_TOKEN` | Bot token for rare monitor |
| `DISCORD_ACLOG_WEBHOOK` | Webhook URL for death/idle alerts |
| `GF_SECURITY_ADMIN_PASSWORD` | Grafana admin |
| `COMMON_RARE_CHANNEL_ID` | Discord channel ID for common rares |
| `GREAT_RARE_CHANNEL_ID` | Discord channel ID for great rares |
| `ACLOG_CHANNEL_ID` | Discord channel ID for the rare bot's status/vortex messages |
| `MONITOR_CHARACTER` | Which character's chat the bot monitors |
### Using Docker (Recommended)
## Deploying Changes
1. Build and start all services:
```bash
docker compose up -d
```
Live backend host: `overlord.snakedesert.se` (SSH user `erik`, key-based auth).
2. Rebuild container after code changes:
```bash
docker compose build --no-cache dereth-tracker
docker compose up -d dereth-tracker
```
3. View logs:
```bash
docker logs mosswartoverlord-dereth-tracker-1
docker logs dereth-db
```
### Without Docker
Start the server using Uvicorn:
### Quick deploy — Python / static file changes
```bash
uvicorn main:app --reload --host 0.0.0.0 --port 8000
```
# Grafana Dashboard UI
```nginx
location /grafana/ {
# Optional: require basic auth on the Grafana UI
auth_basic "Restricted";
auth_basic_user_file /etc/nginx/.htpasswd;
proxy_pass http://127.0.0.1:3000/;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Inject Grafana service account token for anonymous panel embeds
proxy_set_header Authorization "Bearer <YOUR_SERVICE_ACCOUNT_TOKEN>";
# WebSocket support (for live panels)
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_cache_bypass $http_upgrade;
}
```
## NGINX Proxy Configuration
If you cannot reassign the existing `/live` and `/trails` routes, you can namespace this service under `/api` (or any other prefix) and configure NGINX accordingly. Be sure to forward WebSocket upgrade headers so that `/ws/live` and `/ws/position` continue to work. Example:
```nginx
location /api/ {
proxy_pass http://127.0.0.1:8765/;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# WebSocket support
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_cache_bypass $http_upgrade;
}
```
Then the browser client (static/script.js) will fetch `/api/live/` and `/api/trails/` to reach this new server.
- Live Map: `http://localhost:8000/` (or `http://<your-domain>/api/` if behind a prefix)
- Grafana UI: `http://localhost:3000/grafana/` (or `http://<your-domain>/grafana/` if proxied under that path)
### Frontend Configuration
- In `static/script.js`, the constant `API_BASE` controls where live/trails data and WebSocket `/ws/live` are fetched. By default:
```js
const API_BASE = '/api';
```
Update `API_BASE` if you mount the service under a different path or serve it at root.
### Debugging WebSockets
- Server logs now print every incoming WebSocket frame in `main.py`:
- `[WS-PLUGIN RX] <client>: <raw-payload>` for plugin messages on `/ws/position`
- `[WS-LIVE RX] <client>: <parsed-json>` for browser messages on `/ws/live`
- Use these logs to verify messages and troubleshoot handshake failures.
### Styling Adjustments
- Chat input bar is fixed at the bottom of the chat window (`.chat-form { position:absolute; bottom:0; }`).
- Input text and placeholder are white for readability (`.chat-input, .chat-input::placeholder { color:#fff; }`).
- Incoming chat messages forced white via `.chat-messages div { color:#fff !important; }`.
## API Reference
### WebSocket /ws/position
Stream telemetry snapshots over a WebSocket connection. Provide your shared secret either as a query parameter or WebSocket header:
```
ws://<host>:<port>/ws/position?secret=<shared_secret>
```
or
```
X-Plugin-Secret: <shared_secret>
ssh erik@overlord.snakedesert.se \
"cd /home/erik/MosswartOverlord && git pull --ff-only origin master"
# Python changes require a restart:
ssh erik@overlord.snakedesert.se "docker compose restart dereth-tracker"
# Static files (JS/CSS/HTML) are served from the bind-mounted static/ — no restart.
```
After connecting, send JSON messages matching the `TelemetrySnapshot` schema. For example:
⚠️ Uvicorn runs **without** `--reload` in production. Do not add it back — without the `watchfiles` package it falls back to a polling reloader that busy-loops at 100% CPU and eats a whole core.
```json
{
"type": "telemetry",
"character_name": "Dunking Rares",
"char_tag": "moss",
"session_id": "dunk-20250422-xyz",
"timestamp": "2025-04-22T13:45:00Z",
"ew": 123.4,
"ns": 567.8,
"z": 10.2,
"kills": 42,
"deaths": 1,
"prismatic_taper_count": 17,
"vt_state": "Combat",
"kills_per_hour": "N/A",
"onlinetime": "00:05:00"
}
```
Each message above is sent as its own JSON object over the WebSocket (one frame per event). When you want to report a rare spawn, send a standalone `rare` event instead of embedding rare counts in telemetry. For example:
```json
{
"type": "rare",
"timestamp": "2025-04-22T13:48:00Z",
"character_name": "MyCharacter",
"name": "Golden Gryphon",
"ew": 150.5,
"ns": 350.7,
"z": 5.0,
"additional_info": "first sighting of the day"
}
```
### React frontend deploy
### Chat messages
You can also send chat envelopes over the same WebSocket to display messages in the browser. Fields:
- `type`: must be "chat"
- `character_name`: target player name
- `text`: message content
- `color` (optional): CSS color string (e.g. "#ff8800"); if sent as an integer (0xRRGGBB), it will be converted to hex.
Example chat payload:
```json
{
"type": "chat",
"character_name": "MyCharacter",
"text": "Hello world!",
"color": "#88f"
}
```bash
cd frontend && npm run build && cd ..
bash deploy-frontend.sh
git add static/ && git commit -m "deploy frontend" && git push
ssh erik@overlord.snakedesert.se "cd /home/erik/MosswartOverlord && git pull"
# No container restart needed.
```
## Event Payload Formats
### Full rebuild — Dockerfile / pip package / version stamp changes
For a complete reference of JSON payloads accepted by the backend (over `/ws/position`), see the file `EVENT_FORMATS.json` in the project root. It contains example schemas for:
- **Telemetry events** (`type`: "telemetry")
- **Spawn events** (`type`: "spawn")
- **Chat events** (`type`: "chat")
- **Rare events** (`type`: "rare")
- **Inventory events** (`type`: "inventory")
Notes on payload changes:
- Spawn events no longer require the `z` coordinate; if omitted, the server defaults it to 0.0.
Coordinates (`ew`, `ns`, `z`) may be sent as JSON numbers or strings; the backend will coerce them to floats.
- Telemetry events have removed the `latency_ms` field; please omit it from your payloads.
- Inventory events are sent automatically on character login/logout containing complete inventory data.
Each entry shows all required and optional fields, their types, and example values.
### GET /live
Returns active players seen within the last 30 seconds:
```json
{
"players": [ { ... } ]
}
```bash
ssh erik@overlord.snakedesert.se "cd /home/erik/MosswartOverlord && \
git pull --ff-only origin master && \
export BUILD_VERSION=\"\$(date -u +%Y.%-m.%-d.%H%M)-\$(git rev-parse --short HEAD)\" && \
docker compose build --no-cache --build-arg BUILD_VERSION=\$BUILD_VERSION dereth-tracker && \
docker compose up -d dereth-tracker"
```
### GET /history
Retrieve historical snapshots with optional `from` and `to` ISO8601 timestamps:
`BUILD_VERSION` is displayed in the sidebar of the live frontend. Format is CalVer: `YYYY.M.D.HHMM-gitshorthash`.
```
GET /history?from=2025-04-22T12:00:00Z&to=2025-04-22T13:00:00Z
```
## WebSocket Contract
Response:
### `/ws/position` (plugin → backend)
```json
{
"data": [ { ... } ]
}
```
Authenticated via `?secret=<SHARED_SECRET>` or `X-Plugin-Secret` header. Accepts JSON frames with a `type` discriminator:
| `type` | Purpose |
|---|---|
| `telemetry` | Position, kills, session metrics (every 2s per character) |
| `vitals` | Health/stamina/mana/vitae percentages |
| `character_stats` | Full attributes/skills/allegiance (every 10 min) |
| `inventory` / `full_inventory` | Complete inventory dump on login |
| `inventory_delta` | Incremental add/update/remove of a single item |
| `equipment_cantrip_state` | Equipped spell effects |
| `portal` | Discovered portal with coordinates |
| `spawn` | Monster spawn observation |
| `chat` | In-game chat line (any channel) |
| `quest` | Quest timer / progress |
| `rare` | Rare item find notification |
| `nearby_objects` | On-demand radar data (nearby entities) |
| `combat_stats` | Session combat snapshot (Mag-Tools parser output) |
| `share_*` | Cross-machine vital/debuff sharing envelopes |
| `dungeon_map` | Dungeon floor tile data for radar overlay |
See `EVENT_FORMATS.json` for exact per-type schemas.
### `/ws/live` (browser → backend)
Session-cookie authenticated (except for internal Docker network clients, which are trusted by IP). Clients can:
- Send `{"type":"subscribe","message_types":["rare","chat"]}` to filter which events they receive. Without subscribing, all types are forwarded (browser default).
- Send `{"player_name":"Larsson","command":"/radar start"}` to route a command to that character's plugin client.
- Send `{"type":"request_dungeon_map","landblock":"..."}` to pull cached dungeon tile data.
Backend pushes the same firehose (subject to subscription filter) to every browser client.
## HTTP API Reference
See `EVENT_FORMATS.json` for event schemas. Major HTTP endpoints:
- `GET /live` — active players seen in the last 30s
- `GET /history?from=…&to=…` — historical telemetry snapshots
- `GET /trails` — recent player trails for the map
- `GET /spawns/heatmap?hours=N` — aggregated spawn density
- `GET /portals` — discovered portals within retention window
- `GET /inventory/{character}` — current inventory (proxied to inventory-service)
- `GET /character-stats/{character}` — full character attributes/skills
- `GET /combat-stats/{character}` — session + lifetime combat stats
- `GET /vital-sharing/peers` — currently-registered vital sharing peers
- `GET /api-version` — build version stamp
- `GET /server-health` — current Coldeve server status + player count
## Frontend
- **Live Map**: `static/index.html` – Real-time player positions on a map.
- **Inventory Search**: `static/inventory.html` – Search and browse character inventories with advanced filtering.
### React v2 (primary, at `/`)
- Map-first layout with draggable/resizable windows
- Code-split bundles: one chunk per window type, lazy-loaded on open
- Window types: Chat, Stats, Inventory, Character, Radar, CombatStats, CombatPicker, Issues, VitalSharing, QuestStatus, PlayerDashboard
- Per-character inventory version counter — an open inventory window refreshes 2s after its own character's last `inventory_delta`, ignoring unrelated traffic
- Direct DOM pan/zoom on the map (no React state per frame)
- Service worker caches a small whitelist of static assets
- Version badge in the sidebar confirms which build is loaded
### Classic v1 (preserved at `/classic`)
The original vanilla JS frontend with element-pooling optimization is kept for fallback and reference.
## Database Schema
This service uses PostgreSQL with the TimescaleDB extension to store telemetry time-series data,
aggregate character statistics, and a separate inventory database for equipment management.
### Telemetry DB (`dereth`, TimescaleDB)
### Telemetry Database Tables:
| Table | Type | Retention | Purpose |
|---|---|---|---|
| `telemetry_events` | hypertable | 30 days | Position/stats snapshots |
| `spawn_events` | hypertable | 7 days | Monster spawn observations (heatmap source) |
| `rare_events` | regular | forever | Rare find history |
| `portals` | regular | 1 hour | Discovered portals, dedup by rounded coords |
| `char_stats` | regular | forever | Per-character lifetime kill total |
| `rare_stats` | regular | forever | Per-character lifetime rare total |
| `rare_stats_sessions` | regular | forever | Per-session rare count |
| `combat_stats` | regular | forever | Lifetime combat accumulator |
| `combat_stats_sessions` | regular | forever | Per-session combat snapshots |
| `character_stats` | regular | forever | Latest full stats JSON per character |
| `server_status` | regular | forever | Current Coldeve server state (single row) |
- **telemetry_events** (hypertable):
- `id` (PK, serial)
- `character_name` (text, indexed)
- `char_tag` (text, nullable)
- `session_id` (text, indexed)
- `timestamp` (timestamptz, indexed)
- `ew`, `ns`, `z` (float)
- `kills`, `deaths`, `rares_found`, `prismatic_taper_count` (integer)
- `kills_per_hour` (float)
- `onlinetime`, `vt_state` (text)
- Optional metrics: `mem_mb`, `cpu_pct`, `mem_handles`, `latency_ms` (float)
### Inventory DB (`inventory_db`, PostgreSQL)
- **char_stats**:
- `character_name` (text, PK)
- `total_kills` (integer)
Normalized schema: `items`, `item_combat_stats`, `item_requirements`, `item_enhancements`, `item_ratings`, `item_spells`, `item_raw_data`.
- **rare_stats**:
- `character_name` (text, PK)
- `total_rares` (integer)
`items.container_id` stores the in-game ID of the container holding the item (0 = character body). The frontend groups items into packs by this ID.
- **rare_stats_sessions**:
- `character_name`, `session_id` (composite PK)
- `session_rares` (integer)
## Operations & Health
- **spawn_events**:
- `id` (PK, serial)
- `character_name` (text)
- `mob` (text)
- `timestamp` (timestamptz)
- `ew`, `ns`, `z` (float)
### PostgreSQL tuning
`dereth-db` runs with explicit memory overrides in `docker-compose.yml`:
- `shared_buffers=8GB` (was 96GB via auto-tune on a 32GB host — caused thrashing)
- `effective_cache_size=16GB`
- `work_mem=16MB`, `maintenance_work_mem=1GB`
- `max_wal_size=4GB`
- **rare_events**:
- `id` (PK, serial)
- `character_name` (text)
- `name` (text)
- `timestamp` (timestamptz)
- `ew`, `ns`, `z` (float)
### Retention policies
- `telemetry_events`: 30-day drop, daily
- `spawn_events`: 7-day drop, daily
- `portals`: 1-hour cleanup (background task in `main.py`)
- `server_health_checks`: **removed** — was write-only, 850K rows of nothing
- **portals**:
- `id` (PK, serial)
- `portal_name` (text)
- `ns`, `ew`, `z` (float coordinates)
- `discovered_at` (timestamptz, indexed)
- `discovered_by` (text)
- Unique constraint: `ROUND(ns::numeric, 1), ROUND(ew::numeric, 1)`
### Log levels
Both `dereth-tracker` and `inventory-service` run at `LOG_LEVEL=INFO`. Do not set to `DEBUG` in production — it dumps full inventory_delta payloads for every item update (hundreds of KB/sec).
### Inventory Database Tables:
### Host (Proxmox VM)
- 6 vCPU, 32 GiB RAM (of which ~30 GiB is normally free under current load)
- Live host: `overlord.snakedesert.se`
- Reverse proxy: Nginx on the host terminates TLS and strips the `/api/` prefix before forwarding to port 8765
- **items**:
- `id` (PK, serial)
- `character_name` (text, indexed)
- `item_id` (bigint)
- `name` (text)
- `object_class` (integer)
- `icon`, `value`, `burden` (integer)
- `current_wielded_location`, `bonded`, `attuned`, `unique` (various)
- `timestamp` (timestamptz)
- **item_combat_stats**:
- `item_id` (FK to items.id)
- `armor_level`, `max_damage` (integer)
- `damage_bonus`, `attack_bonus` (float)
- Various defense bonuses
- **item_enhancements**:
- `item_id` (FK to items.id)
- `material` (varchar)
- `item_set` (varchar)
- `tinks`, `workmanship` (integer/float)
- **item_spells**:
- `item_id` (FK to items.id)
- `spell_id` (integer)
- `spell_name` (text)
- `is_legendary`, `is_epic` (boolean)
- **item_raw_data**:
- `item_id` (FK to items.id)
- `int_values`, `double_values`, `string_values`, `bool_values` (JSONB)
- `original_json` (JSONB)
### Debug commands
```bash
docker ps
docker logs mosswartoverlord-dereth-tracker-1 --tail 100
docker logs mosswartoverlord-inventory-service-1 --tail 100
docker logs mosswartoverlord-discord-rare-monitor-1 --tail 100
docker exec dereth-db psql -U postgres -d dereth
```
## Contributing
Contributions are welcome! Feel free to open issues or submit pull requests.
Contributions welcome. Please:
- Keep cross-repo protocol changes additive (new optional fields > renames/removes)
- Update both this README and `CLAUDE.md` when workflows change
- Test end-to-end: plugin → backend → browser for any new event type
## Roadmap & TODO
For detailed tasks, migration steps, and future enhancements, see [TODO.md](TODO.md).
### Local Development Database
This service uses PostgreSQL with the TimescaleDB extension. You can configure local development using the provided Docker Compose setup or connect to an external instance:
1. PostgreSQL/TimescaleDB via Docker Compose (recommended):
- Pros:
- Reproducible, isolated environment out-of-the-box
- No need to install Postgres locally
- Aligns development with production setups
- Cons:
- Additional resource usage (memory, CPU)
- Slightly more complex Docker configuration
2. External PostgreSQL instance:
- Pros:
- Leverages existing infrastructure
- No Docker overhead
- Cons:
- Requires manual setup and Timescale extension
- Less portable for new contributors
For detailed architecture notes and ongoing investigations, see `CLAUDE.md` and `docs/plans/`.