373 lines
12 KiB
Markdown
373 lines
12 KiB
Markdown
# Dereth Tracker
|
||
|
||
Dereth Tracker is a real-time telemetry service for the world of Dereth. It collects player data, stores it in a PostgreSQL (TimescaleDB) database for efficient time-series storage, provides a live map interface, and includes a comprehensive inventory management system for tracking and searching character equipment.
|
||
|
||
## Table of Contents
|
||
- [Overview](#overview)
|
||
- [Features](#features)
|
||
- [Requirements](#requirements)
|
||
- [Installation](#installation)
|
||
- [Configuration](#configuration)
|
||
- [Usage](#usage)
|
||
- [API Reference](#api-reference)
|
||
- [Frontend](#frontend)
|
||
- [Database Schema](#database-schema)
|
||
- [Contributing](#contributing)
|
||
|
||
## Overview
|
||
|
||
This project provides:
|
||
- A FastAPI backend with endpoints for receiving and querying telemetry data.
|
||
- PostgreSQL/TimescaleDB-based storage for time-series telemetry and per-character stats.
|
||
- A live, interactive map using static HTML, CSS, and JavaScript.
|
||
- A comprehensive inventory management system with search capabilities.
|
||
- Real-time inventory updates via WebSocket when characters log in/out.
|
||
- A sample data generator script (`generate_data.py`) for simulating telemetry snapshots.
|
||
|
||
## Features
|
||
|
||
- **WebSocket /ws/position**: Stream telemetry snapshots and inventory updates (protected by a shared secret).
|
||
- **GET /live**: Fetch active players seen in the last 30 seconds.
|
||
- **GET /history**: Retrieve historical telemetry data with optional time filtering.
|
||
- **GET /debug**: Health check endpoint.
|
||
- **Live Map**: Interactive map interface with panning, zooming, and sorting.
|
||
- **Inventory Management**:
|
||
- Real-time inventory updates via WebSocket on character login/logout
|
||
- Advanced search across all character inventories
|
||
- Filter by character, equipment type, material, stats, and more
|
||
- Sort by any column with live results
|
||
- Track item properties including spells, armor level, damage ratings
|
||
- **Discord Rare Monitor Bot**: Monitors rare discoveries and posts filtered notifications to Discord channels
|
||
- **Sample Data Generator**: `generate_data.py` sends telemetry snapshots over WebSocket for testing.
|
||
|
||
## Requirements
|
||
|
||
- Python 3.9 or newer (only if running without Docker)
|
||
- pip (only if running without Docker)
|
||
- Docker & Docker Compose (recommended)
|
||
|
||
Python packages (if using local virtualenv):
|
||
|
||
- fastapi
|
||
- uvicorn
|
||
- pydantic
|
||
- databases
|
||
- asyncpg
|
||
- sqlalchemy
|
||
- websockets # required for sample data generator
|
||
|
||
## Installation
|
||
|
||
1. Clone the repository:
|
||
```bash
|
||
git clone https://github.com/yourusername/dereth-tracker.git
|
||
cd dereth-tracker
|
||
```
|
||
2. Create and activate a virtual environment:
|
||
```bash
|
||
python3 -m venv venv
|
||
source venv/bin/activate # Windows: venv\Scripts\activate
|
||
```
|
||
3. Install dependencies:
|
||
```bash
|
||
pip install fastapi uvicorn pydantic websockets
|
||
```
|
||
|
||
## Configuration
|
||
|
||
- Configure the plugin shared secret via the `SHARED_SECRET` environment variable (default in code: `"your_shared_secret"`).
|
||
- The database connection is controlled by the `DATABASE_URL` environment variable (e.g. `postgresql://postgres:password@db:5432/dereth`).
|
||
By default, when using Docker Compose, a TimescaleDB container is provisioned for you.
|
||
- If you need to tune Timescale or Postgres settings (retention, checkpoint, etc.), set the corresponding `DB_*` environment variables as documented in `docker-compose.yml`.
|
||
|
||
## Usage
|
||
|
||
Start the server using Uvicorn:
|
||
|
||
```bash
|
||
uvicorn main:app --reload --host 0.0.0.0 --port 8000
|
||
```
|
||
|
||
# Grafana Dashboard UI
|
||
```nginx
|
||
location /grafana/ {
|
||
# Optional: require basic auth on the Grafana UI
|
||
auth_basic "Restricted";
|
||
auth_basic_user_file /etc/nginx/.htpasswd;
|
||
|
||
proxy_pass http://127.0.0.1:3000/;
|
||
proxy_http_version 1.1;
|
||
proxy_set_header Host $host;
|
||
proxy_set_header X-Real-IP $remote_addr;
|
||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||
proxy_set_header X-Forwarded-Proto $scheme;
|
||
# Inject Grafana service account token for anonymous panel embeds
|
||
proxy_set_header Authorization "Bearer <YOUR_SERVICE_ACCOUNT_TOKEN>";
|
||
# WebSocket support (for live panels)
|
||
proxy_set_header Upgrade $http_upgrade;
|
||
proxy_set_header Connection "upgrade";
|
||
proxy_cache_bypass $http_upgrade;
|
||
}
|
||
```
|
||
|
||
## NGINX Proxy Configuration
|
||
|
||
If you cannot reassign the existing `/live` and `/trails` routes, you can namespace this service under `/api` (or any other prefix) and configure NGINX accordingly. Be sure to forward WebSocket upgrade headers so that `/ws/live` and `/ws/position` continue to work. Example:
|
||
```nginx
|
||
location /api/ {
|
||
proxy_pass http://127.0.0.1:8765/;
|
||
proxy_http_version 1.1;
|
||
proxy_set_header Host $host;
|
||
proxy_set_header X-Real-IP $remote_addr;
|
||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||
proxy_set_header X-Forwarded-Proto $scheme;
|
||
# WebSocket support
|
||
proxy_set_header Upgrade $http_upgrade;
|
||
proxy_set_header Connection "upgrade";
|
||
proxy_cache_bypass $http_upgrade;
|
||
}
|
||
```
|
||
Then the browser client (static/script.js) will fetch `/api/live/` and `/api/trails/` to reach this new server.
|
||
|
||
- Live Map: `http://localhost:8000/` (or `http://<your-domain>/api/` if behind a prefix)
|
||
- Grafana UI: `http://localhost:3000/grafana/` (or `http://<your-domain>/grafana/` if proxied under that path)
|
||
|
||
### Frontend Configuration
|
||
|
||
- In `static/script.js`, the constant `API_BASE` controls where live/trails data and WebSocket `/ws/live` are fetched. By default:
|
||
```js
|
||
const API_BASE = '/api';
|
||
```
|
||
Update `API_BASE` if you mount the service under a different path or serve it at root.
|
||
|
||
### Debugging WebSockets
|
||
|
||
- Server logs now print every incoming WebSocket frame in `main.py`:
|
||
- `[WS-PLUGIN RX] <client>: <raw-payload>` for plugin messages on `/ws/position`
|
||
- `[WS-LIVE RX] <client>: <parsed-json>` for browser messages on `/ws/live`
|
||
- Use these logs to verify messages and troubleshoot handshake failures.
|
||
|
||
### Styling Adjustments
|
||
|
||
- Chat input bar is fixed at the bottom of the chat window (`.chat-form { position:absolute; bottom:0; }`).
|
||
- Input text and placeholder are white for readability (`.chat-input, .chat-input::placeholder { color:#fff; }`).
|
||
- Incoming chat messages forced white via `.chat-messages div { color:#fff !important; }`.
|
||
|
||
## API Reference
|
||
|
||
### WebSocket /ws/position
|
||
Stream telemetry snapshots over a WebSocket connection. Provide your shared secret either as a query parameter or WebSocket header:
|
||
|
||
```
|
||
ws://<host>:<port>/ws/position?secret=<shared_secret>
|
||
```
|
||
or
|
||
```
|
||
X-Plugin-Secret: <shared_secret>
|
||
```
|
||
|
||
After connecting, send JSON messages matching the `TelemetrySnapshot` schema. For example:
|
||
|
||
```json
|
||
{
|
||
"type": "telemetry",
|
||
"character_name": "Dunking Rares",
|
||
"char_tag": "moss",
|
||
"session_id": "dunk-20250422-xyz",
|
||
"timestamp": "2025-04-22T13:45:00Z",
|
||
"ew": 123.4,
|
||
"ns": 567.8,
|
||
"z": 10.2,
|
||
"kills": 42,
|
||
"deaths": 1,
|
||
"prismatic_taper_count": 17,
|
||
"vt_state": "Combat",
|
||
"kills_per_hour": "N/A",
|
||
"onlinetime": "00:05:00"
|
||
}
|
||
```
|
||
|
||
Each message above is sent as its own JSON object over the WebSocket (one frame per event). When you want to report a rare spawn, send a standalone `rare` event instead of embedding rare counts in telemetry. For example:
|
||
|
||
```json
|
||
{
|
||
"type": "rare",
|
||
"timestamp": "2025-04-22T13:48:00Z",
|
||
"character_name": "MyCharacter",
|
||
"name": "Golden Gryphon",
|
||
"ew": 150.5,
|
||
"ns": 350.7,
|
||
"z": 5.0,
|
||
"additional_info": "first sighting of the day"
|
||
}
|
||
```
|
||
|
||
### Chat messages
|
||
You can also send chat envelopes over the same WebSocket to display messages in the browser. Fields:
|
||
- `type`: must be "chat"
|
||
- `character_name`: target player name
|
||
- `text`: message content
|
||
- `color` (optional): CSS color string (e.g. "#ff8800"); if sent as an integer (0xRRGGBB), it will be converted to hex.
|
||
|
||
Example chat payload:
|
||
```json
|
||
{
|
||
"type": "chat",
|
||
"character_name": "MyCharacter",
|
||
"text": "Hello world!",
|
||
"color": "#88f"
|
||
}
|
||
```
|
||
|
||
## Event Payload Formats
|
||
|
||
For a complete reference of JSON payloads accepted by the backend (over `/ws/position`), see the file `EVENT_FORMATS.json` in the project root. It contains example schemas for:
|
||
- **Telemetry events** (`type`: "telemetry")
|
||
- **Spawn events** (`type`: "spawn")
|
||
- **Chat events** (`type`: "chat")
|
||
- **Rare events** (`type`: "rare")
|
||
- **Inventory events** (`type`: "inventory")
|
||
|
||
Notes on payload changes:
|
||
- Spawn events no longer require the `z` coordinate; if omitted, the server defaults it to 0.0.
|
||
Coordinates (`ew`, `ns`, `z`) may be sent as JSON numbers or strings; the backend will coerce them to floats.
|
||
- Telemetry events have removed the `latency_ms` field; please omit it from your payloads.
|
||
- Inventory events are sent automatically on character login/logout containing complete inventory data.
|
||
|
||
Each entry shows all required and optional fields, their types, and example values.
|
||
|
||
### GET /live
|
||
Returns active players seen within the last 30 seconds:
|
||
|
||
```json
|
||
{
|
||
"players": [ { ... } ]
|
||
}
|
||
```
|
||
|
||
### GET /history
|
||
Retrieve historical snapshots with optional `from` and `to` ISO8601 timestamps:
|
||
|
||
```
|
||
GET /history?from=2025-04-22T12:00:00Z&to=2025-04-22T13:00:00Z
|
||
```
|
||
|
||
Response:
|
||
|
||
```json
|
||
{
|
||
"data": [ { ... } ]
|
||
}
|
||
```
|
||
|
||
## Frontend
|
||
|
||
- **Live Map**: `static/index.html` – Real-time player positions on a map.
|
||
- **Inventory Search**: `static/inventory.html` – Search and browse character inventories with advanced filtering.
|
||
|
||
## Database Schema
|
||
|
||
This service uses PostgreSQL with the TimescaleDB extension to store telemetry time-series data,
|
||
aggregate character statistics, and a separate inventory database for equipment management.
|
||
|
||
### Telemetry Database Tables:
|
||
|
||
- **telemetry_events** (hypertable):
|
||
- `id` (PK, serial)
|
||
- `character_name` (text, indexed)
|
||
- `char_tag` (text, nullable)
|
||
- `session_id` (text, indexed)
|
||
- `timestamp` (timestamptz, indexed)
|
||
- `ew`, `ns`, `z` (float)
|
||
- `kills`, `deaths`, `rares_found`, `prismatic_taper_count` (integer)
|
||
- `kills_per_hour` (float)
|
||
- `onlinetime`, `vt_state` (text)
|
||
- Optional metrics: `mem_mb`, `cpu_pct`, `mem_handles`, `latency_ms` (float)
|
||
|
||
- **char_stats**:
|
||
- `character_name` (text, PK)
|
||
- `total_kills` (integer)
|
||
|
||
- **rare_stats**:
|
||
- `character_name` (text, PK)
|
||
- `total_rares` (integer)
|
||
|
||
- **rare_stats_sessions**:
|
||
- `character_name`, `session_id` (composite PK)
|
||
- `session_rares` (integer)
|
||
|
||
- **spawn_events**:
|
||
- `id` (PK, serial)
|
||
- `character_name` (text)
|
||
- `mob` (text)
|
||
- `timestamp` (timestamptz)
|
||
- `ew`, `ns`, `z` (float)
|
||
|
||
- **rare_events**:
|
||
- `id` (PK, serial)
|
||
- `character_name` (text)
|
||
- `name` (text)
|
||
- `timestamp` (timestamptz)
|
||
- `ew`, `ns`, `z` (float)
|
||
|
||
### Inventory Database Tables:
|
||
|
||
- **items**:
|
||
- `id` (PK, serial)
|
||
- `character_name` (text, indexed)
|
||
- `item_id` (bigint)
|
||
- `name` (text)
|
||
- `object_class` (integer)
|
||
- `icon`, `value`, `burden` (integer)
|
||
- `current_wielded_location`, `bonded`, `attuned`, `unique` (various)
|
||
- `timestamp` (timestamptz)
|
||
|
||
- **item_combat_stats**:
|
||
- `item_id` (FK to items.id)
|
||
- `armor_level`, `max_damage` (integer)
|
||
- `damage_bonus`, `attack_bonus` (float)
|
||
- Various defense bonuses
|
||
|
||
- **item_enhancements**:
|
||
- `item_id` (FK to items.id)
|
||
- `material` (varchar)
|
||
- `item_set` (varchar)
|
||
- `tinks`, `workmanship` (integer/float)
|
||
|
||
- **item_spells**:
|
||
- `item_id` (FK to items.id)
|
||
- `spell_id` (integer)
|
||
- `spell_name` (text)
|
||
- `is_legendary`, `is_epic` (boolean)
|
||
|
||
- **item_raw_data**:
|
||
- `item_id` (FK to items.id)
|
||
- `int_values`, `double_values`, `string_values`, `bool_values` (JSONB)
|
||
- `original_json` (JSONB)
|
||
|
||
## Contributing
|
||
|
||
Contributions are welcome! Feel free to open issues or submit pull requests.
|
||
|
||
## Roadmap & TODO
|
||
For detailed tasks, migration steps, and future enhancements, see [TODO.md](TODO.md).
|
||
|
||
### Local Development Database
|
||
This service uses PostgreSQL with the TimescaleDB extension. You can configure local development using the provided Docker Compose setup or connect to an external instance:
|
||
|
||
1. PostgreSQL/TimescaleDB via Docker Compose (recommended):
|
||
- Pros:
|
||
- Reproducible, isolated environment out-of-the-box
|
||
- No need to install Postgres locally
|
||
- Aligns development with production setups
|
||
- Cons:
|
||
- Additional resource usage (memory, CPU)
|
||
- Slightly more complex Docker configuration
|
||
|
||
2. External PostgreSQL instance:
|
||
- Pros:
|
||
- Leverages existing infrastructure
|
||
- No Docker overhead
|
||
- Cons:
|
||
- Requires manual setup and Timescale extension
|
||
- Less portable for new contributors
|