Dereth Tracker

Dereth Tracker is a real-time telemetry service for the world of Dereth. It collects player data, stores it in a PostgreSQL (TimescaleDB) database for efficient time-series storage, provides a live map interface, and includes a comprehensive inventory management system for tracking and searching character equipment.

Table of Contents

• Overview
• Features
• Requirements
• Installation
• Configuration
• Usage
• API Reference
• Frontend
• Database Schema
• Contributing

Overview

This project provides:

• A FastAPI backend with endpoints for receiving and querying telemetry data.
• PostgreSQL/TimescaleDB-based storage for time-series telemetry and per-character stats.
• A live, interactive map using static HTML, CSS, and JavaScript.
• A comprehensive inventory management system with search capabilities.
• Real-time inventory updates via WebSocket when characters log in/out.
• A sample data generator script (generate_data.py) for simulating telemetry snapshots.

Features

• WebSocket /ws/position: Stream telemetry snapshots and inventory updates (protected by a shared secret).
• GET /live: Fetch active players seen in the last 30 seconds.
• GET /history: Retrieve historical telemetry data with optional time filtering.
• GET /debug: Health check endpoint.
• Live Map: Interactive map interface with panning, zooming, and sorting.
• Inventory Management:
  • Real-time inventory updates via WebSocket on character login/logout
  • Advanced search across all character inventories
  • Filter by character, equipment type, material, stats, and more
  • Sort by any column with live results
  • Track item properties including spells, armor level, damage ratings
• Suitbuilder:
  • Equipment optimization across multiple character inventories
  • Constraint-based search for optimal armor combinations
  • Support for primary and secondary armor sets
  • Real-time streaming results during long-running searches
• Portal Tracking:
  • Automatic discovery and tracking of in-game portals
  • 1-hour retention for discovered portals
  • Coordinate-based uniqueness (rounded to 0.1 precision)
  • Real-time portal updates on the map interface
• Discord Rare Monitor Bot: Monitors rare discoveries and posts filtered notifications to Discord channels
• Sample Data Generator: generate_data.py sends telemetry snapshots over WebSocket for testing.

Requirements

• Python 3.9 or newer (only if running without Docker)
• pip (only if running without Docker)
• Docker & Docker Compose (recommended)

Python packages (if using local virtualenv):

• fastapi
• uvicorn
• pydantic
• databases
• asyncpg
• sqlalchemy
• websockets # required for sample data generator

Installation

1. Clone the repository:

   git clone https://github.com/yourusername/dereth-tracker.git
   cd dereth-tracker
   

2. Create and activate a virtual environment:

   python3 -m venv venv
   source venv/bin/activate  # Windows: venv\Scripts\activate
   

3. Install dependencies:

pip install fastapi uvicorn pydantic websockets

## Configuration

- Configure the plugin shared secret via the `SHARED_SECRET` environment variable (default in code: `"your_shared_secret"`).
- The database connection is controlled by the `DATABASE_URL` environment variable (e.g. `postgresql://postgres:password@db:5432/dereth`).
By default, when using Docker Compose, a TimescaleDB container is provisioned for you.
- If you need to tune Timescale or Postgres settings (retention, checkpoint, etc.), set the corresponding `DB_*` environment variables as documented in `docker-compose.yml`.

## Usage

### Using Docker (Recommended)

1. Build and start all services:
```bash
docker compose up -d

2. Rebuild container after code changes:

   docker compose build --no-cache dereth-tracker
   docker compose up -d dereth-tracker
   

3. View logs:

   docker logs mosswartoverlord-dereth-tracker-1
   docker logs dereth-db
   

Without Docker

Start the server using Uvicorn:

uvicorn main:app --reload --host 0.0.0.0 --port 8000

Grafana Dashboard UI

location /grafana/ {
    # Optional: require basic auth on the Grafana UI
    auth_basic           "Restricted";
    auth_basic_user_file /etc/nginx/.htpasswd;

    proxy_pass             http://127.0.0.1:3000/;
    proxy_http_version     1.1;
    proxy_set_header       Host              $host;
    proxy_set_header       X-Real-IP         $remote_addr;
    proxy_set_header       X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header       X-Forwarded-Proto $scheme;
    # Inject Grafana service account token for anonymous panel embeds
    proxy_set_header       Authorization     "Bearer <YOUR_SERVICE_ACCOUNT_TOKEN>";
    # WebSocket support (for live panels)
    proxy_set_header       Upgrade           $http_upgrade;
    proxy_set_header       Connection        "upgrade";
    proxy_cache_bypass     $http_upgrade;
}

NGINX Proxy Configuration

If you cannot reassign the existing /live and /trails routes, you can namespace this service under /api (or any other prefix) and configure NGINX accordingly. Be sure to forward WebSocket upgrade headers so that /ws/live and /ws/position continue to work. Example:

location /api/ {
  proxy_pass         http://127.0.0.1:8765/;
   proxy_http_version 1.1;
  proxy_set_header   Host              $host;
  proxy_set_header   X-Real-IP         $remote_addr;
  proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
  proxy_set_header   X-Forwarded-Proto $scheme;
  # WebSocket support
  proxy_set_header   Upgrade           $http_upgrade;
  proxy_set_header   Connection        "upgrade";
  proxy_cache_bypass $http_upgrade;
}

Then the browser client (static/script.js) will fetch /api/live/ and /api/trails/ to reach this new server.

- Live Map: `http://localhost:8000/` (or `http://<your-domain>/api/` if behind a prefix)
- Grafana UI: `http://localhost:3000/grafana/` (or `http://<your-domain>/grafana/` if proxied under that path)

Frontend Configuration

• In static/script.js, the constant API_BASE controls where live/trails data and WebSocket /ws/live are fetched. By default:

  const API_BASE = '/api';
  

  Update API_BASE if you mount the service under a different path or serve it at root.

Debugging WebSockets

• Server logs now print every incoming WebSocket frame in main.py:
  • [WS-PLUGIN RX] <client>: <raw-payload> for plugin messages on /ws/position
  • [WS-LIVE RX] <client>: <parsed-json> for browser messages on /ws/live
• Use these logs to verify messages and troubleshoot handshake failures.

Styling Adjustments

• Chat input bar is fixed at the bottom of the chat window (.chat-form { position:absolute; bottom:0; }).
• Input text and placeholder are white for readability (.chat-input, .chat-input::placeholder { color:#fff; }).
• Incoming chat messages forced white via .chat-messages div { color:#fff !important; }.

API Reference

WebSocket /ws/position

Stream telemetry snapshots over a WebSocket connection. Provide your shared secret either as a query parameter or WebSocket header:

ws://<host>:<port>/ws/position?secret=<shared_secret>

or

X-Plugin-Secret: <shared_secret>

After connecting, send JSON messages matching the TelemetrySnapshot schema. For example:

{
  "type": "telemetry",
  "character_name": "Dunking Rares",
  "char_tag": "moss",
  "session_id": "dunk-20250422-xyz",
  "timestamp": "2025-04-22T13:45:00Z",
  "ew": 123.4,
  "ns": 567.8,
  "z": 10.2,
  "kills": 42,
  "deaths": 1,
  "prismatic_taper_count": 17,
  "vt_state": "Combat",
  "kills_per_hour": "N/A",
  "onlinetime": "00:05:00"
 }

Each message above is sent as its own JSON object over the WebSocket (one frame per event). When you want to report a rare spawn, send a standalone rare event instead of embedding rare counts in telemetry. For example:

{
  "type": "rare",
  "timestamp": "2025-04-22T13:48:00Z",
  "character_name": "MyCharacter",
  "name": "Golden Gryphon",
  "ew": 150.5,
  "ns": 350.7,
  "z": 5.0,
  "additional_info": "first sighting of the day"
}

Chat messages

You can also send chat envelopes over the same WebSocket to display messages in the browser. Fields:

• type: must be "chat"
• character_name: target player name
• text: message content
• color (optional): CSS color string (e.g. "#ff8800"); if sent as an integer (0xRRGGBB), it will be converted to hex.

Example chat payload:

{
  "type": "chat",
  "character_name": "MyCharacter",
  "text": "Hello world!",
  "color": "#88f"
}

Event Payload Formats

For a complete reference of JSON payloads accepted by the backend (over /ws/position), see the file EVENT_FORMATS.json in the project root. It contains example schemas for:

• Telemetry events (type: "telemetry")
• Spawn events (type: "spawn")
• Chat events (type: "chat")
• Rare events (type: "rare")
• Inventory events (type: "inventory")

Notes on payload changes:

• Spawn events no longer require the z coordinate; if omitted, the server defaults it to 0.0. Coordinates (ew, ns, z) may be sent as JSON numbers or strings; the backend will coerce them to floats.
• Telemetry events have removed the latency_ms field; please omit it from your payloads.
• Inventory events are sent automatically on character login/logout containing complete inventory data.

Each entry shows all required and optional fields, their types, and example values.

GET /live

Returns active players seen within the last 30 seconds:

{
  "players": [ { ... } ]
}

GET /history

Retrieve historical snapshots with optional from and to ISO8601 timestamps:

GET /history?from=2025-04-22T12:00:00Z&to=2025-04-22T13:00:00Z

Response:

{
  "data": [ { ... } ]
}

Frontend

• Live Map: static/index.html – Real-time player positions on a map.
• Inventory Search: static/inventory.html – Search and browse character inventories with advanced filtering.

Database Schema

This service uses PostgreSQL with the TimescaleDB extension to store telemetry time-series data, aggregate character statistics, and a separate inventory database for equipment management.

Telemetry Database Tables:

• telemetry_events (hypertable):

  • id (PK, serial)
  • character_name (text, indexed)
  • char_tag (text, nullable)
  • session_id (text, indexed)
  • timestamp (timestamptz, indexed)
  • ew, ns, z (float)
  • kills, deaths, rares_found, prismatic_taper_count (integer)
  • kills_per_hour (float)
  • onlinetime, vt_state (text)
  • Optional metrics: mem_mb, cpu_pct, mem_handles, latency_ms (float)

• char_stats:

  • character_name (text, PK)
  • total_kills (integer)

• rare_stats:

  • character_name (text, PK)
  • total_rares (integer)

• rare_stats_sessions:

  • character_name, session_id (composite PK)
  • session_rares (integer)

• spawn_events:

  • id (PK, serial)
  • character_name (text)
  • mob (text)
  • timestamp (timestamptz)
  • ew, ns, z (float)

• rare_events:

  • id (PK, serial)
  • character_name (text)
  • name (text)
  • timestamp (timestamptz)
  • ew, ns, z (float)

• portals:

  • id (PK, serial)
  • portal_name (text)
  • ns, ew, z (float coordinates)
  • discovered_at (timestamptz, indexed)
  • discovered_by (text)
  • Unique constraint: ROUND(ns::numeric, 1), ROUND(ew::numeric, 1)

Inventory Database Tables:

• items:

  • id (PK, serial)
  • character_name (text, indexed)
  • item_id (bigint)
  • name (text)
  • object_class (integer)
  • icon, value, burden (integer)
  • current_wielded_location, bonded, attuned, unique (various)
  • timestamp (timestamptz)

• item_combat_stats:

  • item_id (FK to items.id)
  • armor_level, max_damage (integer)
  • damage_bonus, attack_bonus (float)
  • Various defense bonuses

• item_enhancements:

  • item_id (FK to items.id)
  • material (varchar)
  • item_set (varchar)
  • tinks, workmanship (integer/float)

• item_spells:

  • item_id (FK to items.id)
  • spell_id (integer)
  • spell_name (text)
  • is_legendary, is_epic (boolean)

• item_raw_data:

  • item_id (FK to items.id)
  • int_values, double_values, string_values, bool_values (JSONB)
  • original_json (JSONB)

Contributing

Contributions are welcome! Feel free to open issues or submit pull requests.

Roadmap & TODO

For detailed tasks, migration steps, and future enhancements, see TODO.md.

Local Development Database

This service uses PostgreSQL with the TimescaleDB extension. You can configure local development using the provided Docker Compose setup or connect to an external instance:

1. PostgreSQL/TimescaleDB via Docker Compose (recommended):

   • Pros:
     • Reproducible, isolated environment out-of-the-box
     • No need to install Postgres locally
     • Aligns development with production setups
   • Cons:
     • Additional resource usage (memory, CPU)
     • Slightly more complex Docker configuration

2. External PostgreSQL instance:

   • Pros:
     • Leverages existing infrastructure
     • No Docker overhead
   • Cons:
     • Requires manual setup and Timescale extension
     • Less portable for new contributors