Dereth Tracker

Dereth Tracker is a real-time telemetry service for the world of Dereth. It collects player data, stores it in a PostgreSQL (TimescaleDB) database for efficient time-series storage, and provides a live map interface along with a sample data generator for testing.

Table of Contents

• Overview
• Features
• Requirements
• Installation
• Configuration
• Usage
• API Reference
• Frontend
• Database Schema
• Contributing

Overview

• This project provides:
• A FastAPI backend with endpoints for receiving and querying telemetry data.
• PostgreSQL/TimescaleDB-based storage for time-series telemetry and per-character stats.
• A live, interactive map using static HTML, CSS, and JavaScript.
• A sample data generator script (generate_data.py) for simulating telemetry snapshots.
• A sample data generator script (generate_data.py) for simulating telemetry snapshots.

Features

• WebSocket /ws/position: Stream telemetry snapshots (protected by a shared secret).
• GET /live: Fetch active players seen in the last 30 seconds.
• GET /history: Retrieve historical telemetry data with optional time filtering.
• GET /debug: Health check endpoint.
• Live Map: Interactive map interface with panning, zooming, and sorting.
• Sample Data Generator: generate_data.py sends telemetry snapshots over WebSocket for testing.

Requirements

• Python 3.9 or newer (only if running without Docker)
• pip (only if running without Docker)
• Docker & Docker Compose (recommended)

Python packages (if using local virtualenv):

• fastapi
• uvicorn
• pydantic
• databases
• asyncpg
• sqlalchemy
• websockets # required for sample data generator

Installation

1. Clone the repository:

   git clone https://github.com/yourusername/dereth-tracker.git
   cd dereth-tracker
   

2. Create and activate a virtual environment:

   python3 -m venv venv
   source venv/bin/activate  # Windows: venv\Scripts\activate
   

3. Install dependencies:

pip install fastapi uvicorn pydantic websockets

## Configuration

- Configure the plugin shared secret via the `SHARED_SECRET` environment variable (default in code: `"your_shared_secret"`).
- The database connection is controlled by the `DATABASE_URL` environment variable (e.g. `postgresql://postgres:password@db:5432/dereth`).
By default, when using Docker Compose, a TimescaleDB container is provisioned for you.
- If you need to tune Timescale or Postgres settings (retention, checkpoint, etc.), set the corresponding `DB_*` environment variables as documented in `docker-compose.yml`.

## Usage

Start the server using Uvicorn:

```bash
uvicorn main:app --reload --host 0.0.0.0 --port 8000

Grafana Dashboard UI

location /grafana/ {
    # Optional: require basic auth on the Grafana UI
    auth_basic           "Restricted";
    auth_basic_user_file /etc/nginx/.htpasswd;

    proxy_pass             http://127.0.0.1:3000/;
    proxy_http_version     1.1;
    proxy_set_header       Host              $host;
    proxy_set_header       X-Real-IP         $remote_addr;
    proxy_set_header       X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header       X-Forwarded-Proto $scheme;
    # Inject Grafana service account token for anonymous panel embeds
    proxy_set_header       Authorization     "Bearer <YOUR_SERVICE_ACCOUNT_TOKEN>";
    # WebSocket support (for live panels)
    proxy_set_header       Upgrade           $http_upgrade;
    proxy_set_header       Connection        "upgrade";
    proxy_cache_bypass     $http_upgrade;
}

NGINX Proxy Configuration

If you cannot reassign the existing /live and /trails routes, you can namespace this service under /api (or any other prefix) and configure NGINX accordingly. Be sure to forward WebSocket upgrade headers so that /ws/live and /ws/position continue to work. Example:

location /api/ {
  proxy_pass         http://127.0.0.1:8765/;
   proxy_http_version 1.1;
  proxy_set_header   Host              $host;
  proxy_set_header   X-Real-IP         $remote_addr;
  proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
  proxy_set_header   X-Forwarded-Proto $scheme;
  # WebSocket support
  proxy_set_header   Upgrade           $http_upgrade;
  proxy_set_header   Connection        "upgrade";
  proxy_cache_bypass $http_upgrade;
}

Then the browser client (static/script.js) will fetch /api/live/ and /api/trails/ to reach this new server.

- Live Map: `http://localhost:8000/` (or `http://<your-domain>/api/` if behind a prefix)
- Grafana UI: `http://localhost:3000/grafana/` (or `http://<your-domain>/grafana/` if proxied under that path)

Frontend Configuration

• In static/script.js, the constant API_BASE controls where live/trails data and WebSocket /ws/live are fetched. By default:

  const API_BASE = '/api';
  

  Update API_BASE if you mount the service under a different path or serve it at root.

Debugging WebSockets

• Server logs now print every incoming WebSocket frame in main.py:
  • [WS-PLUGIN RX] <client>: <raw-payload> for plugin messages on /ws/position
  • [WS-LIVE RX] <client>: <parsed-json> for browser messages on /ws/live
• Use these logs to verify messages and troubleshoot handshake failures.

Styling Adjustments

• Chat input bar is fixed at the bottom of the chat window (.chat-form { position:absolute; bottom:0; }).
• Input text and placeholder are white for readability (.chat-input, .chat-input::placeholder { color:#fff; }).
• Incoming chat messages forced white via .chat-messages div { color:#fff !important; }.

API Reference

WebSocket /ws/position

Stream telemetry snapshots over a WebSocket connection. Provide your shared secret either as a query parameter or WebSocket header:

ws://<host>:<port>/ws/position?secret=<shared_secret>

or

X-Plugin-Secret: <shared_secret>

After connecting, send JSON messages matching the TelemetrySnapshot schema. For example:

{
  "type": "telemetry",
  "character_name": "Dunking Rares",
  "char_tag": "moss",
  "session_id": "dunk-20250422-xyz",
  "timestamp": "2025-04-22T13:45:00Z",
  "ew": 123.4,
  "ns": 567.8,
  "z": 10.2,
  "kills": 42,
  "deaths": 1,
  "prismatic_taper_count": 17,
  "vt_state": "Combat",
  "kills_per_hour": "N/A",
  "onlinetime": "00:05:00"
 }

Each message above is sent as its own JSON object over the WebSocket (one frame per event). When you want to report a rare spawn, send a standalone rare event instead of embedding rare counts in telemetry. For example:

{
  "type": "rare",
  "timestamp": "2025-04-22T13:48:00Z",
  "character_name": "MyCharacter",
  "name": "Golden Gryphon",
  "ew": 150.5,
  "ns": 350.7,
  "z": 5.0,
  "additional_info": "first sighting of the day"
}

Chat messages

You can also send chat envelopes over the same WebSocket to display messages in the browser. Fields:

• type: must be "chat"
• character_name: target player name
• text: message content
• color (optional): CSS color string (e.g. "#ff8800"); if sent as an integer (0xRRGGBB), it will be converted to hex.

Example chat payload:

{
  "type": "chat",
  "character_name": "MyCharacter",
  "text": "Hello world!",
  "color": "#88f"
}

Event Payload Formats

For a complete reference of JSON payloads accepted by the backend (over /ws/position), see the file EVENT_FORMATS.json in the project root. It contains example schemas for:

• Telemetry events (type: "telemetry")
• Spawn events (type: "spawn")
• Chat events (type: "chat")
• Rare events (type: "rare")

Notes on payload changes:

• Spawn events no longer require the z coordinate; if omitted, the server defaults it to 0.0. Coordinates (ew, ns, z) may be sent as JSON numbers or strings; the backend will coerce them to floats.
• Telemetry events have removed the latency_ms field; please omit it from your payloads.

Each entry shows all required and optional fields, their types, and example values.

GET /live

Returns active players seen within the last 30 seconds:

{
  "players": [ { ... } ]
}

GET /history

Retrieve historical snapshots with optional from and to ISO8601 timestamps:

GET /history?from=2025-04-22T12:00:00Z&to=2025-04-22T13:00:00Z

Response:

{
  "data": [ { ... } ]
}

Frontend

• Live Map: static/index.html – Real-time player positions on a map.

Database Schema

This service uses PostgreSQL with the TimescaleDB extension to store telemetry time-series data and aggregate character statistics. The primary tables are:

• telemetry_events (hypertable):

  • id (PK, serial)
  • character_name (text, indexed)
  • char_tag (text, nullable)
  • session_id (text, indexed)
  • timestamp (timestamptz, indexed)
  • ew, ns, z (float)
  • kills, deaths, rares_found, prismatic_taper_count (integer)
  • kills_per_hour (float)
  • onlinetime, vt_state (text)
  • Optional metrics: mem_mb, cpu_pct, mem_handles, latency_ms (float)

• char_stats:

  • character_name (text, PK)
  • total_kills (integer)

• rare_stats:

  • character_name (text, PK)
  • total_rares (integer)

• rare_stats_sessions:

  • character_name, session_id (composite PK)
  • session_rares (integer)

• spawn_events:

  • id (PK, serial)
  • character_name (text)
  • mob (text)
  • timestamp (timestamptz)
  • ew, ns, z (float)

• rare_events:

  • id (PK, serial)
  • character_name (text)
  • name (text)
  • timestamp (timestamptz)
  • ew, ns, z (float)

Contributing

Contributions are welcome! Feel free to open issues or submit pull requests.

Roadmap & TODO

For detailed tasks, migration steps, and future enhancements, see TODO.md.

Local Development Database

This service uses PostgreSQL with the TimescaleDB extension. You can configure local development using the provided Docker Compose setup or connect to an external instance:

1. PostgreSQL/TimescaleDB via Docker Compose (recommended):

   • Pros:
     • Reproducible, isolated environment out-of-the-box
     • No need to install Postgres locally
     • Aligns development with production setups
   • Cons:
     • Additional resource usage (memory, CPU)
     • Slightly more complex Docker configuration

2. External PostgreSQL instance:

   • Pros:
     • Leverages existing infrastructure
     • No Docker overhead
   • Cons:
     • Requires manual setup and Timescale extension
     • Less portable for new contributors