# Dereth Tracker

Dereth Tracker is a real-time telemetry service for the world of Dereth. It collects player data, stores it in a PostgreSQL (TimescaleDB) database for efficient time-series storage, provides a live map interface, and includes a comprehensive inventory management system for tracking and searching character equipment.

## Table of Contents
- [Overview](#overview)
- [Features](#features)
- [Requirements](#requirements)
- [Installation](#installation)
- [Configuration](#configuration)
- [Usage](#usage)
- [API Reference](#api-reference)
- [Frontend](#frontend)
- [Database Schema](#database-schema)
- [Contributing](#contributing)

## Overview

This project provides:
- A FastAPI backend with endpoints for receiving and querying telemetry data.
- PostgreSQL/TimescaleDB-based storage for time-series telemetry and per-character stats.
- A live, interactive map using static HTML, CSS, and JavaScript.
- A comprehensive inventory management system with search capabilities.
- Real-time inventory updates via WebSocket when characters log in/out.
- A sample data generator script (`generate_data.py`) for simulating telemetry snapshots.

## Features

- **WebSocket /ws/position**: Stream telemetry snapshots and inventory updates (protected by a shared secret).
- **GET /live**: Fetch active players seen in the last 30 seconds.
- **GET /history**: Retrieve historical telemetry data with optional time filtering.
- **GET /debug**: Health check endpoint.
- **Live Map**: Interactive map interface with panning, zooming, and sorting.
- **Inventory Management**: 
  - Real-time inventory updates via WebSocket on character login/logout
  - Advanced search across all character inventories
  - Filter by character, equipment type, material, stats, and more
  - Sort by any column with live results
  - Track item properties including spells, armor level, damage ratings
- **Discord Rare Monitor Bot**: Monitors rare discoveries and posts filtered notifications to Discord channels
- **Sample Data Generator**: `generate_data.py` sends telemetry snapshots over WebSocket for testing.

## Requirements

- Python 3.9 or newer (only if running without Docker)
- pip (only if running without Docker)
- Docker & Docker Compose (recommended)

Python packages (if using local virtualenv):

- fastapi
- uvicorn
- pydantic
- databases
- asyncpg
- sqlalchemy
- websockets  # required for sample data generator

## Installation

1. Clone the repository:
   ```bash
   git clone https://github.com/yourusername/dereth-tracker.git
   cd dereth-tracker
   ```
2. Create and activate a virtual environment:
   ```bash
   python3 -m venv venv
   source venv/bin/activate  # Windows: venv\Scripts\activate
   ```
3. Install dependencies:
   ```bash
  pip install fastapi uvicorn pydantic websockets
   ```

## Configuration

 - Configure the plugin shared secret via the `SHARED_SECRET` environment variable (default in code: `"your_shared_secret"`).
 - The database connection is controlled by the `DATABASE_URL` environment variable (e.g. `postgresql://postgres:password@db:5432/dereth`).
   By default, when using Docker Compose, a TimescaleDB container is provisioned for you.
 - If you need to tune Timescale or Postgres settings (retention, checkpoint, etc.), set the corresponding `DB_*` environment variables as documented in `docker-compose.yml`.

## Usage

Start the server using Uvicorn:

```bash
uvicorn main:app --reload --host 0.0.0.0 --port 8000
```
  
# Grafana Dashboard UI
```nginx
location /grafana/ {
    # Optional: require basic auth on the Grafana UI
    auth_basic           "Restricted";
    auth_basic_user_file /etc/nginx/.htpasswd;

    proxy_pass             http://127.0.0.1:3000/;
    proxy_http_version     1.1;
    proxy_set_header       Host              $host;
    proxy_set_header       X-Real-IP         $remote_addr;
    proxy_set_header       X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header       X-Forwarded-Proto $scheme;
    # Inject Grafana service account token for anonymous panel embeds
    proxy_set_header       Authorization     "Bearer <YOUR_SERVICE_ACCOUNT_TOKEN>";
    # WebSocket support (for live panels)
    proxy_set_header       Upgrade           $http_upgrade;
    proxy_set_header       Connection        "upgrade";
    proxy_cache_bypass     $http_upgrade;
}
``` 

## NGINX Proxy Configuration

If you cannot reassign the existing `/live` and `/trails` routes, you can namespace this service under `/api` (or any other prefix) and configure NGINX accordingly. Be sure to forward WebSocket upgrade headers so that `/ws/live` and `/ws/position` continue to work. Example:
```nginx
location /api/ {
  proxy_pass         http://127.0.0.1:8765/;
   proxy_http_version 1.1;
  proxy_set_header   Host              $host;
  proxy_set_header   X-Real-IP         $remote_addr;
  proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
  proxy_set_header   X-Forwarded-Proto $scheme;
  # WebSocket support
  proxy_set_header   Upgrade           $http_upgrade;
  proxy_set_header   Connection        "upgrade";
  proxy_cache_bypass $http_upgrade;
}
```
Then the browser client (static/script.js) will fetch `/api/live/` and `/api/trails/` to reach this new server.

    - Live Map: `http://localhost:8000/` (or `http://<your-domain>/api/` if behind a prefix)
    - Grafana UI: `http://localhost:3000/grafana/` (or `http://<your-domain>/grafana/` if proxied under that path)

### Frontend Configuration

- In `static/script.js`, the constant `API_BASE` controls where live/trails data and WebSocket `/ws/live` are fetched. By default:
  ```js
  const API_BASE = '/api';
  ```
  Update `API_BASE` if you mount the service under a different path or serve it at root.

### Debugging WebSockets

- Server logs now print every incoming WebSocket frame in `main.py`:
  - `[WS-PLUGIN RX] <client>: <raw-payload>` for plugin messages on `/ws/position`
  - `[WS-LIVE RX]   <client>: <parsed-json>` for browser messages on `/ws/live`
- Use these logs to verify messages and troubleshoot handshake failures.

### Styling Adjustments

- Chat input bar is fixed at the bottom of the chat window (`.chat-form { position:absolute; bottom:0; }`).
- Input text and placeholder are white for readability (`.chat-input, .chat-input::placeholder { color:#fff; }`).
- Incoming chat messages forced white via `.chat-messages div { color:#fff !important; }`.

## API Reference

### WebSocket /ws/position
Stream telemetry snapshots over a WebSocket connection. Provide your shared secret either as a query parameter or WebSocket header:

```
ws://<host>:<port>/ws/position?secret=<shared_secret>
```
or
```
X-Plugin-Secret: <shared_secret>
```

After connecting, send JSON messages matching the `TelemetrySnapshot` schema. For example:

```json
{
  "type": "telemetry",
  "character_name": "Dunking Rares",
  "char_tag": "moss",
  "session_id": "dunk-20250422-xyz",
  "timestamp": "2025-04-22T13:45:00Z",
  "ew": 123.4,
  "ns": 567.8,
  "z": 10.2,
  "kills": 42,
  "deaths": 1,
  "prismatic_taper_count": 17,
  "vt_state": "Combat",
  "kills_per_hour": "N/A",
  "onlinetime": "00:05:00"
 }
```
 
 Each message above is sent as its own JSON object over the WebSocket (one frame per event). When you want to report a rare spawn, send a standalone `rare` event instead of embedding rare counts in telemetry. For example:
 
 ```json
 {
   "type": "rare",
   "timestamp": "2025-04-22T13:48:00Z",
   "character_name": "MyCharacter",
   "name": "Golden Gryphon",
   "ew": 150.5,
   "ns": 350.7,
   "z": 5.0,
   "additional_info": "first sighting of the day"
 }
 ```

### Chat messages
You can also send chat envelopes over the same WebSocket to display messages in the browser. Fields:
- `type`: must be "chat"
- `character_name`: target player name
- `text`: message content
- `color` (optional): CSS color string (e.g. "#ff8800"); if sent as an integer (0xRRGGBB), it will be converted to hex.

Example chat payload:
```json
{
  "type": "chat",
  "character_name": "MyCharacter",
  "text": "Hello world!",
  "color": "#88f"
}
```

## Event Payload Formats

For a complete reference of JSON payloads accepted by the backend (over `/ws/position`), see the file `EVENT_FORMATS.json` in the project root. It contains example schemas for:
  - **Telemetry events** (`type`: "telemetry")
  - **Spawn events** (`type`: "spawn")
  - **Chat events** (`type`: "chat")
  - **Rare events** (`type`: "rare")
  - **Inventory events** (`type`: "inventory")

Notes on payload changes:
  - Spawn events no longer require the `z` coordinate; if omitted, the server defaults it to 0.0.
    Coordinates (`ew`, `ns`, `z`) may be sent as JSON numbers or strings; the backend will coerce them to floats.
  - Telemetry events have removed the `latency_ms` field; please omit it from your payloads.
  - Inventory events are sent automatically on character login/logout containing complete inventory data.

Each entry shows all required and optional fields, their types, and example values.

### GET /live
Returns active players seen within the last 30 seconds:

```json
{
  "players": [ { ... } ]
}
```

### GET /history
Retrieve historical snapshots with optional `from` and `to` ISO8601 timestamps:

```
GET /history?from=2025-04-22T12:00:00Z&to=2025-04-22T13:00:00Z
```

Response:

```json
{
  "data": [ { ... } ]
}
```

## Frontend

- **Live Map**: `static/index.html` – Real-time player positions on a map.
- **Inventory Search**: `static/inventory.html` – Search and browse character inventories with advanced filtering.

## Database Schema

This service uses PostgreSQL with the TimescaleDB extension to store telemetry time-series data,
aggregate character statistics, and a separate inventory database for equipment management. 

### Telemetry Database Tables:

- **telemetry_events** (hypertable):
  - `id` (PK, serial)
  - `character_name` (text, indexed)
  - `char_tag` (text, nullable)
  - `session_id` (text, indexed)
  - `timestamp` (timestamptz, indexed)
  - `ew`, `ns`, `z` (float)
  - `kills`, `deaths`, `rares_found`, `prismatic_taper_count` (integer)
  - `kills_per_hour` (float)
  - `onlinetime`, `vt_state` (text)
  - Optional metrics: `mem_mb`, `cpu_pct`, `mem_handles`, `latency_ms` (float)

- **char_stats**:
  - `character_name` (text, PK)
  - `total_kills` (integer)

- **rare_stats**:
  - `character_name` (text, PK)
  - `total_rares` (integer)

- **rare_stats_sessions**:
  - `character_name`, `session_id` (composite PK)
  - `session_rares` (integer)

- **spawn_events**:
  - `id` (PK, serial)
  - `character_name` (text)
  - `mob` (text)
  - `timestamp` (timestamptz)
  - `ew`, `ns`, `z` (float)

- **rare_events**:
  - `id` (PK, serial)
  - `character_name` (text)
  - `name` (text)
  - `timestamp` (timestamptz)
  - `ew`, `ns`, `z` (float)

### Inventory Database Tables:

- **items**:
  - `id` (PK, serial)
  - `character_name` (text, indexed)
  - `item_id` (bigint)
  - `name` (text)
  - `object_class` (integer)
  - `icon`, `value`, `burden` (integer)
  - `current_wielded_location`, `bonded`, `attuned`, `unique` (various)
  - `timestamp` (timestamptz)

- **item_combat_stats**:
  - `item_id` (FK to items.id)
  - `armor_level`, `max_damage` (integer)
  - `damage_bonus`, `attack_bonus` (float)
  - Various defense bonuses

- **item_enhancements**:
  - `item_id` (FK to items.id)
  - `material` (varchar)
  - `item_set` (varchar)
  - `tinks`, `workmanship` (integer/float)

- **item_spells**:
  - `item_id` (FK to items.id)
  - `spell_id` (integer)
  - `spell_name` (text)
  - `is_legendary`, `is_epic` (boolean)

- **item_raw_data**:
  - `item_id` (FK to items.id)
  - `int_values`, `double_values`, `string_values`, `bool_values` (JSONB)
  - `original_json` (JSONB)

## Contributing

Contributions are welcome! Feel free to open issues or submit pull requests.

## Roadmap & TODO
For detailed tasks, migration steps, and future enhancements, see [TODO.md](TODO.md).

### Local Development Database
This service uses PostgreSQL with the TimescaleDB extension. You can configure local development using the provided Docker Compose setup or connect to an external instance:

1. PostgreSQL/TimescaleDB via Docker Compose (recommended):
   - Pros:
     - Reproducible, isolated environment out-of-the-box
     - No need to install Postgres locally
     - Aligns development with production setups
   - Cons:
     - Additional resource usage (memory, CPU)
     - Slightly more complex Docker configuration

2. External PostgreSQL instance:
   - Pros:
     - Leverages existing infrastructure
     - No Docker overhead
   - Cons:
     - Requires manual setup and Timescale extension
     - Less portable for new contributors