# Project Architecture and Data Model

This document provides an overview of the project files, their roles, and a detailed description of the database architecture and data model.

## Project Structure

Root directory:
- **Dockerfile**: Defines the Python 3.12-slim image, installs dependencies (FastAPI, Uvicorn, SQLAlchemy, databases, TimescaleDB support), and runs the app.
- **docker-compose.yml**: Orchestrates two services:
  - **dereth-tracker**: The FastAPI application container.
  - **db**: A TimescaleDB (PostgreSQL 14 + TimescaleDB extension) container for persistent storage.
- **README.md**: High-level documentation and usage instructions.
- **EVENT_FORMATS.json**: Example JSON payloads for all event types (`telemetry`, `spawn`, `chat`, `rare`).
- **db.py**: Legacy SQLite-based storage (telemetry_log & live_state tables, WAL mode, auto-vacuum).
- **db_async.py**: Async database definitions for PostgreSQL/TimescaleDB:
  - Table schemas (SQLAlchemy Core): `telemetry_events`, `char_stats`, `rare_stats`, `rare_stats_sessions`, `spawn_events`.
  - `init_db_async()`: Creates tables, enables TimescaleDB extension, and configures a hypertable on `telemetry_events`.
- **main.py**: The FastAPI application:
  - HTTP endpoints: `/debug`, `/live`, `/history`, `/trails`.
  - WebSocket endpoints: `/ws/position` (plugin data in), `/ws/live` (browser live updates).
  - Pydantic models: `TelemetrySnapshot`, `SpawnEvent`.
  - In-memory state: `live_snapshots`, WebSocket connection registries.
- **generate_data.py**: Sample WebSocket client that sends synthetic telemetry snapshots.
- **alembic/** & **alembic.ini**: Migration tooling for evolving the database schema.
- **static/**: Frontend assets (HTML, CSS, JavaScript, images) for the live map UI.
- **FIXES.md**, **LESSONSLEARNED.md**, **TODO.md**: Project notes and future work.

## Database Overview

### Technology Stack
- **PostgreSQL** 14 with **TimescaleDB** extension for time-series optimization.
- **databases** library (async) with **SQLAlchemy Core** for schema definitions and queries.
- Environment variable: `DATABASE_URL` controls the connection string.

### Tables and Hypertable
1. **telemetry_events** (hypertable)
   - Columns:
     - `id`: Integer, primary key.
     - `character_name` (String), `char_tag` (String, nullable), `session_id` (String, indexed).
     - `timestamp` (DateTime with TZ, indexed) — partitioning column for the hypertable.
     - `ew`, `ns`, `z`: Float coordinates.
     - `kills`, `deaths`, `rares_found`, `prismatic_taper_count`: Integer metrics.
     - `kills_per_hour`, `onlinetime` (String), `vt_state` (String).
     - Optional: `mem_mb`, `cpu_pct`, `mem_handles`, `latency_ms`.
   - Created via `SELECT create_hypertable('telemetry_events', 'timestamp', if_not_exists=>true, create_default_indexes=>false)`.

2. **char_stats**
   - Tracks cumulative kills per character.
   - Columns: `character_name` (PK), `total_kills` (Integer).

3. **rare_stats**
   - Tracks total rare spawns per character.
   - Columns: `character_name` (PK), `total_rares` (Integer).

4. **rare_stats_sessions**
   - Tracks rarities per session.
   - Columns: composite PK `(character_name, session_id)`, `session_rares` (Integer).

5. **spawn_events**
   - Records individual mob spawn events for heatmapping.
   - Columns: `id` (PK), `character_name` (String), `mob` (String), `timestamp` (DateTime), `ew`, `ns`, `z` (Float).
   - Coordinates (`ew`, `ns`, `z`) can be sent as JSON numbers or strings and are coerced to floats.

6. **rare_events**
   - Records each rare spawn event for future heatmaps and analysis.
   - Columns: `id` (PK), `character_name` (String), `name` (String), `timestamp` (DateTime), `ew`, `ns`, `z` (Float).

### Initialization and Migrations
- On startup (`main.py`), `init_db_async()` is called:
  1. Creates all tables via SQLAlchemy’s `metadata.create_all()`.
  2. Enables TimescaleDB extension.
  3. Converts `telemetry_events` to a hypertable, skipping default index creation to avoid PK/index collisions.
- Alembic is configured for schema migrations (`alembic/` directory).

## Data Ingestion Flow
1. **Plugin** connects to `/ws/position` with a shared secret.
2. Sends JSON frames of types:
   - `telemetry`: parsed into `TelemetrySnapshot`, upserted into `live_snapshots`, persisted to `telemetry_events`, and broadcast to browser clients.
   - `spawn`: parsed into `SpawnEvent`, inserted into `spawn_events`.
   - `rare`: increments `rare_stats` and `rare_stats_sessions` via upsert operations.
   - `chat`: broadcast to browser clients without DB writes.
3. **Browser** connects to `/ws/live` to receive live updates and can send commands to plugins.

## HTTP Query Endpoints
- **GET /live**: returns recent snapshots (last 30s) plus rare counts per character.
- **GET /history**: returns ordered telemetry history with optional time filters.
- **GET /trails**: returns positional trails for a lookback window.

This architecture enables real-time telemetry ingestion, historical time-series analysis, and an interactive front-end map for tracking players and spawn events.