95 lines
No EOL
5 KiB
Markdown
95 lines
No EOL
5 KiB
Markdown
# Project Architecture and Data Model
|
||
|
||
This document provides an overview of the project files, their roles, and a detailed description of the database architecture and data model.
|
||
|
||
## Project Structure
|
||
|
||
Root directory:
|
||
- **Dockerfile**: Defines the Python 3.12-slim image, installs dependencies (FastAPI, Uvicorn, SQLAlchemy, databases, TimescaleDB support), and runs the app.
|
||
- **docker-compose.yml**: Orchestrates two services:
|
||
- **dereth-tracker**: The FastAPI application container.
|
||
- **db**: A TimescaleDB (PostgreSQL 14 + TimescaleDB extension) container for persistent storage.
|
||
- **README.md**: High-level documentation and usage instructions.
|
||
- **EVENT_FORMATS.json**: Example JSON payloads for all event types (`telemetry`, `spawn`, `chat`, `rare`).
|
||
- **db.py**: Legacy SQLite-based storage (telemetry_log & live_state tables, WAL mode, auto-vacuum).
|
||
- **db_async.py**: Async database definitions for PostgreSQL/TimescaleDB:
|
||
- Table schemas (SQLAlchemy Core):
|
||
- `telemetry_events`,
|
||
- `char_stats`,
|
||
- `rare_stats`,
|
||
- `rare_stats_sessions`,
|
||
- `spawn_events`,
|
||
- `rare_events`.
|
||
- `init_db_async()`: Creates tables, enables TimescaleDB extension, and configures a hypertable on `telemetry_events`.
|
||
- **main.py**: The FastAPI application:
|
||
- HTTP endpoints: `/debug`, `/live`, `/history`, `/trails`.
|
||
- WebSocket endpoints: `/ws/position` (plugin data in), `/ws/live` (browser live updates).
|
||
- Pydantic models: `TelemetrySnapshot`, `SpawnEvent`.
|
||
- In-memory state: `live_snapshots`, WebSocket connection registries.
|
||
- **generate_data.py**: Sample WebSocket client that sends synthetic telemetry snapshots.
|
||
- **alembic/** & **alembic.ini**: Migration tooling for evolving the database schema.
|
||
- **static/**: Frontend assets (HTML, CSS, JavaScript, images) for the live map UI.
|
||
- **FIXES.md**, **LESSONSLEARNED.md**, **TODO.md**: Project notes and future work.
|
||
|
||
## Database Overview
|
||
|
||
### Technology Stack
|
||
- **PostgreSQL** 14 with **TimescaleDB** extension for time-series optimization.
|
||
- **databases** library (async) with **SQLAlchemy Core** for schema definitions and queries.
|
||
- Environment variable: `DATABASE_URL` controls the connection string.
|
||
|
||
### Tables and Hypertable
|
||
1. **telemetry_events** (hypertable)
|
||
- Columns:
|
||
- `id`: Integer, primary key.
|
||
- `character_name` (String), `char_tag` (String, nullable), `session_id` (String, indexed).
|
||
- `timestamp` (DateTime with TZ, indexed) — partitioning column for the hypertable.
|
||
- `ew`, `ns`, `z`: Float coordinates.
|
||
- `kills`, `deaths`, `rares_found`, `prismatic_taper_count`: Integer metrics.
|
||
- `kills_per_hour`, `onlinetime` (String), `vt_state` (String).
|
||
- Optional: `mem_mb`, `cpu_pct`, `mem_handles`, `latency_ms`.
|
||
- Created via `SELECT create_hypertable('telemetry_events', 'timestamp', if_not_exists=>true, create_default_indexes=>false)`.
|
||
|
||
2. **char_stats**
|
||
- Tracks cumulative kills per character.
|
||
- Columns: `character_name` (PK), `total_kills` (Integer).
|
||
|
||
3. **rare_stats**
|
||
- Tracks total rare spawns per character.
|
||
- Columns: `character_name` (PK), `total_rares` (Integer).
|
||
|
||
4. **rare_stats_sessions**
|
||
- Tracks rarities per session.
|
||
- Columns: composite PK `(character_name, session_id)`, `session_rares` (Integer).
|
||
|
||
5. **spawn_events**
|
||
- Records individual mob spawn events for heatmapping.
|
||
- Columns: `id` (PK), `character_name` (String), `mob` (String), `timestamp` (DateTime), `ew`, `ns`, `z` (Float).
|
||
- Coordinates (`ew`, `ns`, `z`) can be sent as JSON numbers or strings and are coerced to floats.
|
||
|
||
6. **rare_events**
|
||
- Records each rare spawn event for future heatmaps and analysis.
|
||
- Columns: `id` (PK), `character_name` (String), `name` (String), `timestamp` (DateTime), `ew`, `ns`, `z` (Float).
|
||
|
||
### Initialization and Migrations
|
||
- On startup (`main.py`), `init_db_async()` is called:
|
||
1. Creates all tables via SQLAlchemy’s `metadata.create_all()`.
|
||
2. Enables TimescaleDB extension.
|
||
3. Converts `telemetry_events` to a hypertable, skipping default index creation to avoid PK/index collisions.
|
||
- Alembic is configured for schema migrations (`alembic/` directory).
|
||
|
||
## Data Ingestion Flow
|
||
1. **Plugin** connects to `/ws/position` with a shared secret.
|
||
2. Sends JSON frames of types:
|
||
- `telemetry`: parsed into `TelemetrySnapshot`, upserted into `live_snapshots`, persisted to `telemetry_events`, and broadcast to browser clients.
|
||
- `spawn`: parsed into `SpawnEvent`, inserted into `spawn_events`.
|
||
- `rare`: increments `rare_stats` and `rare_stats_sessions` via upsert operations.
|
||
- `chat`: broadcast to browser clients without DB writes.
|
||
3. **Browser** connects to `/ws/live` to receive live updates and can send commands to plugins.
|
||
|
||
## HTTP Query Endpoints
|
||
- **GET /live**: returns recent snapshots (last 30s) plus rare counts per character.
|
||
- **GET /history**: returns ordered telemetry history with optional time filters.
|
||
- **GET /trails**: returns positional trails for a lookback window.
|
||
|
||
This architecture enables real-time telemetry ingestion, historical time-series analysis, and an interactive front-end map for tracking players and spawn events. |