MosswartOverlord/ARCHITECTURE.md

Project Architecture and Data Model

This document provides an overview of the project files, their roles, and a detailed description of the database architecture and data model.

Project Structure

Root directory:

• Dockerfile: Defines the Python 3.12-slim image, installs dependencies (FastAPI, Uvicorn, SQLAlchemy, databases, TimescaleDB support), and runs the app.
• docker-compose.yml: Orchestrates two services:
  • dereth-tracker: The FastAPI application container.
  • db: A TimescaleDB (PostgreSQL 14 + TimescaleDB extension) container for persistent storage.
• README.md: High-level documentation and usage instructions.
• EVENT_FORMATS.json: Example JSON payloads for all event types (telemetry, spawn, chat, rare).
• db.py: Legacy SQLite-based storage (telemetry_log & live_state tables, WAL mode, auto-vacuum).
• db_async.py: Async database definitions for PostgreSQL/TimescaleDB:
• Table schemas (SQLAlchemy Core):
  • telemetry_events,
  • char_stats,
  • rare_stats,
  • rare_stats_sessions,
  • spawn_events,
  • rare_events.
• init_db_async(): Creates tables, enables TimescaleDB extension, and configures a hypertable on telemetry_events.
• main.py: The FastAPI application:
  • HTTP endpoints: /debug, /live, /history, /trails.
  • WebSocket endpoints: /ws/position (plugin data in), /ws/live (browser live updates).
  • Pydantic models: TelemetrySnapshot, SpawnEvent.
  • In-memory state: live_snapshots, WebSocket connection registries.
• generate_data.py: Sample WebSocket client that sends synthetic telemetry snapshots.
• alembic/ & alembic.ini: Migration tooling for evolving the database schema.
• static/: Frontend assets (HTML, CSS, JavaScript, images) for the live map UI.
• FIXES.md, LESSONSLEARNED.md, TODO.md: Project notes and future work.

Database Overview

Technology Stack

• PostgreSQL 14 with TimescaleDB extension for time-series optimization.
• databases library (async) with SQLAlchemy Core for schema definitions and queries.
• Environment variable: DATABASE_URL controls the connection string.

Tables and Hypertable

1. telemetry_events (hypertable)

   • Columns:
     • id: Integer, primary key.
     • character_name (String), char_tag (String, nullable), session_id (String, indexed).
     • timestamp (DateTime with TZ, indexed) — partitioning column for the hypertable.
     • ew, ns, z: Float coordinates.
     • kills, deaths, rares_found, prismatic_taper_count: Integer metrics.
     • kills_per_hour, onlinetime (String), vt_state (String).
     • Optional: mem_mb, cpu_pct, mem_handles, latency_ms.
   • Created via SELECT create_hypertable('telemetry_events', 'timestamp', if_not_exists=>true, create_default_indexes=>false).

2. char_stats

   • Tracks cumulative kills per character.
   • Columns: character_name (PK), total_kills (Integer).

3. rare_stats

   • Tracks total rare spawns per character.
   • Columns: character_name (PK), total_rares (Integer).

4. rare_stats_sessions

   • Tracks rarities per session.
   • Columns: composite PK (character_name, session_id), session_rares (Integer).

5. spawn_events

   • Records individual mob spawn events for heatmapping.
   • Columns: id (PK), character_name (String), mob (String), timestamp (DateTime), ew, ns, z (Float).
   • Coordinates (ew, ns, z) can be sent as JSON numbers or strings and are coerced to floats.

6. rare_events

   • Records each rare spawn event for future heatmaps and analysis.
   • Columns: id (PK), character_name (String), name (String), timestamp (DateTime), ew, ns, z (Float).

Initialization and Migrations

• On startup (main.py), init_db_async() is called:
  1. Creates all tables via SQLAlchemy’s metadata.create_all().
  2. Enables TimescaleDB extension.
  3. Converts telemetry_events to a hypertable, skipping default index creation to avoid PK/index collisions.
• Alembic is configured for schema migrations (alembic/ directory).

Data Ingestion Flow

1. Plugin connects to /ws/position with a shared secret.
2. Sends JSON frames of types:
   • telemetry: parsed into TelemetrySnapshot, upserted into live_snapshots, persisted to telemetry_events, and broadcast to browser clients.
   • spawn: parsed into SpawnEvent, inserted into spawn_events.
   • rare: increments rare_stats and rare_stats_sessions via upsert operations.
   • chat: broadcast to browser clients without DB writes.
3. Browser connects to /ws/live to receive live updates and can send commands to plugins.

HTTP Query Endpoints

• GET /live: returns recent snapshots (last 30s) plus rare counts per character.
• GET /history: returns ordered telemetry history with optional time filters.
• GET /trails: returns positional trails for a lookback window.

This architecture enables real-time telemetry ingestion, historical time-series analysis, and an interactive front-end map for tracking players and spawn events.