## TODO: Migration & Parity Plan

### Detailed Plan
1. [ ] Review Repository for Data Storage and Event Handling
   - [ ] Scan for SQLite usage (telemetry, spawns, chat, session data)
   - [ ] Identify all event ingestion code paths (WebSocket, HTTP, direct DB inserts)
   - [ ] Locate old or deprecated payload handling
2. [ ] Update Database Access Layer to PostgreSQL/TimescaleDB
   - [ ] Replace SQLite code with SQLAlchemy models & Alembic migrations
   - [ ] Configure TimescaleDB hypertable for telemetry data
   - [ ] Create migration for spawn events table
   - [ ] Set up `DATABASE_URL` and (optional) local Docker Compose service
3. [ ] Refactor Event Ingestion Endpoints and Logic
   - [ ] Modify `/ws/position` to accept new schemas (telemetry, spawn, chat)
   - [ ] Persist telemetry and spawn events to PostgreSQL
   - [ ] Continue broadcasting chat messages without persisting
4. [ ] Update Data Models and API Response Types
   - [ ] Align Pydantic schemas to new event payload structures
   - [ ] Update `/live`, `/history`, `/trails` to query Postgres
   - [ ] Optionally add `GET /spawns` endpoint for spawn data
5. [ ] Migrate or Clean Historical Data
   - [ ] If needed, write script to migrate existing SQLite data to Postgres
   - [ ] Otherwise remove old migration and data transformation code
6. [ ] Refactor Frontend to Query and Visualize New Data (deferred)
7. [ ] Add or Update Grafana Dashboards (deferred)
8. [ ] Testing & Verification (deferred)
9. [ ] Documentation & Developer Instructions
   - [ ] Update README and docs for PostgreSQL/TimescaleDB setup
10. [ ] Maintenance and Future Enhancements
   - [ ] Document data retention and aggregation policies for TimescaleDB

### Phases

### Phase 1: Core Migration & Parity
- [ ] Remove SQLite usage and associated code (`db.py` and direct `sqlite3` calls).
- [ ] Integrate PostgreSQL/TimescaleDB using SQLAlchemy and Alembic for migrations.
  - Set up `DATABASE_URL` environment variable for connection.
  - (Optional) Add a TimescaleDB service in `docker-compose.yml` for local development.
- [ ] Define SQLAlchemy models and create initial Alembic migration:
  - Telemetry table as a TimescaleDB hypertable.
  - Spawn events table.
- [ ] Update backend (`main.py`):
  - Ingest `telemetry` and new `spawn` messages from `/ws/position` WebSocket.
  - Persist telemetry and spawn events to PostgreSQL.
  - Continue broadcasting `chat` messages without persisting.
- [ ] Ensure existing endpoints (`/live`, `/history`, `/trails`) operate against the new database.
- [ ] (Optional) Add retrieval endpoint for spawn events (e.g., `GET /spawns`).

### Phase 2: Frontend & Visualization
- [ ] Update frontend to display spawn events (markers or lists).
- [ ] Expose new telemetry metrics in the UI: `latency_ms`, `mem_mb`, `cpu_pct`, `mem_handles`.

### Phase 3: Dashboards & Monitoring
* [ ] Provision or update Grafana dashboards for:
  - Telemetry performance (TimescaleDB queries, hypertable metrics).
  - Spawn event heatmaps and trends.
  - Rare event heatmaps and trends.

### Phase 4: Documentation & Maintenance
- [ ] Finalize README and developer docs with PostgreSQL setup, migration steps, and usage examples.
- [ ] Document how to add new event types or payload fields, including schema, migrations, and tests.
- [ ] Establish data retention and aggregation policies for TimescaleDB hypertables.