diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..f6144321 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,140 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project Overview + +Dereth Tracker is a real-time telemetry service for game world tracking. It's a FastAPI-based WebSocket and HTTP API service that ingests player position/stats data via plugins and provides live map visualization through a web interface. + +## Key Components + +### Main Service (main.py) +- WebSocket endpoint `/ws/position` receives telemetry and inventory events +- Routes inventory events to inventory service via HTTP +- Handles real-time player tracking and map updates + +### Inventory Service (inventory-service/main.py) +- Separate FastAPI service for inventory management +- Processes inventory JSON into normalized PostgreSQL tables +- Provides search API with advanced filtering and sorting +- Uses comprehensive enum database for translating game IDs to readable names + +### Database Architecture +- **Telemetry DB**: TimescaleDB for time-series player tracking data +- **Inventory DB**: PostgreSQL with normalized schema for equipment data + - `items`: Core item properties + - `item_combat_stats`: Armor level, damage bonuses + - `item_enhancements`: Material, item sets, tinkering + - `item_spells`: Spell names and categories + - `item_raw_data`: Original JSON for complex queries + +## Memories and Known Bugs + +* Fixed: Material names now properly display (e.g., "Gold Celdon Girth" instead of "Unknown_Material_Gold Celdon Girth") +* Fixed: Slot column shows "-" instead of "Unknown" for items without slot data +* Fixed: All 208 items in Larsson's inventory now process successfully (was 186 with 22 SQL type errors) +* Added: Type column in inventory search using object_classes enum for accurate item type classification +* Note: ItemType data is inconsistent in JSON - using ObjectClass as primary source for Type column + +## Recent Fixes (September 2025) + +### Portal Coordinate Rounding Fix ✅ RESOLVED +* **Problem**: Portal insertion failed with duplicate key errors due to coordinate rounding mismatch +* **Root Cause**: Code used 2 decimal places (`ROUND(ns::numeric, 2)`) but database constraint used 1 decimal place +* **Solution**: Changed all portal coordinate checks to use 1 decimal place to match DB constraint +* **Result**: 98% reduction in duplicate key errors (from 600+/min to ~11/min) +* **Location**: `main.py` lines ~1989, 1996, 2025, 2047 + +### Character Display Issues ✅ RESOLVED +* **Problem**: Some characters (e.g., "Crazed n Dazed") not appearing in frontend +* **Root Cause**: Database connection pool exhaustion from portal error spam +* **Solution**: Fixed portal errors to reduce database load +* **Result**: Characters now display correctly after portal fix + +### Docker Container Deployment +* **Issue**: Code changes require container rebuild with `--no-cache` flag +* **Command**: `docker compose build --no-cache dereth-tracker` +* **Reason**: Docker layer caching can prevent updated source code from being copied + +## Current Known Issues + +### Minor Portal Race Conditions +* **Status**: ~11 duplicate key errors per minute (down from 600+) +* **Cause**: Multiple players discovering same portal simultaneously +* **Impact**: Minimal - errors are caught and handled gracefully +* **Handling**: Try/catch in code logs as debug messages and updates portal timestamp +* **Potential Fix**: PostgreSQL ON CONFLICT DO UPDATE (upsert pattern) would eliminate completely + +### Database Initialization Warnings +* **TimescaleDB Hypertable**: `telemetry_events` fails to become hypertable due to primary key constraint +* **Impact**: None - table works as regular PostgreSQL table +* **Warning**: "cannot create a unique index without the column 'timestamp'" + +### Connection Pool Under Load +* **Issue**: Database queries can timeout when connection pool is exhausted +* **Symptom**: Characters may not appear during high error load +* **Mitigation**: Portal error fix significantly reduced this issue + +## Equipment Suit Builder + +### Status: PRODUCTION READY + +Real-time equipment optimization engine for building optimal character loadouts by searching across multiple characters' inventories (mules). Uses Mag-SuitBuilder constraint satisfaction algorithms. + +**Core Features:** +- Multi-character inventory search across 100+ characters, 25,000+ items +- Armor set constraints (primary 5-piece + secondary 4-piece set support) +- Cantrip/ward spell optimization with bitmap-based overlap detection +- Crit damage rating optimization +- Locked slots with set/spell preservation across searches +- Real-time SSE streaming with progressive phase updates +- Suit summary with copy-to-clipboard functionality +- Stable deterministic sorting for reproducible results + +**Access:** `/suitbuilder.html` + +**Architecture Details:** See `docs/plans/2026-02-09-suitbuilder-architecture.md` + +### Known Limitations +- Slot-aware spell filtering not yet implemented (e.g., underclothes have limited spell pools but system treats all slots equally) +- All spells weighted equally (no priority/importance weighting yet) +- See architecture doc for future enhancement roadmap + +## Technical Notes for Development + +### Database Performance +- Connection pool: 5-20 connections (configured in `db_async.py`) +- Under heavy error load, pool exhaustion can cause 2-minute query timeouts +- Portal error fix significantly improved database performance + +### Docker Development Workflow +1. **Code Changes**: Edit source files locally +2. **Rebuild**: `docker compose build --no-cache dereth-tracker` (required for code changes) +3. **Deploy**: `docker compose up -d dereth-tracker` +4. **Debug**: `docker logs mosswartoverlord-dereth-tracker-1` and `docker logs dereth-db` + +### Frontend Architecture +- **Main Map**: `static/index.html` - Real-time player tracking +- **Inventory Search**: `static/inventory.html` - Advanced item filtering +- **Suitbuilder**: `static/suitbuilder.html` - Equipment optimization interface +- **All static files**: Served directly by FastAPI StaticFiles + +### DOM Optimization Status ✅ COMPLETE (September 2025) +* **Achievement**: 100% DOM element reuse with zero element creation after initial render +* **Performance**: ~5ms render time for 69 players, eliminated 4,140+ elements/minute creation +* **Implementation**: Element pooling system with player name mapping for O(1) lookup +* **Monitoring**: Color-coded console output (✨ green = optimized, ⚡ yellow = partial, 🔥 red = poor) +* **Status**: Production ready - achieving perfect element reuse consistently + +**Current Render Stats**: +- ✅ This render: 0 dots created, 69 reused | 0 list items created, 69 reused +- ✅ Lifetime: 69 dots created, 800+ reused | 69 list items created, 800+ reused + +**Remaining TODO**: +- ❌ Fix CSS Grid layout for player sidebar (deferred per user request) +- ❌ Extend optimization to trails and portal rendering +- ❌ Add memory usage tracking + +### WebSocket Endpoints +- `/ws/position`: Plugin telemetry, inventory, portal, rare events (authenticated) +- `/ws/live`: Browser client commands and live updates (unauthenticated) \ No newline at end of file diff --git a/docs/plans/2026-02-09-suitbuilder-architecture.md b/docs/plans/2026-02-09-suitbuilder-architecture.md new file mode 100644 index 00000000..d10c4cdc --- /dev/null +++ b/docs/plans/2026-02-09-suitbuilder-architecture.md @@ -0,0 +1,233 @@ +# Suitbuilder Architecture & Implementation Status + +## Overview + +The Equipment Suit Builder is a real-time optimization engine that finds optimal equipment loadouts for characters by searching across multiple characters' inventories (mules). It uses constraint satisfaction algorithms derived from the original Mag-SuitBuilder (C#) plugin, reimplemented in Python as part of the inventory-service. + +**Status:** Production Ready +**Access:** `/suitbuilder.html` +**Backend:** `inventory-service/suitbuilder.py` +**Frontend:** `static/suitbuilder.js`, `static/suitbuilder.html`, `static/suitbuilder.css` + +--- + +## Key Achievements (2025-2026) + +- Multi-character search across 100+ characters and 25,000+ items +- Mag-SuitBuilder constraint satisfaction algorithms with branch pruning +- Real-time SSE streaming with phase updates and progress tracking +- Spell bitmap-based overlap detection and scoring +- Locked slot constraints with set/spell preservation across searches +- Stable deterministic sorting for reproducible results +- Comprehensive UI with suit summary and copy-to-clipboard +- Individual slot clear buttons for granular control +- Full cantrip/ward selection UI (attributes, weapon skills, magic skills, defenses, wards) + +--- + +## Architecture + +### Core Components + +1. **ConstraintSatisfactionSolver** - Main search engine. Performs recursive depth-first search through equipment buckets, applying constraints and scoring at each level. + +2. **ItemBucket** - Slot-based item organization. Groups items by equipment slot, sorts by priority (armor level, ratings, spells), and handles armor reduction variants. + +3. **SpellBitmapIndex** - O(1) spell overlap detection using bitwise operations. Each spell is assigned a unique bit position, enabling instant duplicate detection. + +4. **SuitState** - Mutable state during recursive search. Tracks equipped items, aggregated spell bitmap, set counts, total armor, and ratings. + +5. **SearchConstraints** - User-defined requirements including character selection, set preferences, required spells, locked slots, and scoring weights. + +6. **ItemPreFilter** - Pre-processing step that removes strictly dominated items (items surpassed in all metrics by another item in the same slot). + +### Data Flow + +``` +User (suitbuilder.html) + | + | POST /inv/suitbuilder/search (constraints JSON) + | + v +Main Service (main.py) + | + | Streaming proxy (dedicated SSE endpoint) + | + v +Inventory Service (inventory-service/main.py) + | + | Routes to suitbuilder module + | + v +ConstraintSatisfactionSolver + | + | 1. Load items from inventory DB + | 2. Create equipment buckets per slot + | 3. Apply armor reduction rules + | 4. Sort buckets (smallest first for pruning) + | 5. Recursive search with constraint checking + | 6. Score and yield suits via SSE stream + | + v +User (real-time suit results appear as found) +``` + +### SSE Streaming Architecture + +The suitbuilder search is a long-running operation (up to 5 minutes). Results stream in real-time via Server-Sent Events (SSE). + +**Main service** (`main.py`) has a dedicated streaming proxy endpoint at `/inv/suitbuilder/search` that uses `httpx.AsyncClient.stream()` to forward chunks without buffering. The generic `/inv/{path:path}` proxy buffers entire responses, which would cause timeouts for SSE streams. + +**Event types streamed:** +- `phase` - Search phase updates (loading, buckets, reducing, sorting, searching) +- `suit` - New suit found with items and score +- `log` - Verbose debug messages +- `stats` - Search statistics (branches explored, suits found, elapsed time) +- `complete` - Search finished +- `error` - Error occurred + +--- + +## Spell System Implementation + +### Spell Bitmap Index + +Uses bitwise operations for O(1) spell overlap detection: +- Each spell is assigned a unique bit position in an integer bitmap +- `spell_bitmap |= item.spell_bitmap` aggregates spells across items in a suit +- `new_spells = item_bitmap & needed_bitmap & ~current_bitmap` detects novel spell contributions +- Prevents duplicate spell slots across items in strict mode + +### Spell Constraint Flow + +1. **UI Collection**: Checkboxes for legendary cantrips and wards + - **Attributes**: Strength, Endurance, Coordination, Quickness, Focus, Willpower + - **Weapon Skills**: Finesse, Heavy, Light, Missile, Two-handed + - **Magic Skills**: War Magic, Void Magic, Creature, Item, Life Magic + - **Defenses**: Magic Resistance, Invulnerability + - **Wards**: Flame, Frost, Acid, Storm, Slashing, Piercing, Bludgeoning, Armor + +2. **Backend Processing**: + - `needed_spell_bitmap = spell_index.get_bitmap(required_spells)` pre-computes target + - During item loading: `item.spell_bitmap = spell_index.get_bitmap(item.spell_names)` + - During search: `_can_get_beneficial_spell_from()` rejects items with only duplicate spells + - Items with spells that contribute nothing new to the suit are pruned + +3. **Scoring**: + - Each fulfilled required spell: +100 points + - Duplicate spells prevented by constraint (not scored negatively, just blocked) + - Encourages maximum spell diversity across the suit + +### Jewelry Special Handling + +Jewelry (rings, bracelets, necklaces, trinkets) receives special treatment: +- Only added to suits if they contribute new required spells +- Prevents jewelry spam when no spell constraints are specified +- Prioritizes spell contribution over raw stats for jewelry slots +- Items with zero beneficial spell contribution are rejected + +--- + +## Recent Features (February 2026) + +### Locked Slots System + +Allows users to lock specific equipment slots with set and spell constraints. Locked slots are preserved when selecting different suit results. + +**Features:** +- Lock populated slots automatically (extracts set + spells from the item) +- Lock empty slots manually (configure set + spell requirements via form) +- Visual distinction (red border, lock icon) +- Constraints sent to backend (locked slots filtered from search buckets) +- Preserved when selecting different suits from results + +**Use Cases:** +- "I already have this Legendary Empyrean Helm, find suits around it" +- "Lock Head slot to Empyrean set with L. Strength + L. Flame Ward" +- Iterative refinement: lock good items, re-search for better accessories + +### Suit Summary Panel + +Copy-paste friendly summary below equipment slots showing all items with their source characters. + +**Format:** +``` +Slot: Item Name [Spell1, Spell2] - Character Name +``` + +**Features:** +- Auto-updates when suit selected or slots cleared +- Copy to clipboard button with visual feedback +- Monospace font for readability +- Shows all equipped items with their source characters + +### Clear Slot Button + +Small "x" button on populated slots for granular control: +- Clears individual slots without affecting locked state +- Removes item from selected suit state +- Updates summary automatically + +--- + +## Known Limitations + +### Slot-Aware Spell Filtering +The system currently treats all slots equally when checking spell availability. In reality, certain cantrips and wards never appear on underclothes (shirt/pants). The search could be optimized by pre-filtering based on realistic spell pools per slot type. + +### Spell Priority Weighting +All required spells are weighted equally (+100 points each). There is no mechanism to mark certain spells as "critical" vs "nice to have." + +--- + +## Future Enhancements Roadmap + +### Phase 1: Spell System Improvements + +#### Spell Priority Weighting +Allow users to mark certain spells as critical vs optional: +- Critical spells: +200 points (must have) +- High priority: +150 points +- Normal priority: +100 points (current default) +- Low priority: +50 points + +#### Slot-Aware Spell Filtering +Pre-filter items based on realistic spell pools per slot type: +- **Underclothes** (Shirt/Pants): Primarily damage rating, rarely cantrips/wards +- **Armor** (Head/Chest/Arms/Legs/Hands/Feet): Full cantrip/ward pool +- **Jewelry** (Rings/Bracelets/Neck): Full cantrip/ward pool +- **Trinket**: Limited or no spells + +Benefits: Faster search (smaller search space), more realistic suit expectations, better UX. + +### Phase 2: New Builders + +#### Weapon Suit Builder +Optimize weapon + shield/offhand combinations: +- Weapon skill requirements (Heavy, Light, Finesse, etc.) +- Damage type optimization +- Special property constraints (Crushing Blow, Armor Rending, etc.) + +#### Accessory-Only Builder +Find best jewelry/trinket combinations when armor is already decided: +- Smaller search space = faster results +- Focus on spell coverage and rating optimization +- Use case: "I have my armor set, find best accessories" + +#### Multi-Suit Batch Generation +Generate multiple viable suits in one search: +- "Show me top 10 suits for different playstyles" +- Variation in set combinations +- Export/compare multiple options + +### Phase 3: Advanced Features + +#### Saved Searches +- Save constraint configurations as named presets +- Quick re-run with updated inventory +- Presets like "Mage Build", "Tank Build", etc. + +#### Suit Comparison +- Side-by-side comparison of 2-3 suits +- Highlight differences in items, spells, and ratings +- Visual scoring breakdown