Server load optimization: spawn_events retention + log spam fix

Database cleanup:
- Converted spawn_events to a TimescaleDB hypertable with 7-day retention.
  Previously a regular table growing unbounded — had reached 482M rows/66GB
  from June 2025. Manual migration copied last 7 days (12M rows) to a new
  hypertable, swapped names, and dropped the old table.
  Result: DB shrunk from 77GB → 12GB.
- Dropped server_health_checks table entirely. It was write-only (850K rows,
  134MB) — only current state in server_status is actually read. Eliminated
  the insert from monitor_server_health().

Telemetry handler cleanup:
- Removed 4 per-message INFO log lines (TELEMETRY_RECEIVED, DB_WRITE_ATTEMPT,
  DB_WRITE_SUCCESS, PROCESSING_COMPLETE). At 60+ chars × every 2s = hundreds
  of log lines/sec. Replaced with single SLOW_* warnings above 500ms/1000ms
  thresholds.
- Removed redundant pool-size introspection (try/except + hasattr) on every
  telemetry message — useless noise in the hot path.
- Removed debug cache-miss and kill-delta logs.

Log level:
- docker-compose.yml: dereth-tracker LOG_LEVEL DEBUG → INFO (was dumping
  entire inventory_delta JSON payloads for every item update).
- inventory-service LOG_LEVEL DEBUG → INFO.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Erik 2026-04-15 10:08:51 +02:00
parent de2cc3a0e3
commit 926e912c57
3 changed files with 45 additions and 101 deletions

View file

@ -165,25 +165,8 @@ portals = Table(
Column("discovered_by", String, nullable=False),
)
# Server health monitoring tables
server_health_checks = Table(
# Time-series data for server health checks
"server_health_checks",
metadata,
Column("id", Integer, primary_key=True),
Column("server_name", String, nullable=False, index=True),
Column("server_address", String, nullable=False),
Column(
"timestamp",
DateTime(timezone=True),
nullable=False,
default=sqlalchemy.func.now(),
),
Column("status", String(10), nullable=False), # 'up' or 'down'
Column("latency_ms", Float, nullable=True),
Column("player_count", Integer, nullable=True),
)
# Server health monitoring: only current state is kept.
# Historical health checks were removed — nothing read from them.
server_status = Table(
# Current server status and uptime tracking
"server_status",
@ -198,13 +181,6 @@ server_status = Table(
Column("last_player_count", Integer, nullable=True),
)
# Index for efficient server health check queries
Index(
"ix_server_health_checks_name_ts",
server_health_checks.c.server_name,
server_health_checks.c.timestamp.desc(),
)
character_stats = Table(
"character_stats",
metadata,
@ -299,6 +275,32 @@ async def init_db_async():
except Exception as e:
print(f"Warning: failed to set retention/compression policies: {e}")
# Ensure spawn_events is a hypertable with a 7-day retention policy.
# This is idempotent — if already a hypertable, create_hypertable is a no-op
# when if_not_exists=TRUE. The existing 482M-row table needed a manual
# migration (see docs/plans/spawn_events_cleanup.md); this block keeps the
# policy alive on subsequent deploys.
try:
with engine.connect().execution_options(isolation_level="AUTOCOMMIT") as conn:
# Try to convert spawn_events to a hypertable if it isn't already.
# migrate_data=FALSE is safe because the manual migration handled it;
# if someone creates a fresh DB, the table is empty and this converts it.
conn.execute(
text(
"SELECT create_hypertable('spawn_events', 'timestamp', "
"if_not_exists => TRUE, migrate_data => FALSE, "
"chunk_time_interval => INTERVAL '1 day')"
)
)
# 7-day retention
conn.execute(
text(
"SELECT add_retention_policy('spawn_events', INTERVAL '7 days', if_not_exists => TRUE)"
)
)
except Exception as e:
print(f"Warning: failed to set spawn_events hypertable/retention: {e}")
# Create unique constraint on rounded portal coordinates
try:
with engine.connect().execution_options(isolation_level="AUTOCOMMIT") as conn: