Server load optimization: spawn_events retention + log spam fix
Database cleanup: - Converted spawn_events to a TimescaleDB hypertable with 7-day retention. Previously a regular table growing unbounded — had reached 482M rows/66GB from June 2025. Manual migration copied last 7 days (12M rows) to a new hypertable, swapped names, and dropped the old table. Result: DB shrunk from 77GB → 12GB. - Dropped server_health_checks table entirely. It was write-only (850K rows, 134MB) — only current state in server_status is actually read. Eliminated the insert from monitor_server_health(). Telemetry handler cleanup: - Removed 4 per-message INFO log lines (TELEMETRY_RECEIVED, DB_WRITE_ATTEMPT, DB_WRITE_SUCCESS, PROCESSING_COMPLETE). At 60+ chars × every 2s = hundreds of log lines/sec. Replaced with single SLOW_* warnings above 500ms/1000ms thresholds. - Removed redundant pool-size introspection (try/except + hasattr) on every telemetry message — useless noise in the hot path. - Removed debug cache-miss and kill-delta logs. Log level: - docker-compose.yml: dereth-tracker LOG_LEVEL DEBUG → INFO (was dumping entire inventory_delta JSON payloads for every item update). - inventory-service LOG_LEVEL DEBUG → INFO. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
de2cc3a0e3
commit
926e912c57
3 changed files with 45 additions and 101 deletions
54
db_async.py
54
db_async.py
|
|
@ -165,25 +165,8 @@ portals = Table(
|
|||
Column("discovered_by", String, nullable=False),
|
||||
)
|
||||
|
||||
# Server health monitoring tables
|
||||
server_health_checks = Table(
|
||||
# Time-series data for server health checks
|
||||
"server_health_checks",
|
||||
metadata,
|
||||
Column("id", Integer, primary_key=True),
|
||||
Column("server_name", String, nullable=False, index=True),
|
||||
Column("server_address", String, nullable=False),
|
||||
Column(
|
||||
"timestamp",
|
||||
DateTime(timezone=True),
|
||||
nullable=False,
|
||||
default=sqlalchemy.func.now(),
|
||||
),
|
||||
Column("status", String(10), nullable=False), # 'up' or 'down'
|
||||
Column("latency_ms", Float, nullable=True),
|
||||
Column("player_count", Integer, nullable=True),
|
||||
)
|
||||
|
||||
# Server health monitoring: only current state is kept.
|
||||
# Historical health checks were removed — nothing read from them.
|
||||
server_status = Table(
|
||||
# Current server status and uptime tracking
|
||||
"server_status",
|
||||
|
|
@ -198,13 +181,6 @@ server_status = Table(
|
|||
Column("last_player_count", Integer, nullable=True),
|
||||
)
|
||||
|
||||
# Index for efficient server health check queries
|
||||
Index(
|
||||
"ix_server_health_checks_name_ts",
|
||||
server_health_checks.c.server_name,
|
||||
server_health_checks.c.timestamp.desc(),
|
||||
)
|
||||
|
||||
character_stats = Table(
|
||||
"character_stats",
|
||||
metadata,
|
||||
|
|
@ -299,6 +275,32 @@ async def init_db_async():
|
|||
except Exception as e:
|
||||
print(f"Warning: failed to set retention/compression policies: {e}")
|
||||
|
||||
# Ensure spawn_events is a hypertable with a 7-day retention policy.
|
||||
# This is idempotent — if already a hypertable, create_hypertable is a no-op
|
||||
# when if_not_exists=TRUE. The existing 482M-row table needed a manual
|
||||
# migration (see docs/plans/spawn_events_cleanup.md); this block keeps the
|
||||
# policy alive on subsequent deploys.
|
||||
try:
|
||||
with engine.connect().execution_options(isolation_level="AUTOCOMMIT") as conn:
|
||||
# Try to convert spawn_events to a hypertable if it isn't already.
|
||||
# migrate_data=FALSE is safe because the manual migration handled it;
|
||||
# if someone creates a fresh DB, the table is empty and this converts it.
|
||||
conn.execute(
|
||||
text(
|
||||
"SELECT create_hypertable('spawn_events', 'timestamp', "
|
||||
"if_not_exists => TRUE, migrate_data => FALSE, "
|
||||
"chunk_time_interval => INTERVAL '1 day')"
|
||||
)
|
||||
)
|
||||
# 7-day retention
|
||||
conn.execute(
|
||||
text(
|
||||
"SELECT add_retention_policy('spawn_events', INTERVAL '7 days', if_not_exists => TRUE)"
|
||||
)
|
||||
)
|
||||
except Exception as e:
|
||||
print(f"Warning: failed to set spawn_events hypertable/retention: {e}")
|
||||
|
||||
# Create unique constraint on rounded portal coordinates
|
||||
try:
|
||||
with engine.connect().execution_options(isolation_level="AUTOCOMMIT") as conn:
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue