SMS-C Operations Guide

← Back to Documentation Index | Main README

Daily operational procedures, monitoring, and maintenance tasks for SMS-C operations teams.

Table of Contents

• Daily Operations
• Monitoring
• Message Tracking
• Route Management
• Frontend Management
• Number Translation Management
• System Maintenance
• Backup and Recovery
• Capacity Planning
• Incident Response

Daily Operations

Morning Health Check

Perform these checks at the start of each day:

1. Check System Status

# API health check
curl https://api.example.com:8443/api/status

# Expected response:
# {"status":"ok","application":"OmniMessage","timestamp":"2025-10-30T08:00:00Z"}

2. Review Prometheus Metrics

Access Prometheus dashboard and check:

• Message throughput (last 24 hours)
• Routing failure rate (should be < 1%)
• Queue backlog (should be < 1000 pending)
• Delivery success rate (should be > 95%)
• Frontend connection status (all expected frontends active)

3. Check Message Queue

Access Web UI: https://sms-admin.example.com/message_queue

Review:

• Total pending messages (should be low)
• Oldest message age (should be < 5 minutes)
• Messages with high delivery attempts (investigate if > 3)
• Dead letter messages (investigate any present)

4. Review Frontend Status

Access Web UI: https://sms-admin.example.com/frontend_status

Verify:

• All expected frontends are active
• No unexpired disconnections
• No frontend errors in last 24 hours

5. Check Application Logs

Access Web UI: https://sms-admin.example.com/logs or check log files

Look for:

• Error-level messages
• Routing failures
• Charging failures
• Database connection issues
• Cluster node problems

Message Volume Monitoring

Check Hourly Message Counts:

Use Prometheus query:

# Messages received per hour
increase(sms_c_message_received_count[1h])

# Messages delivered per hour
increase(sms_c_delivery_succeeded_count[1h])

# Calculate delivery rate
rate(sms_c_delivery_succeeded_count[1h]) / rate(sms_c_message_received_count[1h])

Expected Patterns:

• Business hours: Higher volume
• Nights/weekends: Lower volume
• Delivery rate: Should be > 95%

Alert Conditions:

• Sudden drop in messages ( > 50% decrease)
• Sudden spike in messages ( > 200% increase)
• Delivery rate drop below 90%

Monitoring

Key Metrics to Watch

Message Processing Metrics

Message Received Count (sms_c_message_received_count):

• What: Total messages entering system
• Alert: Sudden drop or spike
• Query: rate(sms_c_message_received_count[5m])

Message Processing Duration (sms_c_message_processing_stop_duration):

• What: End-to-end processing time
• Alert: p95 > 1000ms
• Query: histogram_quantile(0.95, sms_c_message_processing_stop_duration)

Routing Metrics

Routing Failures (sms_c_routing_failed_count):

• What: Messages that couldn't be routed
• Alert: Any failures ( > 0)
• Query: increase(sms_c_routing_failed_count[5m])

Route Matched (sms_c_routing_route_matched_count):

• What: Which routes are being used
• Alert: High-priority routes not matching
• Query: sms_c_routing_route_matched_count

Delivery Metrics

Delivery Success Rate:

• What: Percentage of successful deliveries
• Alert: Rate < 95%
• Query: rate(sms_c_delivery_succeeded_count[5m]) / rate(sms_c_delivery_queued_count[5m])

Delivery Attempts (sms_c_delivery_succeeded_attempt_count):

• What: Retries needed for delivery
• Alert: p95 > 2 (too many retries)
• Query: histogram_quantile(0.95, sms_c_delivery_succeeded_attempt_count)

Queue Metrics

Queue Size (sms_c_queue_size_size):

• What: Total messages in queue
• Alert: Size > 10,000
• Query: sms_c_queue_size_size

Oldest Message Age (sms_c_queue_oldest_message_age_seconds):

• What: Age of oldest pending message
• Alert: Age > 300 seconds
• Query: sms_c_queue_oldest_message_age_seconds

Dashboard Setup

Operational Dashboard Panels:

1. Message Throughput (Graph)

   • Messages received (5-minute rate)
   • Messages delivered (5-minute rate)
   • Time range: Last 24 hours

2. Queue Status (Single Stats)

   • Current pending messages
   • Oldest message age
   • Failed message count

3. Delivery Performance (Graph)

   • Success rate over time
   • Failure rate over time
   • Time range: Last 24 hours

4. Routing Status (Table)

   • Route ID
   • Match count (last hour)
   • Destination SMSC
   • Priority

5. Frontend Status (Table)

   • Frontend name
   • Status (active/expired)
   • Last seen
   • Message count (last hour)

6. System Health (Single Stats)

   • API response time (p95)
   • Database query time (p95)
   • ENUM lookup time (p95)

Alert Configuration

Critical Alerts (Immediate Response Required):

# No route found - messages cannot be delivered
- alert: RoutingFailures
  expr: increase(sms_c_routing_failed_count[5m]) > 0
  severity: critical
  description: "{{ $value }} messages failed routing in last 5 minutes"

# Queue building up - processing falling behind
- alert: QueueBacklog
  expr: sms_c_queue_size_pending > 10000
  severity: critical
  description: "Queue has {{ $value }} pending messages"

# Messages aging - delivery stuck
- alert: OldMessagesInQueue
  expr: sms_c_queue_oldest_message_age_seconds > 300
  severity: critical
  description: "Oldest message is {{ $value }} seconds old"

# Frontend disconnected - no delivery path
- alert: FrontendDisconnected
  expr: sms_c_frontend_status_count{status="disconnected"} > 0
  severity: critical
  description: "{{ $value }} frontends disconnected"

Warning Alerts (Investigation Needed):

# Delivery success rate dropping
- alert: LowDeliveryRate
  expr: rate(sms_c_delivery_succeeded_count[10m]) / rate(sms_c_delivery_queued_count[10m]) < 0.90
  severity: warning
  description: "Delivery success rate is {{ $value }}"

# Too many delivery retries
- alert: HighRetryRate
  expr: histogram_quantile(0.95, sms_c_delivery_succeeded_attempt_count) > 2
  severity: warning
  description: "95th percentile delivery attempts: {{ $value }}"

# ENUM lookups slow or failing
- alert: SlowEnumLookups
  expr: histogram_quantile(0.95, sms_c_enum_lookup_stop_duration) > 5000
  severity: warning
  description: "ENUM lookups taking > 5 seconds"

# Low ENUM cache hit rate
- alert: LowEnumCacheHitRate
  expr: rate(sms_c_enum_cache_hit_count[10m]) / (rate(sms_c_enum_cache_hit_count[10m]) + rate(sms_c_enum_cache_miss_count[10m])) < 0.70
  severity: warning
  description: "ENUM cache hit rate: {{ $value }}"

Message Tracking

Find Specific Message

By Message ID:

1. Web UI: Navigate to /message_queue
2. Enter message ID in search box
3. View full details and event history

Via API:

curl https://api.example.com:8443/api/messages/12345

By Phone Number:

1. Web UI: Navigate to /message_queue
2. Enter phone number in search box
3. View all messages for that number

Track Message Lifecycle

View Event History:

1. Web UI: Click on message in queue, view "Events" section
2. API: GET /api/events/12345

Common Event Sequence:

1. message_inserted - Message created
   ↓
2. number_translated - Numbers normalized (if configured)
   ↓
3. message_routed - Routing decision made
   ↓
4. charging_attempted - Charging check (if enabled)
   ↓
5. message_delivered - Successfully delivered

Failed Delivery Sequence:

1. message_inserted
   ↓
2. message_routed
   ↓
3. delivery_attempt_1 - First attempt failed
   ↓
4. delivery_attempt_2 - Second attempt failed (2min delay)
   ↓
5. delivery_attempt_3 - Third attempt failed (4min delay)
   ↓
6. message_dead_letter - Exceeded retry limit

Check Delivery Status

Awaiting Delivery:

• Status: "queued" (ready to send now) or "backoff" (waiting out a retry)
• deliver_after: nil/past for queued, a future timestamp for backoff
• delivery_attempts: 0 for a fresh queued message, higher for backoff

Delivered Messages:

• Status: "delivered" (delivery confirmed) or "sent" (submitted, no DLR yet)
• deliver_time: Timestamp of delivery
• dest_smsc: Frontend that delivered

Undelivered / Failed Messages:

• Status: "backoff" with high delivery_attempts — still retrying (capped at a 30-minute interval, until the message's expires is reached)
• Status: "expired", "dropped", "balance_rejected", or "auto_replied" — terminal
• Check event log for failure reasons

Location-Based Message Routing

The SMS-C supports location-based message retrieval, allowing frontends to automatically receive messages destined for subscribers registered at their location.

How It Works:

When a frontend queries for pending messages using get_messages_for_smsc(smsc_name), the system returns messages in two ways:

1. Explicit Routing - Messages where dest_smsc explicitly matches the frontend name
2. Location-Based Routing - Messages where:
   • dest_smsc is null (not explicitly routed)
   • destination_msisdn has an active location record
   • The location's location field matches the frontend name
   • The location has not expired

Example Scenario:

A subscriber with MSISDN +447700900123 registers at frontend uk_gateway:

# Subscriber registers (creates location record)
POST /api/locations
{
  "msisdn": "+447700900123",
  "imsi": "234150123456789",
  "location": "uk_gateway",
  "expires": "2025-11-01T12:00:00Z"
}

When a message arrives for this subscriber without explicit routing:

# Message submitted without dest_smsc
POST /api/messages
{
  "source_msisdn": "+15551234567",
  "destination_msisdn": "+447700900123",
  "message_body": "Hello",
  "source_smsc": "api"
  # Note: dest_smsc is null
}

The uk_gateway frontend will automatically receive this message when it polls:

# Frontend polls for messages
GET /api/messages/queue?smsc=uk_gateway

# Returns the message even though dest_smsc is null
# because the destination subscriber is registered at uk_gateway

Location Requirements:

For location-based routing to work:

• The locations table must have an entry for the destination_msisdn
• The location field must match the querying SMSC name
• The expires timestamp must be in the future

Monitoring Location-Based Routing:

Check location records:

# Via API
GET /api/locations/{msisdn}

# Check if location is expired
# expires field should be > current time

Common Issues:

• Message not delivered: Check if location has expired
• Wrong frontend: Verify location field matches expected frontend name
• Location not found: Subscriber may need to re-register

Manual Interventions

Retry Failed Message:

# Reset delivery_attempts and deliver_after
curl -X PATCH https://api.example.com:8443/api/messages/12345 \
  -H "Content-Type: application/json" \
  -d '{
    "delivery_attempts": 0,
    "deliver_after": "2025-10-30T12:00:00Z"
  }'

Change Destination:

# Route to different SMSC
curl -X PATCH https://api.example.com:8443/api/messages/12345 \
  -H "Content-Type: application/json" \
  -d '{
    "dest_smsc": "backup_gateway"
  }'

Delete Stuck Message:

curl -X DELETE https://api.example.com:8443/api/messages/12345

Route Management

View Current Routes

Web UI: Navigate to /sms_routing

Via API:

# List all routes
curl https://api.example.com:8443/api/routes

Check Route Usage:

Prometheus query:

# Messages routed by each route (last hour)
increase(sms_c_routing_route_matched_count[1h])

Add New Route

Web UI:

1. Navigate to /sms_routing
2. Click "Add New Route"
3. Fill in fields:
   • Calling Prefix: Source number prefix (optional)
   • Called Prefix: Destination number prefix (required for geographic routing)
   • Source SMSC: Source system filter (optional)
   • Dest SMSC: Destination gateway (required unless auto-reply/drop)
   • On-Net Only: Restrict this route to on-net destinations only (requires Diameter/HSS)
   • Priority: Route priority (1-255, lower = higher priority)
   • Weight: Load balancing weight (1-100)
   • Description: Human-readable description
   • Enabled: Check to activate immediately
4. Click "Save Route"

Example: Geographic Route:

• Called Prefix: +44
• Dest SMSC: uk_gateway
• Priority: 50
• Weight: 100
• Description: "UK routing"

Example: Load Balanced Route:

Create two routes with same criteria but different weights:

Route 1:

• Called Prefix: +44
• Dest SMSC: uk_primary
• Priority: 50
• Weight: 70
• Description: "UK primary (70%)"

Route 2:

• Called Prefix: +44
• Dest SMSC: uk_backup
• Priority: 50
• Weight: 30
• Description: "UK backup (30%)"

Example: On-Net Only Route:

Restrict an external carrier or internal application to only send to on-net subscribers. Messages to off-net destinations are dropped.

• Source SMSC: carrier_smpp_bind
• Dest SMSC: local_msc
• On-Net Only: checked
• Priority: 50
• Weight: 100
• Description: "Carrier X — on-net only"

This requires Diameter/HSS to be enabled. If the HSS is unavailable, messages are dropped (fail-closed).

Test Routes

Routing Simulator:

1. Navigate to /simulator
2. Enter test parameters:
   • Calling Number: +15551234567
   • Called Number: +447700900000
   • Source SMSC: (optional)
   • Source Type: (optional)
3. Click "Simulate Routing"
4. Review results:
   • Selected Route: Which route was chosen
   • All Matches: Which routes matched criteria
   • Evaluation: Why each route matched or didn't match

Test Before Production:

• Test all new routes in simulator
• Verify correct route is selected
• Check priority ordering
• Validate weight distribution

Modify Existing Route

Web UI:

1. Navigate to /sms_routing
2. Find route in list
3. Click "Edit"
4. Modify fields
5. Click "Save Route"

Common Modifications:

• Disable Route: Uncheck "Enabled" (temporary removal)
• Adjust Weight: Change load balance distribution
• Change Priority: Reorder route evaluation
• Update Destination: Switch to different SMSC

Delete Route

Web UI:

1. Navigate to /sms_routing
2. Find route in list
3. Click "Delete"
4. Confirm deletion

Warning: Deleting routes is permanent. Consider disabling instead.

Export/Import Routes

Export Routes (Backup):

1. Navigate to /sms_routing
2. Click "Export Routes"
3. Save JSON file

Import Routes:

1. Navigate to /sms_routing
2. Click "Import Routes"
3. Select JSON file
4. Choose import mode:
   • Merge: Add to existing routes
   • Replace: Delete all and import

Use Cases:

• Backup before major changes
• Copy routes between environments
• Disaster recovery
• Configuration versioning

Frontend Management

Monitor Frontend Connections

Web UI: Navigate to /frontend_status

Check:

• All expected frontends are "active"
• Last seen times are recent ( < 90 seconds)
• No unexpected expired frontends

Via API:

# Get active frontends
curl https://api.example.com:8443/api/frontends/active

# Get statistics
curl https://api.example.com:8443/api/frontends/stats

Investigate Disconnections

Frontend Expired:

1. Check frontend logs for errors
2. Verify network connectivity to SMS-C
3. Confirm frontend is running
4. Check frontend registration logic (should re-register every 60s)

Registration Not Showing:

1. Verify frontend is calling POST /api/frontends/register
2. Check API logs for registration errors
3. Verify JSON payload format
4. Test registration manually with curl

Example Manual Registration:

curl -X POST https://api.example.com:8443/api/frontends/register \
  -H "Content-Type: application/json" \
  -d '{
    "frontend_name": "test_gateway",
    "frontend_type": "smpp",
    "ip_address": "10.0.1.50",
    "hostname": "gateway.example.com"
  }'

View Frontend History

Web UI:

1. Navigate to /frontend_status
2. Find frontend in list
3. Click "History"
4. Review past registrations

Via API:

curl https://api.example.com:8443/api/frontends/history/uk_gateway

Use Cases:

• Investigate connection reliability
• Track frontend uptime patterns
• Identify configuration changes

Number Translation Management

Number translation rules are managed via config/runtime.exs. Changes require application restart.

View Active Translation Rules

Check configuration file:

cat config/runtime.exs | grep -A 20 "translation_rules:"

Common Translation Tasks

Add Country Code to Local Numbers:

Edit config/runtime.exs:

%{
  calling_prefix: nil,
  called_prefix: nil,
  source_smsc: nil,
  calling_match: "^(\d{10})$",
  calling_replace: "+1\1",
  called_match: "^(\d{10})$",
  called_replace: "+1\1",
  priority: 100,
  description: "Add +1 to 10-digit US numbers",
  enabled: true
}

Normalize International Format:

%{
  calling_prefix: nil,
  called_prefix: nil,
  source_smsc: nil,
  calling_match: "^00(\d+)$",
  calling_replace: "+\1",
  called_match: "^00(\d+)$",
  called_replace: "+\1",
  priority: 10,
  description: "Convert 00 prefix to +",
  enabled: true
}

Carrier-Specific Code Stripping:

%{
  calling_prefix: nil,
  called_prefix: "101",
  source_smsc: "carrier_a",
  calling_match: nil,
  calling_replace: nil,
  called_match: "^101(\d+)$",
  called_replace: "\1",
  priority: 5,
  description: "Strip carrier code from carrier A",
  enabled: true
}

Test Translation Rules

After configuration changes:

1. Restart application to load new rules
2. Submit test message with source/destination that should match
3. Check event log for number_translated event
4. Verify numbers were transformed correctly

Disable Translation Rule

Set enabled: false in rule:

%{
  ...
  enabled: false
}

Restart application.

System Maintenance

Database Maintenance

Check Database Size:

Use your database management tools to monitor CDR storage size:

• MySQL/MariaDB: Query information_schema.tables for database size
• PostgreSQL: Use pg_database_size() function or \l+ command in psql

Cleanup Old CDR Records:

CDR records should be archived and purged periodically based on your retention policy:

• Configure automatic archiving based on business requirements (typically 30-90 days in operational database)
• Archive older records to data warehouse or cold storage
• Delete archived records from operational database in batches to avoid lock contention

Optimize Tables:

Periodically optimize database tables to maintain performance:

• MySQL/MariaDB: Run OPTIMIZE TABLE command during low-traffic periods
• PostgreSQL: Run VACUUM ANALYZE regularly (or enable autovacuum)

Run Weekly during low-traffic period to maintain optimal performance.

Mnesia Database Maintenance

Check Mnesia Table Size:

# In IEx console
:mnesia.table_info(:sms_route, :size)
:mnesia.table_info(:translation_rule, :size)

Backup Mnesia Tables:

# Export routes (Web UI)
# Navigate to /sms_routing
# Click "Export Routes"

# Or via Mnesia backup
:mnesia.backup("/var/backups/sms_c/mnesia_backup.bup")

Restore Mnesia:

# Via Web UI import
# Or restore backup:
:mnesia.restore("/var/backups/sms_c/mnesia_backup.bup", [])

Log Rotation

Configure logrotate for application logs:

# /etc/logrotate.d/sms_c
/var/log/sms_c/*.log {
    daily
    rotate 30
    compress
    delaycompress
    notifempty
    create 0644 sms_user sms_group
    sharedscripts
    postrotate
        systemctl reload sms_c || true
    endscript
}

Restart Application

Graceful Restart (zero downtime in cluster):

# Restart one node at a time
systemctl restart sms_c

# Wait for node to join cluster
# Repeat for each node

Emergency Restart (all nodes):

systemctl restart sms_c

After Restart:

• Verify all frontends reconnect
• Check Prometheus for metric continuity
• Monitor logs for errors
• Verify message processing resumes

Backup and Recovery

What to Backup

1. Configuration Files:

   • config/runtime.exs
   • config/config.exs
   • config/prod.exs (if exists)

2. Routing Tables (Mnesia):

   • Export via Web UI
   • Or Mnesia backup command

3. SQL CDR Database:

   • Daily full backup
   • Transaction log backups (continuous)

4. TLS Certificates:

   • priv/cert/*.crt
   • priv/cert/*.key

Backup Procedures

Daily Configuration Backup:

#!/bin/bash
# /opt/sms_c/scripts/backup_config.sh

BACKUP_DIR="/var/backups/sms_c/$(date +%Y%m%d)"
mkdir -p $BACKUP_DIR

# Backup configuration
cp -r /opt/sms_c/config $BACKUP_DIR/

# Backup certificates
cp -r /opt/sms_c/priv/cert $BACKUP_DIR/

# Set permissions
chmod 600 $BACKUP_DIR/cert/*

echo "Configuration backup completed: $BACKUP_DIR"

Database Backup:

#!/bin/bash
# /opt/sms_c/scripts/backup_database.sh

BACKUP_DIR="/var/backups/sms_c/database"
DATE=$(date +%Y%m%d_%H%M%S)

mkdir -p $BACKUP_DIR

# Backup SQL CDR database
# MySQL/MariaDB: Use mysqldump with --single-transaction for consistency
# PostgreSQL: Use pg_dump -F c for custom format

# Example structure (adapt to your database):
# - Use appropriate backup tool (mysqldump, pg_dump)
# - Enable transaction-safe backups for consistency
# - Compress output to save space
# - Configure retention period (e.g., 30 days)

# Remove old backups
find $BACKUP_DIR -name "sms_c_*.gz" -mtime +30 -delete

echo "Database backup completed: sms_c_${DATE}"

Routing Table Backup:

#!/bin/bash
# /opt/sms_c/scripts/backup_routes.sh

BACKUP_DIR="/var/backups/sms_c/routes"
DATE=$(date +%Y%m%d)

mkdir -p $BACKUP_DIR

# Export via API
curl https://api.example.com:8443/api/routes/export \
  > $BACKUP_DIR/routes_${DATE}.json

echo "Routes backup completed: routes_${DATE}.json"

Schedule Backups (crontab):

# Daily at 2 AM
0 2 * * * /opt/sms_c/scripts/backup_config.sh
0 2 * * * /opt/sms_c/scripts/backup_database.sh
0 2 * * * /opt/sms_c/scripts/backup_routes.sh

Recovery Procedures

Restore Configuration:

# Stop application
systemctl stop sms_c

# Restore config files
cp -r /var/backups/sms_c/20251030/config/* /opt/sms_c/config/

# Restore certificates
cp -r /var/backups/sms_c/20251030/cert/* /opt/sms_c/priv/cert/

# Start application
systemctl start sms_c

Restore SQL CDR Database:

Use appropriate restore tools for your database:

• MySQL/MariaDB: Decompress and pipe to mysql client
• PostgreSQL: Use pg_restore with custom format dumps

Important: Stop the SMS-C application before restoring database to prevent data conflicts.

Restore Routing Tables:

1. Navigate to Web UI /sms_routing
2. Click "Import Routes"
3. Select backup JSON file
4. Choose "Replace" mode
5. Confirm import

Capacity Planning

Monitor Growth Trends

Message Volume Trend:

Prometheus query (30-day average):

avg_over_time(sms_c_message_received_count[30d])

Database Growth Rate:

-- Monthly data growth
SELECT
  DATE_FORMAT(inserted_at, '%Y-%m') AS month,
  COUNT(*) AS message_count,
  ROUND(SUM(LENGTH(message_body)) / 1024 / 1024, 2) AS data_mb
FROM message_queues
GROUP BY month
ORDER BY month DESC
LIMIT 12;

Capacity Indicators

CPU Usage:

• Normal: < 50% average
• High: > 70% sustained
• Critical: > 90%

Memory Usage:

• Normal: < 70% of available
• High: > 80%
• Critical: > 90%

Disk Usage:

• Normal: < 60% full
• High: > 75%
• Critical: > 85%

Queue Depth:

• Normal: < 1000 pending
• High: > 5000 pending
• Critical: > 10,000 pending

Scaling Recommendations

When to Scale Vertically (Upgrade Resources):

• CPU consistently > 70%
• Memory consistently > 80%
• Single-node bottleneck

When to Scale Horizontally (Add Nodes):

• CPU > 50% on all nodes
• Message volume > 5,000 msg/sec
• Geographic distribution needed
• High availability required

Database Scaling:

• Read replicas for reporting queries
• Connection pooling optimization
• Index optimization
• Partition large tables by date

Incident Response

Severity Levels

Critical (Immediate Response):

• No messages being delivered
• All frontends disconnected
• Database unavailable
• API completely down

High (Response within 1 hour):

• Delivery success rate < 80%
• Multiple frontends disconnected
• Routing failures > 10%
• Queue backlog growing

Medium (Response within 4 hours):

• Single frontend disconnected
• Delivery success rate 80-95%
• Slow message processing
• ENUM lookups failing

Low (Response within 24 hours):

• Minor performance degradation
• Single route issue
• Non-critical warning alerts

Incident Checklist

1. Assess Severity:

• Check Prometheus alerts
• Review dashboard metrics
• Check message queue status
• Verify frontend connections

2. Gather Information:

• Recent configuration changes?
• Recent deployments?
• External dependencies status (OCS, DNS)?
• Error messages in logs?

3. Immediate Actions:

• Stop ongoing changes
• Roll back recent deployments if suspected cause
• Enable verbose logging if needed
• Notify stakeholders

4. Investigation:

• Review application logs
• Check system resource usage
• Examine database performance
• Test external dependencies

5. Resolution:

• Apply fix
• Test in simulator
• Deploy to production
• Monitor for improvement

6. Post-Incident:

• Document root cause
• Update monitoring/alerts
• Implement preventive measures
• Update runbooks

Common Incidents

High Queue Backlog:

1. Check delivery success rate
2. Verify frontends are connected and polling
3. Check database performance
4. Review Prometheus for bottlenecks
5. Consider increasing batch size/interval

Routing Failures:

1. Review routing configuration
2. Test in routing simulator
3. Check for missing routes
4. Verify catch-all route exists
5. Check event logs for failure reasons

Frontend Disconnections:

1. Check frontend system status
2. Verify network connectivity
3. Review frontend logs
4. Test manual API registration
5. Check firewall rules

Slow Message Processing:

1. Check database query performance
2. Review batch worker configuration
3. Verify adequate resources (CPU/Memory)
4. Check for ENUM lookup delays
5. Review charging system performance

For detailed troubleshooting procedures, see the Troubleshooting Guide.