Skip to main content

SMS-C Prometheus Metrics Documentation

← Back to Documentation Index | Main README

Overview​

This document describes all Prometheus metrics exposed by the SMS-C system. These metrics are designed for operations staff to monitor system health, performance, and troubleshoot issues.

Accessing Metrics​

The Prometheus metrics endpoint is available at:

http://localhost:9568/metrics

This endpoint exposes metrics in Prometheus text format that can be scraped by a Prometheus server. The metrics are updated in real-time as the system processes messages.

Metric Naming Convention​

All metrics follow the pattern: sms_c.<category>.<metric_name>.<type>

Categories:

  • license - License status metrics
  • message - Message processing metrics
  • routing - Routing decision metrics
  • enum - ENUM/NAPTR lookup metrics
  • delivery - Message delivery metrics
  • queue - Queue management metrics
  • charging - Billing/charging metrics
  • mnesia - Database metrics
  • frontend - Frontend connection metrics
  • location - Location/registration metrics
  • phoenix.endpoint - HTTP API request metrics
  • vm - Erlang VM system metrics

License Metrics​

sms_c_license_status​

Type: Gauge

Description: Current license status of the OmniMessage SMS-C system.

Values:

  • 1 - Valid license
  • 0 - Invalid/expired license

Labels: None

Product Name: omnimessage

Use Case: Monitor license validity to ensure system is operating with a valid license. When invalid, messages are still received but routed to destination "NOLICENCE" instead of normal routing.

Behavior When License Invalid:

  • Inbound messages are accepted and stored
  • Message destination (dest_smsc) is automatically set to "NOLICENCE"
  • Normal routing is bypassed
  • UI and monitoring remain accessible
  • Database and all services remain operational

Alerting:

- alert: SMS_C_License_Invalid
  expr: sms_c_license_status == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "SMS-C license invalid or expired"
    description: "License status is invalid - messages being routed to NOLICENCE"

Example Prometheus Queries:

# Check if license is valid
sms_c_license_status == 1

# Alert on invalid license
sms_c_license_status == 0

# Count messages routed to NOLICENCE (indicates license issue)
sms_c_routing_route_matched_count{dest_smsc="NOLICENCE"}

Message Processing Metrics​

sms_c_message_received_count​

Type: Counter

Description: Total number of messages received by the SMS-C from all sources.

Labels:

  • source_smsc: Name of the source SMSC that sent the message
  • source_type: Type of source connection (ims, circuit_switched, smpp)
  • message_type: Type of message (sms, mms)

Use Case: Monitor incoming message volume by source and type. Use to detect traffic patterns, identify busy periods, and spot anomalies in message flow.

Alerting: Set alerts for sudden drops (potential source connectivity issues) or spikes (potential attack/spam).

sms_c_message_validated_count​

Type: Counter

Description: Total number of message validations performed.

Labels:

  • valid: Whether validation passed (true or false)

Use Case: Track validation success/failure rates. High failure rates may indicate malformed messages or integration issues.

Alerting: Alert when validation failure rate exceeds threshold (e.g., > 5% failures).

sms_c_message_processing_stop_duration​

Type: Histogram

Description: Time taken to process a message from receipt to completion (includes validation, routing, and queueing).

Unit: Milliseconds

Buckets: 10, 50, 100, 250, 500, 1000, 2500, 5000 ms

Labels:

  • success: Whether processing succeeded (true or false)

Use Case: Monitor end-to-end message processing performance. Identify slowdowns in the processing pipeline.

Alerting: Alert when p95 or p99 latency exceeds SLA thresholds.

Routing Metrics​

sms_c_routing_route_matched_count​

Type: Counter

Description: Total number of times a specific route was matched and selected for message routing.

Labels:

  • route_id: Unique identifier of the matched route
  • dest_smsc: Destination SMSC selected by the route
  • priority: Priority value of the matched route

Use Case: Understand which routes are being used most frequently. Identify underutilized or overloaded routes. Useful for capacity planning and route optimization.

Alerting: Alert if high-priority routes are rarely matched (may indicate routing misconfiguration).

sms_c_routing_failed_count​

Type: Counter

Description: Total number of routing failures where no suitable route could be found.

Labels:

  • reason: Failure reason (no_route_found, validation_failed, etc.)

Use Case: Track routing failures to identify configuration gaps or unexpected traffic patterns.

Alerting: Alert on any routing failures as they indicate messages cannot be delivered.

sms_c_routing_action_count​

Type: Counter

Description: Total number of special routing actions taken.

Labels:

  • action: Type of action (drop, auto_reply, forward)
  • route_id: Route that triggered the action

Use Case: Monitor drop rules (anti-spam), auto-reply usage, and forwarding patterns.

Alerting: Alert on unexpected spikes in drop actions (may indicate spam attack).

sms_c_routing_stop_duration​

Type: Histogram

Description: Time taken to evaluate all routes and select the best match.

Unit: Milliseconds

Buckets: 1, 5, 10, 25, 50, 100, 250, 500 ms

Labels:

  • dest_smsc: Selected destination SMSC

Use Case: Monitor routing engine performance. Slow routing indicates too many routes or complex matching logic.

Alerting: Alert when routing takes consistently longer than expected (e.g., p95 > 50ms).

ENUM/NAPTR Lookup Metrics​

sms_c_enum_cache_hit_count​

Type: Counter

Description: Total number of ENUM lookups served from cache (did not require DNS query).

Labels:

  • domain: ENUM domain queried

Use Case: Monitor cache effectiveness. High cache hit rates reduce DNS load and improve performance.

Alerting: Alert if cache hit rate drops below threshold (may indicate cache issues or unusual traffic).

sms_c_enum_cache_miss_count​

Type: Counter

Description: Total number of ENUM lookups that required a DNS query (not in cache).

Labels:

  • domain: ENUM domain queried

Use Case: Track cache misses to understand cache effectiveness. Use with hit count to calculate hit rate.

Calculation: cache_hit_rate = hits / (hits + misses)

sms_c_enum_cache_size_size​

Type: Gauge

Description: Current number of entries in the ENUM cache.

Use Case: Monitor cache size to ensure it's not growing unbounded. Help tune cache TTL settings.

Alerting: Alert if cache size exceeds expected bounds (may indicate memory leak).

sms_c_enum_lookup_stop_duration​

Type: Histogram

Description: Time taken to complete an ENUM lookup (including DNS query if not cached).

Unit: Milliseconds

Buckets: 10, 50, 100, 250, 500, 1000, 2500, 5000 ms

Labels:

  • domain: ENUM domain queried
  • success: Whether lookup succeeded (true or false)
  • cache_hit: Whether result was served from cache (true or false)

Use Case: Monitor ENUM lookup performance. Identify slow DNS servers or network issues.

Alerting: Alert when p95 lookup time exceeds timeout threshold.

sms_c_enum_naptr_records_record_count​

Type: Histogram

Description: Number of NAPTR records returned by a successful ENUM lookup.

Buckets: 0, 1, 2, 3, 5, 10

Labels:

  • domain: ENUM domain queried

Use Case: Understand ENUM record distribution. Most lookups should return 1-3 records.

Alerting: Alert if frequently returning 0 records (DNS configuration issue).

Delivery Metrics​

sms_c_delivery_queued_count​

Type: Counter

Description: Total number of messages queued for delivery to a destination SMSC.

Labels:

  • dest_smsc: Destination SMSC name

Use Case: Monitor message flow to each destination. Useful for capacity planning.

Alerting: Compare with delivery success/failure counts to detect accumulation.

sms_c_delivery_attempted_count​

Type: Counter

Description: Total number of delivery attempts made (includes retries).

Labels:

  • dest_smsc: Destination SMSC name

Use Case: Track delivery attempt volume. High attempt count relative to queued count indicates retry behavior.

sms_c_delivery_succeeded_count​

Type: Counter

Description: Total number of messages successfully delivered to destination SMSC.

Labels:

  • dest_smsc: Destination SMSC name

Use Case: Track successful deliveries per destination. Primary success metric.

Alerting: Alert if success rate drops below SLA threshold.

Calculation: success_rate = succeeded / queued

sms_c_delivery_failed_count​

Type: Counter

Description: Total number of messages that failed delivery after all retry attempts.

Labels:

  • dest_smsc: Destination SMSC name
  • reason: Failure reason

Use Case: Track delivery failures to identify problematic destinations or failure patterns.

Alerting: Alert on elevated failure rates or specific failure reasons.

sms_c_delivery_dead_letter_count​

Type: Counter

Description: Total number of messages moved to dead letter queue (undeliverable).

Labels:

  • reason: Reason for dead letter (e.g., max_retries_exceeded, expired)

Use Case: Monitor undeliverable messages requiring manual intervention.

Alerting: Alert on any dead letter events as they represent complete delivery failure.

sms_c_delivery_succeeded_attempt_count​

Type: Histogram

Description: Number of delivery attempts required before successful delivery.

Buckets: 1, 2, 3, 5, 10

Labels:

  • dest_smsc: Destination SMSC name

Use Case: Understand retry behavior. Most deliveries should succeed on first attempt.

Alerting: Alert if average attempt count exceeds 2 (indicates destination reliability issues).

sms_c_delivery_failed_attempt_count​

Type: Histogram

Description: Number of delivery attempts made before final failure.

Buckets: 1, 2, 3, 5, 10

Labels:

  • dest_smsc: Destination SMSC name

Use Case: Understand how many retries occur before giving up.

sms_c_delivery_time_delta_delta_ms​

Type: Histogram

Description: Time from message submission to delivery confirmation. This metric captures the complete end-to-end latency from when a message enters the SMS-C to when delivery is confirmed, with detailed labels for analysis by source, destination, and retry behavior.

Unit: Milliseconds

Buckets: 50, 100, 250, 500, 1000, 2000, 5000, 10000, 20000, 60000, 600000 ms (50ms to 10 minutes)

Labels:

  • source_smsc: Source SMSC that submitted the message
  • dest_smsc: Destination SMSC that delivered the message
  • delivery_attempts: Number of delivery attempts required (0 = first attempt success)

Use Case: Analyze end-to-end delivery performance by source-destination pair. Identify slow paths, problematic SMSC combinations, and correlate latency with retry behavior. Essential for SLA monitoring and capacity planning.

Alerting:

  • Alert when p95 exceeds SLA threshold for specific source/dest combinations
  • Alert when messages consistently require multiple attempts

Example Prometheus Queries:

# P95 delivery time by source and destination SMSC
histogram_quantile(0.95,
  sum by (source_smsc, dest_smsc, le) (
    rate(sms_c_delivery_time_delta_delta_ms_bucket[5m])
  )
)

# P99 delivery time overall
histogram_quantile(0.99,
  sum by (le) (
    rate(sms_c_delivery_time_delta_delta_ms_bucket[5m])
  )
)

# Average delivery time by source SMSC
sum by (source_smsc) (rate(sms_c_delivery_time_delta_delta_ms_sum[5m])) /
sum by (source_smsc) (rate(sms_c_delivery_time_delta_delta_ms_count[5m]))

# Delivery time for messages requiring retries vs first-attempt success
histogram_quantile(0.95,
  sum by (le) (rate(sms_c_delivery_time_delta_delta_ms_bucket{delivery_attempts="0"}[5m]))
)
histogram_quantile(0.95,
  sum by (le) (rate(sms_c_delivery_time_delta_delta_ms_bucket{delivery_attempts!="0"}[5m]))
)

# Slowest source-destination pairs (by p95)
topk(10,
  histogram_quantile(0.95,
    sum by (source_smsc, dest_smsc, le) (
      rate(sms_c_delivery_time_delta_delta_ms_bucket[1h])
    )
  )
)

# Percentage of messages delivered within 2 seconds
sum(rate(sms_c_delivery_time_delta_delta_ms_bucket{le="2000"}[5m])) /
sum(rate(sms_c_delivery_time_delta_delta_ms_count[5m])) * 100

# Messages delivered within 2 seconds by destination
sum by (dest_smsc) (rate(sms_c_delivery_time_delta_delta_ms_bucket{le="2000"}[5m])) /
sum by (dest_smsc) (rate(sms_c_delivery_time_delta_delta_ms_count[5m])) * 100

sms_c_delivery_time_by_smsc_delta_ms​

Type: Histogram

Description: Time from message submission to delivery confirmation, labelled by source and destination SMSC only. This is a lower-cardinality companion to sms_c_delivery_time_delta_delta_ms — it omits the delivery_attempts label so average delivery time per source/destination pair can be read directly without aggregating across attempt counts. Emitted from the same [:sms_c, :delivery, :time_delta] event.

Unit: Milliseconds

Buckets: 50, 100, 250, 500, 1000, 2000, 5000, 10000, 20000, 60000, 600000 ms (50ms to 10 minutes)

Labels:

  • source_smsc: Source SMSC that submitted the message
  • dest_smsc: Destination SMSC that delivered the message

Use Case: Dashboard the average end-to-end delivery time per source/destination pair. Prefer this metric over sms_c_delivery_time_delta_delta_ms for average/SLA panels where the attempt-count breakdown is not needed; use the _delta variant when you need to slice by retry behaviour.

Example Prometheus Queries:

# Average delivery time (ms) by source/destination SMSC
sum by (source_smsc, dest_smsc) (rate(sms_c_delivery_time_by_smsc_delta_ms_sum[5m])) /
sum by (source_smsc, dest_smsc) (rate(sms_c_delivery_time_by_smsc_delta_ms_count[5m]))

# Overall average delivery time
sum(rate(sms_c_delivery_time_by_smsc_delta_ms_sum[5m])) /
sum(rate(sms_c_delivery_time_by_smsc_delta_ms_count[5m]))

# P95 delivery time by source/destination SMSC
histogram_quantile(0.95,
  sum by (source_smsc, dest_smsc, le) (
    rate(sms_c_delivery_time_by_smsc_delta_ms_bucket[5m])
  )
)

Queue Metrics​

sms_c_queue_size_size​

Type: Gauge

Description: Current total number of messages in queue (all states combined).

Labels:

  • queue_type: Type of queue (message_queue, dead_letter)

Use Case: Monitor queue depth to detect backlogs or processing issues.

Alerting: Alert when queue size exceeds capacity thresholds.

sms_c_queue_size_pending​

Type: Gauge

Description: Current number of messages pending delivery (not yet attempted).

Labels:

  • queue_type: Type of queue

Use Case: Monitor pending message count. High pending counts indicate processing delays.

Alerting: Alert when pending count exceeds threshold for extended period.

sms_c_queue_size_failed​

Type: Gauge

Description: Current number of messages in failed state (awaiting retry).

Labels:

  • queue_type: Type of queue

Use Case: Monitor failed message accumulation. Indicates delivery issues.

Alerting: Alert on elevated failed count as it impacts delivery rates.

sms_c_queue_size_delivered​

Type: Gauge

Description: Current number of delivered messages awaiting cleanup/removal from queue.

Labels:

  • queue_type: Type of queue

Use Case: Monitor cleanup lag. High counts indicate cleanup process is falling behind.

Alerting: Alert if delivered messages accumulate significantly.

sms_c_queue_oldest_message_age_seconds​

Type: Gauge

Description: Age (in seconds) of the oldest message currently in pending state.

Labels:

  • queue_type: Type of queue

Use Case: Detect message aging and processing stalls. Critical for SLA monitoring.

Alerting: Alert when oldest message age exceeds SLA threshold (e.g., > 300 seconds).

Charging Metrics​

Charging is performed over the Diameter Ro interface (Credit-Control). The account label carries the charged subscriber's MSISDN (the message sender). A charging_failed event is emitted both for a definitive out-of-balance decline (reason: :no_credit) and for an OCS error (transport/protocol). Only a :no_credit decline rejects a message (status :balance_rejected); OCS errors fail open and the message is delivered.

sms_c_charging_requested_count​

Type: Counter

Description: Total number of charging requests (Diameter Ro CCR Event-Requests) sent to the OCS.

Labels:

  • account: Charged subscriber MSISDN (message sender)

Use Case: Track charging volume per account. Useful for billing reconciliation.

sms_c_charging_succeeded_count​

Type: Counter

Description: Total number of successful charging operations.

Labels:

  • account: Account identifier charged

Use Case: Monitor charging success rate per account.

Calculation: success_rate = succeeded / requested

sms_c_charging_failed_count​

Type: Counter

Description: Total number of failed charging operations.

Labels:

  • account: Account identifier
  • reason: Failure reason

Use Case: Identify charging failures that may impact revenue or require account intervention.

Alerting: Alert on elevated charging failure rates.

sms_c_charging_succeeded_duration​

Type: Histogram

Description: Time taken to complete a successful charging request.

Unit: Milliseconds

Buckets: 10, 50, 100, 250, 500, 1000, 2500, 5000 ms

Labels:

  • account: Account identifier

Use Case: Monitor billing system performance. Slow charging can delay message delivery.

Alerting: Alert when p95 charging time exceeds threshold.

System Health Metrics​

sms_c_mnesia_table_size_record_count​

Type: Gauge

Description: Current number of records in each Mnesia database table. Polled every 10 seconds.

Labels:

  • table: Table name

Tracked Tables:

  • sms_route - SMS routing rules
  • message_store - Message queue (pending, delivered, failed messages)
  • location_store - Subscriber location/registration data
  • frontend_store - Frontend/SMSC registrations
  • translation_rule - Number translation rules
  • cell_tower_store - Cell tower location data
  • message_events - Message event logs

Use Case: Monitor database growth. Detect unexpected data accumulation. Capacity planning.

Alerting: Alert on unexpected table growth rates or when approaching capacity limits.

Example Prometheus Queries:

# All table sizes
sms_c_mnesia_table_size_record_count

# Message queue size
sms_c_mnesia_table_size_record_count{table="message_store"}

# Active subscriber registrations
sms_c_mnesia_table_size_record_count{table="location_store"}

# Routing rules count
sms_c_mnesia_table_size_record_count{table="sms_route"}

sms_c_frontend_status_count​

Type: Gauge

Description: Number of frontends in each connection status.

Labels:

  • frontend_name: Frontend identifier
  • status: Connection status (connected, disconnected)

Use Case: Monitor frontend connectivity. Detect connection failures.

Alerting: Alert when expected frontends disconnect.

sms_c_location_registered_count​

Type: Counter

Description: Total number of location/subscriber registrations received by the system.

Labels:

  • location: Frontend/SMSC name where subscriber is registered
  • ims_capable: Whether the subscriber supports IMS (true/false)

Use Case: Monitor subscriber registration activity. Track IMS vs non-IMS subscribers. Detect registration storms or failures.

Alerting: Set alerts for:

  • Registration rate drops (may indicate network issues)
  • Unusual spikes in registrations
  • High ratio of non-IMS registrations (legacy device influx)

Example Query:

# Registration rate per minute
rate(sms_c_location_registered_count[1m])

# IMS vs non-IMS registration ratio
sum(rate(sms_c_location_registered_count{ims_capable="true"}[5m])) /
sum(rate(sms_c_location_registered_count[5m]))

sms_c_location_active_registrations_count​

Type: Gauge

Description: Current number of active subscriber registrations, grouped by location (frontend/SMSC) and IMS capability. This metric is polled every 10 seconds and reflects the current state of the location store.

Labels:

  • location: Frontend/SMSC name where subscribers are registered
  • ims_capable: Whether the registrations are IMS capable (true/false)

Use Case: Monitor current subscriber distribution across frontends. Track IMS adoption. Identify load imbalances between frontends. Capacity planning for each SMSC instance.

Alerting: Set alerts for:

  • Sudden drops in registrations for a location (may indicate frontend issues)
  • Imbalanced distribution across frontends
  • Total registrations approaching capacity limits

Example Prometheus Queries:

# Total active registrations
sum(sms_c_location_active_registrations_count)

# Active registrations by location
sum by (location) (sms_c_location_active_registrations_count)

# Active IMS-capable registrations by location
sum by (location) (sms_c_location_active_registrations_count{ims_capable="true"})

# IMS adoption rate per location
sum by (location) (sms_c_location_active_registrations_count{ims_capable="true"}) /
sum by (location) (sms_c_location_active_registrations_count) * 100

# Total IMS vs non-IMS registrations
sum by (ims_capable) (sms_c_location_active_registrations_count)

# Locations with most registrations
topk(10, sum by (location) (sms_c_location_active_registrations_count))

# Registration distribution as percentage of total
sum by (location) (sms_c_location_active_registrations_count) /
sum(sms_c_location_active_registrations_count) * 100

HTTP API Request Metrics​

phoenix_endpoint_stop_duration​

Type: Distribution (Histogram)

Description: HTTP request processing duration in milliseconds, from request start to response completion.

Labels:

  • route: API endpoint route (e.g., /api/messages, /api/frontends)

Buckets: 10ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s

Use Case: Monitor API performance. Identify slow endpoints. Track response time SLAs.

Alerting: Set alerts for:

  • P95 latency > 500ms for critical endpoints
  • P99 latency > 1s for any endpoint
  • Increasing latency trends

Example Query:

# P95 response time by endpoint
histogram_quantile(0.95,
  rate(phoenix_endpoint_stop_duration_bucket[5m]))

# Requests slower than 1 second
sum(rate(phoenix_endpoint_stop_duration_bucket{le="1000"}[5m]))

phoenix_endpoint_stop_count​

Type: Counter

Description: Total number of HTTP requests completed, categorized by route and HTTP status code.

Labels:

  • route: API endpoint route
  • status: HTTP status code (200, 201, 400, 404, 500, etc.)

Use Case: Monitor API request volume and success rates. Track error rates by endpoint.

Alerting: Set alerts for:

  • Error rate > 5% for any endpoint
  • 5xx errors on critical endpoints
  • Sudden drops in request volume

Example Query:

# Request rate per endpoint
sum by (route) (rate(phoenix_endpoint_stop_count[5m]))

# Error rate by endpoint
sum by (route) (rate(phoenix_endpoint_stop_count{status=~"5.."}[5m])) /
sum by (route) (rate(phoenix_endpoint_stop_count[5m]))

# Success rate
sum(rate(phoenix_endpoint_stop_count{status=~"2.."}[5m])) /
sum(rate(phoenix_endpoint_stop_count[5m]))

phoenix_router_dispatch_exception_count​

Type: Counter

Description: Total number of exceptions/errors raised during HTTP request processing.

Labels:

  • route: API endpoint route where exception occurred
  • kind: Type of exception (error, exit, throw)

Use Case: Track application errors. Identify problematic endpoints. Monitor system stability.

Alerting: Set alerts for any non-zero value on critical endpoints.

Example Query:

# Exception rate by endpoint
rate(phoenix_router_dispatch_exception_count[5m])

# Total exceptions in last hour
increase(phoenix_router_dispatch_exception_count[1h])

Erlang VM Metrics​

vm_memory_total​

Type: Gauge

Description: Total memory allocated by the Erlang VM in bytes.

Use Case: Monitor overall memory usage. Detect memory leaks. Plan capacity.

Alerting: Alert when memory usage > 80% of available system memory.

vm_memory_processes​

Type: Gauge

Description: Memory used by Erlang processes in bytes.

Use Case: Track process memory consumption. Most common source of memory growth.

Alerting: Alert on sustained high growth rate.

vm_total_run_queue_lengths_total​

Type: Gauge

Description: Total number of processes waiting to be scheduled across all CPU schedulers.

Use Case: Measure system load. High values indicate CPU saturation.

Alerting: Alert when consistently > 10 * number of CPU cores.

vm_system_counts_process_count​

Type: Gauge

Description: Current number of processes running in the VM.

Use Case: Monitor process creation patterns. Detect process leaks.

Alerting: Alert when approaching process limit (default 262,144).

Metric Collection and Polling​

The system automatically collects the following metrics every 10 seconds:

  • Queue sizes and ages
  • Mnesia table sizes
  • ENUM cache statistics

All other metrics are event-driven and emitted when the corresponding action occurs.

Common Monitoring Patterns​

Delivery Success Rate by Destination​

Track the success rate of message delivery for each destination SMSC:

Formula: (sms_c_delivery_succeeded_count) / (sms_c_delivery_queued_count)

Interpretation: Should be > 95% for healthy destinations. Lower rates indicate delivery issues.

End-to-End Message Latency​

Monitor total time from message receipt to delivery:

Metrics:

  • sms_c_message_processing_stop_duration (processing)
  • sms_c_delivery_time_delta_delta_ms (delivery)

Interpretation: Sum represents total user-facing latency.

ENUM Cache Effectiveness​

Measure how well the ENUM cache is performing:

Formula: (sms_c_enum_cache_hit_count) / (sms_c_enum_cache_hit_count + sms_c_enum_cache_miss_count)

Interpretation: Should be > 80% after warm-up. Lower rates may indicate short TTL or high traffic variance.

Route Utilization​

Identify which routes handle the most traffic:

Metric: sms_c_routing_route_matched_count grouped by route_id

Interpretation: Use to identify hot routes for optimization and capacity planning.

Queue Backlog Trend​

Monitor if message queue is growing (backlog) or shrinking (catching up):

Metrics:

  • sms_c_queue_size_pending (current pending)
  • sms_c_queue_oldest_message_age_seconds (age trending)

Interpretation: Growing pending count + increasing age = backlog forming.

Retry Rate​

Understand how often delivery retries are required:

Metric: sms_c_delivery_succeeded_attempt_count histogram percentiles

Interpretation: If p95 > 1, most messages require retries. Indicates destination reliability issues.

AlertConditionSeverityDescription
High Routing Failure Raterouting_failed_count increaseCriticalMessages cannot be routed
Queue Backlogqueue_size_pending > thresholdWarningMessages accumulating
Old Messages in Queuequeue_oldest_message_age_seconds > 300CriticalSLA violation
Delivery Failure Spikedelivery_failed_count spikeHighDestination issues
Dead Letter Eventsdelivery_dead_letter_count > 0HighUndeliverable messages
ENUM Lookup Timeoutsenum_lookup_stop_duration p95 > 5000msWarningDNS issues
Low Cache Hit RateENUM cache hit rate < 0.7WarningCache ineffective
Frontend Disconnectedfrontend_status_count{status="disconnected"} > 0HighConnectivity loss
Charging Failurescharging_failed_count > thresholdHighBilling issues
Slow Message Processingmessage_processing_stop_duration p95 > 1000msWarningPerformance degradation

Dashboard Recommendations​

Operations Dashboard​

Purpose: Real-time system health monitoring

Panels:

  1. Message throughput (received/processed/delivered per minute)
  2. Queue sizes (pending, failed, delivered)
  3. Delivery success rate by destination
  4. p95 processing and delivery latency
  5. Active frontends status
  6. Current alerts

Performance Dashboard​

Purpose: System performance analysis

Panels:

  1. Message processing duration histogram
  2. Routing duration histogram
  3. ENUM lookup duration histogram
  4. Charging duration histogram
  5. Delivery attempts distribution
  6. Cache hit rates

Business Dashboard​

Purpose: Traffic and usage analysis

Panels:

  1. Messages by source SMSC
  2. Messages by destination SMSC
  3. Route utilization heatmap
  4. Auto-reply and drop action counts
  5. ENUM usage statistics
  6. Charging volume by account

Metric Retention​

Recommended Prometheus retention settings:

  • Raw metrics: 15 days
  • 5-minute aggregates: 90 days
  • 1-hour aggregates: 2 years

This provides detailed recent history while maintaining long-term trends for capacity planning.

Troubleshooting with Metrics​

Scenario: Messages Not Being Delivered​

Investigation Steps:

  1. Check sms_c_message_received_count - Are messages being received?
  2. Check sms_c_routing_failed_count - Are they being routed?
  3. Check sms_c_delivery_queued_count - Are they being queued?
  4. Check sms_c_delivery_failed_count - Are delivery attempts failing?
  5. Check dest_smsc labels to identify problematic destination

Scenario: Slow Message Processing​

Investigation Steps:

  1. Check sms_c_message_processing_stop_duration histogram - Overall processing time
  2. Check sms_c_routing_stop_duration - Is routing slow?
  3. Check sms_c_enum_lookup_stop_duration - Are ENUM lookups slow?
  4. Check sms_c_charging_succeeded_duration - Is charging slow?
  5. Identify bottleneck and investigate specific component

Scenario: Growing Message Queue​

Investigation Steps:

  1. Check sms_c_queue_size_pending trend - Is it growing?
  2. Check sms_c_delivery_attempted_count - Are delivery attempts happening?
  3. Check sms_c_delivery_failed_count - Are they failing?
  4. Check sms_c_delivery_time_delta_delta_ms - Is delivery taking too long?
  5. Check dest_smsc labels to identify slow destinations

Prometheus Query Examples​

Message Throughput​

Messages Received Per Second (5-minute average):

rate(sms_c_message_received_count[5m])

Messages Received Per Minute (1-hour average):

rate(sms_c_message_received_count[1h]) * 60

Total Messages Today:

increase(sms_c_message_received_count[24h])

Messages by Source Type:

sum by (source_type) (rate(sms_c_message_received_count[5m]))

Messages by Source SMSC:

sum by (source_smsc) (rate(sms_c_message_received_count[5m]))

Delivery Performance​

Delivery Success Rate (Percentage):

(rate(sms_c_delivery_succeeded_count[5m]) / rate(sms_c_delivery_queued_count[5m])) * 100

Delivery Failure Rate (Percentage):

(rate(sms_c_delivery_failed_count[5m]) / rate(sms_c_delivery_queued_count[5m])) * 100

Average Delivery Attempts (p95):

histogram_quantile(0.95, sms_c_delivery_succeeded_attempt_count_bucket)

Delivery Success by Destination:

sum by (dest_smsc) (rate(sms_c_delivery_succeeded_count[5m]))

Delivery Failure Reasons:

sum by (reason) (rate(sms_c_delivery_failed_count[5m]))

Time to Delivery (p95):

histogram_quantile(0.95, sms_c_delivery_time_delta_delta_ms_bucket)

Time to Delivery (p99):

histogram_quantile(0.99, sms_c_delivery_time_delta_delta_ms_bucket)

Queue Metrics​

Current Pending Messages:

sms_c_queue_size_pending

Failed Messages Awaiting Retry:

sms_c_queue_size_failed

Oldest Message Age (Minutes):

sms_c_queue_oldest_message_age_seconds / 60

Queue Growth Rate (Messages/Hour):

rate(sms_c_queue_size_size[1h]) * 3600

Messages Entering Queue:

rate(sms_c_delivery_queued_count[5m])

Messages Leaving Queue:

rate(sms_c_delivery_succeeded_count[5m]) + rate(sms_c_delivery_failed_count[5m])

Queue Backlog (Entering - Leaving):

rate(sms_c_delivery_queued_count[5m]) - (rate(sms_c_delivery_succeeded_count[5m]) + rate(sms_c_delivery_failed_count[5m]))

Routing Performance​

Routing Success Rate:

(1 - (rate(sms_c_routing_failed_count[5m]) / (rate(sms_c_routing_route_matched_count[5m]) + rate(sms_c_routing_failed_count[5m])))) * 100

Most Used Routes:

topk(10, sum by (route_id, dest_smsc) (rate(sms_c_routing_route_matched_count[1h])))

Routing Latency (p50, p95, p99):

histogram_quantile(0.50, sms_c_routing_stop_duration_bucket)
histogram_quantile(0.95, sms_c_routing_stop_duration_bucket)
histogram_quantile(0.99, sms_c_routing_stop_duration_bucket)

Routing Failures Per Minute:

rate(sms_c_routing_failed_count[5m]) * 60

Drop Actions Per Hour:

increase(sms_c_routing_action_count{action="drop"}[1h])

Auto-Reply Actions Per Hour:

increase(sms_c_routing_action_count{action="auto_reply"}[1h])

ENUM Performance​

ENUM Cache Hit Rate:

rate(sms_c_enum_cache_hit_count[5m]) / (rate(sms_c_enum_cache_hit_count[5m]) + rate(sms_c_enum_cache_miss_count[5m]))

ENUM Cache Hit Percentage:

(rate(sms_c_enum_cache_hit_count[5m]) / (rate(sms_c_enum_cache_hit_count[5m]) + rate(sms_c_enum_cache_miss_count[5m]))) * 100

ENUM Lookup Latency (p95):

histogram_quantile(0.95, sms_c_enum_lookup_stop_duration_bucket)

ENUM Lookups Per Second (Cached vs Uncached):

# Cached (fast)
rate(sms_c_enum_cache_hit_count[5m])

# Uncached (requires DNS query)
rate(sms_c_enum_cache_miss_count[5m])

Average NAPTR Records Returned:

rate(sms_c_enum_naptr_records_record_count_sum[5m]) / rate(sms_c_enum_naptr_records_record_count_count[5m])

ENUM Cache Size:

sms_c_enum_cache_size_size

Processing Performance​

Message Processing Latency (p95):

histogram_quantile(0.95, sms_c_message_processing_stop_duration_bucket)

Message Processing Latency (p99):

histogram_quantile(0.99, sms_c_message_processing_stop_duration_bucket)

Processing Failures:

rate(sms_c_message_processing_stop_duration_count{success="false"}[5m])

Validation Failure Rate:

rate(sms_c_message_validated_count{valid="false"}[5m]) / rate(sms_c_message_validated_count[5m])

Charging Metrics​

Charging Success Rate:

rate(sms_c_charging_succeeded_count[5m]) / rate(sms_c_charging_requested_count[5m])

Charging Failures Per Minute:

rate(sms_c_charging_failed_count[5m]) * 60

Charging Latency (p95):

histogram_quantile(0.95, sms_c_charging_succeeded_duration_bucket)

Charging Volume by Account:

sum by (account) (rate(sms_c_charging_requested_count[1h]))

Frontend Health​

Active Frontends:

sum(sms_c_frontend_status_count{status="connected"})

Disconnected Frontends:

sum(sms_c_frontend_status_count{status="disconnected"})

Frontends by Name:

sum by (frontend_name) (sms_c_frontend_status_count{status="connected"})

System Health​

Mnesia Table Sizes:

sms_c_mnesia_table_size_record_count

Route Count:

sms_c_mnesia_table_size_record_count{table="sms_route"}

Translation Rule Count:

sms_c_mnesia_table_size_record_count{table="translation_rule"}

Grafana Dashboard Examples​

Dashboard 1: Real-Time Operations​

Purpose: Monitor current system activity and health.

Panels:

  1. Message Throughput (Graph)

    • Query: rate(sms_c_message_received_count[5m])
    • Query: rate(sms_c_delivery_succeeded_count[5m])
    • Unit: messages/second
    • Legend: {{source_type}}

  2. Delivery Success Rate (Gauge)

    • Query: (rate(sms_c_delivery_succeeded_count[5m]) / rate(sms_c_delivery_queued_count[5m])) * 100
    • Unit: percent (0-100)
    • Thresholds:
      • Red: < 90
      • Yellow: 90-95
      • Green: > 95

  3. Queue Depth (Graph)

    • Query: sms_c_queue_size_pending
    • Query: sms_c_queue_size_failed
    • Unit: messages
    • Legend: {{queue_type}}

  4. Oldest Message Age (Stat)

    • Query: sms_c_queue_oldest_message_age_seconds / 60
    • Unit: minutes
    • Thresholds:
      • Green: < 5
      • Yellow: 5-10
      • Red: > 10

  5. Active Frontends (Stat)

    • Query: sum(sms_c_frontend_status_count{status="connected"})
    • Unit: count
    • Color: Blue

  6. Routing Failures (Graph)

    • Query: rate(sms_c_routing_failed_count[5m]) * 60
    • Unit: failures/minute
    • Alert threshold: > 0

Dashboard 2: Performance Analysis​

Purpose: Analyze system performance and identify bottlenecks.

Panels:

  1. End-to-End Latency (Graph)

    • Query: histogram_quantile(0.50, sms_c_message_processing_stop_duration_bucket) (p50)
    • Query: histogram_quantile(0.95, sms_c_message_processing_stop_duration_bucket) (p95)
    • Query: histogram_quantile(0.99, sms_c_message_processing_stop_duration_bucket) (p99)
    • Unit: milliseconds
    • Legend: Percentile

  2. Component Latencies (Bar Gauge)

    • Routing: histogram_quantile(0.95, sms_c_routing_stop_duration_bucket)
    • ENUM: histogram_quantile(0.95, sms_c_enum_lookup_stop_duration_bucket)
    • Charging: histogram_quantile(0.95, sms_c_charging_succeeded_duration_bucket)
    • Delivery: histogram_quantile(0.95, sms_c_delivery_time_delta_delta_ms_bucket)
    • Unit: milliseconds
    • Horizontal bars

  3. Delivery Attempts Distribution (Heatmap)

    • Query: sms_c_delivery_succeeded_attempt_count_bucket
    • Shows how many attempts are typically needed
    • Color scale: Blue (1 attempt) to Red (many attempts)

  4. ENUM Cache Performance (Graph)

    • Hit Rate: rate(sms_c_enum_cache_hit_count[5m]) / (rate(sms_c_enum_cache_hit_count[5m]) + rate(sms_c_enum_cache_miss_count[5m]))
    • Cache Size: sms_c_enum_cache_size_size
    • Dual Y-axis (rate vs size)

  5. Processing Success Rate (Gauge)

    • Query: (rate(sms_c_message_processing_stop_duration_count{success="true"}[5m]) / rate(sms_c_message_processing_stop_duration_count[5m])) * 100
    • Unit: percent
    • Thresholds:
      • Red: < 95
      • Yellow: 95-99
      • Green: > 99

Dashboard 3: Traffic Analysis​

Purpose: Analyze message traffic patterns and routing distribution.

Panels:

  1. Messages by Source Type (Pie Chart)

    • Query: sum by (source_type) (increase(sms_c_message_received_count[1h]))
    • Shows distribution: IMS vs CS vs SMPP

  2. Messages by Source SMSC (Bar Chart)

    • Query: sum by (source_smsc) (rate(sms_c_message_received_count[1h]))
    • Top 10 sources
    • Horizontal bars

  3. Route Utilization (Table)

    • Columns:
      • Route ID
      • Destination SMSC
      • Messages (1h): sum by (route_id, dest_smsc) (increase(sms_c_routing_route_matched_count[1h]))
      • Priority
      • Success Rate
    • Sorted by message count

  4. Delivery by Destination (Graph)

    • Query: sum by (dest_smsc) (rate(sms_c_delivery_succeeded_count[5m]))
    • Unit: messages/second
    • Stacked area chart
    • Legend: {{dest_smsc}}

  5. Drop/Auto-Reply Actions (Stat)

    • Dropped: increase(sms_c_routing_action_count{action="drop"}[1h])
    • Auto-Replied: increase(sms_c_routing_action_count{action="auto_reply"}[1h])
    • Side by side stats

  6. Hourly Traffic Pattern (Graph)

    • Query: rate(sms_c_message_received_count[1h]) * 3600
    • Time range: Last 7 days
    • Shows daily patterns

Dashboard 4: Capacity & Resources​

Purpose: Monitor resource usage and capacity limits.

Panels:

  1. Queue Capacity (Graph)

    • Current: sms_c_queue_size_size
    • Capacity line: Fixed value based on system limits
    • Shows utilization trend

  2. Database Table Growth (Graph)

    • Messages: sms_c_mnesia_table_size_record_count{table="sms_route"}
    • Translations: sms_c_mnesia_table_size_record_count{table="translation_rule"}
    • Trend over last 30 days

  3. Message Backlog Trend (Graph)

    • Query: rate(sms_c_delivery_queued_count[5m]) - (rate(sms_c_delivery_succeeded_count[5m]) + rate(sms_c_delivery_failed_count[5m]))
    • Positive = backlog growing
    • Negative = catching up

  4. Peak Traffic (Stat)

    • Query: max_over_time(rate(sms_c_message_received_count[5m])[24h:])
    • Shows highest 5m rate in last 24h
    • Unit: messages/second

  5. Capacity Utilization (Gauge)

    • Query: (rate(sms_c_message_received_count[5m]) / MAX_CAPACITY) * 100
    • Replace MAX_CAPACITY with your system limit
    • Unit: percent
    • Thresholds:
      • Green: < 70
      • Yellow: 70-85
      • Red: > 85

Dashboard 5: SLA Compliance​

Purpose: Track SLA metrics and compliance.

Panels:

  1. SLA Compliance (Gauge)

    • Delivery Success: (rate(sms_c_delivery_succeeded_count[1h]) / rate(sms_c_delivery_queued_count[1h])) * 100
    • Target line at 99%
    • Thresholds:
      • Red: < 95
      • Yellow: 95-99
      • Green: >= 99

  2. Messages Delivered Within SLA (Stat)

    • Query: count(sms_c_delivery_time_delta_delta_ms_bucket{le="5000"}) / count(sms_c_delivery_time_delta_delta_ms_bucket)
    • Shows percentage delivered within 5 seconds
    • Unit: percent

  3. SLA Violations (Counter)

    • Messages exceeding 5 minutes: increase(sms_c_queue_oldest_message_age_seconds{} > 300)[24h:]
    • Should be 0

  4. Uptime (Stat)

    • Query: up{job="sms-c"}
    • Binary: 1 = up, 0 = down
    • Shows current status

  5. Daily Success Rate Trend (Graph)

    • Query: avg_over_time((rate(sms_c_delivery_succeeded_count[1h]) / rate(sms_c_delivery_queued_count[1h]))[24h:1h])
    • Time range: Last 30 days
    • SLA line at 99%

Alert Rule Examples​

Critical Alerts​

Routing Failures:

alert: RoutingFailuresDetected
expr: increase(sms_c_routing_failed_count[5m]) > 0
for: 2m
labels:
  severity: critical
annotations:
  summary: "{{ $value }} routing failures in last 5 minutes"
  description: "Messages cannot be routed. Check routing configuration."

Queue Backlog:

alert: MessageQueueBacklog
expr: sms_c_queue_size_pending > 10000
for: 5m
labels:
  severity: critical
annotations:
  summary: "Message queue has {{ $value }} pending messages"
  description: "Queue is backing up. Check delivery performance."

Old Messages in Queue:

alert: OldMessagesInQueue
expr: sms_c_queue_oldest_message_age_seconds > 300
for: 2m
labels:
  severity: critical
annotations:
  summary: "Oldest message is {{ $value }} seconds old"
  description: "Messages not being delivered. Check frontends."

All Frontends Disconnected:

alert: NoActiveFrontends
expr: sum(sms_c_frontend_status_count{status="connected"}) == 0
for: 1m
labels:
  severity: critical
annotations:
  summary: "No frontends connected"
  description: "No delivery path available. Check frontend connectivity."

Dead Letter Queue Growing:

alert: DeadLetterMessagesIncreasing
expr: rate(sms_c_delivery_dead_letter_count[10m]) > 0
for: 5m
labels:
  severity: critical
annotations:
  summary: "{{ $value }} messages moved to dead letter queue"
  description: "Messages are becoming undeliverable. Investigate failures."

Warning Alerts​

Low Delivery Success Rate:

alert: LowDeliverySuccessRate
expr: (rate(sms_c_delivery_succeeded_count[10m]) / rate(sms_c_delivery_queued_count[10m])) < 0.95
for: 10m
labels:
  severity: warning
annotations:
  summary: "Delivery success rate is {{ $value | humanizePercentage }}"
  description: "Success rate below 95%. Investigate delivery failures."

High Retry Rate:

alert: HighDeliveryRetryRate
expr: histogram_quantile(0.95, sms_c_delivery_succeeded_attempt_count_bucket) > 2
for: 15m
labels:
  severity: warning
annotations:
  summary: "95th percentile delivery attempts: {{ $value }}"
  description: "Messages requiring multiple attempts. Check destination reliability."

Slow Message Processing:

alert: SlowMessageProcessing
expr: histogram_quantile(0.95, sms_c_message_processing_stop_duration_bucket) > 1000
for: 10m
labels:
  severity: warning
annotations:
  summary: "95th percentile processing time: {{ $value }}ms"
  description: "Message processing is slow. Check system resources."

ENUM Lookups Failing:

alert: HighEnumFailureRate
expr: rate(sms_c_enum_lookup_stop_duration_count{success="false"}[10m]) > 0.1
for: 10m
labels:
  severity: warning
annotations:
  summary: "ENUM lookup failure rate: {{ $value }}"
  description: "DNS lookups failing. Check DNS servers."

Low ENUM Cache Hit Rate:

alert: LowEnumCacheHitRate
expr: rate(sms_c_enum_cache_hit_count[10m]) / (rate(sms_c_enum_cache_hit_count[10m]) + rate(sms_c_enum_cache_miss_count[10m])) < 0.70
for: 30m
labels:
  severity: warning
annotations:
  summary: "ENUM cache hit rate: {{ $value | humanizePercentage }}"
  description: "Low cache efficiency. May indicate unique number traffic."

Charging Failures:

alert: ChargingFailuresDetected
expr: rate(sms_c_charging_failed_count[10m]) > 0.05
for: 10m
labels:
  severity: warning
annotations:
  summary: "Charging failure rate: {{ $value }}"
  description: "Charging system errors. Check OCS connectivity."

Additional Notes​

  • All duration metrics use nanosecond precision internally but are converted to milliseconds for reporting
  • Counter metrics are cumulative and should be used with rate() or increase() functions in Prometheus queries
  • Gauge metrics represent instantaneous values at collection time
  • Histogram metrics provide percentile calculations (p50, p95, p99) and can be used to create heatmaps
  • All metrics include default labels added by Prometheus (instance, job, etc.)
  • When creating dashboards, use appropriate time ranges: 5m for real-time, 1h for trends, 24h+ for capacity planning
  • Set up recording rules in Prometheus for frequently-used complex queries to improve dashboard performance
  • Use variable templating in Grafana for dynamic dashboards (select dest_smsc, source_smsc, etc.)
Last updated on