Skip to main content

Monitoring & Metrics Guide

Prometheus Metrics, Grafana Dashboards, and Alerting

OmniSGW by Omnitouch Network Services

Table of Contents​

  1. Overview
  2. Metrics Exporter
  3. Available Metrics
  4. Prometheus Configuration
  5. Grafana Dashboards
  6. Alerting Rules
  7. Troubleshooting

Overview​

OmniSGW exposes Prometheus-compatible metrics for comprehensive monitoring of network operations, session management, and system health.

Metrics Architecture​

Metrics Exporter​

Accessing Metrics​

Metrics are exposed at the configured HTTP endpoint:

# Default endpoint (if configured)
curl http://127.0.0.40:42068/metrics

# Export to file
curl http://127.0.0.40:42068/metrics > metrics.txt

# Real-time monitoring
watch -n 5 'curl -s http://127.0.0.40:42068/metrics | head -30'

For metrics endpoint configuration (bind address, port, and poll interval), see the Configuration Guide.

Metrics Format​

Metrics are in Prometheus text format:

# HELP teid_registry_count Total number of allocated TEIDs
# TYPE teid_registry_count gauge
teid_registry_count 1234

# HELP s11_inbound_messages_total Total number of inbound S11 messages
# TYPE s11_inbound_messages_total counter
s11_inbound_messages_total{message_type="create_session_request"} 5432
s11_inbound_messages_total{message_type="delete_session_request"} 5100
s11_inbound_messages_total{message_type="modify_bearer_request"} 12000

Available Metrics​

Session Management Metrics​

Active Sessions:

teid_registry_count
├── Description: Active S11/S5S8 TEID allocations
├── Type: Gauge
├── Range: 0 to max licensed capacity
└── Example: 1234 (1234 active sessions)

seid_registry_count
├── Description: Active PFCP sessions (per SGW-U peer)
├── Type: Gauge
├── Labels: peer_ip
└── Example: seid_registry_count{peer_ip="10.0.0.30"} 1234

active_ue_sessions
├── Description: Total active UE sessions
├── Type: Gauge
└── Example: 5000

active_bearers
├── Description: Total active bearers (default + dedicated)
├── Type: Gauge
└── Example: 5500 (1 default + 0.1 dedicated per session)

charging_id_registry_count
├── Description: Active charging IDs
├── Type: Gauge
└── Example: 5000

Message Counters​

S11 (MME Interface):

s11_inbound_messages_total
├── Type: Counter (increasing)
├── Labels: message_type
├── Values:
│   ├── create_session_request
│   ├── delete_session_request
│   ├── modify_bearer_request
│   ├── create_bearer_request
│   ├── delete_bearer_request
│   ├── release_access_bearers_request
│   ├── downlink_data_notification
│   └── echo_request
└── Example: s11_inbound_messages_total{message_type="create_session_request"} 5432

S5/S8 (PGW Interface):

s5s8_inbound_messages_total
├── Type: Counter
├── Labels: message_type
├── Values: (same as S11 request types)
└── Example: s5s8_inbound_messages_total{message_type="create_session_request"} 4500

Sxa (SGW-U Interface):

sxa_inbound_messages_total
├── Type: Counter
├── Labels: message_type
├── Values:
│   ├── session_establishment_request
│   ├── session_modification_request
│   ├── session_deletion_request
│   ├── session_report_request
│   ├── association_setup_request
│   └── heartbeat_request
└── Example: sxa_inbound_messages_total{message_type="session_report_request"} 67000

Performance Metrics​

Message Latency:

s11_inbound_duration_seconds
├── Type: Histogram (with buckets)
├── Description: S11 message processing time
├── Percentiles: _count, _sum, _bucket
└── Example: s11_inbound_duration_seconds_bucket{le="0.1"} 5000

s5s8_inbound_duration_seconds
├── Type: Histogram
├── Description: S5/S8 message processing time

sxa_inbound_duration_seconds
├── Type: Histogram
├── Description: Sxa message processing time

PFCP Association:

pfcp_association_status
├── Type: Gauge
├── Values: 1 (associated) or 0 (not associated)
├── Labels: peer_ip, node_id
└── Example: pfcp_association_status{peer_ip="10.0.0.30"} 1

pfcp_heartbeat_latency_ms
├── Type: Gauge
├── Description: Heartbeat round-trip time
├── Labels: peer_ip
└── Example: pfcp_heartbeat_latency_ms{peer_ip="10.0.0.30"} 15

Error Metrics​

Protocol Errors:

s11_inbound_errors_total
├── Type: Counter
├── Labels: error_type
├── Values:
│   ├── parse_error
│   ├── validation_error
│   ├── timeout
│   └── other
└── Example: s11_inbound_errors_total{error_type="timeout"} 12

s5s8_inbound_errors_total
├── Type: Counter
├── Description: S5/S8 errors

sxa_inbound_errors_total
├── Type: Counter
├── Description: Sxa errors

Create Session Failures:

create_session_response_cause
├── Type: Counter
├── Labels: cause_code
├── Values: (3GPP cause codes)
├── Examples:
│   ├── cause_code="0": Success
│   ├── cause_code="16": No resources available
│   ├── cause_code="25": Semantics error
│   └── cause_code="49": No matching rule

Prometheus Configuration​

Installation​

# Download Prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.45.0/prometheus-2.45.0.linux-amd64.tar.gz
tar xzf prometheus-2.45.0.linux-amd64.tar.gz
cd prometheus-2.45.0.linux-amd64

Configuration File​

prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    monitor: 'sgw-c-prod'

scrape_configs:
  - job_name: 'sgw-c'
    static_configs:
      - targets: ['127.0.0.40:42068']
        labels:
          instance: 'sgw-c-prod-01'

  - job_name: 'sgw-c-backup'
    static_configs:
      - targets: ['127.0.0.41:42068']
        labels:
          instance: 'sgw-c-prod-02'

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['127.0.0.50:9093']

Starting Prometheus​

./prometheus --config.file=prometheus.yml \
             --storage.tsdb.path=/var/lib/prometheus \
             --web.console.libraries=consoles \
             --web.console.templates=console_templates

Accessing Prometheus​

http://localhost:9090

Grafana Dashboards​

Installation​

# Docker (easiest)
docker run -d \
  --name=grafana \
  -p 3000:3000 \
  -e GF_SECURITY_ADMIN_PASSWORD=admin \
  grafana/grafana

Adding Data Source​

  1. Open Grafana: http://localhost:3000
  2. Configuration → Data Sources
  3. Add → Prometheus
  4. URL: http://prometheus:9090

Dashboard: Session Overview​

Panels:

Row 1:
├── Active Sessions (Gauge)
├── Active Bearers (Gauge)
├── S11 Messages/sec (Graph)
└── S5/S8 Messages/sec (Graph)

Row 2:
├── Sxa Messages/sec (Graph)
├── S11 Latency p95 (Graph)
├── S5/S8 Latency p95 (Graph)
└── Sxa Latency p95 (Graph)

Row 3:
├── S11 Errors/min (Graph)
├── S5/S8 Errors/min (Graph)
├── Sxa Errors/min (Graph)
└── PFCP Associations (Status)

Dashboard: Interface Health​

Panels:

Row 1:
├── S11 Peer Status (Status)
├── S5/S8 Peer Status (Status)
├── SGW-U Peer Status (Status list)
└── System Load (Gauge)

Row 2:
├── S11 Message Rate (Graph)
├── S5/S8 Message Rate (Graph)
├── Sxa Message Rate (Graph)
└── Error Rate (Graph)

Row 3:
├── Message Latency Histogram (Heatmap)
├── Session Creation Rate (Graph)
├── Session Termination Rate (Graph)
└── Bearer Creation Rate (Graph)

Dashboard: Capacity Planning​

Panels:

Row 1:
├── Sessions vs Capacity (Gauge + Threshold)
├── Bearers vs Capacity (Gauge + Threshold)
├── PFCP Sessions Distribution (Bar Chart)
└── Sessions by APN (Pie Chart)

Row 2:
├── Session Growth Trend (Graph)
├── Bearer Growth Trend (Graph)
├── Peak Session Time (Heatmap)
└── Session Duration Distribution (Histogram)

Dashboard Query Examples​

Active Sessions:

teid_registry_count

Session Creation Rate:

rate(s11_inbound_messages_total{message_type="create_session_request"}[5m])

S11 Latency (95th percentile):

histogram_quantile(0.95, rate(s11_inbound_duration_seconds_bucket[5m]))

Error Rate:

rate(s11_inbound_errors_total[5m]) +
rate(s5s8_inbound_errors_total[5m]) +
rate(sxa_inbound_errors_total[5m])

PFCP Association Status:

pfcp_association_status{peer_ip=~"10.0.0.3[0-2]"}

Alerting Rules​

Alert Rules File​

sgw-c-alerts.yml:

groups:
  - name: sgw-c-alerts
    interval: 30s
    rules:
      # Session capacity alerts
      - alert: SGWCapacityHigh
        expr: (teid_registry_count / 100000) > 0.8
        for: 5m
        annotations:
          summary: "SGW session capacity above 80%"
          description: "Sessions: {{ $value }} of 100000"

      # Interface health alerts
      - alert: S11PeerDown
        expr: absent(s11_inbound_messages_total) > 0
        for: 2m
        annotations:
          summary: "S11 interface unreachable"

      - alert: PGWPeerDown
        expr: create_session_response_cause{cause_code="49"} > 100
        for: 2m
        annotations:
          summary: "PGW-C peer unreachable"

      - alert: SGWUAssociationDown
        expr: pfcp_association_status == 0
        for: 1m
        annotations:
          summary: "SGW-U association lost"
          description: "Peer: {{ $labels.peer_ip }}"

      # Message latency alerts
      - alert: S11LatencyHigh
        expr: histogram_quantile(0.95, rate(s11_inbound_duration_seconds_bucket[5m])) > 1
        for: 5m
        annotations:
          summary: "S11 latency above 1 second"
          description: "p95: {{ $value }}s"

      - alert: S5S8LatencyHigh
        expr: histogram_quantile(0.95, rate(s5s8_inbound_duration_seconds_bucket[5m])) > 1
        for: 5m
        annotations:
          summary: "S5/S8 latency above 1 second"

      # Error rate alerts
      - alert: S11ErrorRate
        expr: rate(s11_inbound_errors_total[5m]) > 10
        for: 3m
        annotations:
          summary: "High S11 error rate"
          description: "{{ $value }} errors/sec"

      - alert: SessionEstablishmentFailure
        expr: rate(create_session_response_cause{cause_code!="0"}[5m]) > 20
        for: 3m
        annotations:
          summary: "High session establishment failure rate"
          description: "{{ $value }} failures/sec"

Configuring AlertManager​

alertmanager.yml:

global:
  resolve_timeout: 5m

route:
  receiver: 'sgw-alerts'
  group_by: ['alertname', 'instance']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h

receivers:
  - name: 'sgw-alerts'
    webhook_configs:
      - url: 'http://slack-webhook-url'
    email_configs:
      - to: 'noc@example.com'
        from: 'sgw-alerts@example.com'
        smarthost: 'smtp.example.com:587'

Alert Notification Examples​

Slack Integration:

🚨 SGW Capacity High
Severity: Warning
Active Sessions: 85,000 / 100,000 (85%)
Time: 2025-12-10 15:30:00 UTC
Action: Monitor for capacity increase

Email Integration:

Subject: [ALERT] S11 Peer Unreachable

SGW-C S11 interface has not received messages for 2 minutes.
This may indicate:
- MME network connectivity issue
- SGW-C restart required
- S11 port configuration changed

Immediate Action Required: Check S11 status

Troubleshooting​

Metrics Not Appearing​

Problem: Metrics endpoint empty or 404

Diagnosis:

# Check metrics endpoint reachable
curl -v http://127.0.0.40:42068/metrics

# Check logs for metrics exporter errors
tail -f /var/log/sgw_c/sgw_c.log | grep -i metric

# Verify configuration
cat config/runtime.exs | grep metrics

Solutions:

  1. Restart SGW-C process
  2. Verify metrics IP/port not blocked by firewall
  3. Check bind address configuration
  4. Ensure sufficient memory for metrics collection

Missing Metrics for Specific Interface​

Problem: S11 metrics show but S5/S8 or Sxa missing

Diagnosis:

  1. Check interface is configured
  2. Verify interface is active
  3. Monitor logs for connection errors

Solution:

  • Verify peer reachability
  • Check interface binding
  • Review configuration

High Memory Usage​

Problem: Metrics exporter consuming excessive memory

Diagnosis:

# Check process memory
ps aux | grep sgw_c | grep -v grep | awk '{print $6}'

# Monitor growth over time
watch -n 5 'ps aux | grep sgw_c'

Solutions:

  1. Reduce metrics poll interval
  2. Limit number of metric samples
  3. Implement metric retention policy
  4. Scale to multiple instances

Best Practices​

Metrics Collection​

  • Scrape Interval: 15-30 seconds for balance
  • Retention: 15-30 days of metrics storage
  • Aggregation: Pre-aggregate high-cardinality metrics
  • Sampling: Use percentiles for latency, not raw values

Dashboard Design​

  • Context: Include time range, instance, peer info
  • Layering: Overview → Detail → Debug
  • Alerting: Visual thresholds on capacity graphs
  • Correlation: Link related metrics

Alerting Strategy​

  • Hierarchy: Critical → Warning → Info
  • Escalation: Page on-call for critical alerts
  • Threshold Tuning: Baseline then +20% for warning
  • Regular Testing: Test alert paths monthly

Monitoring Guide - OmniSGW Metrics and Observability

Developed by Omnitouch Network Services

Documentation Version: 1.0 Last Updated: 2025-12-10

Last updated on