Skip to main content

OmniPGW Monitoring & Metrics Guide

Prometheus Integration and Operational Monitoring

by Omnitouch Network Services

Table of Contents​

  1. Overview
  2. Metrics Endpoint
  3. Available Metrics
  4. Prometheus Configuration
  5. Grafana Dashboards
  6. Alerting
  7. Performance Monitoring
  8. Troubleshooting Metrics

Overview​

OmniPGW provides two complementary monitoring approaches:

1. Real-Time Web UI (covered briefly here, detailed in respective interface docs)

  • Live session viewer
  • PFCP peer status
  • Diameter peer connectivity
  • Individual session inspection

2. Prometheus Metrics (main focus of this document)

  • Historical trends and analysis
  • Alerting and notifications
  • Performance metrics
  • Capacity planning

This document focuses on Prometheus metrics. For Web UI details, see:

Prometheus Metrics Overview​

OmniPGW exposes Prometheus-compatible metrics for comprehensive monitoring of system health, performance, and capacity. This enables operations teams to:

  • Monitor System Health - Track active sessions, allocations, and errors
  • Capacity Planning - Understand resource utilization trends
  • Performance Analysis - Measure message handling latency
  • Alerting - Proactive notification of issues
  • Debugging - Identify root causes of problems

Monitoring Architecture​

Metrics Endpoint​

Configuration​

Enable metrics in config/runtime.exs:

config :pgw_c,
  metrics: %{
    enabled: true,
    ip_address: "0.0.0.0",  # Bind to all interfaces
    port: 9090,              # HTTP port
    registry_poll_period_ms: 5_000  # Poll interval
  }

Accessing Metrics​

HTTP Endpoint:

http://<omnipgw_ip>:<port>/metrics

Example:

curl http://10.0.0.20:9090/metrics

Output Format​

Metrics are exposed in Prometheus text format:

# HELP teid_registry_count The number of TEID registered to sessions
# TYPE teid_registry_count gauge
teid_registry_count 150

# HELP address_registry_count The number of addresses registered to sessions
# TYPE address_registry_count gauge
address_registry_count 150

# HELP s5s8_inbound_messages_total The total number of messages received from S5/S8 peers
# TYPE s5s8_inbound_messages_total counter
s5s8_inbound_messages_total{message_type="create_session_request"} 1523
s5s8_inbound_messages_total{message_type="delete_session_request"} 1487

Available Metrics​

OmniPGW exposes the following metric categories:

Session Metrics​

Active Session Counts:

Metric NameTypeDescription
teid_registry_countGaugeActive S5/S8 sessions (TEID count)
seid_registry_countGaugeActive PFCP sessions (SEID count)
session_id_registry_countGaugeActive Gx sessions (Diameter Session-ID count)
session_registry_countGaugeActive sessions (IMSI, EBI pairs)
address_registry_countGaugeAllocated UE IP addresses
charging_id_registry_countGaugeActive charging IDs (see Data CDR Format for CDR billing records)
sxb_sequence_number_registry_countGaugePending PFCP responses (awaiting response)
s5s8_sequence_number_registry_countGaugePending S5/S8 responses (awaiting response)
sxb_peer_registry_countGaugeNumber of registered PFCP peer processes

Usage:

# Current active sessions
teid_registry_count

# Session creation rate (per second)
rate(teid_registry_count[5m])

# Peak sessions in last hour
max_over_time(teid_registry_count[1h])

Message Counters​

S5/S8 (GTP-C) Messages:

Metric NameTypeLabelsDescription
s5s8_inbound_messages_totalCountermessage_typeTotal inbound S5/S8 messages
s5s8_outbound_messages_totalCountermessage_typeTotal outbound S5/S8 messages
s5s8_inbound_errors_totalCountermessage_typeS5/S8 processing errors

Message Types:

  • create_session_request
  • create_session_response
  • delete_session_request
  • delete_session_response
  • create_bearer_request
  • delete_bearer_request

Sxb (PFCP) Messages:

Metric NameTypeLabelsDescription
sxb_inbound_messages_totalCountermessage_typeTotal inbound PFCP messages
sxb_outbound_messages_totalCountermessage_typeTotal outbound PFCP messages
sxb_inbound_errors_totalCountermessage_typePFCP inbound processing errors
sxb_outbound_errors_totalCountermessage_typePFCP outbound processing errors

Message Types:

  • association_setup_request
  • association_setup_response
  • heartbeat_request
  • heartbeat_response
  • session_establishment_request
  • session_establishment_response
  • session_modification_request
  • session_deletion_request

Gx (Diameter) Messages:

Metric NameTypeLabelsDescription
gx_inbound_messages_totalCountermessage_typeTotal inbound Diameter messages
gx_outbound_messages_totalCountermessage_typeTotal outbound Diameter messages
gx_inbound_errors_totalCountermessage_typeDiameter inbound processing errors
gx_outbound_errors_totalCountermessage_typeDiameter outbound processing errors
gx_outbound_responses_totalCountermessage_type, result_code_class, diameter_hostDiameter responses sent, categorized by result code class and peer host

Message Types:

  • gx_CCA (Credit-Control-Answer)
  • gx_CCR (Credit-Control-Request)
  • gx_RAA (Re-Auth-Answer)
  • gx_RAR (Re-Auth-Request)

Result Code Classes (for gx_outbound_responses_total):

  • 2xxx - Success responses (e.g., 2001 DIAMETER_SUCCESS)
  • 3xxx - Protocol errors (e.g., 3001 DIAMETER_COMMAND_UNSUPPORTED)
  • 4xxx - Transient failures (e.g., 4001 DIAMETER_AUTHENTICATION_REJECTED)
  • 5xxx - Permanent failures (e.g., 5012 DIAMETER_UNABLE_TO_COMPLY)

Usage Examples:

# Monitor Gx response success rate
sum(rate(gx_outbound_responses_total{result_code_class="2xxx"}[5m])) /
sum(rate(gx_outbound_responses_total[5m])) * 100

# Track failures by PCRF host
rate(gx_outbound_responses_total{result_code_class!="2xxx"}[5m]) by (diameter_host)

# Count total successful Re-Auth-Answer messages
gx_outbound_responses_total{message_type="gx_RAA",result_code_class="2xxx"}

# Alert on high failure rate to specific PCRF
rate(gx_outbound_responses_total{result_code_class=~"4xxx|5xxx",diameter_host="pcrf.example.com"}[5m]) > 0.1

Error Handling:

Metric NameTypeLabelsDescription
rescues_totalCountermodule, functionTotal rescue blocks hit (exception handling)

Latency Metrics​

Inbound Message Processing Duration:

Metric NameTypeLabelsDescription
s5s8_inbound_handling_durationHistogramrequest_message_typeS5/S8 message handling time (includes decode, encode, send, receive)
sxb_inbound_handling_durationHistogramrequest_message_typePFCP message handling time (includes decode, encode, send, receive)
gx_inbound_handling_durationHistogramrequest_message_typeDiameter message handling time (includes decode, encode, send, receive)

Outbound Transaction Duration:

Metric NameTypeLabelsDescription
s5s8_outbound_transaction_durationHistogramrequest_message_typeS5/S8 request-response round-trip time
sxb_outbound_transaction_durationHistogramrequest_message_typePFCP request-response round-trip time (excludes encode/decode)
gx_outbound_transaction_durationHistogramrequest_message_typeDiameter request-response round-trip time (includes encode/decode)

Buckets (seconds):

  • Values: 0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0, 5.0
  • (100µs, 500µs, 1ms, 5ms, 10ms, 50ms, 100ms, 500ms, 1s, 5s)

Usage:

# 95th percentile S5/S8 latency
histogram_quantile(0.95,
  rate(s5s8_inbound_handling_duration_bucket[5m])
)

# Average PFCP latency
rate(sxb_inbound_handling_duration_sum[5m]) /
rate(sxb_inbound_handling_duration_count[5m])

UPF Health Monitoring​

UPF Peer Metrics:

Metric NameTypeLabelsDescription
upf_peers_totalGauge-Total number of registered UPF peers
upf_peers_healthyGauge-Number of healthy UPF peers (associated + heartbeats OK)
upf_peers_unhealthyGauge-Number of unhealthy UPF peers
upf_peers_associatedGauge-Number of UPF peers with active PFCP association
upf_peers_unassociatedGauge-Number of UPF peers without PFCP association
upf_peer_healthyGaugepeer_ipHealth status of specific UPF (1=healthy, 0=unhealthy)
upf_peer_missed_heartbeatsGaugepeer_ipConsecutive missed heartbeats for specific UPF

Usage:

# Monitor UPF pool health
upf_peers_healthy / upf_peers_total

# Alert on unhealthy UPFs
upf_peers_unhealthy > 0

# Track specific UPF health
upf_peer_healthy{peer_ip="10.98.0.20"}

# Identify UPFs with heartbeat issues
upf_peer_missed_heartbeats > 2

Alerting Examples:

# Alert when UPF goes down
- alert: UPF_Peer_Down
  expr: upf_peer_healthy == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "UPF {{ $labels.peer_ip }} is down"
    description: "UPF peer not responding to PFCP heartbeats"

# Alert when multiple UPFs are down
- alert: UPF_Pool_Degraded
  expr: (upf_peers_healthy / upf_peers_total) < 0.5
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "UPF pool degraded"
    description: "Only {{ $value | humanizePercentage }} of UPFs are healthy"

# Warning on missed heartbeats
- alert: UPF_Heartbeat_Issues
  expr: upf_peer_missed_heartbeats > 2
  for: 30s
  labels:
    severity: warning
  annotations:
    summary: "UPF {{ $labels.peer_ip }} heartbeat issues"
    description: "{{ $value }} consecutive missed heartbeats"

P-CSCF Health Monitoring​

P-CSCF Server Metrics:

Metric NameTypeLabelsDescription
pcscf_fqdns_totalGauge-Total P-CSCF FQDNs being monitored
pcscf_fqdns_resolvedGauge-P-CSCF FQDNs successfully resolved via DNS
pcscf_fqdns_failedGauge-P-CSCF FQDNs that failed DNS resolution
pcscf_servers_totalGauge-Total P-CSCF servers discovered
pcscf_servers_healthyGaugefqdnHealthy P-CSCF servers per FQDN
pcscf_servers_unhealthyGaugefqdnUnhealthy P-CSCF servers per FQDN

See: P-CSCF Monitoring Guide for detailed IMS health tracking.

License Metrics​

License Status:

Metric NameTypeDescription
license_statusGaugeCurrent license status (1 = valid, 0 = invalid)

Usage:

# Check if license is valid
license_status == 1

# Alert on invalid license
license_status == 0

Alerting Example:

- alert: PGW_C_License_Invalid
  expr: license_status == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "PGW-C license invalid or expired"
    description: "License status is invalid - create session requests are being blocked"

Impact of Invalid License:

When the license is invalid or the license server is unreachable, Create Session Requests will be rejected with GTP-C cause code "No resources available" (73). This is visible in packet captures as shown below:

License Invalid - No Resources Available

Wireshark capture showing Create Session Response with "No resources available" cause when license is invalid

Notes:

  • Product name registered with license server: omnipgwc
  • License server URL is configured in config/runtime.exs under :license_client
  • When license is invalid (license_status == 0), create session requests are blocked with GTP-C cause code 73 (No resources available)
  • UI and monitoring remain accessible regardless of license status
  • Diameter, GTP-C, and PFCP peers continue to maintain connections
  • Existing sessions are not affected - only new session creation is blocked

System Metrics​

Erlang VM Metrics:

Metric NameTypeDescription
vm_memory_totalGaugeTotal VM memory (bytes)
vm_memory_processesGaugeMemory used by processes
vm_memory_systemGaugeMemory used by system
vm_system_process_countGaugeTotal Erlang processes
vm_system_port_countGaugeTotal open ports

Prometheus Configuration​

Scrape Configuration​

Add OmniPGW to Prometheus prometheus.yml:

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'omnipgw'
    static_configs:
      - targets: ['10.0.0.20:9090']
        labels:
          instance: 'omnipgw-01'
          environment: 'production'
          site: 'datacenter-1'

Multiple OmniPGW Instances​

scrape_configs:
  - job_name: 'omnipgw'
    static_configs:
      - targets:
          - '10.0.0.20:9090'
          - '10.0.0.21:9090'
          - '10.0.0.22:9090'
        labels:
          environment: 'production'

Service Discovery​

Kubernetes:

scrape_configs:
  - job_name: 'omnipgw'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        action: keep
        regex: omnipgw
      - source_labels: [__meta_kubernetes_pod_ip]
        target_label: __address__
        replacement: '${1}:9090'

Verification​

Test scrape:

# Check Prometheus targets
curl http://prometheus:9090/api/v1/targets

# Query a metric
curl 'http://prometheus:9090/api/v1/query?query=teid_registry_count'

Grafana Dashboards​

Dashboard Setup​

1. Add Prometheus Data Source:

Configuration → Data Sources → Add data source → Prometheus
URL: http://prometheus:9090

2. Import Dashboard:

Create a new dashboard or import from JSON.

Key Panels​

Panel 1: Active Sessions​

# Query
teid_registry_count

# Panel Type: Gauge
# Thresholds:
#   Green: < 5000
#   Yellow: 5000-8000
#   Red: > 8000

Panel 2: Session Rate​

# Query
rate(s5s8_inbound_messages_total{message_type="create_session_request"}[5m])

# Panel Type: Graph
# Unit: requests/sec

Panel 3: IP Pool Utilization​

# Query (for /24 subnet with 254 IPs)
(address_registry_count / 254) * 100

# Panel Type: Gauge
# Unit: percent (0-100)
# Thresholds:
#   Green: < 70%
#   Yellow: 70-85%
#   Red: > 85%

Panel 4: Message Latency (95th Percentile)​

# Query
histogram_quantile(0.95,
  rate(s5s8_inbound_handling_duration_bucket{request_message_type="create_session_request"}[5m])
)

# Panel Type: Graph
# Unit: milliseconds

Panel 5: Error Rate​

# Query
rate(s5s8_inbound_errors_total[5m])

# Panel Type: Graph
# Unit: errors/sec
# Alert Threshold: > 0.1

Panel 6: Gx Response Success Rate​

# Query: Calculate percentage of successful Gx responses
sum(rate(gx_outbound_responses_total{result_code_class="2xxx"}[5m])) /
sum(rate(gx_outbound_responses_total[5m])) * 100

# Panel Type: Gauge
# Unit: percent (0-100)
# Thresholds:
#   Green: > 95%
#   Yellow: 90-95%
#   Red: < 90%

Alternative - Breakdown by Result Code Class:

# Query: Show response counts by result code class
sum(rate(gx_outbound_responses_total[5m])) by (result_code_class)

# Panel Type: Pie Chart or Bar Chart
# Legend: {{ result_code_class }}

Alternative - Per-PCRF Response Status:

# Query: Show responses by PCRF host
sum(rate(gx_outbound_responses_total[5m])) by (diameter_host, result_code_class)

# Panel Type: Stacked Bar Chart
# Legend: {{ diameter_host }} - {{ result_code_class }}

Panel 7: UPF Health Status​

# Query: Overall pool health percentage
(upf_peers_healthy / upf_peers_total) * 100

# Panel Type: Gauge
# Unit: percent (0-100)
# Thresholds:
#   Green: 100%
#   Yellow: 50-99%
#   Red: < 50%

Alternative - Per-UPF Status:

# Query: Individual UPF health
upf_peer_healthy

# Panel Type: Stat
# Mappings:
#   1 = "UP" (Green)
#   0 = "DOWN" (Red)

Complete Dashboard Example​

{
  "dashboard": {
    "title": "OmniPGW - Operations Dashboard",
    "panels": [
      {
        "title": "Active Sessions",
        "targets": [
          {
            "expr": "teid_registry_count",
            "legendFormat": "Active Sessions"
          }
        ],
        "type": "graph"
      },
      {
        "title": "Session Creation Rate",
        "targets": [
          {
            "expr": "rate(s5s8_inbound_messages_total{message_type=\"create_session_request\"}[5m])",
            "legendFormat": "Sessions/sec"
          }
        ],
        "type": "graph"
      },
      {
        "title": "IP Pool Utilization",
        "targets": [
          {
            "expr": "(address_registry_count / 254) * 100",
            "legendFormat": "Pool Usage %"
          }
        ],
        "type": "gauge"
      },
      {
        "title": "Message Latency (p95)",
        "targets": [
          {
            "expr": "histogram_quantile(0.95, rate(s5s8_inbound_handling_duration_bucket[5m]))",
            "legendFormat": "S5/S8 p95"
          },
          {
            "expr": "histogram_quantile(0.95, rate(sxb_inbound_handling_duration_bucket[5m]))",
            "legendFormat": "PFCP p95"
          }
        ],
        "type": "graph"
      }
    ]
  }
}

Alerting​

Alert Rules​

Create omnipgw_alerts.yml:

groups:
  - name: omnipgw
    interval: 30s
    rules:
      # Session Count Alerts
      - alert: OmniPGW_HighSessionCount
        expr: teid_registry_count > 8000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "OmniPGW high session count"
          description: "{{ $value }} active sessions (threshold: 8000)"

      - alert: OmniPGW_SessionCountCritical
        expr: teid_registry_count > 9500
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "OmniPGW session count critical"
          description: "{{ $value }} active sessions approaching capacity"

      # IP Pool Alerts
      - alert: OmniPGW_IPPoolUtilizationHigh
        expr: (address_registry_count / 254) * 100 > 80
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "OmniPGW IP pool utilization high"
          description: "IP pool {{ $value }}% utilized"

      - alert: OmniPGW_IPPoolExhausted
        expr: address_registry_count >= 254
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "OmniPGW IP pool exhausted"
          description: "No IPs available for allocation"

      # Error Rate Alerts
      - alert: OmniPGW_HighErrorRate
        expr: rate(s5s8_inbound_errors_total[5m]) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "OmniPGW high error rate"
          description: "{{ $value }} errors/sec on S5/S8 interface"

      - alert: OmniPGW_GxErrorRate
        expr: rate(gx_inbound_errors_total[5m]) > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "OmniPGW Gx errors"
          description: "{{ $value }} Diameter errors/sec"

      # Gx Response Alerts
      - alert: OmniPGW_GxResponseFailureRate
        expr: |
          sum(rate(gx_outbound_responses_total{result_code_class!="2xxx"}[5m])) /
          sum(rate(gx_outbound_responses_total[5m])) > 0.1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "OmniPGW high Gx response failure rate"
          description: "{{ $value | humanizePercentage }} of Gx responses are failures (non-2xxx result codes)"

      - alert: OmniPGW_GxPCRFFailures
        expr: rate(gx_outbound_responses_total{result_code_class=~"4xxx|5xxx"}[5m]) by (diameter_host) > 0.05
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: "PCRF {{ $labels.diameter_host }} receiving failure responses"
          description: "{{ $value }} failure responses/sec to PCRF {{ $labels.diameter_host }}"

      # UPF Health Alerts
      - alert: OmniPGW_UPF_PeerDown
        expr: upf_peer_healthy == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "UPF peer {{ $labels.peer_ip }} down"
          description: "UPF not responding to PFCP heartbeats"

      - alert: OmniPGW_UPF_PoolDegraded
        expr: (upf_peers_healthy / upf_peers_total) < 0.5
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "UPF pool degraded"
          description: "{{ $value | humanizePercentage }} of UPFs are healthy (< 50%)"

      - alert: OmniPGW_UPF_HeartbeatFailures
        expr: upf_peer_missed_heartbeats > 2
        for: 30s
        labels:
          severity: warning
        annotations:
          summary: "UPF {{ $labels.peer_ip }} heartbeat failures"
          description: "{{ $value }} consecutive missed heartbeats"

      - alert: OmniPGW_UPF_AllDown
        expr: upf_peers_healthy == 0 and upf_peers_total > 0
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "All UPF peers down"
          description: "No healthy UPFs available for session creation"

      # Latency Alerts
      - alert: OmniPGW_HighLatency
        expr: |
          histogram_quantile(0.95,
            rate(s5s8_inbound_handling_duration_bucket[5m])
          ) > 100000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "OmniPGW high message latency"
          description: "p95 latency {{ $value }}µs (> 100ms)"

      # System Alerts
      - alert: OmniPGW_HighMemoryUsage
        expr: vm_memory_total > 2000000000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "OmniPGW high memory usage"
          description: "VM using {{ $value | humanize }}B memory"

      - alert: OmniPGW_HighProcessCount
        expr: vm_system_process_count > 100000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "OmniPGW high process count"
          description: "{{ $value }} Erlang processes (potential leak)"

AlertManager Configuration​

# alertmanager.yml
global:
  resolve_timeout: 5m

route:
  receiver: 'ops-team'
  group_by: ['alertname', 'instance']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 12h

  routes:
    - match:
        severity: critical
      receiver: 'pagerduty'

    - match:
        severity: warning
      receiver: 'slack'

receivers:
  - name: 'ops-team'
    email_configs:
      - to: 'ops@example.com'

  - name: 'slack'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'
        channel: '#omnipgw-alerts'
        title: 'OmniPGW Alert: {{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'

  - name: 'pagerduty'
    pagerduty_configs:
      - service_key: 'YOUR_PAGERDUTY_KEY'

Performance Monitoring​

Key Performance Indicators (KPIs)​

Throughput Queries​

Session Setup Rate:

rate(s5s8_inbound_messages_total{message_type="create_session_request"}[5m])

Session Teardown Rate:

rate(s5s8_inbound_messages_total{message_type="delete_session_request"}[5m])

Net Session Growth:

rate(s5s8_inbound_messages_total{message_type="create_session_request"}[5m]) -
rate(s5s8_inbound_messages_total{message_type="delete_session_request"}[5m])

Latency Analysis​

Message Processing Latency (Percentiles):

# p50 (Median)
histogram_quantile(0.50,
  rate(s5s8_inbound_handling_duration_bucket[5m])
)

# p95
histogram_quantile(0.95,
  rate(s5s8_inbound_handling_duration_bucket[5m])
)

# p99
histogram_quantile(0.99,
  rate(s5s8_inbound_handling_duration_bucket[5m])
)

Latency Breakdown by Message Type:

histogram_quantile(0.95,
  rate(s5s8_inbound_handling_duration_bucket[5m])
) by (request_message_type)

Session Growth Trend (24h):

teid_registry_count -
teid_registry_count offset 24h

Capacity Remaining:

# For max capacity of 10,000 sessions
10000 - teid_registry_count

Time to Capacity Exhaustion:

# Days until capacity exhausted (based on 1h growth rate)
(10000 - teid_registry_count) /
(rate(teid_registry_count[1h]) * 86400)

Troubleshooting Metrics​

Identifying Issues​

Issue: High Session Rejection Rate​

Query:

rate(s5s8_inbound_errors_total[5m]) by (message_type)

Action:

  • Check error logs
  • Verify PCRF connectivity (Gx errors)
  • Check IP pool exhaustion

Issue: Slow Session Setup​

Query:

histogram_quantile(0.95,
  rate(s5s8_inbound_handling_duration_bucket{request_message_type="create_session_request"}[5m])
)

Action:

  • Check Gx latency (PCRF response time)
  • Check PFCP latency (PGW-U response time)
  • Review system resource usage

Issue: PCRF Policy Failures​

Queries:

# Overall Gx response failure rate
sum(rate(gx_outbound_responses_total{result_code_class!="2xxx"}[5m])) /
sum(rate(gx_outbound_responses_total[5m])) * 100

# Breakdown by PCRF host
sum(rate(gx_outbound_responses_total[5m])) by (diameter_host, result_code_class)

# Specific result code classes
rate(gx_outbound_responses_total{result_code_class="5xxx"}[5m]) by (diameter_host)

Action:

  • Check PCRF connectivity and health
  • Review subscriber profiles in PCRF (5xxx errors often indicate policy issues)
  • Verify Diameter peer configuration
  • Check PCRF logs for corresponding errors
  • For 5012 (DIAMETER_UNABLE_TO_COMPLY), review Re-Auth-Request handling

Issue: Memory Leak Suspected​

Queries:

# Total memory trend
rate(vm_memory_total[1h])

# Process memory trend
rate(vm_memory_processes[1h])

# Process count trend
rate(vm_system_process_count[1h])

Action:

  • Check for stale sessions
  • Review registry counts
  • Restart if leak confirmed

Debugging Queries​

Find Peak Session Time:

max_over_time(teid_registry_count[24h])

Compare Current vs. Historical:

teid_registry_count /
avg_over_time(teid_registry_count[7d])

Identify Anomalies:

abs(
  teid_registry_count -
  avg_over_time(teid_registry_count[1h])
) > 100

Best Practices​

Metric Collection​

  1. Scrape Interval: 15-30 seconds (balance granularity vs. load)
  2. Retention: 15+ days for historical analysis
  3. Labels: Use consistent labeling (instance, environment, site)

Dashboard Design​

  1. Overview Dashboard - High-level KPIs for NOC
  2. Detailed Dashboards - Per-interface deep dive
  3. Troubleshooting Dashboard - Error metrics and logs

Alert Design​

  1. Avoid Alert Fatigue - Only alert on actionable issues
  2. Escalation - Warning → Critical with escalating severity
  3. Context - Include runbook links in alert descriptions

Configuration and Setup​

Interface Metrics​

Specialized Monitoring​

Back to Operations Guide

OmniPGW Monitoring Guide - by Omnitouch Network Services

Last updated on