Skip to main content

OmniPGW Troubleshooting Guide

Troubleshooting Procedures and Common Issues

by Omnitouch Network Services

Table of Contents​

  1. Overview
  2. Troubleshooting Tools
  3. Session Establishment Issues
  4. PFCP / User Plane Issues
  5. Diameter (Gx/Gy) Issues
  6. IP Allocation Issues
  7. Quick Reference

Overview​

This guide provides step-by-step troubleshooting procedures for common OmniPGW operational issues. Each issue includes:

  • Symptom: What you'll observe
  • Likely Causes: Common root causes
  • Diagnosis: How to confirm the cause
  • Resolution: Step-by-step fix
  • Prevention: How to avoid recurrence

Troubleshooting Tools​

Web UI​

Access: http://<omnipgw_ip>:4000

Key Pages:

  • /pgw_sessions - Real-time session viewer (search by IMSI, IP, MSISDN, APN)
  • /diameter - Diameter peer status (Gx PCRF, Gy OCS)
  • /pfcp_peers - PFCP peer status (PGW-U connectivity)
  • /logs - Real-time log streaming with filtering

Prometheus Metrics​

Access: http://<omnipgw_ip>:9090/metrics

Key Metrics:

  • teid_registry_count - Active sessions
  • address_registry_count - Allocated UE IPs
  • sxb_inbound_errors_total - PFCP errors
  • gx_inbound_errors_total - Diameter Gx errors
  • gy_inbound_errors_total - Diameter Gy errors

See Monitoring Guide for complete metrics reference.

Log Analysis​

Web UI: Access /logs page and use search filters

Common Log Filters:

  • "create_session_request" - Session establishment
  • "Credit Control" - Gx/Gy interactions
  • "PFCP Session" - User plane programming
  • "error" or "ERROR" - Error messages
  • "timeout" - Timeout issues

Session Establishment Issues​

Issue: Create Session Request Rejected with "No Resources Available"​

Symptom:

  • SGW-C receives Create Session Response with cause "No resources available" (73)
  • All new session attempts fail
  • Existing sessions continue working
  • Logs: [PGW-C] Create Session Request blocked - invalid license

License Invalid - No Resources Available

Wireshark capture showing Create Session Response with "No resources available" cause

Likely Cause:

  • Invalid or expired OmniPGW license
  • License server unreachable

Diagnosis:

  1. Check license metric:

    license_status
    • Value of 0 indicates invalid license

  2. Check logs for license warnings:

    • Search for "license" or "License"
    • Look for "Unable to contact license server" messages

  3. Verify license server connectivity:

    • Check configured URL in config/runtime.exs under :license_client
    • Default: https://localhost:10443/api

Resolution:

  1. Verify license server is reachable:

    curl -k https://<license_server_ip>:10443/api/status

  2. Check license configuration in config/runtime.exs:

    config :license_client,
      license_server_api_urls: ["https://<license_server_ip>:10443/api"],
      licensee: "Your Company Name"

  3. Verify product is licensed:

    • Product name: omnipgwc
    • Contact Omnitouch to verify license status

  4. Restart OmniPGW after configuration changes

Prevention:

  • Monitor license_status metric with critical alerts
  • Ensure license server high availability
  • Set up license expiry alerts before expiration

Issue: Create Session Request Rejected (Other Causes)​

Symptom:

  • SGW-C receives Create Session Response with error cause
  • Users cannot establish PDN connections
  • Metric: s5s8_inbound_errors_total increasing

Likely Causes:

  1. IP pool exhausted
  2. PCRF (Gx) unreachable or rejecting policy
  3. PGW-U (PFCP) unavailable
  4. Invalid APN configuration

Diagnosis:

  1. Check IP pool utilization:

    address_registry_count
    • If equals configured pool size, pool is exhausted

  2. Check PCRF connectivity:

    • Web UI → /diameter page
    • Look for PCRF peer status = "disconnected"
    • Logs: Search "Credit Control Answer" for errors

  3. Check PFCP peer status:

    • Web UI → /pfcp_peers page
    • Look for "Association: DOWN"
    • Metric: pfcp_peer_associated = 0

  4. Check APN configuration:

    • Review config/runtime.exs under ue.apn_map
    • Verify requested APN exists in configuration

Resolution:

For IP Pool Exhaustion:

  1. Identify stale sessions: Web UI → /pgw_sessions, look for old sessions
  2. Expand IP pool in config/runtime.exs: 
    config :pgw_c,
      ue: %{
        subnet_map: %{
          "internet" => "10.0.0.0/23"  # Changed from /24 to /23 (doubles capacity)
        }
      }
  3. Restart OmniPGW
  4. Verify: curl http://<ip>:9090/metrics | grep address_registry_count

For PCRF Connectivity Issues:

  1. Check network connectivity: ping <pcrf_ip>
  2. Verify PCRF Diameter service: telnet <pcrf_ip> 3868
  3. Check config/runtime.exs Diameter peer configuration
  4. Restart OmniPGW if config changed
  5. Verify via Web UI → /diameter (peer should show "connected")

For PFCP Issues:

Prevention:

  • Monitor IP pool utilization with alerts at 80%
  • Monitor PCRF connectivity with Diameter peer alerts
  • Implement session cleanup for idle sessions

Issue: Sessions Stuck in Intermediate State​

Symptom:

  • Session appears in Web UI but incomplete
  • Metrics show growing session count but no user traffic
  • Delete Session Request fails or times out

Likely Causes:

  1. PFCP Session Establishment failed but S5/S8 session created
  2. PCRF CCR-Initial timed out
  3. Create Bearer Request (dedicated bearer) failed
  4. Network disruption during session setup

Diagnosis:

  1. Search for session in Web UI:

    • /pgw_sessions → Search by IMSI
    • Check if pfcp_seid is present (if missing, PFCP failed)
    • Check if gx_session_id is present (if missing, Gx failed)

  2. Check logs for the IMSI:

    • Filter logs by IMSI
    • Look for "Session Establishment Request" (PFCP)
    • Look for "Credit Control Request" (Gx)
    • Look for timeout or error messages

  3. Check metrics:

    # Sessions with TEID but no PFCP session
    teid_registry_count - seid_registry_count

    # Sessions with TEID but no Gx session
    teid_registry_count - session_id_registry_count

Resolution:

  1. For PFCP establishment failures:

    • Check PGW-U health and logs
    • Verify PFCP association: Web UI → /pfcp_peers
    • Send Delete Session Request from SGW-C to cleanup

  2. For Gx timeout issues:

    • Check PCRF latency: histogram_quantile(0.95, rate(gx_outbound_transaction_duration_bucket[5m]))
    • Increase Gx timeout in config/runtime.exs if needed
    • Send Delete Session Request to cleanup

  3. Manual cleanup (last resort):

    • Currently requires OmniPGW restart to clear stuck sessions
    • Monitor teid_registry_count before/after restart to confirm cleanup

Prevention:

  • Monitor PFCP and Gx latency metrics
  • Implement session timeout/cleanup for incomplete sessions
  • Alert on registry count mismatches

PFCP / User Plane Issues​

Issue: PFCP Association Down​

Symptom:

  • Web UI → /pfcp_peers shows "Association: DOWN"
  • All new session establishments fail
  • Metric: pfcp_peer_associated = 0
  • Logs: "PFCP heartbeat timeout" or "Association Setup failed"

Likely Causes:

  1. PGW-U unreachable (network issue)
  2. PGW-U crashed or restarted
  3. PFCP configuration mismatch (IP, port)
  4. Firewall blocking UDP 8805

Diagnosis:

  1. Check network connectivity:

    ping <pgw_u_ip>
    nc -u -v <pgw_u_ip> 8805

  2. Check PFCP configuration:

    • Review config/runtime.exs under upf.peer_list
    • Verify IP address and node ID match PGW-U configuration

  3. Check PGW-U status:

    • Access PGW-U logs
    • Verify PGW-U is running: systemctl status omnipgw_u (or equivalent)

  4. Check metrics:

    # Heartbeat failures
    pfcp_consecutive_heartbeat_failures

    # PFCP error rate
    rate(sxb_inbound_errors_total[5m])

Resolution:

  1. For network issues:

    • Verify routing: traceroute <pgw_u_ip>
    • Check firewall rules: Ensure UDP 8805 allowed
    • Check security groups (if cloud deployment)

  2. For PGW-U crashes:

    • Restart PGW-U service
    • Wait 30 seconds for association re-establishment
    • Verify via Web UI → /pfcp_peers (should show "Association: UP")

  3. For configuration issues:

    • Correct config/runtime.exs PFCP peer configuration
    • Restart OmniPGW
    • Verify association established

Prevention:

  • Monitor pfcp_peer_associated metric with critical alerts
  • Monitor pfcp_consecutive_heartbeat_failures (alert at > 2)
  • Implement redundant PGW-U instances
  • Enable PFCP keepalive/heartbeat (should be default)

Issue: PFCP Session Modification Failures​

Symptom:

  • Dedicated bearer creation fails
  • QoS policy updates (from PCRF RAR) fail
  • Logs: "Session Modification Request failed"
  • Metric: sxb_inbound_errors_total{message_type="session_modification_response"} increasing

Likely Causes:

  1. Invalid PFCP rules (PDR/FAR/QER references)
  2. PGW-U resource exhaustion
  3. Rule ID conflicts
  4. PGW-U software bug

Diagnosis:

  1. Check logs:

    • Filter for "Session Modification" and SEID
    • Look for error cause codes in PFCP response
    • Common causes: "Rule ID already exists", "Out of resources"

  2. Check PGW-U logs:

    • Look for PFCP processing errors
    • Check resource utilization (CPU, memory)

  3. Check session state in Web UI:

    • /pgw_sessions → Find session by IMSI
    • Review pdr_map, far_map, qer_map for conflicts
    • Look for duplicate IDs

Resolution:

  1. For rule conflicts:

    • Delete and recreate dedicated bearer
    • If persistent, Delete Session and have UE reconnect

  2. For PGW-U resource issues:

    • Check PGW-U capacity (sessions, PDRs, throughput)
    • Scale PGW-U if needed
    • Reduce session load on affected PGW-U instance

  3. For software bugs:

    • Capture full session state (Web UI session details)
    • Capture PFCP message logs
    • Report to vendor with reproduction steps

Prevention:

  • Monitor PGW-U resource utilization
  • Test dedicated bearer creation in staging
  • Monitor sxb_inbound_errors_total with alerts

Diameter (Gx/Gy) Issues​

Issue: PCRF Peer Disconnected (Gx)​

Symptom:

  • Web UI → /diameter shows PCRF peer "disconnected"
  • Sessions created without QoS policies (default QCI=5 applied)
  • Logs: "Diameter peer connection failed" or "CER/CEA timeout"

Likely Causes:

  1. PCRF unreachable (network issue)
  2. PCRF service down
  3. Diameter configuration mismatch (Origin-Host, Realm)
  4. Firewall blocking TCP 3868

Diagnosis:

  1. Check network connectivity:

    ping <pcrf_ip>
    telnet <pcrf_ip> 3868

  2. Check Diameter configuration:

    • Review config/runtime.exs under diameter.peer_list
    • Verify host, realm, ip match PCRF configuration
    • Check origin_host matches what PCRF expects

  3. Check PCRF logs:

    • Look for CER (Capabilities-Exchange-Request) from PGW-C
    • Look for rejection reasons

  4. Check metrics:

    # Diameter connection errors
    diameter_peer_connected{peer="<pcrf_host>"}

Resolution:

  1. For network issues:

    • Verify routing to PCRF
    • Check firewall rules: Ensure TCP 3868 allowed
    • Test connectivity: nc -v <pcrf_ip> 3868

  2. For PCRF service down:

    • Restart PCRF service
    • Wait for automatic reconnection (30s retry interval)
    • Verify via Web UI → /diameter

  3. For configuration mismatch:

    • Correct config/runtime.exs Diameter configuration: 
      config :pgw_c,
        diameter: %{
          host: "pgw-c.epc.mnc999.mcc999.3gppnetwork.org",  # Must match PCRF config
          realm: "epc.mnc999.mcc999.3gppnetwork.org",
          peer_list: [
            %{
              host: "pcrf.epc.mnc999.mcc999.3gppnetwork.org",
              realm: "epc.mnc999.mcc999.3gppnetwork.org",
              ip: "192.168.1.100",
              initiate_connection: true
            }
          ]
        }
    • Restart OmniPGW
    • Verify connection established

Prevention:

  • Monitor Diameter peer connectivity with critical alerts
  • Implement redundant PCRF instances (if supported)
  • Document Diameter configuration in runbook

Issue: CCR/CCA Timeouts (Gx Policy Requests)​

Symptom:

  • Session establishment slow (> 5 seconds)
  • Logs: "Credit Control Request timeout"
  • Metric: gx_outbound_transaction_duration very high (> 5s)
  • Sessions created with default QoS (fallback behavior)

Likely Causes:

  1. PCRF overloaded
  2. PCRF database slow
  3. Network latency
  4. PCRF software issue

Diagnosis:

  1. Check Gx latency:

    # P95 latency
    histogram_quantile(0.95, rate(gx_outbound_transaction_duration_bucket[5m]))

    # P99 latency (outliers)
    histogram_quantile(0.99, rate(gx_outbound_transaction_duration_bucket[5m]))

  2. Check PCRF health:

    • Access PCRF monitoring dashboards
    • Check CPU, memory, database connections
    • Review PCRF logs for slow queries

  3. Check network latency:

    ping -c 100 <pcrf_ip> | tail -1  # Check avg latency

  4. Check logs:

    • Count CCR/CCA exchanges: Filter "Credit Control"
    • Measure time between "Sending CCR" and "Received CCA"

Resolution:

  1. For PCRF overload:

    • Scale PCRF (add instances)
    • Reduce CCR message size if possible
    • Tune PCRF thread pools/workers

  2. For network latency:

    • Investigate network path (routers, switches)
    • Consider co-locating PGW-C and PCRF

  3. Temporary workaround (increase timeout):

    • Edit config/runtime.exs: 
      config :pgw_c,
        diameter: %{
          transaction_timeout_ms: 10000  # Increase from default 5000
        }
    • Restart OmniPGW
    • Note: This only masks the issue; fix root cause

Prevention:

  • Monitor Gx latency with alerts (warning > 1s, critical > 5s)
  • Capacity plan PCRF for expected session rate
  • Test PCRF performance under load

Issue: OCS Peer Disconnected (Gy)​

Symptom:

  • Web UI → /diameter shows OCS peer "disconnected"
  • Sessions cannot be charged (online charging fails)
  • Logs: "Gy peer connection failed"

Diagnosis and Resolution:

Similar to PCRF Peer Disconnected, but for Gy interface.

Key differences:

  • Port: Typically TCP 3868 (same as Gx)
  • Impact: Charging fails, sessions may be rejected or allowed without charging (depends on config)
  • Configuration: Check diameter.peer_list for OCS entry

See: Diameter Gy Interface for Gy-specific troubleshooting

IP Allocation Issues​

Issue: IP Pool Exhausted​

Symptom:

  • Create Session Request rejected with cause "No resources available"
  • Metric: address_registry_count equals configured pool size
  • Web UI → /pgw_sessions shows many active sessions
  • Logs: "IP allocation failed: pool exhausted"

Likely Causes:

  1. Pool too small for subscriber base
  2. Sessions not releasing IPs (Delete Session failures)
  3. Rapid session churn without cleanup
  4. IP address leak

Diagnosis:

  1. Check pool utilization:

    # For /24 subnet (254 IPs)
    (address_registry_count / 254) * 100

  2. Check configured pool size:

    • Review config/runtime.exs under ue.subnet_map
    • Example: "10.0.0.0/24" = 254 usable IPs

  3. Compare session count to IP count:

    # Should be approximately equal
    teid_registry_count
    address_registry_count

  4. Review active sessions:

    • Web UI → /pgw_sessions
    • Sort by session start time
    • Look for very old sessions (potential leaks)

Resolution:

Immediate (expand pool):

  1. Edit config/runtime.exs (each APN maps to a list of CIDR strings): 
    config :pgw_c,
      ue: %{
        subnet_map: %{
          "internet" => ["10.0.0.0/22"]  # 1022 IPs (was /24 = 254 IPs)
        }
      }
  2. Restart OmniPGW
  3. Verify: Sessions can now establish

Long-term (cleanup):

  1. Identify stale sessions in Web UI
  2. Coordinate with SGW-C to send Delete Session Requests
  3. Implement session timeout policy on PCRF/SGW
  4. Monitor address_registry_count to verify pool freed up after cleanup

Prevention:

  • Monitor IP pool utilization with alerts:
    • Warning: > 70%
    • Critical: > 85%
  • Trend analysis to predict exhaustion
  • Implement session idle timeout
  • Regular session audits

Issue: Duplicate IP Address Assigned​

Symptom:

  • UE reports IP address conflict
  • Logs: "IP already allocated" warning
  • Two sessions in Web UI with same IP address

Likely Causes:

  1. Software bug (rare)
  2. Database inconsistency after crash
  3. Manual intervention error

Diagnosis:

  1. Search for IP in Web UI:

    • /pgw_sessions → Search by IP address
    • Check if multiple IMSIs have same IP

  2. Check logs:

    • Search for IP address
    • Look for "IP allocation" events

Resolution:

  1. Identify affected sessions:

    • Note both IMSIs with duplicate IP

  2. Delete one session:

    • Coordinate with SGW-C to send Delete Session Request for one IMSI
    • Prefer deleting the newer session

  3. UE reconnects:

    • UE should automatically reconnect
    • Will receive new unique IP

  4. If persistent:

    • Restart OmniPGW to rebuild IP registry
    • All sessions will be lost (coordinate maintenance window)

Prevention:

  • Monitor for duplicate allocations (no built-in metric currently)
  • Regular database integrity checks (if applicable)

Issue: IPv4v6 Session on an APN With No IPv6 Pool​

Symptom:

  • A UE requests an IPv4v6 PDN but the matched APN has only an IPv4 CIDR configured in ue.subnet_map
  • An emergency (SOS) APN comes up IPv4-only (no IPv6 in the PAA); logs show a warning like "No IPv6 pool for emergency APN ...; granting IPv4-only"
  • A regular APN has the session rejected; logs show an error like "No IPv6 pool for APN ...; rejecting IPv4v6 request"

Likely Causes:

  1. The APN intentionally has no IPv6 range (common on IPv4-only deployments)
  2. An IPv6 CIDR was omitted from the APN's ue.subnet_map entry by mistake

Behaviour:

  • AddressRegistry.register_new_ipv6/1 returns {:error, :no_ipv6_pool} (it does not crash on an empty pool).
  • Emergency (SOS) APNs (name contains "sos") degrade to IPv4-only — IPv4 address kept, IPv6 cleared, PDN type set to IPv4, and the P-CSCF is still returned in the PCO so emergency IMS registration works.
  • Regular APNs are rejected (no silent downgrade), surfacing the unsupported request / misconfiguration.

Resolution:

  1. If the APN should offer dual-stack, add an IPv6 CIDR to its ue.subnet_map entry and restart OmniPGW.
  2. If the IPv4-only behaviour is intended (e.g. an SOS APN on an IPv4-only network), no action is required.

See IPv4v6 request when the APN has no IPv6 pool and UE IP Allocation → Issue 4.

Quick Reference​

Common Prometheus Queries​

# Active sessions
teid_registry_count

# Session setup rate (per second)
rate(s5s8_inbound_messages_total{message_type="create_session_request"}[5m])

# IP pool utilization (for /24 subnet)
(address_registry_count / 254) * 100

# P95 session setup latency
histogram_quantile(0.95, rate(s5s8_inbound_handling_duration_bucket{request_message_type="create_session_request"}[5m]))

# Error rate
rate(s5s8_inbound_errors_total[5m])

# PCRF latency
histogram_quantile(0.95, rate(gx_outbound_transaction_duration_bucket[5m]))

# PFCP association status
pfcp_peer_associated

Common Log Filters (Web UI)​

FilterPurpose
IMSIFind all logs for specific subscriber
"create_session"Session establishment flow
"delete_session"Session teardown flow
"Credit Control"Gx PCRF interactions
"PFCP Session"User plane programming
"error"All error messages
"timeout"Timeout issues
"Association"PFCP association events

Health Check Commands​

# Check service status
systemctl status omnipgw_c

# Check web UI
curl http://<omnipgw_ip>:4000

# Check metrics endpoint
curl http://<omnipgw_ip>:9090/metrics

# Check active sessions
curl http://<omnipgw_ip>:9090/metrics | grep teid_registry_count

# Check PFCP association
curl http://<omnipgw_ip>:9090/metrics | grep pfcp_peer_associated

# Check IP pool usage
curl http://<omnipgw_ip>:9090/metrics | grep address_registry_count

Back to Operations Guide

OmniPGW Troubleshooting Guide - by Omnitouch Network Services

Last updated on