Skip to main content

Monitoring Guide

Table of Contents​

  1. Overview
  2. Statistics Monitoring
  3. Capacity Monitoring
  4. Performance Metrics
  5. Alerting and Thresholds
  6. Capacity Planning
  7. Troubleshooting Performance Issues

Overview​

Effective monitoring of OmniUPF is critical for maintaining service quality, preventing capacity exhaustion, and troubleshooting performance issues. OmniUPF provides comprehensive real-time metrics through its Web UI and REST API.

Monitoring Categories​

CategoryPurposeUpdate FrequencyKey Metrics
Packet StatisticsTrack packet processing rates and errorsReal-timeRX/TX packets, drops, protocol breakdown
Interface StatisticsMonitor N3/N6 traffic distributionReal-timeN3 RX/TX, N6 RX/TX
XDP StatisticsTrack kernel datapath performanceReal-timeXDP processed, passed, dropped, aborted
Route StatisticsMonitor packet routing decisionsReal-timeFIB lookups, cache hits/misses
eBPF Map CapacityPrevent resource exhaustionEvery 10sMap usage percentages, used vs. capacity
Buffer StatisticsTrack packet buffering during mobilityEvery 5sBuffered packets, buffer age, FAR count

Statistics Monitoring​

N3/N6 Interface Statistics​

N3/N6 interface statistics provide visibility into traffic distribution between the RAN (N3) and Data Network (N6).

Metrics:

  • RX N3: Packets received from RAN (uplink GTP-U traffic)
  • TX N3: Packets transmitted to RAN (downlink GTP-U traffic)
  • RX N6: Packets received from Data Network (downlink native IP)
  • TX N6: Packets transmitted to Data Network (uplink native IP)
  • Total: Aggregate packet count across all interfaces

Expected Behavior:

  • RX N3 ≈ TX N6: Uplink packets flow from RAN to Data Network
  • RX N6 ≈ TX N3: Downlink packets flow from Data Network to RAN
  • Significant imbalance may indicate:
    • Asymmetric traffic (downloads >> uploads)
    • Packet drops or forwarding errors
    • Routing misconfigurations

N3/N6 Interface Statistics

XDP Statistics​

XDP (eXpress Data Path) statistics show kernel-level packet processing performance.

Metrics:

  • Aborted: XDP program encountered an error (should always be 0)
  • Drop: Packets intentionally dropped by XDP program
  • Pass: Packets passed to network stack for further processing
  • Redirect: Packets directly redirected to output interface
  • TX: Packets transmitted via XDP

Interpretation:

  • Aborted > 0: Critical issue with eBPF program or kernel compatibility
  • Drop > 0: Policy-based drops or invalid packets
  • Pass high: Most packets processed in network stack (normal)
  • Redirect high: Packets forwarded directly (optimal performance)

Packet Statistics​

Detailed packet protocol breakdown and processing counters.

Protocol Counters:

  • RX ARP: Address Resolution Protocol packets
  • RX GTP ECHO: GTP-U Echo Request/Response (keepalive)
  • RX GTP OTHER: Other GTP control messages
  • RX GTP PDU: GTP-U encapsulated user data (main traffic)
  • RX GTP UNEXP: Unexpected GTP packet types
  • RX ICMP: Internet Control Message Protocol (ping, errors)
  • RX ICMP6: ICMPv6 packets
  • RX IP4: IPv4 packets
  • RX IP6: IPv6 packets
  • RX OTHER: Other protocols
  • RX TCP: Transmission Control Protocol packets
  • RX UDP: User Datagram Protocol packets

Use Cases:

  • Monitor GTP-U PDU count: Primary user traffic indicator
  • Check ICMP for connectivity: Network reachability testing
  • Track TCP vs UDP ratio: Application traffic patterns
  • Detect unexpected protocols: Security or misconfiguration issues

Route Statistics​

FIB (Forwarding Information Base) lookup statistics for routing decisions.

IPv4 FIB Lookup:

  • Cache: Cached route lookups (fast path)
  • OK: Successful route lookups

IPv6 FIB Lookup:

  • Cache: Cached IPv6 route lookups
  • OK: Successful IPv6 route lookups

Performance Indicators:

  • High Cache Hit Rate: Indicates good routing cache performance
  • High OK Count: Confirms routing tables are correctly configured
  • Low or Zero Lookups: May indicate traffic not flowing or routing bypass

Capacity Monitoring​

eBPF Map Capacity​

eBPF map capacity monitoring prevents session establishment failures due to resource exhaustion.

Critical eBPF Maps​

far_map (Forwarding Action Rules):

  • Capacity: 131,070 entries
  • Key Size: 4 B (FAR ID)
  • Value Size: 16 B (forwarding parameters)
  • Memory Usage: ~2.6 MB
  • Criticality: High - Used for all packet forwarding decisions

pdr_map_downlin (Downlink PDRs - IPv4):

  • Capacity: 131,070 entries
  • Key Size: 4 B (UE IPv4 address)
  • Value Size: 208 B (PDR info)
  • Memory Usage: ~27 MB
  • Criticality: Critical - Session establishment fails if full

pdr_map_downlin_ip6 (Downlink PDRs - IPv6):

  • Capacity: 131,070 entries
  • Key Size: 16 B (UE IPv6 address)
  • Value Size: 208 B (PDR info)
  • Memory Usage: ~29 MB
  • Criticality: Critical - IPv6 session establishment fails if full

pdr_map_teid_ip (Uplink PDRs):

  • Capacity: 131,070 entries
  • Key Size: 4 B (TEID)
  • Value Size: 208 B (PDR info)
  • Memory Usage: ~27 MB
  • Criticality: Critical - Uplink traffic fails if full

qer_map (QoS Enforcement Rules):

  • Capacity: 65,535 entries
  • Key Size: 4 B (QER ID)
  • Value Size: 32 B (QoS parameters)
  • Memory Usage: ~2.3 MB
  • Criticality: Medium - QoS enforcement only

urr_map (Usage Reporting Rules):

  • Capacity: 131,070 entries
  • Key Size: 4 B (URR ID)
  • Value Size: 16 B (volume counters)
  • Memory Usage: ~2.6 MB
  • Criticality: Low - Affects charging only

eBPF Map Capacity View

Capacity Thresholds​

ThresholdAction Required
0-50% (Green)Normal operation - No action required
50-70% (Yellow)Caution - Monitor growth trends, plan capacity increase
70-90% (Amber)Warning - Schedule capacity increase within 1 week
90-100% (Red)Critical - Immediate action required, new sessions will fail

Capacity Increase Procedure​

Before increasing capacity:

  1. Review current usage trends
  2. Estimate future growth rate
  3. Calculate required capacity

Steps to increase map capacity:

  1. Stop OmniUPF service
  2. Update UPF configuration file with new map sizes
  3. Restart OmniUPF service
  4. Verify new capacity in Capacity view
  5. Monitor for successful session establishment

Note: Changing eBPF map capacity requires UPF restart and clears all existing sessions.

Performance Metrics​

For detailed information about all Prometheus metrics exposed by OmniUPF, see the Metrics Reference.

Packet Processing Rate​

Calculation:

Packet Rate (pps) = (Packet Count Delta) / (Time Delta in seconds)

Example:

  • Initial RX packets: 7,000
  • After 10 seconds: 17,000
  • Packet Rate = (17,000 - 7,000) / 10 = 1,000 pps

Performance Targets:

  • Small UPF: 10,000 - 100,000 pps
  • Medium UPF: 100,000 - 1,000,000 pps
  • Large UPF: 1,000,000 - 10,000,000 pps

Bottleneck Indicators:

  • XDP aborted count increasing
  • High CPU utilization
  • Packet drops increasing
  • Latency increasing

Throughput Calculation​

Calculation:

Throughput (Mbps) = (Byte Count Delta × 8) / (Time Delta in seconds × 1,000,000)

Example:

  • Initial RX bytes: 500 MB
  • After 60 seconds: 800 MB
  • Throughput = (300 MB × 8) / (60 × 1,000,000) = 40 Mbps

Per-direction byte counters are exposed directly as Prometheus metrics (upf_n3_rx_bytes, upf_n3_tx_bytes, upf_n6_rx_bytes, upf_n6_tx_bytes) and as JSON at /api/v1/xdp_stats. N3 is the GTP-U side (gNB), N6 is the SGi side (data network). In Grafana the calculation above reduces to a rate():

rate(upf_n3_tx_bytes[1m]) * 8   # downlink bits/sec toward the UE
rate(upf_n3_rx_bytes[1m]) * 8   # uplink bits/sec from the UE

See the Metrics Reference for the full list.

Capacity Planning:

  • Monitor peak throughput times (e.g., evening hours)
  • Compare to link capacity (N3/N6 interface speeds)
  • Plan for 2x peak throughput for headroom

Drop Rate​

Calculation:

Drop Rate (%) = (Dropped Packets / Total RX Packets) × 100

Acceptable Thresholds:

  • < 0.1%: Excellent (normal packet loss due to errors)
  • 0.1% - 1%: Good (minor issues or rate limiting)
  • 1% - 5%: Poor (investigate QoS or capacity issues)
  • > 5%: Critical (major forwarding or capacity problem)

Common Drop Causes:

  • QER rate limiting (MBR exceeded)
  • eBPF map lookup failures
  • Invalid TEIDs or UE IPs
  • Routing errors

Alerting and Thresholds​

Critical Alerts (Immediate response required):

  • eBPF map capacity > 90%
  • XDP aborted count > 0
  • Drop rate > 5%
  • UPF health check failed

Warning Alerts (Response within 1 hour):

  • eBPF map capacity > 70%
  • Drop rate > 1%
  • Packet rate approaching link capacity
  • Buffer TTL exceeded (packets older than 30s)

Informational Alerts (Monitor trends):

  • eBPF map capacity > 50%
  • Buffered packet count increasing
  • New PFCP associations established/released
  • URR volume thresholds exceeded

Alert Configuration​

Alerts can be configured via:

  1. Prometheus Metrics: Export metrics for external monitoring (see Metrics Reference for complete list)
  2. Log Monitoring: Parse OmniUPF logs for error patterns
  3. REST API Polling: Periodically query /map_info, /packet_stats endpoints
  4. Web UI Monitoring: Manual monitoring via Statistics and Capacity pages

Capacity Planning​

Session Capacity Estimation​

Calculate maximum sessions:

Max Sessions = min(
  PDR Map Capacity / 2,  # Downlink + Uplink PDRs per session
  FAR Map Capacity / 2,  # Downlink + Uplink FARs per session
  QER Map Capacity       # Optional, one QER per session
)

Example:

  • PDR Map Capacity: 131,070
  • FAR Map Capacity: 131,070
  • QER Map Capacity: 65,535

Max Sessions = min(131,070 / 2, 131,070 / 2, 65,535) = 65,535 sessions

Memory Capacity​

Calculate total eBPF map memory:

Memory = Σ (Map Capacity × (Key Size + Value Size))

Example Configuration:

  • PDR maps: 3 × 131,070 × 212 B = 83.3 MB
  • FAR map: 131,070 × 20 B = 2.6 MB
  • QER map: 65,535 × 36 B = 2.3 MB
  • URR map: 131,070 × 20 B = 2.6 MB
  • Total: ~91 MB of kernel memory

Kernel Memory Considerations:

  • Ensure sufficient locked memory limit (ulimit -l)
  • Reserve 2x estimated usage for safety margin
  • Monitor kernel memory availability

Traffic Capacity​

Calculate required throughput capacity:

  1. Estimate average session throughput:

    • Video streaming: ~5 Mbps
    • Web browsing: ~1 Mbps
    • VoIP: ~0.1 Mbps

  2. Calculate aggregate throughput:

    Total Throughput = Sessions × Average Session Throughput

  3. Add headroom:

    Required Capacity = Total Throughput × 2  # 100% headroom

Example:

  • 10,000 concurrent sessions
  • Average 2 Mbps per session
  • Total: 20 Gbps
  • Required capacity: 40 Gbps (N3 + N6 interfaces)

Growth Planning​

Trend Analysis:

  1. Record daily peak session count
  2. Calculate weekly growth rate
  3. Extrapolate to capacity limit

Growth Rate Formula:

Weeks to Capacity = (Capacity - Current Usage) / (Weekly Growth)

Example:

  • Current sessions: 30,000
  • Capacity: 65,535 sessions
  • Weekly growth: 2,000 sessions
  • Weeks to capacity: (65,535 - 30,000) / 2,000 = 17.8 weeks

Action: Plan capacity upgrade in 12 weeks (leaving 5 weeks buffer).

Troubleshooting Performance Issues​

High Packet Drop Rate​

Symptoms: Drop rate > 1%, user complaints of poor connectivity

Diagnosis:

  1. Check Statistics → Packet Statistics
  2. Identify if drops are protocol-specific
  3. Review XDP Statistics for XDP drops vs. aborts

Common Causes:

  • QER Rate Limiting: Check QER MBR values vs. actual traffic
  • Invalid TEIDs: Verify uplink PDR TEID matches gNB assignment
  • Unknown UE IPs: Verify downlink PDR exists for UE IP
  • Buffer Overflow: Check buffer statistics

Resolution:

  • Increase QER MBR if rate limiting
  • Verify SMF has created correct PDRs
  • Clear buffers if overflow detected

XDP Processing Errors​

Symptoms: XDP aborted > 0

Diagnosis:

  1. Navigate to Statistics → XDP Statistics
  2. Check aborted counter
  3. Review OmniUPF logs for eBPF errors

Common Causes:

  • eBPF program verification failure
  • Kernel version incompatibility
  • eBPF map access errors
  • Memory corruption

Resolution:

  • Restart OmniUPF service
  • Check kernel version meets minimum requirements (Linux 5.4+)
  • Review eBPF program logs
  • Contact support if issue persists

Capacity Exhaustion​

Symptoms: Session establishment failures, map capacity at 100%

Diagnosis:

  1. Navigate to Capacity page
  2. Identify which map is at 100%
  3. Check if sessions are stuck (not being deleted)

Immediate Mitigation:

  1. Identify stale sessions (check Sessions page)
  2. Request SMF to delete old sessions
  3. Clear buffers to free FAR entries

Long-term Resolution:

  1. Increase eBPF map capacity
  2. Schedule UPF restart with larger maps
  3. Implement session cleanup policies

Performance Degradation​

Symptoms: High latency, low throughput, CPU saturation

Diagnosis:

  1. Check packet rate vs. historical baseline
  2. Review XDP statistics for processing delays
  3. Monitor CPU utilization on UPF host
  4. Check N3/N6 interface utilization

Common Causes:

  • Traffic exceeding UPF capacity
  • Insufficient CPU cores for packet processing
  • Network interface bottleneck
  • eBPF map hash collisions

Resolution:

  • Scale UPF horizontally (add more instances)
  • Upgrade CPU or enable RSS (Receive Side Scaling)
  • Upgrade network interfaces to higher speed
  • Tune eBPF map hash function
Last updated on