Skip to main content

XDP Attachment Modes for OmniUPF

Table of Contents​

  1. Overview
  2. XDP Mode Comparison
  3. Generic Mode (Default)
  4. Native Mode (Recommended for Production)
  5. Offload Mode (SmartNIC)
  6. Enabling Native XDP on Proxmox VE
  7. Enabling Native XDP on Other Hypervisors
  8. Verifying XDP Mode
  9. Troubleshooting XDP Issues

Overview​

OmniUPF uses XDP (eXpress Data Path) for high-performance packet processing. XDP is a Linux kernel technology that allows packet processing programs (eBPF) to run at the earliest possible point in the network stack, providing microsecond-level latency and millions of packets per second throughput.

The XDP attachment mode determines where in the packet path the eBPF program executes:

Choosing the right XDP mode significantly impacts OmniUPF performance and determines whether you can achieve production-grade packet processing.

XDP Mode Comparison​

AspectGeneric ModeNative ModeOffload Mode
Attach PointLinux network stackNetwork driverNIC hardware
Performance~1-2 Mpps~5-10 Mpps~10-40 Mpps
Latency~100 μs~10 μs~1 μs
CPU UsageHighMediumLow
NIC RequirementsAny NICXDP-capable driverSmartNIC with XDP support
Hypervisor SupportAll hypervisorsMost (requires multi-queue)Rare (PCI passthrough)
Use CaseTesting, developmentProduction (recommended)High-throughput edge sites
Configurationxdp_attach_mode: genericxdp_attach_mode: nativexdp_attach_mode: offload

Recommendation: Use native mode for production deployments. Generic mode is only suitable for testing.

Generic Mode (Default)​

Description​

Generic XDP runs the eBPF program in the Linux network stack after the driver has processed the packet. This is the slowest XDP mode but works with any network interface.

Performance Characteristics​

  • Throughput: ~1-2 million packets per second (Mpps)
  • Latency: ~100 microseconds per packet
  • CPU Overhead: High (packet copied to kernel stack before XDP)

When to Use​

  • Development and testing only
  • Lab environments where performance doesn't matter
  • Initial deployment to verify functionality before optimizing

Configuration​

# /etc/omniupf/runtime.exs
xdp_interfaces = "eth0"
xdp_attach_mode = "generic"    # Default mode

Warning: Generic mode is not suitable for production. It will bottleneck at high packet rates and waste CPU resources.

Description​

Native XDP runs the eBPF program inside the network driver, before packets reach the Linux network stack. This provides near-hardware performance while maintaining kernel-level flexibility.

Performance Characteristics​

  • Throughput: ~5-10 million packets per second (Mpps) per core
  • Latency: ~10 microseconds per packet
  • CPU Overhead: Low (packet processed at driver level)
  • Scaling: Linear scaling with CPU cores and NIC queues

When to Use​

  • Production deployments (recommended)
  • Carrier-grade networks requiring high throughput
  • Edge computing scenarios with performance requirements
  • Any deployment where performance matters

NIC Driver Requirements​

Native XDP requires a network driver with XDP support. Most modern NICs support native XDP:

Physical NICs (bare metal):

  • Intel: ixgbe (10G), i40e (40G), ice (100G)
  • Broadcom: bnxt_en
  • Mellanox: mlx4_en, mlx5_core
  • Netronome: nfp (with offload support)
  • Marvell: mvneta, mvpp2

Virtual NICs (hypervisors):

  • VirtIO: virtio_net (KVM, Proxmox, OpenStack) ✓
  • VMware: vmxnet3 ✓
  • Microsoft: hv_netvsc (Hyper-V) ✓
  • Amazon: ena (AWS) ✓
  • SR-IOV: ixgbevf, i40evf (PCI passthrough) ✓

Note: VirtualBox does not support native XDP (use generic mode only).

Configuration​

# /etc/omniupf/runtime.exs
xdp_interfaces = "eth0"
xdp_attach_mode = "native"

Multi-Queue Requirement: For optimal performance, enable multi-queue on virtual NICs (see Proxmox section below).

Offload Mode (SmartNIC)​

Description​

Offload XDP runs the eBPF program directly on the NIC hardware (SmartNIC), completely bypassing the CPU for packet processing. This provides the highest performance but requires specialized hardware.

Performance Characteristics​

  • Throughput: ~10-40 million packets per second (Mpps)
  • Latency: ~1 microsecond per packet
  • CPU Overhead: Near-zero (processing on NIC)

When to Use​

  • Ultra-high-throughput deployments (10G+ per UPF instance)
  • Edge sites with hardware acceleration
  • Cost-sensitive deployments (reduce CPU requirements)

Hardware Requirements​

Only Netronome Agilio SmartNICs currently support XDP offload:

  • Netronome Agilio CX 10G/25G/40G/100G

Note: Offload mode requires bare metal or PCI passthrough - not available in standard VM configurations.

Configuration​

# /etc/omniupf/runtime.exs
xdp_interfaces = "eth0"
xdp_attach_mode = "offload"

Enabling Native XDP on Proxmox VE​

Proxmox VE uses VirtIO network devices for VMs, which support native XDP via the virtio_net driver. However, you must enable multi-queue for optimal performance.

Step 1: Understanding the Requirement​

Why Multi-Queue Matters:

  • Single queue (default): All network traffic processed by one CPU core → bottleneck
  • Multi-queue: Traffic distributed across multiple CPU cores → linear scaling

Step 2: Enable Multi-Queue in Proxmox​

Option A: Via Proxmox Web UI​

  1. Shutdown the VM completely (not just reboot)

    • Select your VM in the Proxmox web interface
    • Click Shutdown

  2. Edit Network Device

    • Go to Hardware tab
    • Click on your network device (e.g., net0)
    • Click Edit

  3. Set Multiqueue

    • Find the "Multiqueue" field
    • Set to 8 (or match your vCPU count, max 16)
    • Click OK

  4. Start the VM

    • Click Start

Option B: Via Proxmox Command Line​

# SSH to your Proxmox host

# Find your VM ID
qm list

# Set multi-queue (replace XXX with your VM ID)
qm set XXX -net0 virtio=XX:XX:XX:XX:XX:XX,bridge=vmbr0,queues=8

# Example for VM 191 with MAC BC:24:11:1D:BA:00
qm set 191 -net0 virtio=BC:24:11:1D:BA:00,bridge=vmbr0,queues=8

# Shutdown the VM
qm shutdown XXX

# Wait for shutdown, then start
qm start XXX

Queue Count Recommendations:

  • 4 queues: Minimum for production (good for 2-4 vCPU VMs)
  • 8 queues: Recommended for most deployments (4-8 vCPU VMs)
  • 16 queues: Maximum for high-performance (8+ vCPU VMs)

Step 3: Verify Multi-Queue Inside VM​

After VM restart, SSH into the VM and verify:

# Check queue configuration
ethtool -l eth0

# Expected output:
# Channel parameters for eth0:
# Combined:	8        <-- Should match your configured value

# Count actual queues
ls -1d /sys/class/net/eth0/queues/rx-* | wc -l
ls -1d /sys/class/net/eth0/queues/tx-* | wc -l

# Both should show 8 (or your configured value)

Step 4: Enable Native XDP in OmniUPF​

Edit the OmniUPF configuration:

# Edit config file
sudo nano /etc/omniupf/runtime.exs

Change XDP mode:

# Before
xdp_attach_mode = "generic"

# After
xdp_attach_mode = "native"

Restart OmniUPF:

sudo systemctl restart omniupf

Step 5: Verify Native XDP is Active​

Check logs:

# View startup logs
journalctl -u omniupf --since "1 minute ago" | grep -i "xdp\|attach"

# Expected output:
# xdp_attach_mode:native
# XDPAttachMode:native
# Attached XDP program to iface "eth0" (index 2)

Check via API:

# Query configuration
curl -s http://localhost:8080/api/v1/config | grep xdp_attach_mode

# Expected output:
# "xdp_attach_mode": "native",

Common Proxmox Issues​

Issue: "Failed to attach XDP program"

Solution:

  • Verify multi-queue is enabled (ethtool -l eth0)
  • Check kernel version: uname -r (must be ≥ 5.15)
  • Ensure VirtIO driver loaded: lsmod | grep virtio_net

Issue: Only 1 queue despite configuration

Solution:

  • VM must be fully shutdown (not rebooted) for queue changes
  • Use qm shutdown XXX && sleep 5 && qm start XXX
  • Verify in Proxmox config: grep net0 /etc/pve/qemu-server/XXX.conf

Issue: Performance not improving with native mode

Solution:

  • Check CPU pinning (avoid oversubscription)
  • Monitor top - CPU usage should spread across cores
  • Verify XDP stats: curl http://localhost:8080/api/v1/xdp_stats

Enabling Native XDP on Other Hypervisors​

VMware ESXi / vSphere​

VMware uses vmxnet3 driver which supports native XDP.

Requirements:

  • ESXi 6.7 or later
  • vmxnet3 driver version 1.4.16+ in VM
  • VM hardware version 14 or later

Enable Multi-Queue:

  1. Power off the VM

  2. Edit VM settings:

    • Right-click VM → Edit Settings
    • Network Adapter → Advanced
    • Set Receive Side Scaling to Enabled

  3. Edit .vmx file (optional, for more queues):

    ethernet0.pnicFeatures = "4"
    ethernet0.multiqueue = "8"

  4. Start VM and verify:

    ethtool -l ens192  # Check queue count

Configure OmniUPF (in runtime.exs):

xdp_interfaces = "ens192"    # VMware typically uses ens192
xdp_attach_mode = "native"

KVM / libvirt (Raw)​

Enable Multi-Queue via virsh:

# Edit VM configuration
virsh edit your-vm-name

Add to network interface section:

<interface type='network'>
  <source network='default'/>
  <model type='virtio'/>
  <driver name='vhost' queues='8'/>
</interface>

Restart VM and verify:

ethtool -l eth0

Microsoft Hyper-V​

Hyper-V uses hv_netvsc driver which supports native XDP.

Requirements:

  • Windows Server 2016 or later
  • Linux Integration Services 4.3+ in VM
  • Generation 2 VM

Enable Multi-Queue:

PowerShell on Hyper-V host:

# Set VMQ (Virtual Machine Queue) - Hyper-V's multi-queue
Set-VMNetworkAdapter -VMName "YourVM" -VrssEnabled $true -VmmqEnabled $true

Configure OmniUPF (in runtime.exs):

xdp_interfaces = "eth0"
xdp_attach_mode = "native"

VirtualBox​

Warning: VirtualBox does NOT support native XDP.

Reason: VirtualBox network drivers (e1000, virtio-net) do not implement XDP hooks.

Workaround: Use generic mode only:

xdp_attach_mode = "generic"    # Only option for VirtualBox

Verifying XDP Mode​

After configuring native XDP, verify it's working correctly:

1. Check OmniUPF Logs​

# View recent logs
journalctl -u omniupf --since "5 minutes ago" | grep -i xdp

# Look for:
# ✓ "xdp_attach_mode:native"
# ✓ "Attached XDP program to iface"
# ✗ "Failed to attach" or "falling back to generic"

2. Check via API​

# Query configuration endpoint
curl -s http://localhost:8080/api/v1/config | jq .xdp_attach_mode

# Expected output:
# "native"

3. Check XDP Statistics​

# View XDP processing stats
curl -s http://localhost:8080/api/v1/xdp_stats | jq

# Example output:
{
  "xdp_aborted": 0,      # Should be 0 (errors)
  "xdp_drop": 1234,      # Dropped packets
  "xdp_pass": 5678,      # Passed to stack
  "xdp_redirect": 9012,  # Redirected packets
  "xdp_tx": 3456         # Transmitted packets
}

4. Verify Driver Support​

# Check if driver supports XDP
ethtool -i eth0 | grep driver

# For Proxmox/KVM: Should show "virtio_net"
# For VMware: Should show "vmxnet3"
# For Hyper-V: Should show "hv_netvsc"

5. Performance Test​

Compare packet processing before and after:

# Monitor packet rate
watch -n 1 'curl -s http://localhost:8080/api/v1/packet_stats | jq .rx_packets'

# Generic mode: ~1-2 Mpps
# Native mode: ~5-10 Mpps (5-10x improvement)

Troubleshooting XDP Issues​

Issue: "Failed to attach XDP program" on Startup​

Symptoms:

Error: failed to attach XDP program to interface eth0

Diagnosis:

  1. Check driver support:

    ethtool -i eth0 | grep driver

    # If driver is not virtio_net/vmxnet3/hv_netvsc, native XDP won't work

  2. Check kernel version:

    uname -r

    # Must be >= 5.15 for reliable XDP support

  3. Check for existing XDP programs:

    ip link show eth0 | grep xdp

    # If another XDP program is attached, unload it first
    ip link set dev eth0 xdp off

Solution:

  • Update kernel to 5.15+ if older
  • Ensure virtio_net driver is loaded: modprobe virtio_net
  • Fall back to generic mode if driver doesn't support native XDP

Issue: Native Mode Falls Back to Generic​

Symptoms:

Warning: falling back to generic XDP mode

Diagnosis:

Check dmesg for driver errors:

dmesg | grep -i xdp | tail -20

Common causes:

  1. Driver doesn't support native XDP:

    • VirtualBox drivers (no native XDP support)
    • Older NIC drivers

  2. Multi-queue not enabled:

    • Check: ethtool -l eth0
    • Should show > 1 combined queue

  3. Kernel XDP support disabled:

    # Check if XDP is enabled in kernel
    grep XDP /boot/config-$(uname -r)

    # Should show:
    # CONFIG_XDP_SOCKETS=y
    # CONFIG_BPF=y

Solution:

  • Enable multi-queue (see Proxmox section)
  • Update to supported driver
  • Rebuild kernel with XDP support if necessary

Issue: Performance Not Improving with Native Mode​

Symptoms: Native mode enabled but packet rate same as generic mode

Diagnosis:

  1. Verify multi-queue distribution:

    # Check per-queue statistics
    ethtool -S eth0 | grep rx_queue

    # Traffic should be distributed across multiple queues

  2. Check CPU utilization:

    # Monitor CPU usage per core
    mpstat -P ALL 1

    # Should see load spread across multiple CPUs

  3. Verify XDP is actually running in native mode:

    # Check bpftool (if available)
    sudo bpftool net list

    # Should show XDP attached to interface

Solution:

  • Increase queue count (8-16 queues)
  • Enable CPU pinning to prevent core migration
  • Check for CPU oversubscription on hypervisor

Issue: XDP Program Aborted (xdp_aborted > 0)​

Symptoms:

curl http://localhost:8080/api/v1/xdp_stats
{
  "xdp_aborted": 1234,  # Non-zero indicates errors
  ...
}

Diagnosis:

XDP aborted means the eBPF program hit an error during execution.

  1. Check eBPF verifier logs:

    dmesg | grep -i bpf | tail -20

  2. Check for map size limits:

    # eBPF maps may be full
    curl http://localhost:8080/api/v1/map_info

    # Look for maps at 100% capacity

Solution:

  • Increase eBPF map sizes in configuration
  • Check for corrupted packets causing eBPF errors
  • Verify Linux kernel eBPF support is complete

Issue: Multi-Queue Not Working on Proxmox​

Symptoms: ethtool -l eth0 shows only 1 queue despite configuration

Diagnosis:

  1. Check Proxmox VM config:

    # On Proxmox host
    grep net0 /etc/pve/qemu-server/YOUR_VM_ID.conf

    # Should show: queues=8

  2. Verify VM was fully shutdown:

    # On Proxmox host
    qm status YOUR_VM_ID

    # Must show "status: stopped" before starting

Solution:

# On Proxmox host
# Force shutdown and restart
qm shutdown YOUR_VM_ID
sleep 10
qm start YOUR_VM_ID

# Then check inside VM
ethtool -l eth0

Important: Changes to queue count require a full VM shutdown, not just a reboot from inside the VM.

Issue: Permission Denied When Attaching XDP​

Symptoms:

Error: permission denied when attaching XDP program

Diagnosis:

XDP operations require CAP_NET_ADMIN and CAP_SYS_ADMIN capabilities.

Solution:

  1. Run OmniUPF as root (or with capabilities):

    sudo systemctl restart omniupf

  2. If using systemd, verify service file has capabilities:

    # /lib/systemd/system/omniupf.service
    [Service]
    CapabilityBoundingSet=CAP_NET_ADMIN CAP_SYS_ADMIN CAP_NET_RAW
    AmbientCapabilities=CAP_NET_ADMIN CAP_SYS_ADMIN CAP_NET_RAW

  3. If using Docker, run with --privileged:

    docker run --privileged -v /sys/fs/bpf:/sys/fs/bpf ...

Performance Impact Summary​

Real-world performance comparison for OmniUPF packet processing:

ScenarioGeneric ModeNative ModeImprovement
Packet Rate1.5 Mpps8.2 Mpps5.5x faster
Latency95 μs12 μs8x lower
CPU Usage (1 Gbps)85% (1 core)15% (distributed)5x more efficient
Max Throughput~1.2 Gbps~10 Gbps8x higher

Recommendation: Always use native mode with multi-queue enabled for production deployments.

Hardware Recommendations for XDP​

⚠️ IMPORTANT: Before purchasing any hardware, consult with Omnitouch support to confirm it's 100% compatible with your specific configuration and deployment requirements.

Known Good NICs for Native XDP​

These NICs are verified to support native XDP mode with OmniUPF:

ModelSpeedDriverXDP SupportNotes
Intel X52010GbEixgbeNative ✓Proven, widely available, good price/performance
Intel X71010/40GbEi40eNative ✓Excellent multi-queue support
Intel E810100GbEiceNative ✓Latest generation, best performance
Intel i3501GbEigbNative ✓ (kernel 5.10+)Good for lower bandwidth needs

Mellanox/NVIDIA NICs (High Performance)​

ModelSpeedDriverXDP SupportNotes
ConnectX-425/50/100GbEmlx5Native ✓High throughput, good for edge computing
ConnectX-525/50/100GbEmlx5Native ✓Excellent performance, hardware acceleration
ConnectX-650/100/200GbEmlx5Native ✓Latest generation, best for ultra-high throughput
BlueField-2100/200GbEmlx5Native ✓SmartNIC with DPU capabilities

Broadcom NICs​

ModelSpeedDriverXDP SupportNotes
BCM57xxx series10/25/50GbEbnxt_enNative ✓Common in Dell/HP servers

Virtual NICs (VM Deployments)​

PlatformNIC TypeDriverXDP SupportMulti-QueueNotes
Proxmox/KVMVirtIOvirtio_netNative ✓Yes (configurable)Best for VMs
VMware ESXivmxnet3vmxnet3Native ✓YesRequires ESXi 6.7+
Hyper-VSynthetic NIChv_netvscNative ✓YesWindows Server 2016+
AWSENAenaNative ✓YesEC2 metal instances
VirtualBoxAnyvariousGeneric only ❌NoNot recommended for production

NICs with Hardware Offload Support​

True XDP hardware offload (eBPF runs on NIC):

VendorModelSpeedNotes
NetronomeAgilio CX 10G10GbEOnly confirmed XDP offload support
NetronomeAgilio CX 25G25GbERequires special firmware
NetronomeAgilio CX 40G40GbEVery expensive (~$2,500-5,000)
NetronomeAgilio CX 100G100GbEEnterprise-grade only

Note: Hardware offload NICs are rare, expensive, and require bare metal deployment. Most deployments should use native XDP instead.

Tested Configurations​

These configurations have been verified with OmniUPF in production:

Budget Option (1-10 Gbps)​

  • NIC: Intel X520 (10GbE dual-port)
  • Mode: Native XDP
  • Throughput: ~8-10 Gbps per UPF instance
  • Cost: ~$100-200 (used/refurbished)

Mid-Range (10-50 Gbps)​

  • NIC: Intel X710 (40GbE) or Mellanox ConnectX-4 (25GbE)
  • Mode: Native XDP
  • Throughput: ~25-40 Gbps per UPF instance
  • Cost: ~$300-800

High-End (50-100+ Gbps)​

  • NIC: Mellanox ConnectX-5/6 (100GbE)
  • Mode: Native XDP
  • Throughput: ~80-100 Gbps per UPF instance
  • Cost: ~$1,000-2,500

VM Deployments (Proxmox/KVM)​

  • NIC: VirtIO with 8-16 queues
  • Mode: Native XDP
  • Throughput: ~5-10 Gbps per UPF instance
  • Cost: No additional hardware cost

What NOT to Buy​

Avoid these for production OmniUPF deployments:

NIC/PlatformReasonAlternative
Realtek NICsNo XDP support, poor Linux driversIntel i350 or better
VirtualBoxNo native XDP supportMigrate to Proxmox/KVM
Consumer-grade NICsLimited queue support, unreliableServer-grade Intel/Mellanox
Very old NICs (<2014)No XDP driver supportIntel X520 or newer

Pre-Purchase Checklist​

Before buying hardware, verify:

  1. ✅ Driver Support: Check if Linux driver supports XDP

    # On similar system
    modinfo <driver_name> | grep -i xdp

  2. ✅ Kernel Version: Ensure kernel ≥ 5.15 for reliable XDP

    uname -r

  3. ✅ Multi-Queue: Verify NIC supports multiple queues (RSS/VMDq)

  4. ✅ PCI Bandwidth: Ensure PCIe slot has sufficient lanes

    • 10GbE: PCIe 2.0 x4 minimum
    • 40GbE: PCIe 3.0 x8 minimum
    • 100GbE: PCIe 3.0 x16 or PCIe 4.0 x8

  5. ✅ Deployment Type:

    • Bare metal: Physical NIC required
    • VM: VirtIO or SR-IOV support needed
    • Container: Host NIC configuration inherited

⚠️ Don't buy hardware based solely on this guide - always confirm with Omnitouch support first!

Additional Resources​

Quick Reference​

Proxmox Native XDP Setup (TL;DR)​

# On Proxmox host:
qm set <VM_ID> -net0 virtio=<MAC>,bridge=vmbr0,queues=8
qm shutdown <VM_ID> && sleep 10 && qm start <VM_ID>

# Inside VM:
ethtool -l eth0  # Verify 8 queues
sudo nano /etc/omniupf/runtime.exs  # Set: xdp_attach_mode = "native"
sudo systemctl restart omniupf
journalctl -u omniupf --since "1 min ago" | grep xdp  # Verify native mode

Verify XDP Mode is Active​

# Check configuration
curl -s http://localhost:8080/api/v1/config | grep xdp_attach_mode

# Check statistics
curl -s http://localhost:8080/api/v1/xdp_stats | jq

# Check queues
ethtool -l eth0
Last updated on