Skip to main content

OmniHSS Troubleshooting Guide

← Back to Operations Guide

Table of Contents​

Troubleshooting Overview​

General Troubleshooting Approach​

Information to Gather​

Before troubleshooting any issue, collect:

  1. Subscriber Information (if subscriber-specific)

    • IMSI
    • MSISDN (phone number)
    • Last known state
    • Error messages from device

  2. Timing Information

    • When did the issue start?
    • Is it intermittent or constant?
    • Time of last successful operation

  3. Scope of Impact

    • Single subscriber or multiple?
    • Specific network or all networks?
    • Specific service (data/voice) or both?

  4. System State

    • Check Control Panel for system status
    • Review Diameter peer status
    • Verify database connectivity

Authentication Failures​

Symptoms​

  • Subscriber cannot attach to network
  • "Authentication rejected" errors
  • Repeated authentication attempts

Common Causes and Solutions​

Cause 1: Incorrect Key Set​

Symptoms:

  • Consistent authentication failure for specific subscriber
  • Works for other subscribers with same profile

Diagnostic Steps:

  1. Query subscriber to verify key_set_id:

    curl -k https://hss.example.com:8443/api/subscriber/imsi/[IMSI]

  2. Verify key set exists and has correct values:

    curl -k https://hss.example.com:8443/api/key_set/[KEY_SET_ID]

  3. Compare Ki and OPC values with SIM card documentation

Solution:

  • Update subscriber with correct key set
  • If keys are correct, SIM card may be faulty

Cause 2: SQN Out of Sync​

Symptoms:

  • Authentication fails after previously working
  • Error: "SQN synchronization failure"
  • Works intermittently

Diagnostic Steps:

  1. Check subscriber state for SQN value in database
  2. Look for SQN-related errors in logs
  3. Verify subscriber's key set SQN value

Solution:

  • SQN will automatically resynchronize after subscriber sends AUTS
  • If persistent, reset SQN to 0 in key set (requires subscriber re-attach)

Warning: Resetting SQN can cause security issues. Only do during maintenance.

Cause 3: Subscriber Disabled​

Symptoms:

  • Authentication rejected immediately
  • No authentication vectors generated

Diagnostic Steps:

  1. Check subscriber enabled status:

    curl -k https://hss.example.com:8443/api/subscriber/imsi/[IMSI]

  2. Verify enabled field is true

Solution:

  • Enable subscriber: 
    curl -k -X PUT https://hss.example.com:8443/api/subscriber/[ID] \
      -H "Content-Type: application/json" \
      -d '{"subscriber": {"enabled": true}}'

Cause 4: Missing EPC Profile​

Symptoms:

  • Subscriber lookup succeeds but authentication fails
  • Error: "No EPC profile assigned"

Diagnostic Steps:

  1. Check subscriber's epc_profile_id field
  2. Verify EPC profile exists: 
    curl -k https://hss.example.com:8443/api/epc/profile/[PROFILE_ID]

Solution:

Authentication Troubleshooting Flowchart​

Diameter Connectivity Issues​

Symptoms​

  • Diameter peers showing as disconnected in Control Panel
  • "No route to host" errors
  • Services failing for all subscribers

Common Causes and Solutions​

Cause 1: Network Connectivity​

Symptoms:

  • Peer never connects
  • Connection timeout errors
  • Ping fails to peer

Diagnostic Steps:

  1. Verify network connectivity from OmniHSS to peer:

    ping [PEER_IP]

  2. Check if Diameter port is reachable:

    telnet [PEER_IP] 3868

  3. Verify firewall rules allow Diameter traffic (port 3868)

Solution:

  • Fix network routing
  • Update firewall rules
  • Verify peer is running and listening

Cause 2: Incorrect Diameter Configuration​

Symptoms:

  • Connection attempts fail
  • CER/CEA exchange fails
  • Peer rejects connection

Diagnostic Steps:

  1. Review runtime.exs Diameter configuration:

    • Verify peer origin_host matches peer's expected value
    • Check origin_realm configuration
    • Verify peer IP address is correct

  2. Check logs for CER/CEA errors

  3. Verify peer's configuration expects OmniHSS's origin_host

Solution:

  • Update runtime.exs with correct Diameter configuration
  • Restart OmniHSS after configuration change
  • Coordinate with peer administrator to verify settings

Cause 3: Certificate Issues (TLS Diameter)​

Symptoms:

  • Connection fails during TLS handshake
  • Certificate validation errors
  • "Certificate expired" or "Certificate invalid" errors

Diagnostic Steps:

  1. Verify certificate files exist in priv/cert/

  2. Check certificate expiration:

    openssl x509 -in priv/cert/diameter.crt -noout -dates

  3. Verify certificate chain is complete

  4. Check peer's certificate if mutual TLS

Solution:

  • Renew expired certificates
  • Install correct certificate chain
  • Update certificate files and restart OmniHSS

Cause 4: Peer Application Support Mismatch​

Symptoms:

  • Peer connects but doesn't support required applications
  • Capabilities exchange succeeds but operations fail
  • "Application not supported" errors

Diagnostic Steps:

  1. Check Control Panel Diameter page for peer applications
  2. Verify peer supports required application (S6a, Cx, Sh, etc.)
  3. Review CER/CEA exchange in logs

Solution:

  • Verify peer configuration includes required Diameter applications
  • Check that peer type matches expected functionality:
    • MME must support S6a (16777251)
    • S-CSCF must support Cx (16777216)
    • P-GW must support Gx (16777238)

Diameter Troubleshooting Flowchart​

Database Issues​

Symptoms​

  • API returns 500 errors
  • Control Panel fails to load
  • "Database connection failed" errors
  • Slow query performance

Common Causes and Solutions​

Cause 1: Database Server Down​

Symptoms:

  • All API calls fail
  • Control Panel shows error
  • "Connection refused" errors

Diagnostic Steps:

  1. Test database connectivity:

    # Connect to the database
    psql -h [DB_HOST] -U [DB_USER] -d [DB_NAME]

  2. Check database service status on database server

  3. Verify network connectivity to database server

Solution:

  • Start database service
  • Fix database server issues
  • Verify network routing to database server

Cause 2: Incorrect Database Credentials​

Symptoms:

  • "Authentication failed" errors
  • OmniHSS can't connect at startup

Diagnostic Steps:

  1. Review database configuration in runtime.exs
  2. Test credentials manually with database client
  3. Check database user permissions

Solution:

  • Update database configuration in runtime.exs
  • Grant correct permissions to database user
  • Restart OmniHSS after configuration change

Cause 3: Connection Pool Exhausted​

Symptoms:

  • Intermittent 500 errors
  • "No available connections" errors
  • High load periods trigger failures

Diagnostic Steps:

  1. Check current connection count in database
  2. Review database pool size in runtime.exs
  3. Monitor connection usage during peak load

Solution:

  • Increase pool size in runtime.exs configuration
  • Investigate connection leaks if pool exhausts repeatedly
  • Consider database scaling if load is consistently high

Cause 4: Slow Queries​

Symptoms:

  • API responses very slow
  • Timeouts on subscriber lookups
  • Database CPU high

Diagnostic Steps:

  1. Query database for slow query log
  2. Identify specific slow queries
  3. Check for missing indexes
  4. Verify subscriber count and table sizes

Solution:

  • Optimize slow queries
  • Add missing indexes
  • Consider database performance tuning
  • Plan for database scaling if needed

Database Troubleshooting Flowchart​

EPC Registration Failures​

Symptoms​

  • Subscriber cannot attach to LTE network
  • MME rejects attachment
  • No PDN session established

Common Causes and Solutions​

Cause 1: Roaming Denied​

Symptoms:

  • Subscriber works on home network but fails when roaming
  • "Roaming not allowed" errors
  • Works for some networks but not others

Diagnostic Steps:

  1. Check subscriber's roaming_profile_id
  2. Query roaming profile and rules
  3. Verify MCC/MNC of visited network
  4. Check if roaming rule exists for that network

Solution:

Cause 2: Missing APN Configuration​

Symptoms:

  • Attachment succeeds but PDN session fails
  • "Unknown APN" errors from MME
  • Subscriber can't get data connection

Diagnostic Steps:

  1. Check EPC profile has APN profiles linked
  2. Verify APN identifier matches what device requests
  3. Query APN profile configuration

Solution:

  • Link APN profiles to subscriber's EPC profile
  • Ensure APN name matches device configuration
  • Verify APN QoS profile exists

Cause 3: MME Not Connected​

Symptoms:

  • All subscribers fail to attach
  • No communication with MME
  • Diameter peer down

Diagnostic Steps:

  1. Check Control Panel Diameter page
  2. Verify MME peer status is "Connected"
  3. Check MME supports S6a application

Solution:

Cause 4: Subscriber State Corruption​

Symptoms:

  • Subscriber shows as attached but can't attach again
  • State doesn't match reality
  • Detach and re-attach fails

Diagnostic Steps:

  1. Query subscriber state from database
  2. Check for stale MME assignments
  3. Verify last update timestamp

Solution:

  • Clear subscriber state (detach procedure)
  • Reset serving MME in subscriber state
  • May require subscriber power cycle

EPC Registration Troubleshooting Flowchart​

IMS Registration Failures​

Symptoms​

  • Subscriber can't register for VoLTE
  • "IMS registration failed" on device
  • Data works but voice doesn't

Common Causes and Solutions​

Cause 1: IMS Disabled for Subscriber​

Symptoms:

  • Subscriber has data but no IMS
  • Registration rejected immediately

Diagnostic Steps:

  1. Query subscriber and check ims_enabled field
  2. Verify subscriber has ims_profile_id assigned

Solution:

Cause 2: S-CSCF Not Connected​

Symptoms:

  • All IMS registrations fail
  • No IMS-related Diameter traffic

Diagnostic Steps:

  1. Check Control Panel Diameter page
  2. Verify S-CSCF peer connected
  3. Check S-CSCF supports Cx application

Solution:

Cause 3: Missing or Invalid IFC Template​

Symptoms:

  • Registration fails during User-Authorization-Answer
  • IFC-related errors in logs

Diagnostic Steps:

  1. Query subscriber's IMS profile
  2. Verify IFC template is present
  3. Check IFC XML syntax

Solution:

Cause 4: Roaming Denied for IMS​

Symptoms:

  • IMS works on home network
  • Fails when roaming
  • Data roaming works but not IMS

Diagnostic Steps:

  1. Check roaming profile IMS action
  2. Verify roaming rules have correct ims_action

Solution:

  • Update roaming rules to allow IMS
  • Or update roaming profile default IMS action

IMS Registration Troubleshooting Flowchart​

VoLTE Call Failures​

Symptoms​

  • IMS registration succeeds but calls fail
  • One-way audio
  • Call drops immediately
  • "Call failed" error on device

Common Causes and Solutions​

Cause 1: P-CSCF Not Connected​

Symptoms:

  • Registration works but calls fail
  • Media authorization fails

Diagnostic Steps:

  1. Check Control Panel Diameter page
  2. Verify P-CSCF peer connected
  3. Check P-CSCF supports Rx application (OmniHSS PCRF function)

Solution:

Cause 2: Missing Media Authorization​

Symptoms:

  • Call setup starts but fails
  • AAR/AAA exchange fails
  • Rx interface errors

Diagnostic Steps:

  1. Check logs for Rx Diameter messages
  2. Verify AAR (AA-Request) received
  3. Check AAA (AA-Answer) response

Solution:

  • Verify P-CSCF is sending AAR for media authorization
  • Check OmniHSS Rx application configuration
  • Verify subscriber has active IMS registration

Cause 3: QoS/Bearer Issues​

Symptoms:

  • Call connects but no audio
  • One-way audio
  • Quality issues

Diagnostic Steps:

  1. Check APN QoS profile for voice APN
  2. Verify QCI is set correctly (typically QCI 1 for voice)
  3. Check P-GW is connected for Gx (PCRF function)

Solution:

VoLTE Call Troubleshooting Flowchart​

Roaming Issues​

Symptoms​

  • Subscriber works at home but not when roaming
  • Some roaming networks work, others don't
  • Roaming data works but not voice (or vice versa)

Common Causes and Solutions​

Cause 1: No Roaming Profile Assigned​

Symptoms:

  • Roaming fails for subscriber
  • Other subscribers roam successfully

Diagnostic Steps:

  1. Query subscriber's roaming_profile_id
  2. Check if field is null

Solution:

Cause 2: Roaming Denied by Policy​

Symptoms:

  • Roaming fails consistently on specific network
  • Error indicates policy rejection

Diagnostic Steps:

  1. Identify visited network MCC/MNC from subscriber device or MME
  2. Query subscriber's roaming profile
  3. Check roaming rules for matching MCC/MNC
  4. Check profile's default action

Solution:

  • Add roaming rule to allow visited network: 
    curl -k -X POST https://hss.example.com:8443/api/roaming/rule \
      -H "Content-Type: application/json" \
      -d '{
        "roaming_rule": {
          "name": "Allow Visited Network",
          "mcc": "310",
          "mnc": "410",
          "data_action": "allow",
          "ims_action": "allow"
        }
      }'

Cause 3: Data Allowed but IMS Denied​

Symptoms:

  • Data roaming works
  • Voice/IMS roaming fails
  • Split service availability

Diagnostic Steps:

  1. Query roaming rules for visited network
  2. Check data_action vs ims_action values
  3. Verify roaming profile default actions

Solution:

  • Update roaming rule to allow IMS:
    • Set ims_action: "allow"
  • Or update profile's ims_action_if_no_rules_match to "allow"

See Roaming Documentation for detailed configuration.

EIR Problems​

Symptoms​

  • Devices blocked unexpectedly
  • Stolen devices not blocked
  • EIR check failing

Common Causes and Solutions​

Cause 1: Incorrect IMEI Regex​

Symptoms:

  • Wrong devices blocked/allowed
  • Rule matches incorrectly

Diagnostic Steps:

  1. Query EIR rules
  2. Identify which rule is matching
  3. Test regex pattern against actual IMEI
  4. Check rule priority/order

Solution:

  • Update EIR rule with correct regex
  • Test regex thoroughly before applying
  • Consider rule order (first match wins)

Cause 2: MME Not Sending S13 Requests​

Symptoms:

  • EIR check never happens
  • All devices allowed regardless of rules

Diagnostic Steps:

  1. Check if MME is configured to use S13 interface
  2. Verify MME Diameter peer connected
  3. Check for S13 application support
  4. Review MME configuration

Solution:

  • Configure MME to perform EIR checks via S13
  • Verify Diameter peer supports S13 application (16777252)
  • Contact MME administrator if needed

Cause 3: No Default Rule​

Symptoms:

  • Devices not matching any rule have unexpected behavior

Diagnostic Steps:

  1. Query all EIR rules
  2. Check if catch-all rule exists
  3. Verify rule ordering

Solution:

  • Add default rule with regex .* to match all IMEIs
  • Set appropriate action (whitelist or blacklist)
  • Ensure specific rules are checked before catch-all

Performance Problems​

Symptoms​

  • Slow API responses
  • Diameter request timeouts
  • High CPU or memory usage
  • Control Panel slow to load

Common Causes and Solutions​

Cause 1: High Database Load​

Symptoms:

  • All operations slow
  • Database CPU high
  • Query timeouts

Diagnostic Steps:

  1. Check database server resource usage
  2. Identify slow queries
  3. Check for missing indexes
  4. Monitor query patterns

Solution:

  • Optimize slow queries
  • Add database indexes
  • Increase database resources
  • Consider database scaling
  • See Database Issues

Cause 2: High Subscriber Count​

Symptoms:

  • Performance degraded over time
  • Slowness correlates with subscriber growth
  • List operations especially slow

Diagnostic Steps:

  1. Query total subscriber count
  2. Check table sizes
  3. Review query execution plans
  4. Monitor resource usage trends

Solution:

  • Plan capacity upgrade
  • Optimize queries for large datasets
  • Consider pagination for large results
  • Implement caching if needed

Cause 3: Diameter Peer Issues​

Symptoms:

  • Diameter operations slow
  • Timeouts on specific peer
  • Some peers fast, others slow

Diagnostic Steps:

  1. Check Control Panel Diameter page
  2. Identify slow peer
  3. Test network latency to peer
  4. Check peer resource usage

Solution:

  • Investigate peer performance issues
  • Check network path for congestion
  • Consider adding redundant peers
  • Increase Diameter timeout if needed

Cause 4: Memory Issues​

Symptoms:

  • OmniHSS memory usage high
  • Out of memory errors
  • Performance degrades over time

Diagnostic Steps:

  1. Check OmniHSS memory usage on Application page
  2. Monitor memory trend
  3. Check for memory leaks
  4. Review Erlang VM settings

Solution:

  • Restart OmniHSS to clear temporary condition
  • Investigate memory leak if usage continuously grows
  • Adjust Erlang VM memory settings in runtime.exs
  • Plan for hardware upgrade if consistently high

Subscriber State Issues​

Symptoms​

  • Subscriber shows as attached but isn't
  • Stale state information
  • Location information incorrect
  • Can't detach subscriber

Common Causes and Solutions​

Cause 1: MME Crash/Restart​

Symptoms:

  • Subscriber shows serving MME that is no longer serving
  • Subscriber can't attach after MME restart
  • State is stale

Diagnostic Steps:

  1. Check subscriber state for serving MME
  2. Verify if MME has restarted
  3. Check MME's last connection time

Solution:

  • Wait for subscriber to attach again (state will update)
  • Or manually clear subscriber state
  • MME should send Cancel-Location on restart

Cause 2: Network Detach Not Received​

Symptoms:

  • Subscriber powered off but shows as attached
  • PDN sessions remain in database
  • Location not cleared

Diagnostic Steps:

  1. Check subscriber's last_seen timestamp
  2. Verify if old state (hours or days old)
  3. Check if subscriber device is reachable

Solution:

  • State will clear when subscriber attaches again
  • Or wait for state timeout (if implemented)
  • Manual cleanup may be required for very stale state

Cause 3: Database Corruption​

Symptoms:

  • Inconsistent state across tables
  • Foreign key violations
  • State doesn't make sense

Diagnostic Steps:

  1. Query subscriber state directly from database
  2. Check for orphaned records
  3. Verify referential integrity

Solution:

  • Identify and fix inconsistent data
  • May require manual database cleanup
  • Contact support if corruption is widespread

API Issues​

Symptoms​

  • API returns errors
  • Slow API responses
  • Cannot create/update entities
  • 500 errors

Common Causes and Solutions​

Cause 1: Invalid Request Data​

Symptoms:

  • 400 or 422 errors
  • Validation error messages
  • Field rejected

Diagnostic Steps:

  1. Review error response for specific field errors
  2. Check API request format
  3. Verify required fields present
  4. Check data types

Solution:

  • Fix request data to match API reference
  • Ensure all required fields included
  • Verify foreign key references exist (profile IDs, etc.)

Cause 2: Foreign Key Constraint​

Symptoms:

  • Cannot create subscriber
  • Error: "key_set_id does not exist"
  • Referenced entity not found

Diagnostic Steps:

  1. Identify which foreign key is failing
  2. Verify referenced entity exists:
    • key_set_id → key sets
    • epc_profile_id → EPC profiles
    • ims_profile_id → IMS profiles

Solution:

Cause 3: Database Connectivity​

Symptoms:

  • 500 errors
  • All API calls fail
  • Database connection errors

Solution:

Diagnostic Tools and Commands​

Control Panel Quick Checks​

  1. System Overview

    • URL: https://[hostname]:7443/overview
    • Check: Subscriber counts, active sessions, system status

  2. Diameter Status

    • URL: https://[hostname]:7443/diameter
    • Check: All critical peers connected

  3. Application Health

    • URL: https://[hostname]:7443/application
    • Check: Memory usage, process count, uptime

API Diagnostic Commands​

Check System Health:

curl -k https://hss.example.com:8443/api/status

Query Subscriber:

# By IMSI
curl -k https://hss.example.com:8443/api/subscriber/imsi/001001123456789

# By MSISDN
curl -k https://hss.example.com:8443/api/subscriber/msisdn/14155551234

# By ID
curl -k https://hss.example.com:8443/api/subscriber/1

List All Subscribers:

curl -k https://hss.example.com:8443/api/subscriber

Check Profile Configuration:

# EPC Profile
curl -k https://hss.example.com:8443/api/epc/profile/1

# IMS Profile
curl -k https://hss.example.com:8443/api/ims/profile/1

# Roaming Profile
curl -k https://hss.example.com:8443/api/roaming/profile/1

Network Diagnostic Commands​

Test Diameter Port Connectivity:

telnet [PEER_IP] 3868

Check TLS Certificate:

openssl s_client -connect [hostname]:8443 -showcerts

Test Database Connectivity:

psql -h [DB_HOST] -U [DB_USER] -d [DB_NAME] -c "SELECT COUNT(*) FROM subscriber;"

Log Analysis​

Search Logs for Specific IMSI:

grep "001001123456789" /var/log/omnihss/omnihss.log

Find Authentication Failures:

grep "authentication.*fail" /var/log/omnihss/omnihss.log

Check Diameter Peer Events:

grep "Diameter peer" /var/log/omnihss/omnihss.log

Find Database Errors:

grep -i "database.*error" /var/log/omnihss/omnihss.log

Escalation Guidelines​

When to Escalate​

Escalate to engineering/vendor support when:

  1. System-wide failures that cannot be resolved with documented procedures
  2. Data corruption or inconsistent database state
  3. Suspected software bugs or unexpected behavior
  4. Performance issues that cannot be resolved with tuning
  5. Security incidents or unauthorized access
  6. Questions about undocumented behavior

Information to Provide​

When escalating, include:

  1. Detailed symptoms - What is failing, when, for whom
  2. Steps taken - What troubleshooting you've already done
  3. Logs - Relevant log excerpts showing the issue
  4. Configuration - Relevant portions of runtime.exs (redact sensitive data)
  5. Environment - OmniHSS version, database version, OS version
  6. Impact - How many subscribers affected, business impact
  7. Subscriber examples - Specific IMSIs showing the problem

Critical vs Non-Critical​

Critical Issues (Escalate Immediately):

  • System completely down
  • All subscribers unable to attach
  • Database corruption
  • Security breach

Non-Critical Issues (Document and Escalate During Business Hours):

  • Single subscriber issues that can be worked around
  • Performance degradation that's manageable
  • Enhancement requests
  • Documentation questions

Common Error Messages Reference​

Authentication Errors​

Error MessageCauseSolution
"Authentication vectors generation failed"Missing or invalid key setCheck key set configuration
"SQN synchronization failure"SQN out of syncWait for resync
"Subscriber not found"Invalid IMSIVerify IMSI, provision subscriber
"Subscriber disabled"enabled=falseEnable subscriber

Diameter Errors​

Error MessageCauseSolution
"Diameter peer connection timeout"Network issueCheck network connectivity
"CER/CEA exchange failed"Configuration mismatchVerify Diameter config
"Application not supported"Peer doesn't support required appCheck peer applications
"TLS handshake failed"Certificate issueCheck certificates

Database Errors​

Error MessageCauseSolution
"Connection refused"Database downStart database
"Authentication failed"Wrong credentialsFix credentials
"No connections available"Pool exhaustedIncrease pool size
"Query timeout"Slow queryOptimize queries

API Errors​

Error MessageCauseSolution
"key_set_id does not exist"Invalid foreign keyCreate key set first
"IMSI has already been taken"Duplicate IMSIUse different IMSI or delete existing
"Validation error"Invalid inputCheck field format and requirements

← Back to Operations Guide | Next: API Reference →

Last updated on