Troubleshooting Guide
This document covers common operational issues encountered with OmniMSC deployments and their resolution steps. For configuration reference, see Configuration Reference. For system metrics and alerting, see Metrics Reference. For real-time status and subscriber inspection, see Control Panel Guide.
SS7 Connectivity
STP Link Down
Symptoms: No new calls are established, location updates do not complete, the sctp_link_down alarm is raised, and omnimsc_peer_status shows 0 for the affected STP peer.
Investigation: Verify that the SCTP association to the STP is established. Check that the remote IP address and port match the STP configuration. Confirm that SCTP protocol 132 is permitted through any firewalls between the MSC and STP. Verify the STP routing context matches the value configured in OmniMSC. Check that the M3UA ASP state machine has progressed through ASPUP and ASPAC to the ACTIVE state. If the association establishes but M3UA does not reach ACTIVE, check the M3UA logs for ERR chunks indicating a routing context or traffic mode mismatch.
Resolution: Correct the SCTP endpoint configuration (IP, port), firewall rules, or M3UA routing context as indicated by the investigation. Once the SCTP association re-establishes and M3UA reaches ACTIVE, the alarm clears automatically and signaling resumes.
MAP Dialogue Timeout
Symptoms: Location updates hang without completing, SMS delivery fails, the hlr_unreachable alarm may be raised. MAP operations time out without receiving a TC-CONTINUE or TC-END from the HLR.
Investigation: Confirm that the HLR is reachable at the network level. Verify that point codes are correctly configured on both ends. Check that Global Title translation is correctly configured if GT-based SCCP routing is in use. Verify that M3UA has an active route to the HLR point code. Inspect logs for TC-BEGIN messages being sent without corresponding TC-CONTINUE responses.
Resolution: Correct point code or Global Title configuration as needed. If the HLR is reachable but not responding, the issue is on the HLR side. Verify that the HLR has a return route to the MSC point code.
Wrong DPC for MAP Responses
Symptoms: The SMSc never receives the MT-SMS acknowledgement. MAP ForwardSM responses are sent but do not arrive at the originating SMSc. The MT-SMS delivery appears to complete from the MSC perspective but the SMSc retransmits the request.
Investigation: Check the Destination Point Code (DPC) in the M3UA DATA message carrying the MAP response. The response DPC must match the Originating Point Code (OPC) from the incoming MAP request, which is the SMSc point code. If the response is being sent to the STP or HLR point code instead of the SMSc, the routing_info OPC is not being used for the response DPC.
Resolution: Ensure that the MAP response uses the OPC from the incoming routing_info as the DPC for the return message. This guarantees the response reaches the correct SMSc point code rather than a default route.
Call Issues
One-Way Audio
Symptoms: Call is established (both parties hear ringing and answer) but audio flows in only one direction, or not at all.
Investigation: Verify that the MGCP CRCX (Create Connection) and MDCX (Modify Connection) messages are being sent to the media gateway and that the gateway responds with 200 OK. Check that the MGW is reachable and healthy. Inspect the SDP connection address (c= line) in the SIP 200 OK to confirm it contains a routable IP address, not a private or loopback address. If behind NAT, verify that the external_ip parameter is configured. Confirm that the RTP port range is open in both directions through any firewalls.
Resolution: Correct the MGW connectivity, SDP address configuration, or firewall rules as identified. For NAT deployments, configure the external_ip parameter in the SIP configuration.
SIP BYE Not Sent on Call Clear
Symptoms: When the A-interface connection is released (e.g., radio link failure, BSC clear), the SIP peer is not notified. The remote SIP endpoint keeps the session open until its own timer expires.
Investigation: Verify that the connection_lost event is being sent to the CC FSMs when MSC-A releases the connection. Each active call transaction must receive the release indication so that the CC FSM can trigger SIP BYE on the outgoing leg.
Resolution: Ensure that the MSC-A release handler iterates all active transactions and delivers the connection_lost event. The CC FSM should transition to the release state and send SIP BYE upon receiving this event.
Assignment Complete to Wrong CC FSM
Symptoms: A new call in an existing connection (e.g., a second MO call while the first is held) receives the Assignment Complete for the wrong transaction, causing call state corruption.
Investigation: Verify that the active_trans reference is updated when a new call starts while the connection is already in the communicating state. The Assignment Complete is delivered to whatever transaction is marked as active_trans at the time of receipt.
Resolution: Ensure that the MSC-A process updates active_trans to the new transaction when processing the second CM Service Request or call setup in the communicating state.
DTMF Not Working
Symptoms: In-call DTMF tones are not recognized by the remote party or IVR system. The subscriber presses digits but no action occurs.
Investigation: Verify that SIP INFO messages are being sent with content type application/dtmf-relay when DTMF events are received from the radio side. Confirm that the SIP peer supports the INFO method for DTMF relay. Check the SIP peer configuration for DTMF handling preferences.
Resolution: Ensure SIP INFO-based DTMF relay is enabled and that the peer is configured to accept it. If the peer requires RFC 2833 telephone-event RTP packets instead of SIP INFO, adjust the DTMF relay method accordingly.
SMS Issues
MT-SMS Delivery Timeout (TC1)
Symptoms: MT-SMS delivery initiated by the SMSc times out. The MAP ForwardSM operation does not complete within the TC1 timer. The SMSc retransmits the delivery attempt.
Investigation: Check that the register_mt_sms operation uses a synchronous call to ensure the DTAP CP-DATA is sent before the MAP response. Verify that the DTAP Transaction Identifier (TI) is correctly allocated for the MT direction. Confirm that SMS PDUs are being sent on SAPI 3 (the dedicated signaling channel for SMS) rather than SAPI 0.
Resolution: Ensure synchronous MT-SMS registration, correct TI allocation with the TI flag set for network-originated transactions, and SAPI 3 usage for SMS DTAP messages.
MT-SMS Re-Delivery Loop
Symptoms: The MT-SMS handler process crashes and restarts repeatedly, causing the same SMS to be re-delivered in a loop. The subscriber may receive duplicate messages.
Investigation: Check the child_spec of the MT-SMS handler process in the supervision tree. If the restart strategy is set to permanent, the supervisor will restart the handler unconditionally after every termination, including normal completion. This causes re-delivery because the restarted handler re-initiates the delivery.
Resolution: Set the MT-SMS handler child_spec to restart: :temporary so that normal termination does not trigger a restart. Only abnormal crashes should cause a restart, and the SMSc will handle re-delivery via its own retry mechanism.
MAP Response Not Reaching SMSc
Symptoms: MT-SMS delivery completes on the radio side (subscriber receives the message) but the SMSc does not receive the MAP ForwardSM response. The SMSc treats the delivery as failed and may re-deliver.
Investigation: Check the DPC in the M3UA DATA message carrying the MAP ForwardSM response. The DPC must match the SMSc point code, which is the OPC from the incoming MAP request (routing_info[:opc]). If the response is routed to the wrong point code, the SMSc never sees it.
Resolution: Ensure the MAP response DPC is derived from routing_info[:opc] of the incoming request, not from a default or HLR-associated point code.
Authentication
Auth Failure -- MAC Mismatch
Symptoms: Subscriber authentication fails with reason mac_failure. The UE sends Authentication Failure with cause "MAC failure" indicating the network authentication check failed on the USIM.
Investigation: This indicates a mismatch between the subscriber key (Ki/K) stored in the USIM and the key stored in the HLR/AuC. The authentication vector generated by the AuC does not match what the USIM computes. This can occur after SIM replacement, HLR migration, or provisioning errors.
Resolution: Verify that the subscriber key in the HLR/AuC matches the key programmed into the USIM. Re-provision the subscriber profile in the HLR if necessary. This is not an MSC-side issue -- the MSC faithfully relays vectors from the HLR.
Auth Resync Loop
Symptoms: The subscriber enters an authentication resynchronization cycle that repeats without resolution. The UE sends Authentication Failure with cause "SQN failure" and an AUTS parameter. The MSC relays the resync to the HLR, but subsequent authentication attempts also fail with SQN failure.
Investigation: OmniMSC permits a maximum of 2 resynchronization attempts per authentication cycle. Check whether the resync count has been exceeded. Verify that the HLR correctly processes the AUTS parameter and advances the SQN. If the HLR SQN is significantly out of range from the USIM SQN, a single resync may not be sufficient.
Resolution: If the resync loop persists after 2 attempts, the subscriber must be re-provisioned in the HLR. The HLR SQN must be reset to align with the USIM SQN. Contact HLR operations to resolve the SQN discrepancy.
Subscriber Issues
TMSI Unknown on Paging Response
Symptoms: A subscriber responds to paging using a TMSI that is not recognized by the MSC. This can occur after MSC restart or TMSI table corruption.
Investigation: When the MSC receives a Paging Response with an unknown TMSI, it does not reject the subscriber. Instead, the MSC sends an Identity Request to obtain the IMSI directly from the mobile station. This is normal recovery behavior per 3GPP TS 24.008.
Resolution: No action required. The MSC automatically resolves the unknown TMSI by requesting the subscriber's IMSI. If this occurs frequently, investigate whether TMSI reallocation is functioning correctly during location updates.
CM Service Request for Unknown IMSI
Symptoms: A subscriber sends a CM Service Request but is not registered in the VLR. The MSC does not have subscriber data for this IMSI.
Investigation: OmniMSC triggers an implicit location update when it receives a CM Service Request from an unregistered subscriber. The MSC initiates the full location update procedure (authentication, ciphering, MAP UPDATE LOCATION to HLR) before proceeding with the requested service.
Resolution: No action required. The implicit location update is automatic. If the location update itself fails (HLR unreachable, authentication failure), the CM Service Request is rejected and the subscriber must retry.
Subscriber Stuck in VLR
Symptoms: A subscriber record remains in the VLR despite the subscriber being unreachable or having moved to another MSC. Paging attempts for this subscriber consistently fail.
Investigation: Check whether the subscriber has completed a location update at another MSC (the HLR would have sent a MAP CANCEL LOCATION, which removes the VLR record). If the cancel was missed or the VLR record is otherwise stale, manual removal may be necessary.
Resolution: Use the DELETE /api/subscribers/{id} endpoint to purge the subscriber record from the VLR. The subscriber will re-register via location update when they next access the network.
Performance
Overload Rejection
Symptoms: New calls and location updates are rejected with GSM cause 42 (switching equipment congestion). The overload alarm is active.
Investigation: Check the current overload thresholds and the system metrics that are exceeding them. The four monitored metrics are: active calls (default max 10,000), registered subscribers (default max 50,000), BEAM process count (default max 500,000), and paging rate (default max 1,000/sec). Determine which metric is triggering the overload condition.
Resolution: If the thresholds are appropriate for the hardware, investigate the source of the overload: abnormal call volume, paging storms, or process leaks. If the thresholds are too conservative for the deployment, increase them in the overload configuration. See Configuration Reference.
High Memory Usage
Symptoms: BEAM memory consumption grows over time or spikes unexpectedly. The system may become unresponsive if memory is exhausted.
Investigation: Check ETS table sizes for the major in-memory stores: VLR subscriber table, MPTY conference contexts, and TCAP transaction table. Large numbers of stale entries in any of these tables can cause memory growth. Also check for process leaks -- a rising BEAM process count without corresponding subscriber or call growth indicates processes that are not terminating correctly.
Resolution: Identify and resolve the source of the growth. Stale VLR entries can be purged via the subscriber API. TCAP transaction timeouts should clean up abandoned dialogues automatically. If process leaks are identified, investigate the supervision tree for processes with incorrect restart strategies or missing timeout handling.
SIP Issues
For SIP peer configuration and OPTIONS keepalive details, see SIP Trunking.
SIP Peer Showing UNKNOWN Status
Symptoms: A SIP peer appears with UNKNOWN status in the control panel or metrics, rather than UP or DOWN.
Investigation: OmniMSC determines SIP peer status via periodic OPTIONS keepalive messages. If OPTIONS keepalive is not configured for the peer, or if the peer has never responded to an OPTIONS request, the status remains UNKNOWN. Check whether the peer configuration includes keepalive settings. Verify that the peer is reachable and responds to SIP OPTIONS with 200 OK.
Resolution: Enable OPTIONS keepalive for the peer in the SIP configuration. If keepalive is enabled but the peer does not respond, investigate network connectivity and confirm the peer supports the OPTIONS method.
re-INVITE Retransmissions from Peer
Symptoms: A SIP peer sends re-INVITE requests during an established call (for codec renegotiation, session refresh, or hold/unhold) and retransmits them because the MSC does not respond. The peer may eventually terminate the call due to the unanswered re-INVITE.
Investigation: Verify that the MSC SIP stack handles in-dialogue re-INVITE messages when the call is in the active state. If the re-INVITE arrives during a state transition or is not matched to the correct dialogue, it may be silently dropped.
Resolution: Ensure the SIP transaction layer correctly matches in-dialogue re-INVITE to the existing call leg and that the CC FSM processes the re-INVITE in the active state. The MSC should respond with 200 OK containing an updated SDP answer.
Session Timer Expiry
Symptoms: Established calls are unexpectedly disconnected after a fixed interval (typically 1800 seconds). The SIP peer sends BYE with reason "Session Timer Expired."
Investigation: SIP session timers (RFC 4028) require periodic re-INVITE or UPDATE to keep the session alive. Check the Session-Expires header negotiation during call setup. Verify that the MSC respects the Min-SE (Minimum Session Expires) value proposed by the peer. If the MSC is the refresher, confirm that it sends re-INVITE or UPDATE before the session timer expires.
Resolution: Ensure the MSC participates in session timer negotiation and performs session refresh as required by the negotiated role (refresher or refreshee). If session timers are not needed in the deployment, the peer may need to be configured to not require them.
3GPP Specification References
| Specification | Title | Relevance |
|---|---|---|
| TS 24.008 | Mobile Radio Interface Layer 3 | Authentication, identity, DTAP procedures |
| TS 29.002 | MAP Specification | HLR operations, point code routing |
| TS 48.008 | MSC-BSS Interface (BSSMAP) | Assignment, cipher mode, paging |
| TS 23.018 | Handover Procedures | Inter-MSC handover |
| TS 22.101 | Service Principles | Emergency call, overload |