Skip to main content

Troubleshooting Guide

Common issues and solutions

OmniMessage Connectivity Issues​

Since the SMPP Gateway is stateless and depends entirely on OmniMessage Core, connectivity problems with OmniMessage are the most critical issues.

Symptoms of OmniMessage Disconnection​

  • No outbound messages: Queue builds up, messages not being sent
  • No inbound messages: Partners can't submit messages
  • Timeouts: API calls timing out or hanging
  • Logs show: "Connection refused", "Timeout", "HTTP 503", "Connection reset"

Diagnosis​

1. Check OmniMessage Availability:

# Test connectivity
curl -k -v https://omnimessage-core.example.com:8443/api/system/health

# Test from gateway host specifically
ssh gateway-server 'curl -k https://omnimessage-core.example.com:8443/api/system/health'

2. Check Configured API URL:

# Review the configuration
grep -A1 'api_base_url' /opt/omnimessage-smpp/config/runtime.exs

# Check for network connectivity
ping omnimessage-core.example.com
nc -zv omnimessage-core.example.com 8443

3. Check Gateway Logs for API Errors:

# Look for API-related errors
sudo journalctl -u omnimessage-smpp -f | grep -i 'api\|omnimessage\|connect'

# Search logs for recent errors
sudo journalctl -u omnimessage-smpp -n 200 | grep -i error

Solutions​

If OmniMessage is down:

  1. Contact OmniMessage operations team
  2. Pending messages will accumulate in the queue
  3. Gateway will keep retrying (see SMPP_POLL_INTERVAL)
  4. Check OmniMessage status page or monitoring

If OmniMessage is up but gateway can't reach it:

  1. Check firewall rules allow outbound HTTPS
  2. Check DNS resolution: nslookup omnimessage-core.example.com
  3. Check network routing: traceroute omnimessage-core.example.com
  4. Verify SSL certificates if using HTTPS

If API URL is misconfigured:

  1. Edit /opt/omnimessage-smpp/config/runtime.exs
  2. Verify api_base_url is correct (must be HTTPS for production)
  3. Restart gateway: sudo systemctl restart omnimessage-smpp

Connection Problems​

Connection Won't Establish​

Symptoms:

  • Status shows "Disconnected" (red)
  • No successful bind in logs
  • Repeated connection attempts

Possible Causes & Solutions:

1. Network Connectivity Issues​

Check:

# Test DNS resolution
nslookup smpp.carrier.com

# Test connectivity
ping -c 3 smpp.carrier.com

# Test port
telnet smpp.carrier.com 2775
# or
nc -zv smpp.carrier.com 2775

Solutions:

  • If DNS fails: Use IP address instead of hostname in configuration
  • If ping fails: Check firewall rules, contact carrier
  • If port fails: Verify correct port number, check firewall

2. Incorrect Credentials​

Check:

  • Logs show "bind failed" or "authentication error"
  • Web UI: SMPP → Client Peers → verify system_id and password

Solutions:

  • Confirm credentials with carrier
  • Check for typos (case-sensitive)
  • Update configuration and reconnect

3. IP Not Whitelisted​

Check:

  • Connection rejected immediately
  • Carrier logs show unauthorized IP

Solutions:

  • Confirm your gateway's public IP: 
    curl ifconfig.me
  • Request carrier add IP to whitelist
  • Verify IP hasn't changed (dynamic IP)

4. Firewall Blocking​

Check:

# Check if port is open
sudo iptables -L -n | grep 2775

# Check UFW (Ubuntu/Debian)
sudo ufw status | grep 2775

# Check firewalld (RHEL/CentOS)
sudo firewall-cmd --list-ports | grep 2775

Solutions:

# Ubuntu/Debian
sudo ufw allow out 2775/tcp

# RHEL/CentOS
sudo firewall-cmd --permanent --add-port=2775/tcp
sudo firewall-cmd --reload

5. Pinned Source Address/Port Unavailable​

Only applies when the client bind sets bind_source_ip and/or bind_source_port (see CONFIGURATION.md).

Check:

  • Logs show the connection failing to open the local socket
  • bind_source_ip is not an address held by this host
  • Another process already holds the configured bind_source_port

Solutions:

  • Confirm bind_source_ip is a local address: ip addr show
  • An invalid bind_source_ip is logged and ignored — the OS then picks the source IP, which may not match the carrier's whitelist. Correct the address if delivery is failing.
  • The pinned port is opened with SO_REUSEADDR, so a reconnect rebinds immediately even while the old socket is in TIME_WAIT. If the port is still unavailable, check whether another service uses it: sudo ss -ltnp | grep <port>

Connection Keeps Dropping​

Symptoms:

  • Connection established but frequently disconnects
  • smpp_disconnection_total metric increasing
  • Logs show repeated reconnections

Possible Causes & Solutions:

1. Network Instability​

Check:

# Monitor packet loss
ping -c 100 smpp.carrier.com | grep loss

# Check network errors
netstat -s | grep -i error

Solutions:

  • Contact carrier about network issues
  • Check with ISP if on your end
  • Consider backup connection/route

Check:

  • Logs show "enquire_link timeout"
  • Connection drops after periods of inactivity

Solutions:

  • Default timeout is 30 seconds
  • Verify network allows keepalive packets
  • Check for aggressive firewalls timing out idle connections

3. TPS Limit Exceeded​

Check:

  • High message rate at disconnect time
  • Carrier throttling messages

Solutions:

  • Review tps_limit setting
  • Reduce TPS to 70-80% of carrier maximum
  • Spread traffic across multiple binds

4. Carrier Server Issues​

Check:

  • Check carrier service status
  • Contact carrier support

Solutions:

  • Wait for carrier to resolve
  • Configure backup carrier if available

Message Delivery Problems​

Messages Not Being Sent​

Symptoms:

  • Messages stuck in queue
  • smpp_messages_sent_total not increasing
  • Connection shows connected

Possible Causes & Solutions:

1. Wrong dest_smsc Routing​

Check:

  • Web UI → Queue → Check message dest_smsc field
  • Compare with connection name in SMPP → Live Status

Solutions:

  • Messages route based on dest_smsc field
  • Verify backend is setting correct dest_smsc
  • If dest_smsc is NULL, check default routing

2. Messages Scheduled for Future​

Check:

  • Web UI → Queue → Check deliver_after field
  • Messages with future timestamp won't send yet

Explanation:

  • Retry system sets deliver_after for failed messages
  • Messages wait until that time before retry

Solutions:

  • Wait for scheduled time
  • If urgent, contact backend team to reset timestamp

3. TPS Limit Too Low​

Check:

  • Large queue buildup
  • Messages sending very slowly

Solutions:

  • Increase tps_limit in configuration
  • Verify carrier can handle higher rate
  • See CONFIGURATION.md

4. Queue Worker Not Running​

Check:

  • Service status
  • Logs for errors

Solutions:

# Restart service
sudo systemctl restart omnimessage-smpp

# Check logs
sudo journalctl -u omnimessage-smpp -f

5. Frontend Not Registered Under the Expected Name​

OmniMessage Core only routes a message to this gateway if a peer is registered under a frontend name equal to the message's dest_smsc. Each peer registers under its routing key (see Frontend Registration):

  • Client peers: the configured queue, or the bare peer name if no queue is set.
  • Server binds: the bind name.

Check:

  • Logs for Successfully registered SMPP peer frontend '<name>' entries — confirm <name> matches the dest_smsc on stuck messages
  • For binds sharing a queue, expect a single registration under the shared queue name, not one per peer

Solutions:

  • Ensure the backend sets dest_smsc to the routing key (the peer's queue, else its name), not to {hostname}_{peer} (an earlier naming scheme that no longer applies)
  • If using a shared queue, route messages to the queue name rather than an individual peer name
  • Confirm the peer is enabled — disabled peers are not registered

High Delivery Failure Rate​

Symptoms:

  • smpp_delivery_failures_total increasing
  • Logs show "submit_sm_resp" with error status
  • Messages not reaching recipients

Possible Causes & Solutions:

1. Invalid Destination Numbers​

Check:

  • Logs for specific error codes
  • Review message destination format

Common Error Codes:

  • 0x0000000B - Invalid destination
  • 0x00000001 - Invalid message length
  • 0x00000003 - Invalid command

Solutions:

  • Validate number format (E.164 recommended)
  • Check number includes country code
  • Verify with carrier requirements

2. Invalid Message Content​

Check:

  • Message length
  • Special characters
  • Encoding

Solutions:

  • GSM-7: Max 160 characters
  • UCS-2: Max 70 characters
  • Remove unsupported characters
  • Check encoding settings

3. Carrier Rejection​

Check:

  • Specific error codes from carrier
  • Patterns in rejected messages

Solutions:

  • Contact carrier for rejection reason
  • May need content filtering
  • Check for spam/abuse patterns

4. Expired Messages​

Check:

  • Message expires timestamp
  • Delivery attempt timing

Solutions:

  • Increase message validity period
  • Reduce retry delay for time-sensitive messages

Web UI Problems​

Can't Access Web Dashboard​

Symptoms:

Possible Causes & Solutions:

1. Service Not Running​

Check:

sudo systemctl status omnimessage-smpp

Solutions:

# If stopped, start it
sudo systemctl start omnimessage-smpp

# Check logs for errors
sudo journalctl -u omnimessage-smpp -n 50

2. Firewall Blocking Port 8087​

Check:

sudo ufw status | grep 8087
# or
sudo firewall-cmd --list-ports | grep 8087

Solutions:

# Ubuntu/Debian
sudo ufw allow 8087/tcp

# RHEL/CentOS
sudo firewall-cmd --permanent --add-port=8087/tcp
sudo firewall-cmd --reload

3. SSL Certificate Issues​

Check:

  • Browser shows security warning
  • Certificate expired or invalid

Solutions:

  • Accept security exception (if self-signed)
  • Install valid SSL certificate
  • Check certificate files exist: 
    ls -l /opt/omnimessage-smpp/priv/cert/

4. Wrong URL​

Check:

  • Verify using HTTPS (not HTTP)
  • Verify correct server IP/hostname
  • Verify port 8087

Web UI Shows Errors​

Symptoms:

  • Page loads but shows errors
  • Functions don't work
  • Data not displaying

Solutions:

  1. Clear Browser Cache:

    • Ctrl+F5 (hard refresh)
    • Clear browser cache and cookies

  2. Check Browser Console:

    • Press F12
    • Check Console tab for JavaScript errors
    • Report to support if errors found

  3. Try Different Browser:

    • Test in Chrome, Firefox, Edge
    • Isolate browser-specific issues

  4. Check Service Logs:

    sudo journalctl -u omnimessage-smpp -f

Metrics Problems​

Prometheus Metrics Not Available​

Symptoms:

  • curl http://localhost:4000/metrics fails
  • Prometheus can't scrape metrics
  • Empty or error response

Possible Causes & Solutions:

1. Service Not Running​

Check:

sudo systemctl status omnimessage-smpp

Solutions:

sudo systemctl start omnimessage-smpp

2. Port Not Accessible​

Check:

# Test locally
curl http://localhost:4000/metrics

# Test remotely
curl http://your-server-ip:4000/metrics

Solutions:

  • If local works but remote doesn't: Check firewall
  • Open port 4000 in firewall for Prometheus server

3. Wrong Endpoint​

Verify:

  • Endpoint is /metrics (not /prometheus or /stats)
  • Port is 4000 (not 8087)

Metrics Show Unexpected Values​

Symptoms:

  • Counters reset to zero
  • Gauges show wrong values
  • Missing metrics for some binds

Solutions:

  1. Service Restart Resets Counters:

    • Counters reset on service restart
    • This is normal behavior
    • Use increase() or rate() in Prometheus queries

  2. New Binds Not Showing:

    • Metrics only appear after first event
    • Send test message to populate metrics
    • Check bind is enabled and connected

  3. Stale Metrics:

    • Old binds may still show in metrics
    • Restart service to clear stale entries
    • Or use Prometheus relabeling to filter

Performance Problems​

High CPU Usage​

Check:

top -p $(pgrep -f omnimessage-smpp)

Possible Causes:

  • Very high message volume
  • Too many connections
  • Configuration issue

Solutions:

  • Check message rate is within capacity
  • Review TPS limits
  • Contact support if sustained high CPU

High Memory Usage​

Check:

ps aux | grep omnimessage-smpp

Possible Causes:

  • Large message queue in memory
  • Memory leak (rare)

Solutions:

  • Restart service to clear memory
  • Check message queue size
  • Contact support if memory grows continuously

Slow Message Processing​

Symptoms:

  • Messages take long to send
  • Queue building up
  • Low message rate

Check:

  1. TPS limits - may be too restrictive
  2. queue_check_frequency - may be too high
  3. Backend API response time - may be slow
  4. Network latency to carrier

Solutions:

  • Increase TPS if carrier allows
  • Decrease queue_check_frequency for faster polling
  • Optimize backend API
  • Check network latency

Configuration Problems​

Configuration File Syntax Errors​

Symptoms:

  • Service won't start after config change
  • Logs show "syntax error" or "parse error"

Check:

# Validate Elixir syntax
/opt/omnimessage-smpp/bin/omnimessage-smpp eval "File.read!('config/runtime.exs')"

Common Mistakes:

  • Missing comma between map entries
  • Mismatched quotes (" vs ')
  • Unmatched brackets or braces
  • Missing import Config at top

Solutions:

  • Restore from backup
  • Carefully review syntax
  • Use text editor with Elixir syntax highlighting

Changes Not Taking Effect​

Symptoms:

  • Modified configuration but no change in behavior
  • Old settings still active

Solutions:

# Configuration changes require restart
sudo systemctl restart omnimessage-smpp

# Verify restart succeeded
sudo systemctl status omnimessage-smpp

# Check logs for errors
sudo journalctl -u omnimessage-smpp -n 50

Emergency Recovery​

Complete System Failure​

Steps:

  1. Check basic system health:

    # Disk space
    df -h

    # Memory
    free -h

    # CPU load
    uptime

  2. Check service status:

    sudo systemctl status omnimessage-smpp

  3. Review recent logs:

    sudo journalctl -u omnimessage-smpp -n 200

  4. Try service restart:

    sudo systemctl restart omnimessage-smpp

  5. If restart fails:

    • Check configuration syntax
    • Verify SSL certificates exist
    • Check file permissions
    • Review logs for specific error

  6. Restore from backup (if needed):

    # Restore config
    sudo cp /opt/omnimessage-smpp/config/runtime.exs.backup \
      /opt/omnimessage-smpp/config/runtime.exs

    # Restart
    sudo systemctl restart omnimessage-smpp

  7. Contact support if unresolved

Getting Help​

Information to Gather​

Before contacting support, collect:

  1. Version: cat /opt/omnimessage-smpp/VERSION
  2. Recent Logs: 
    sudo journalctl -u omnimessage-smpp -n 200 > /tmp/smpp-logs.txt
  3. Configuration (sanitize passwords): 
    sudo cp /opt/omnimessage-smpp/config/runtime.exs /tmp/config.exs
    # Edit /tmp/config.exs to remove passwords before sending
  4. Metrics Output: 
    curl http://localhost:4000/metrics > /tmp/metrics.txt
  5. System Info: 
    uname -a > /tmp/system-info.txt
    free -h >> /tmp/system-info.txt
    df -h >> /tmp/system-info.txt

Contact Support​

Last updated on