SMS-C Troubleshooting Guide

← Back to Documentation Index | Main README

Comprehensive guide for diagnosing and resolving common SMS-C issues.

Diagnostic Tools
Message Delivery Issues
Routing Problems
Performance Issues
Database Problems
Frontend Connection Issues
Charging/Billing Issues
ENUM Lookup Problems
Cluster Issues
API Problems
Web UI Issues
System Resource Issues

Diagnostic Tools

Quick Health Check

# 1. Check API status
curl https://api.example.com:8443/api/status

# 2. Check Prometheus metrics endpoint
curl https://api.example.com:9568/metrics | grep sms_c

# 3. Check application logs
tail -f /var/log/sms_c/application.log

# 4. Check process status
systemctl status sms_c

# 5. Check SQL CDR database connectivity (MySQL/MariaDB)
mysql -u sms_user -p -h db.example.com -e "SELECT 1"

# For PostgreSQL:
# psql -U sms_user -h db.example.com -d sms_c_prod -c "SELECT 1"

Log Analysis

View Recent Errors:

# Last 100 error-level log entries
tail -1000 /var/log/sms_c/application.log | grep "\[error\]"

# Search for specific error patterns
grep "routing_failed" /var/log/sms_c/application.log

# Find SQL database errors
grep -i "database\|sql\|ecto" /var/log/sms_c/application.log | grep error

Monitor Logs in Real-Time:

# Follow logs with filter
tail -f /var/log/sms_c/application.log | grep -E "(error|warning|critical)"

Metric Queries

Check Message Processing Rate:

# Messages per second
rate(sms_c_message_received_count[5m])

# Delivery success rate
rate(sms_c_delivery_succeeded_count[5m]) / rate(sms_c_delivery_queued_count[5m])

Check Queue Status:

# Current queue depth
sms_c_queue_size_pending

# Oldest message age (seconds)
sms_c_queue_oldest_message_age_seconds

Check System Performance:

# Message processing latency (p95)
histogram_quantile(0.95, sms_c_message_processing_stop_duration_bucket)

# Routing latency (p95)
histogram_quantile(0.95, sms_c_routing_stop_duration_bucket)

Message Delivery Issues

Messages Not Being Delivered

Symptoms:

Messages stuck in "queued"/"backoff" status
High undelivered message count
No delivery notifications

Diagnostic Steps:

Check Frontend Connections:

curl https://api.example.com:8443/api/frontends/active

Expected: List of active frontends Problem: Empty list or missing expected frontends

Check Message Queue:

Access Web UI: /message_queue

Filter by status: "queued" or "backoff"
Check dest_smsc value
Verify deliver_after is not in future

Check Routing:

Access Web UI: /simulator

Test with actual message parameters
Verify route matches and destination is correct

Check Frontend Polling:

Review frontend system logs:

Is frontend querying /api/messages?
Is frontend sending smsc header correctly?

Solutions:

No Frontends Connected:

# Check frontend system status
systemctl status frontend_service

# Verify frontend can reach API
curl -k https://api.example.com:8443/api/status

# Manually register frontend
curl -X POST https://api.example.com:8443/api/frontends/register \
  -H "Content-Type: application/json" \
  -d '{
    "frontend_name": "test_gateway",
    "frontend_type": "smpp",
    "ip_address": "10.0.1.50"
  }'

Messages Routed to Wrong SMSC:

Review routing configuration
Check route priorities
Test in routing simulator
Verify frontend name matches dest_smsc in messages

Messages Scheduled for Future:

Check deliver_after timestamp
Reset if needed:

curl -X PATCH https://api.example.com:8443/api/messages/12345 \
  -H "Content-Type: application/json" \
  -d '{"deliver_after": "2025-10-30T12:00:00Z"}'

Messages Failing with Retries

Symptoms:

delivery_attempts counter increasing
Messages with high attempt count ( > 3)
Exponential backoff delays

Diagnostic Steps:

Check Event Log:

curl https://api.example.com:8443/api/events/12345

Look for:

Delivery failure events
Error descriptions
Retry timestamps

Check Frontend Logs:

Why is frontend failing to deliver?
Network errors?
Protocol errors?
Downstream system unavailable?

Solutions:

Temporary Network Issues:

Wait for retry (automatic)
Monitor for successful delivery

Persistent Failures:

# Route to alternate gateway
curl -X PATCH https://api.example.com:8443/api/messages/12345 \
  -H "Content-Type: application/json" \
  -d '{"dest_smsc": "backup_gateway"}'

# Reset retry counter
curl -X PATCH https://api.example.com:8443/api/messages/12345 \
  -H "Content-Type: application/json" \
  -d '{"delivery_attempts": 0, "deliver_after": "2025-10-30T12:00:00Z"}'

Invalid Destination Number:

Verify number format
Check number translation rules
Delete message if truly invalid

Dead Letter Messages

Symptoms:

deadletter: true in message
Messages past expiration time
Status still "queued"/"backoff" (failed deliveries are not dead-lettered; they stop being served once past expires)

Diagnostic Steps:

Find Dead Letter Messages:

Access Web UI: /message_queue

Filter by expired status
Check expiration timestamps

Check Why Expired:

Review event log
Check delivery attempt history
Verify routing was successful

Solutions:

Extend Expiration:

# Add 24 hours to expiration
curl -X PATCH https://api.example.com:8443/api/messages/12345 \
  -H "Content-Type: application/json" \
  -d '{"expires": "2025-10-31T12:00:00Z", "deadletter": false}'

Routing Problems

No Route Found

Symptoms:

Error: no_route_found
sms_c_routing_failed_count metric increasing
Event log shows "routing_failed"

Diagnostic Steps:

Check Routes Exist:

Access Web UI: /sms_routing

Verify routes are configured
Check at least one route is enabled

Test Routing:

Access Web UI: /simulator

Enter message parameters (calling number, called number, source SMSC)
Review evaluation results
Check why routes didn't match

Check Route Criteria:

Prefix matches required?
Source SMSC filter too restrictive?
All routes disabled?

Solutions:

No Routes Configured:

Add catch-all route:

Calling Prefix: (empty)
Called Prefix: (empty)
Source SMSC: (empty)
Dest SMSC: default_gateway
Priority: 255
Weight: 100
Enabled: ✓
Description: Catch-all default route

Routes Too Specific:

Add broader route:

Called Prefix: +
Dest SMSC: international_gateway
Priority: 200
Weight: 100
Enabled: ✓
Description: International catch-all

All Routes Disabled:

Enable appropriate routes via Web UI
Check configuration didn't accidentally disable routes

Wrong Route Selected

Symptoms:

Messages routed to unexpected destination
Wrong gateway receiving traffic
Load balance not distributing as expected

Diagnostic Steps:

Use Routing Simulator:

Access Web UI: /simulator

Test with actual message parameters
Review "All Matches" section
Check priority and specificity scores

Check Route Priorities:

Lower number = higher priority
Routes evaluated in priority order
Within same priority, weights apply

Check Route Specificity:

Specificity scoring:

Longer called prefix: +100 points per character
Longer calling prefix: +50 points per character
Source SMSC specified: +25 points
Source type specified: +10 points
ENUM domain specified: +15 points

Solutions:

Adjust Priorities:

Make specific route higher priority:

Premium Route:
  Called Prefix: +1555
  Priority: 10 (high priority)

General Route:
  Called Prefix: +1
  Priority: 50 (lower priority)

Adjust Weights:

Change load balance distribution:

Primary (70%):
  Weight: 70

Backup (30%):
  Weight: 30

Add More Specific Route:

Override general route for specific case:

Specific Route:
  Called Prefix: +15551234
  Dest SMSC: dedicated_gateway
  Priority: 1

General Route:
  Called Prefix: +1
  Dest SMSC: general_gateway
  Priority: 50

Auto-Reply Not Working

Symptoms:

Auto-reply route configured but not triggering
No reply messages being sent
Event log missing auto-reply event

Diagnostic Steps:

Check Route Configuration:

auto_reply: true
auto_reply_message contains text
Route is enabled
Route matches message criteria

Test in Simulator:

Verify route is selected
Check for "auto_reply" indication

Check Event Log:

curl https://api.example.com:8443/api/events/12345 | grep auto_reply

Solutions:

Route Not Matching:

Broaden criteria (remove filters)
Check priority (should be higher than normal routes)
Verify enabled status

Message Not Set:

Edit route, add message:

Auto-Reply: ✓
Auto-Reply Message: "Thank you for your message. We will respond soon."

Wrong Priority:

Auto-reply routes should have high priority (low number):

Auto-Reply Route:
  Priority: 10

Normal Route:
  Priority: 50

Performance Issues

High Message Processing Latency

Symptoms:

sms_c_message_processing_stop_duration p95 > 1000ms
Slow API responses
Queue building up

Diagnostic Steps:

Check Component Latencies:

# Routing latency
histogram_quantile(0.95, sms_c_routing_stop_duration_bucket)

# ENUM lookup latency
histogram_quantile(0.95, sms_c_enum_lookup_stop_duration_bucket)

# Charging latency
histogram_quantile(0.95, sms_c_charging_succeeded_duration_bucket)

# Delivery latency
histogram_quantile(0.95, sms_c_delivery_succeeded_duration_bucket)

Check System Resources:

# CPU usage
top -b -n 1 | grep sms_c

# Memory usage
ps aux | grep beam.smp


**Solutions**:

**Routing Slow** (Many routes):
- Reduce number of enabled routes
- Combine similar routes
- Optimize route criteria

**ENUM Lookups Slow**:
- Check DNS server latency
- Increase timeout
- Use faster/closer DNS servers
- Disable ENUM if not needed

**Charging Slow**:
- Check OCS performance
- Increase OCS timeout
- Disable charging if not needed
- Use async charging

**Database Slow**:
- Increase connection pool size
- Add indexes
- Optimize queries
- Upgrade database resources

**Configuration Changes**:

```elixir
# config/config.exs
# Increase batch size for throughput
config :sms_c,
  batch_insert_batch_size: 200,
  batch_insert_flush_interval_ms: 200

# Increase database pool
config :sms_c, SmsC.Repo,
  pool_size: 50

Low Message Throughput

Symptoms:

Processing < 100 msg/sec
Using async API but still slow
High API response times

Diagnostic Steps:

Check Batch Worker:

# In production console (iex)
SmsC.Messaging.BatchInsertWorker.stats()

Look for:

current_queue_size near max
flush_errors > 0
last_flush_duration_ms very high

Check Bottlenecks:

# Database query time
ecto_pools_query_time

# Connection pool queue time
ecto_pools_queue_time

Solutions:

Database Bottleneck:

Increase pool size:

config :sms_c, SmsC.Repo,
  pool_size: 50  # Increase from 20

Batch Configuration:

Tune for throughput:

config :sms_c,
  batch_insert_batch_size: 200,  # Larger batches
  batch_insert_flush_interval_ms: 200  # Longer interval

Use Async Endpoint:

# High throughput: use /create_async
curl -X POST https://api.example.com:8443/api/messages/create_async

# NOT: /api/messages (synchronous)

Queue Backlog Growing

Symptoms:

sms_c_queue_size_pending increasing
Oldest message age increasing
Processing can't keep up with incoming rate

Diagnostic Steps:

Check Incoming vs Delivery Rate:

# Incoming rate
rate(sms_c_message_received_count[5m])

# Delivery rate
rate(sms_c_delivery_succeeded_count[5m])

Check Frontend Capacity:

Are frontends polling frequently enough?
Are frontends processing messages fast enough?
Any frontend errors?

Check Delivery Success Rate:

rate(sms_c_delivery_succeeded_count[5m]) / rate(sms_c_delivery_attempted_count[5m])

Solutions:

Frontends Not Polling:

Check frontend connectivity
Verify polling interval (should be 5-10 seconds)
Restart frontend services

Frontends Too Slow:

Add more frontend instances
Optimize frontend processing
Increase frontend concurrency

High Retry Rate:

Investigate delivery failures
Fix downstream issues
Route to alternate gateways

Temporary Spike:

Wait for queue to drain
Monitor until normal
Consider capacity upgrades if recurring

Database Problems

Connection Failures

Symptoms:

Error: "unable to connect to database"
API returning 500 errors
Application won't start

Diagnostic Steps:

Check SQL CDR Database Status:

# MySQL/MariaDB
systemctl status mysql

# PostgreSQL
systemctl status postgresql

# Test connectivity (MySQL/MariaDB)
mysql -u sms_user -p -h db.example.com -e "SELECT 1"

# Test connectivity (PostgreSQL)
psql -U sms_user -h db.example.com -d sms_c_prod -c "SELECT 1"

Check Network:

# Ping database host
ping db.example.com

# Check port connectivity (MySQL/MariaDB: 3306, PostgreSQL: 5432)
telnet db.example.com 3306
# or
telnet db.example.com 5432

Check Credentials:

# Verify environment variables
echo $DB_USERNAME
echo $DB_HOSTNAME
echo $DB_PORT

# Try manual connection with same credentials (MySQL/MariaDB)
mysql -u $DB_USERNAME -p$DB_PASSWORD -h $DB_HOSTNAME

# For PostgreSQL:
# psql -U $DB_USERNAME -h $DB_HOSTNAME -d sms_c_prod

Solutions:

Database Down:

# Start database (MySQL/MariaDB)
systemctl start mysql

# Start database (PostgreSQL)
systemctl start postgresql

Wrong Credentials:

Update configuration:

export DB_USERNAME=correct_user
export DB_PASSWORD=correct_password

# Restart application
systemctl restart sms_c

Network Issue:

Check firewall rules
Verify security groups (cloud)
Check VPN/network connectivity

Connection Pool Exhausted:

Increase pool size:

config :sms_c, SmsC.Repo,
  pool_size: 50  # Increase from current value

Slow Queries

Symptoms:

Database query time high
API responses slow
Connection pool queue building up

Diagnostic Steps:

Check Slow Query Log:

-- MySQL/MariaDB: Enable slow query log
SET GLOBAL slow_query_log = 'ON';
SET GLOBAL long_query_time = 1;  -- Log queries > 1 second

-- View slow queries (MySQL/MariaDB)
SELECT * FROM mysql.slow_log ORDER BY query_time DESC LIMIT 10;

-- PostgreSQL: Enable slow query log in postgresql.conf
-- log_min_duration_statement = 1000  # milliseconds
-- Then check PostgreSQL logs

Check Missing Indexes:

-- Check table indexes
SHOW INDEX FROM message_queues;

-- Expected indexes:
-- - source_smsc
-- - dest_smsc
-- - send_time
-- - inserted_at

Check Table Stats:

-- Table sizes (MySQL/MariaDB)
SELECT
  table_name,
  table_rows,
  ROUND(data_length / 1024 / 1024, 2) AS data_mb,
  ROUND(index_length / 1024 / 1024, 2) AS index_mb
FROM information_schema.tables
WHERE table_schema = 'sms_c_prod';

-- Table sizes (PostgreSQL)
-- SELECT schemaname, tablename,
--   pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size
-- FROM pg_tables WHERE schemaname = 'public';

Solutions:

Missing Indexes:

CREATE INDEX idx_message_queues_source_smsc ON message_queues(source_smsc);
CREATE INDEX idx_message_queues_dest_smsc ON message_queues(dest_smsc);
CREATE INDEX idx_message_queues_send_time ON message_queues(send_time);
CREATE INDEX idx_message_queues_status ON message_queues(status);

Table Fragmentation:

-- MySQL/MariaDB
OPTIMIZE TABLE message_queues;
OPTIMIZE TABLE frontend_registrations;

-- PostgreSQL
-- VACUUM ANALYZE message_queues;
-- VACUUM ANALYZE frontend_registrations;

Too Much Data:

Clean up old records:

-- Delete delivered messages older than 30 days
DELETE FROM message_queues
WHERE status = 'delivered'
AND deliver_time < DATE_SUB(NOW(), INTERVAL 30 DAY)
LIMIT 10000;

Disk Space Full

Symptoms:

Error: "Disk full"
Cannot write to database
Application crashes

Diagnostic Steps:

Check Disk Usage:

df -h

# Check SQL database directory (MySQL/MariaDB)
du -sh /var/lib/mysql

# Check SQL database directory (PostgreSQL)
du -sh /var/lib/postgresql

Find Large Files:

# Find largest files (MySQL/MariaDB)
find /var/lib/mysql -type f -exec du -h &#123;&#125; + | sort -rh | head -20

# Find largest files (PostgreSQL)
find /var/lib/postgresql -type f -exec du -h &#123;&#125; + | sort -rh | head -20

# Check log files
du -sh /var/log/sms_c/*

Solutions:

Clean Old Data:

-- Delete old messages
DELETE FROM message_queues
WHERE inserted_at < DATE_SUB(NOW(), INTERVAL 90 DAY)
LIMIT 100000;

Rotate Logs:

# Force logrotate
logrotate -f /etc/logrotate.d/sms_c

# Clear old log files
find /var/log/sms_c -name "*.log.*" -mtime +30 -delete

Expand Disk:

Resize volume (cloud)
Add new disk and extend volume
Move data to larger disk

Frontend Connection Issues

Frontend Not Showing as Active

Symptoms:

Frontend status shows "expired"
Frontend not in active list
Messages not being delivered to frontend

Diagnostic Steps:

Check Registration:

curl https://api.example.com:8443/api/frontends/active | grep frontend_name

Check Frontend Logs:

Is frontend calling /api/frontends/register?
Any API errors?
Registration frequency (should be every 60s)

Check API Logs:

grep "frontend.*register" /var/log/sms_c/application.log | tail -20

Solutions:

Frontend Not Registering:

Test manual registration:

curl -X POST https://api.example.com:8443/api/frontends/register \
  -H "Content-Type: application/json" \
  -d '&#123;
    "frontend_name": "uk_gateway",
    "frontend_type": "smpp",
    "ip_address": "10.0.1.50"
  &#125;'

If successful, problem is in frontend code/configuration.

Registration Timing Out:

Frontends expire after 90 seconds. Ensure registration every 60 seconds:

# Frontend should call register every 60 seconds
while True:
    register_with_smsc()
    time.sleep(60)

Network Issues:

Check firewall between frontend and API
Verify DNS resolution
Test with curl from frontend server

Frontend Repeatedly Connecting/Disconnecting

Symptoms:

Frontend status flipping between active/expired
High registration count in history
Unstable connection

Diagnostic Steps:

Check Frontend Health:

Is frontend process stable?
Any crashes or restarts?
Resource issues (CPU/memory)?

Check Network Stability:

# Check packet loss
ping -c 100 api.example.com

# Check connection resets
netstat -s | grep -i reset

Check Registration Timing:

Too frequent? (every few seconds)
Too infrequent? ( > 90 seconds)

Solutions:

Frontend Unstable:

Fix frontend application issues
Increase frontend resources
Check frontend logs for errors

Network Issues:

Check for intermittent connectivity
Review firewall logs
Check load balancer health checks

Wrong Registration Interval:

Correct interval:

REGISTRATION_INTERVAL = 60  # seconds

Charging/Billing Issues

Charging Failures

Charging is performed over Diameter Ro (Credit-Control). Note charging fails open: an OCS error or an unconfigured Diameter stack logs a failure but still delivers the message. Only a definitive out-of-balance answer rejects a message (status :balance_rejected).

Symptoms:

sms_c_charging_failed_count increasing
Event log shows "charging_failed" or "balance_rejected"
Messages marked with status :balance_rejected (out of balance), or log warnings about the Diameter stack not being enabled

Diagnostic Steps:

Check the Diameter Ro peer state (CER/CEA established to the DRA/OCS):

# In a remote console
:diameter.service_info(:omnimessage, :connections)

Confirm the stack is enabled and the :ro application is registered:

grep -E "diameter_enabled|application_name: :ro" config/runtime.exs

A log line Charging requested but the Diameter Ro stack is not enabled/configured means charging is being skipped (messages allowed) — enable the stack.

Check the rating profile matches the rating context configured on the OCS:

grep rating_profile config/runtime.exs

Solutions:

OCS / DRA Unreachable:

Verify IP reachability and that the DRA has this node provisioned as a Diameter peer (Origin-Host/Realm).
Confirm the CCR Service-Context-Id (rating_profile) matches the OCS's SMS rating profile.

Out-of-balance handling (config :sms_c, :charging):

config :sms_c, :charging,
  out_of_balance_action: :reject,    # or :auto_reply
  out_of_balance_reply_text: "...",
  rating_profile: "sms@sms-c"

Disable Charging Temporarily:

config :sms_c,
  default_charging_enabled: false

Restart application.

Account Issues:

Check account exists in OCS
Verify account has balance
Check rating plans are configured

Charging Too Slow

Symptoms:

sms_c_charging_succeeded_duration p95 > 500ms
Message processing slow when charging enabled
Fast when charging disabled

Diagnostic Steps:

Check Charging Latency:

histogram_quantile(0.95, sms_c_charging_succeeded_duration_bucket)

Check OCS / DRA performance (Diameter Ro round-trip). High sms_c_charging_succeeded_duration usually points at OCS rating latency or a slow/flapping DRA peer.
Check Network Latency to the DRA:

# Ping the DRA / OCS host
ping -c 10 dra01.example.com

Solutions:

OCS Slow:

Optimize OCS configuration
Add OCS resources
Use faster rating engine

Network Latency:

Deploy OCS closer to SMS-C
Use direct network path
Avoid VPN/tunnels if possible

Timeout Too Low:

Increase timeout:

config :sms_c,
  ocs_timeout: 5000  # 5 seconds

ENUM Lookup Problems

ENUM Lookups Failing

Symptoms:

sms_c_enum_lookup_stop_duration showing failures
Event log shows ENUM errors
Routes with enum_result_domain not matching

Diagnostic Steps:

Check ENUM Configuration:

grep -A 10 "enum_" config/runtime.exs

Test DNS Connectivity:

# Test DNS server
dig @8.8.8.8 e164.arpa

# Test ENUM query
# For +15551234567:
dig @8.8.8.8 NAPTR 7.6.5.4.3.2.1.5.5.5.1.e164.arpa

Check DNS Server:

# Is custom DNS reachable?
ping 10.0.1.53

# Test port
nc -zv 10.0.1.53 53

Solutions:

DNS Server Unreachable:

Use alternate DNS:

config :sms_c,
  enum_dns_servers: [
    &#123;"8.8.8.8", 53&#125;,  # Google Public DNS
    &#123;"1.1.1.1", 53&#125;   # Cloudflare DNS
  ]

ENUM Domain Wrong:

Update domain:

config :sms_c,
  enum_domains: ["e164.arpa"]  # Use standard domain

Timeout Too Short:

Increase timeout:

config :sms_c,
  enum_timeout: 10000  # 10 seconds

Disable ENUM (if not needed):

config :sms_c,
  enum_enabled: false

ENUM Cache Issues

Symptoms:

Low cache hit rate (< 70%)
Cache size growing unbounded
Memory usage high

Diagnostic Steps:

Check Cache Stats:

# Cache hit rate
rate(sms_c_enum_cache_hit_count[5m]) / (rate(sms_c_enum_cache_hit_count[5m]) + rate(sms_c_enum_cache_miss_count[5m]))

# Cache size
sms_c_enum_cache_size_size

Check Traffic Pattern:

Are numbers repeating?
Cache TTL appropriate?

Solutions:

Low Hit Rate (Expected):

Traffic to unique numbers (normal)
Monitor but don't alarm if < 70%

Cache Growing:

Clear cache via NAPTR Test page or restart application.

High Memory Usage:

Expected with large cache
Monitor overall system memory
Consider TTL adjustment

Cluster Issues

Node Can't Join Cluster

Symptoms:

Single node running
Cluster queries returning only local results
Erlang distribution errors

Diagnostic Steps:

Check Node Names:

# In IEx console
Node.self()
# Expected: :sms@node1.example.com

Node.list()
# Expected: List of other nodes

Check Erlang Cookie:

# Check cookie file
cat ~/.erlang.cookie

# Verify same on all nodes

Check Network:

# Can nodes reach each other?
ping node2.example.com

# Check ports
nc -zv node2.example.com 4369
nc -zv node2.example.com 9100-9200

Solutions:

Cookie Mismatch:

Set same cookie on all nodes:

export ERLANG_COOKIE=same_secret_value_here

# Or update ~/.erlang.cookie
echo "same_secret_value_here" > ~/.erlang.cookie
chmod 400 ~/.erlang.cookie

Firewall Blocking:

Open required ports:

# EPMD
iptables -A INPUT -p tcp --dport 4369 -j ACCEPT

# Erlang distribution
iptables -A INPUT -p tcp --dport 9100:9200 -j ACCEPT

DNS Issues:

Use IP addresses instead of hostnames:

config :sms_c,
  cluster_nodes: [
    :"sms@10.0.1.10",
    :"sms@10.0.1.11"
  ]

Cluster Split Brain

Symptoms:

Nodes running but disconnected
Different data on different nodes
Mnesia inconsistencies

Diagnostic Steps:

Check Node Connectivity:

# On each node (IEx)
Node.list()

Check Mnesia:

:mnesia.system_info(:running_db_nodes)

Solutions:

Reconnect Nodes:

# Stop all nodes
systemctl stop sms_c

# Start one node first
systemctl start sms_c  # On node1

# Wait for it to fully start, then start others
systemctl start sms_c  # On node2
systemctl start sms_c  # On node3

Mnesia Inconsistency:

Export routes from correct node
Stop all nodes
Delete Mnesia directory
Start nodes
Import routes

API Problems

API Not Responding

Symptoms:

Connection timeout
Connection refused
No response

Diagnostic Steps:

Check API Process:

# Is application running?
systemctl status sms_c

# Check listening ports
netstat -tlnp | grep 8443

Check Firewall:

# Check iptables
iptables -L -n | grep 8443

# Test local connectivity
curl -k https://localhost:8443/api/status

Check TLS Configuration:

# Check certificate exists
ls -l priv/cert/server.crt priv/cert/server.key

# Check certificate validity
openssl x509 -in priv/cert/server.crt -noout -dates

Solutions:

Application Not Running:

systemctl start sms_c

Firewall Blocking:

# Allow API port
iptables -A INPUT -p tcp --dport 8443 -j ACCEPT

Certificate Issues:

Generate new certificate (see Configuration Guide).

Wrong Port:

Check configuration:

grep "port:" config/runtime.exs

API Returning 500 Errors

Symptoms:

Internal Server Error
500 status code
Error in logs

Diagnostic Steps:

Check Application Logs:

tail -100 /var/log/sms_c/application.log | grep "\[error\]"

Check Database:

mysql -u sms_user -p -e "SELECT 1"

Check Resources:

# Memory
free -h

# CPU
top -b -n 1

# Disk
df -h

Solutions:

Database Unavailable:

Start database
Fix connection configuration

Out of Memory:

Restart application
Increase system memory
Check for memory leaks

Application Error:

Check specific error in logs
Fix configuration issue
Restart application

Web UI Issues

Can't Access Web UI

Symptoms:

Connection timeout
404 Not Found
Page won't load

Diagnostic Steps:

Check Application Status:

systemctl status sms_c

Check Port:

netstat -tlnp | grep 80

Check URL:

Correct hostname?
Correct port?
HTTP vs HTTPS?

Solutions:

Wrong Port:

Check configuration:

grep "control_panel" config/runtime.exs

Access on correct port (default: 80 or 4000).

Application Not Running:

systemctl start sms_c

Firewall:

iptables -A INPUT -p tcp --dport 80 -j ACCEPT

LiveView Not Updating

Symptoms:

Page loads but doesn't update
Data is stale
WebSocket errors in browser console

Diagnostic Steps:

Check Browser Console:

Open Developer Tools (F12)
Look for WebSocket errors
Check network tab for failed requests

Check Proxy Configuration:

If using reverse proxy, ensure WebSocket support:

location /live &#123;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
&#125;

Solutions:

WebSocket Blocked:

Configure proxy for WebSocket
Check firewall
Check browser extensions

Refresh Page:

Hard refresh (Ctrl+F5)
Clear browser cache

System Resource Issues

High CPU Usage

Symptoms:

CPU consistently > 80%
System slow
Application unresponsive

Diagnostic Steps:

Check Process:

top -b -n 1 | grep beam.smp

Check Metrics:

# Message processing rate
rate(sms_c_message_received_count[5m])

# Routing operations
rate(sms_c_routing_route_matched_count[5m])

Solutions:

High Traffic:

Scale horizontally (add nodes)
Scale vertically (add CPU)

Inefficient Routing:

Reduce number of routes
Optimize route criteria

Too Many ENUM Lookups:

Check cache hit rate
Consider disabling if not needed

High Memory Usage

Symptoms:

Memory usage > 90%
Application crashes
Out of memory errors

Diagnostic Steps:

Check Memory:

free -h

ps aux | grep beam.smp

Check Cache Sizes:

sms_c_enum_cache_size_size

Solutions:

ENUM Cache Too Large:

Clear cache
Reduce TTL
Disable ENUM if not needed

Batch Queue Growing:

# Check worker stats (IEx)
SmsC.Messaging.BatchInsertWorker.stats()

If queue is large, flush manually or restart.

Add Memory:

Scale vertically
Add swap (temporary)

Memory Leak:

Restart application
Report issue for investigation

For additional assistance, consult:

Operations Guide - Daily procedures
Configuration Guide - Configuration options
Metrics Guide - Monitoring setup
Application logs - /var/log/sms_c/application.log

Table of Contents​

Diagnostic Tools​

Quick Health Check​

Log Analysis​

Metric Queries​

Message Delivery Issues​

Messages Not Being Delivered​

Messages Failing with Retries​

Dead Letter Messages​

Routing Problems​

No Route Found​

Wrong Route Selected​

Auto-Reply Not Working​

Performance Issues​

High Message Processing Latency​

Low Message Throughput​

Queue Backlog Growing​

Database Problems​

Connection Failures​

Slow Queries​

Disk Space Full​

Frontend Connection Issues​

Frontend Not Showing as Active​

Frontend Repeatedly Connecting/Disconnecting​

Charging/Billing Issues​

Charging Failures​

Charging Too Slow​

ENUM Lookup Problems​

ENUM Lookups Failing​

ENUM Cache Issues​

Cluster Issues​

Node Can't Join Cluster​

Cluster Split Brain​

API Problems​

API Not Responding​

API Returning 500 Errors​

Web UI Issues​

Can't Access Web UI​

LiveView Not Updating​

System Resource Issues​

High CPU Usage​

High Memory Usage​

Table of Contents

Diagnostic Tools

Quick Health Check

Log Analysis

Metric Queries

Message Delivery Issues

Messages Not Being Delivered

Messages Failing with Retries

Dead Letter Messages

Routing Problems

No Route Found

Wrong Route Selected

Auto-Reply Not Working

Performance Issues

High Message Processing Latency

Low Message Throughput

Queue Backlog Growing

Database Problems

Connection Failures

Slow Queries

Disk Space Full

Frontend Connection Issues

Frontend Not Showing as Active

Frontend Repeatedly Connecting/Disconnecting

Charging/Billing Issues

Charging Failures

Charging Too Slow

ENUM Lookup Problems

ENUM Lookups Failing

ENUM Cache Issues

Cluster Issues

Node Can't Join Cluster

Cluster Split Brain

API Problems

API Not Responding

API Returning 500 Errors

Web UI Issues

Can't Access Web UI

LiveView Not Updating

System Resource Issues

High CPU Usage

High Memory Usage