Skip to main content

SMPP Message Cache

Overview​

The SMPP Message Cache is a local persistence layer that allows the SMPP gateway to continue accepting inbound messages even when the backend API is unavailable. Messages are cached locally in Mnesia and automatically delivered to the API when connectivity is restored, using intelligent retry logic with exponential backoff.

Features​

  • Resilient Message Acceptance - Continue accepting SMPP messages during API outages
  • Persistent Storage - Uses Mnesia with :disc_copies for durability across restarts
  • Automatic Retry - Background workers automatically attempt delivery with exponential backoff
  • Per-Bind Configuration - Enable/disable caching independently for each SMPP bind
  • Overflow Protection - FIFO eviction when cache reaches configured size limit
  • Failed Message Retention - Permanently failed messages kept for manual review
  • Real-Time Monitoring - LiveView dashboard with cache statistics and metrics
  • Prometheus Metrics - Full metrics export for monitoring and alerting

Architecture​

Message Flow​

With Cache Enabled​

SMPP Client → submit_sm → Gateway Server
                             ↓
                      Cache in Mnesia (immediate response)
                             ↓
                      CacheFlushWorker (background)
                             ↓
                        Backend API

With Cache Disabled​

SMPP Client → submit_sm → Gateway Server
                             ↓
                        Backend API (direct)

Components​

  1. MessageCache Module (lib/sms_c/smpp/message_cache.ex)

    • Core caching logic
    • Overflow handling
    • Query functions for LiveView and workers

  2. CacheFlushWorker (lib/sms_c/smpp/cache_flush_worker.ex)

    • GenServer per bind with caching enabled
    • Polls for messages ready for retry
    • Implements exponential backoff
    • Marks permanently failed messages

  3. Mnesia Table (:smpp_message_cache)

    • Persistent storage with :disc_copies
    • Indexed by bind_name, next_retry_at, and status
    • Survives application restarts

Configuration​

Global Settings​

Edit config/runtime.exs:

config :omnimessage_smpp,
  # How often flush workers poll for messages (milliseconds)
  cache_flush_interval: 10_000,

  # Maximum retry attempts before marking as failed_permanent
  cache_max_retry_attempts: 10,

  # Exponential backoff multiplier
  cache_backoff_multiplier: 2,

  # Mnesia storage type (:disc_copies or :ram_copies)
  mnesia_storage_type: :disc_copies

Per-Bind Configuration​

Each SMPP bind (client or server) can be configured independently:

config :omnimessage_smpp, :binds, [
  %{
    name: "my_smpp_bind",
    mode: :server,
    system_id: "username",
    password: "password",

    # Cache configuration
    cache_enabled: true,         # Enable caching (default: false)
    cache_max_size: 10_000,      # Max messages to cache (default: 10,000)
    cache_retry_interval: 60     # Base retry interval in seconds (default: 60)
  }
]

Environment Variables​

# Global cache settings
CACHE_FLUSH_INTERVAL=10000              # Flush poll interval (ms)
CACHE_MAX_RETRY_ATTEMPTS=10             # Max retries before permanent failure
CACHE_BACKOFF_MULTIPLIER=2              # Exponential backoff multiplier
MNESIA_STORAGE_TYPE=disc_copies         # Mnesia storage type

Retry Behavior​

Exponential Backoff​

When message delivery fails, the retry interval doubles with each attempt:

Base interval: 60 seconds
Backoff multiplier: 2

Retry 0: 60s   (1 minute)
Retry 1: 120s  (2 minutes)
Retry 2: 240s  (4 minutes)
Retry 3: 480s  (8 minutes)
Retry 4: 960s  (16 minutes)
Retry 5: 1920s (32 minutes)
...
Retry 9: 30,720s (8.5 hours)

Permanent Failure​

After 10 failed attempts (by default), messages are marked as failed_permanent and:

  • Remain in the cache for manual review
  • Stop being retried automatically
  • Appear in the "Failed Permanent" section of the cache dashboard
  • Can be manually retried or cleared

Status Transitions​

:pending → :delivering → SUCCESS (deleted from cache)
                      → FAILURE → :pending (retry with backoff)
                               → :failed_permanent (after max retries)

Monitoring​

LiveView Dashboard​

Access the cache dashboard at http://your-server:4000/smpp → "Message Cache" tab

Summary Cards:

  • Total Cached - All messages currently in cache
  • Pending Delivery - Messages waiting for retry
  • Failed Permanent - Messages that exceeded max retries

Per-Bind Table:

  • Bind name
  • Cached message count
  • Pending / Failed breakdown
  • Max cache size
  • Utilization percentage (with visual progress bar)

Prometheus Metrics​

# Current cache size per bind
smpp_cache_size{bind_name="my_bind",mode="server"} 42

# Total successfully delivered messages
smpp_cache_delivered_total{bind_name="my_bind"} 1234

# Total retry attempts
smpp_cache_retry_total{bind_name="my_bind"} 56

# Total permanent failures
smpp_cache_permanent_failures_total{bind_name="my_bind"} 2

# Total overflow events (messages dropped)
smpp_cache_overflow_total{bind_name="my_bind"} 0

Log Messages​

INFO  Message -123456789 cached for my_smpp_bind
INFO  Successfully delivered cached message -123456789, API ID: 99999
WARN  Failed to deliver message -123456789 (retry 3/10), next retry at 2026-02-01 12:34:56Z
ERROR Message -123456789 exceeded max retries (10), marking as failed_permanent
WARN  Cache overflow for my_smpp_bind: deleted oldest message

Operations​

Enable/Disable Caching​

Via LiveView UI​

  1. Navigate to http://your-server:4000/smpp
  2. Go to "Client Peers" or "Server Peers" tab
  3. Edit the bind
  4. Toggle "Cache Enabled" checkbox
  5. Save changes

Via IEx Console​

# Enable caching for a bind
SmsC.SMPPConfig.update_server_peer("my_bind", "username", "password",
  cache_enabled: true,
  cache_max_size: 10_000,
  cache_retry_interval: 60
)

# Disable caching
SmsC.SMPPConfig.update_server_peer("my_bind", "username", "password",
  cache_enabled: false
)

Monitor Cache Status​

# Get global summary
SmsC.SMPP.MessageCache.get_cache_summary()
# => %{total_cached: 42, pending: 40, failed: 2}

# Get per-bind breakdown
SmsC.SMPP.MessageCache.get_cache_by_bind()
# => [
#   %{bind_name: "bind1", total: 30, pending: 28, failed: 2, max_size: 10_000},
#   %{bind_name: "bind2", total: 12, pending: 12, failed: 0, max_size: 10_000}
# ]

# Count messages for specific bind
SmsC.SMPP.MessageCache.count_cached_messages("my_bind")
# => 42

Manual Interventions​

Clear Failed Messages​

# Get all failed messages for a bind
{:atomic, failed_messages} = :mnesia.transaction(fn ->
  :mnesia.match_object({:smpp_message_cache, :_, "my_bind", :_, :_, :_, :_, :_, :_, :failed_permanent})
end)

# Delete them
Enum.each(failed_messages, fn {_, {bind_name, msg_id}, _, _, _, _, _, _, _, _} ->
  SmsC.SMPP.MessageCache.delete_cache_record(bind_name, msg_id)
end)

Force Retry of Failed Message​

# Reset a failed_permanent message to pending
SmsC.SMPP.MessageCache.update_cache_record("my_bind", -123456, %{
  status: :pending,
  retry_count: 0,
  next_retry_at: DateTime.utc_now(),
  last_error: nil
})

Troubleshooting​

Cache Full / Overflow Events​

Symptom: cache_overflow_total metric increasing, oldest messages being dropped

Cause: Cache size limit reached

Solutions:

  1. Increase cache_max_size for the bind
  2. Investigate why API delivery is failing (check API logs, network)
  3. Manually clear old failed messages
  4. Check if flush interval is too slow

Messages Not Being Delivered​

Symptom: Messages stuck in :pending status

Possible Causes:

  1. API is down

    • Check API availability
    • Verify backend API logs
    • Check network connectivity

  2. next_retry_at is in the future

    • Messages will be retried when next_retry_at is reached
    • Check exponential backoff schedule

  3. Flush worker not running

    # Check if workers are running
    Supervisor.which_children(SmsC.SMPP.Supervisor)

  4. Cache disabled

    • Verify cache_enabled: true in bind configuration

High Retry Counts​

Symptom: Many messages with high retry_count values

Investigation:

# Find messages with high retry counts
{:atomic, messages} = :mnesia.transaction(fn ->
  :mnesia.match_object({:smpp_message_cache, :_, "my_bind", :_, :_, :_, :_, :_, :_, :_})
end)

high_retry = Enum.filter(messages, fn {_, _, _, _, _, _, retry_count, _, _, _} ->
  retry_count >= 5
end)

# Check last_error for each
Enum.each(high_retry, fn {_, _, _, msg_id, _, _, retry_count, _, last_error, _} ->
  IO.puts("Message #{msg_id}: #{retry_count} retries, error: #{inspect(last_error)}")
end)

Mnesia Disk Space​

Symptom: Disk space filling up

Check Mnesia directory:

ls -lh Mnesia.*
du -sh Mnesia.*

Clean up:

  1. Clear old failed messages (see Manual Interventions above)
  2. Reduce cache_max_size per bind
  3. Enable cache overflow (ensure proper FIFO eviction)

Performance Considerations​

Memory Usage​

  • Each cached message uses approximately 500-1000 bytes (depending on message size)
  • 10,000 messages ≈ 5-10 MB memory
  • With :disc_copies, data is also written to disk

CPU Usage​

  • Flush workers poll every 10 seconds by default (configurable)
  • Batch processing (100 messages per cycle) reduces overhead
  • Concurrent delivery (max 10 concurrent API calls per worker)

Disk I/O​

  • :disc_copies writes to disk on every transaction
  • For very high throughput (>1000 msg/sec), consider:
    • Using :ram_copies (loses persistence)
    • Increasing flush intervals
    • Scaling horizontally
Scenariocache_max_sizecache_flush_interval
Low volume (<100 msg/sec)10,00010,000ms
Medium volume (100-500 msg/sec)50,0005,000ms
High volume (>500 msg/sec)100,0003,000ms

Recovery Scenarios​

Application Restart​

  1. Mnesia automatically loads :disc_copies tables from disk
  2. Cached messages remain intact
  3. Flush workers restart and continue processing

Database Migration​

When upgrading from a version without cache support:

  1. Migration automatically adds cache fields to existing binds
  2. Default values: cache_enabled: false, cache_max_size: 10_000, cache_retry_interval: 60
  3. No data loss
  4. Cache table created on first run

API Outage Recovery​

  1. Messages accumulate in cache during outage
  2. When API recovers, flush workers automatically deliver
  3. Oldest messages delivered first (FIFO)
  4. Exponential backoff prevents API overload during recovery

Best Practices​

  1. Enable Caching by Default - Prevents message loss during outages
  2. Monitor Metrics - Set up alerts on cache_permanent_failures_total and cache_overflow_total
  3. Size Appropriately - Set cache_max_size based on expected outage duration
  4. Review Failed Messages - Regularly check failed_permanent messages for patterns
  5. Test Failover - Simulate API outages to verify cache behavior
  6. Adjust Retry Intervals - Tune based on API recovery time patterns
  7. Use Persistent Storage - Keep mnesia_storage_type: disc_copies in production

See Also​

Last updated on