Performance Tuning Guide

← Back to Documentation Index | Main README

This guide explains how to optimize SMS-C performance for different workload scenarios.

Performance Overview

SMS-C delivers 1,750 messages/second throughput using Mnesia for in-memory message storage with automatic SQL database archiving for CDR retention.

Key Performance Metrics

Measured on Intel i7-8650U @ 1.90GHz (8 cores):

Operation	Throughput	Latency (avg)	Improvement
Message Insert (with routing)	1,750 msg/sec	0.58ms	21x faster than SQL
Message Insert (simple)	1,750 msg/sec	0.57ms	21x faster than SQL
Get Messages for SMSC	800 msg/sec	1.25ms	In-memory query
Memory per Insert	62 KB	-	50% reduction

Capacity: ~150 million messages per day on single node

Table of Contents

• Message Storage Architecture
• Mnesia Optimization
• CDR Archiving Configuration
• Query Optimization
• Benchmarking

Message Storage Architecture

SMS-C uses a dual-storage architecture for optimal performance:

Active Message Store (Mnesia)

• Purpose: Ultra-fast message insertion, routing, and delivery
• Storage: In-memory with disk persistence (disc_copies)
• Performance: 1,750 msg/sec insert throughput, 0.58ms latency
• Retention: Configurable (default: 24 hours)
• Clustering: Supports distributed Mnesia for horizontal scaling

CDR Archive (SQL Database)

• Purpose: Long-term message history and reporting
• Storage: SQL database (MySQL/MariaDB or PostgreSQL) for durable archival
• Performance: Batched writes to minimize database load
• Retention: Permanent (or per data retention policy)
• Queries: Analytics, reporting, compliance

Data Flow

Mnesia Optimization

Message Retention Configuration

# config/runtime.exs
config :sms_c,
  message_retention_hours: 24  # Default: 24 hours

Tuning Guidelines:

• High volume (>1M msg/day): 12-24 hours retention

  • Minimizes Mnesia table size
  • Faster queries
  • More frequent archiving to MySQL

• Medium volume (100K-1M msg/day): 24-48 hours retention

  • Good balance for most deployments
  • Adequate buffer for retry logic

• Low volume (<100K msg/day): 48-168 hours retention

  • Longer message history in fast storage
  • Less frequent archiving

Mnesia Table Indices

MessageStore automatically creates indices on:

• status - For filtering pending/delivered messages
• dest_smsc - For SMSC-specific queries
• expires - For expiration handling
• destination_msisdn - For subscriber queries
• source_msisdn - For subscriber queries

Mnesia Disc Persistence

Messages are stored as disc_copies providing:

• ✅ In-memory performance
• ✅ Automatic disk persistence
• ✅ Crash recovery
• ✅ No data loss on restart

CDR Archiving Configuration

The BatchInsertWorker handles CDR archiving to MySQL using batched writes:

# config/runtime.exs
config :sms_c,
  batch_insert_batch_size: 100,           # CDR batch size
  batch_insert_flush_interval_ms: 100     # Auto-flush interval

CDR Tuning Guidelines

High Volume Archiving

batch_insert_batch_size: 200
batch_insert_flush_interval_ms: 200

• Larger batches reduce MySQL load
• Higher latency for CDR writes (acceptable for archiving)

Balanced (Recommended)

batch_insert_batch_size: 100
batch_insert_flush_interval_ms: 100

• Good balance for most deployments
• CDRs written within 100ms

Real-time CDR Requirements

batch_insert_batch_size: 20
batch_insert_flush_interval_ms: 20

• Faster CDR writes for compliance
• More MySQL write operations

Query Optimization

Using Mnesia Indices Effectively

Queries that use indexed fields are fastest:

# Fast queries (use indices)
MessageStore.list(status: :pending)
MessageStore.list(dest_smsc: "gateway-1")
Messaging.get_messages_for_smsc("gateway-1")

# Slower queries (full table scan)
MessageStore.list(limit: :infinity)  # Returns all messages

MySQL Connection Pool

For CDR queries and archiving, configure MySQL connection pool:

# config/runtime.exs
config :sms_c, SmsC.Repo,
  pool_size: 10  # Increase for heavy CDR reporting

Guidelines:

• Standard deployment: pool_size: 10
• Heavy CDR reporting: pool_size: 20-30
• Archiving only: pool_size: 5

Benchmarking

Running Benchmarks

The project includes Benchee-based benchmarks for performance testing:

# Raw SMS API benchmark (compares sync vs async)
mix run benchmarks/raw_sms_bench.exs

# General message API benchmark
mix run benchmarks/message_api_bench.exs

Interpreting Results

Example output:

Name                                       ips        average  deviation         median         99th %
submit_message_raw_async (batch)        4.65 K        0.22 ms    ±41.72%       0.184 ms        0.55 ms
submit_message_raw (sync)             0.0696 K       14.36 ms    ±33.42%       12.57 ms       33.71 ms

Key metrics:

• ips: Iterations per second (higher is better)
• average: Average execution time (lower is better)
• median: Middle value, more representative than average for skewed distributions
• 99th %: 99th percentile latency (important for SLA compliance)

Performance Baseline

Expected performance on modern hardware (Intel i7-8650U, 8 cores):

Metric	insert_message (Mnesia)	Previous (MySQL)
Throughput (with routing)	1,750 msg/sec	83 msg/sec
Throughput (simple)	1,750 msg/sec	89 msg/sec
Response Time (avg)	0.58ms	16ms
Response Time (p99)	<5ms	30ms
Memory per operation	62 KB	121 KB
Performance Gain	21x faster	-

Key Improvements:

• ✅ Removed duplicate number translation calls
• ✅ Async post-processing (routing, charging, events)
• ✅ Mnesia in-memory storage vs MySQL disk I/O
• ✅ 50% memory reduction

Monitoring

Runtime Statistics

Check batch worker statistics:

SmsC.Messaging.BatchInsertWorker.stats()

Returns:

%{
  total_enqueued: 10000,
  total_flushed: 9900,
  total_batches: 99,
  current_queue_size: 100,
  flush_errors: 0,
  last_flush_at: ~U[2025-10-22 12:34:56Z],
  last_flush_count: 100,
  last_flush_duration_ms: 45
}

Key Metrics to Monitor

1. Queue Size: current_queue_size - Should be below batch_size most of the time
2. Flush Duration: last_flush_duration_ms - Should be < 100ms for batch_size=100
3. Flush Errors: flush_errors - Should be 0 or very low
4. Throughput: total_flushed / uptime - Should match expected load

Alerts

Set up monitoring alerts for:

• Queue size consistently at max (indicates backpressure)
• Flush duration increasing (database performance degradation)
• Flush errors > 0 (database connectivity issues)
• Throughput below expected (performance degradation)

Troubleshooting

Symptom: Low Throughput

Possible causes:

1. Database connection pool exhausted: Increase pool_size
2. Slow database: Check query performance, add indexes
3. Network latency: Optimize network path to database
4. Batch size too small: Increase batch_insert_batch_size

Symptom: High Latency

Possible causes:

1. Flush interval too high: Reduce batch_insert_flush_interval_ms
2. Batch size too high: Reduce batch_insert_batch_size
3. Database slow writes: Check disk I/O, optimize tables
4. Using async API when you need sync: Switch to synchronous endpoint

Symptom: Memory Issues

Possible causes:

1. Queue backing up: Messages accumulating faster than flushing
2. Batch size too large: Reduce batch_insert_batch_size
3. Flush failures: Check flush_errors in stats
4. Need to restart worker: Supervisor.terminate_child/2 and restart

Best Practices

1. Start with defaults (100/100ms) and tune based on observed behavior
2. Monitor in production for at least 1 week before optimizing
3. Test configuration changes in staging with production-like load
4. Use benchmarks to validate configuration changes
5. Document your tuning decisions for future reference
6. Set up alerts before optimizing to catch regressions
7. Consider time zones - peak load varies by region

Example Configurations

Configuration: High-Volume Aggregator

# config/prod.exs
config :sms_c,
  batch_insert_batch_size: 200,
  batch_insert_flush_interval_ms: 200

config :sms_c, SmsC.Repo,
  pool_size: 50

Configuration: Enterprise Real-Time Messaging

# config/prod.exs
config :sms_c,
  batch_insert_batch_size: 20,
  batch_insert_flush_interval_ms: 10

config :sms_c, SmsC.Repo,
  pool_size: 20

Configuration: Development/Testing

# config/dev.exs
config :sms_c,
  batch_insert_batch_size: 10,
  batch_insert_flush_interval_ms: 50

config :sms_c, SmsC.Repo,
  pool_size: 5

Further Reading

• Ecto Performance Guide
• Benchee Documentation
• Phoenix Under Pressure