Skip to main content

Data Retention Policy Guide

Overview​

The RAN Monitor application now includes a comprehensive Data Retention Policy system that allows you to manage how long performance metrics, configuration data, and alarm records are stored in InfluxDB. This guide covers everything you need to know about managing data retention.

🎯 Quick Start​

Accessing the Retention Policy Dashboard​

  1. Navigate to the Control Panel: https://localhost:9443
  2. Click on Data Retention in the navigation menu
  3. View and manage retention settings for all configured eNodeBs

Setting a Custom Retention Period​

  1. Find the eNodeB in the list
  2. Update the "Retention Period" field (in hours)
  3. The setting saves immediately
  4. Falls back to global default if left empty

Cleaning Old Data​

  1. Click Clean Old Data button to remove records older than the retention period
  2. Or click Clear All Data to delete all records for that eNodeB (use with caution!)

Screenshot​

Data Retention Dashboard

The Data Retention dashboard showing retention settings and record counts for each eNodeB

📋 Features​

Global Retention Settings​

  • Default Retention Period: 720 hours (30 days)
  • Configurable: Change in config/config.exs
  • Fallback: Applied to all eNodeBs without custom settings

Per-eNodeB Retention​

  • Override Global: Set custom retention for specific eNodeBs
  • Database Stored: Persisted in the airscales table
  • Real-Time: Takes effect immediately

Automatic Cleanup​

  • Scheduled: Runs every hour automatically
  • Background Worker: RanMonitor.Data.RetentionCleanupWorker
  • Per-eNodeB: Respects individual retention settings
  • Logged: All cleanups are logged for audit trail

Data Visibility​

  • Record Counts: See how many records per measurement type:
    • Performance Metrics
    • Configuration
    • Alarms
  • Total Summary: View total records per eNodeB
  • Real-Time: Updated on page refresh

🖥️ User Interface​

Dashboard Layout​

Data Retention Dashboard

The Data Retention dashboard showing global settings, per-eNodeB retention periods, and record counts

Layout Overview:

┌─────────────────────────────────────────────────────────────┐
│                   Data Retention Policy                      │
│                                                              │
│  ┌────────────────────────────────────────────────────────┐ │
│  │ GLOBAL SETTINGS                                        │ │
│  │                                                        │ │
│  │ Default Retention: 30 days  | Total Records: 1.2M     │ │
│  │ Auto-Cleanup: ✓ Enabled (runs hourly)                 │ │
│  └────────────────────────────────────────────────────────┘ │
│                                                              │
│  ┌────────────────────────────────────────────────────────┐ │
│  │ eNodeB RETENTION SETTINGS                              │ │
│  │                                                        │ │
│  │ ┌─ SITE-01 ────────────────────────────────────────┐ │ │
│  │ │ Status: REGISTERED                               │ │ │
│  │ │ Retention: 720 hours (30 days)                   │ │ │
│  │ │                                                  │ │ │
│  │ │ Data Records:                                    │ │ │
│  │ │   Performance Metrics: 250,000                   │ │ │
│  │ │   Configuration: 5,000                           │ │ │
│  │ │   Alarms: 15,000                                 │ │ │
│  │ │   ────────────────────                           │ │ │
│  │ │   Total: 270,000                                 │ │ │
│  │ │                                                  │ │ │
│  │ │ [Clean Old Data] [Clear All Data]                │ │ │
│  │ └──────────────────────────────────────────────────┘ │ │
│  │                                                        │ │
│  │ (More eNodeBs below...)                               │ │
│  └────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘

Status Indicators​

  • Green (✓): eNodeB registered and active
  • Red (✗): eNodeB pending or unregistered
  • Disabled: Cannot modify settings for non-registered eNodeBs

Action Buttons​

ButtonActionEffect
Clean Old DataRemove old recordsDeletes records older than retention period
Clear All DataComplete wipeDeletes ALL records (⚠️ use with caution!)
RefreshUpdate displayRe-fetches record counts and settings

⚙️ Configuration​

Global Retention Configuration​

Edit config/config.exs:

config :ran_monitor,
  ecto_repos: [RanMonitor.Repo],
  generators: [context_app: :ran_monitor],
  data_retention_hours: 720  # 30 days, adjust as needed

Supported Time Values​

PeriodHoursDaysRecommended For
1 hour10.04Testing only
1 day241Short-term metrics
7 days1687Weekly reports
14 days33614Bi-weekly reports
30 days72030Monthly reports (default)
90 days216090Long-term trends
180 days4320180Bi-annual reports
1 year8760365Annual reports

Environment Variables​

Optionally override at runtime:

export DATA_RETENTION_HOURS=1440  # 60 days
mix phx.server

🔄 How It Works​

Data Retention Flow​

1. DATA INSERTION
   ├─ Performance Metrics → InfluxDB
   ├─ Configuration Data → InfluxDB
   └─ Alarms → InfluxDB

2. AUTOMATIC CLEANUP (Hourly)
   ├─ RetentionCleanupWorker triggers
   ├─ For each eNodeB:
   │  ├─ Get effective retention (per-eNodeB or global)
   │  ├─ Calculate cutoff timestamp
   │  └─ Delete records older than cutoff
   └─ Log results

3. MANUAL CLEANUP (On Demand)
   ├─ User clicks button in UI
   ├─ Retention policy applied
   ├─ Records deleted immediately
   └─ Success/error notification shown

4. MONITORING
   └─ Record counts displayed in UI

Retention Logic​

Per-eNodeB Effective Retention:

effective_retention = case airscale.retention_hours do
  nil -> Config.data_retention_hours()      # Use global (720h)
  hours -> hours                             # Use per-eNodeB custom value
end

Example:

  • Global default: 720 hours (30 days)
  • eNodeB "SITE-01" custom: 168 hours (7 days)
  • eNodeB "SITE-02" custom: nil → uses global 720 hours

Cleanup Process​

Cutoff Timestamp = Now - (retention_hours * 3600 seconds)

Example with 30-day retention:
├─ Current: 2025-12-11 10:00:00
├─ Retention: 720 hours (30 days)
├─ Cutoff: 2025-11-11 10:00:00
└─ Delete all records with timestamp < cutoff

📊 Monitoring & Logging​

Log Entries​

The system logs all retention activities. Look for:

[RetentionCleanupWorker] Starting retention cleanup worker
[RetentionCleanupWorker] Cleaning data for SITE-01 (retention: 720h)
[RetentionCleanupWorker] Deleted 15,000 records for SITE-01
[RetentionCleanupWorker] Cleanup cycle complete: 5 successful, 0 failed, 75,000 total deleted

Monitoring Record Counts​

Real-time Visibility:

  1. Open Data Retention dashboard
  2. View current record counts per measurement per eNodeB
  3. Click "Refresh" to update counts

Historical Tracking:

  • Check application logs for cleanup summaries
  • Monitor InfluxDB disk usage over time
  • Set up alerts based on record count growth

🛠️ Advanced Usage​

Programmatic Access​

Use the retention policy service in your code:

alias RanMonitor.Data.RetentionPolicy
alias RanMonitor.Database.Nokia

# Get effective retention for an eNodeB
airscale = Nokia.get_airscale!(1)
hours = RetentionPolicy.get_retention_hours(airscale)
# => 720 (or custom value if set)

# Get record counts for an eNodeB
counts = RetentionPolicy.get_record_counts("SITE-01")
# => %{"PerformanceMetrics" => 250000, "Configuration" => 5000, "Alarms" => 15000}

# Get total records
total = RetentionPolicy.get_total_record_count("SITE-01")
# => 270000

# Delete old records manually
{:ok, deleted_count} = RetentionPolicy.delete_old_records("SITE-01", 720)
# => {:ok, 50000}  (50k records deleted)

# Clear all records for an eNodeB
{:ok, deleted_count} = RetentionPolicy.clear_all_records("SITE-01")
# => {:ok, 270000}  (all 270k records deleted)

Adjusting Cleanup Interval​

Edit lib/ran_monitor/data/retention_cleanup_worker.ex:

# Change from 1 hour (3600000ms) to 30 minutes (1800000ms)
@cleanup_interval_ms 1800000  # 30 minutes

Then recompile:

mix compile

Database-Level Queries​

View retention settings directly:

SELECT name, retention_hours FROM airscales;

Update retention via database:

UPDATE airscales
SET retention_hours = 168
WHERE name = 'SITE-01';

📈 Best Practices​

Retention Period Selection​

Short-term (< 7 days)

  • Use for: Testing, staging environments
  • Not recommended for: Production
  • Risk: May delete important historical data

Standard (7-30 days)

  • Use for: Production deployments with typical storage
  • Best for: Most use cases
  • Balance: Good history with manageable storage

Long-term (> 30 days)

  • Use for: Trend analysis, compliance requirements
  • Cost: Higher storage requirements
  • Benefit: Extended historical data
Use CaseRetentionReason
Daily reports7-14 daysWeekly review cycles
Weekly reports30-60 daysMonthly summaries
Monthly reports90 daysQuarterly analysis
Trend analysis180-365 daysLong-term patterns
ComplianceAs requiredLegal/regulatory

Storage Considerations​

Estimate storage needs:

  • 1000 records ≈ 1-5 KB (depending on measurement type)
  • 1 million records ≈ 1-5 GB
  • Retention period × collection rate = total storage

Monitor growth with:

# Check InfluxDB bucket size
influx bucket list

# Or check disk usage
df -h /path/to/influxdb/data

🔐 Security & Compliance​

Data Privacy​

  • No encryption at rest by default
  • Network access controlled via InfluxDB security
  • Access logs available in application logs

Compliance​

  • Audit trail: All cleanups logged with timestamp
  • Data integrity: Soft deletes, no hard wipes at application level
  • Retention proof: Logs show what was retained/deleted

Recommendations​

  1. Enable InfluxDB authentication for production
  2. Monitor cleanup logs regularly
  3. Set retention carefully to balance compliance and storage
  4. Backup before bulk operations if critical data
  5. Test retention policies in staging first

🐛 Troubleshooting​

Issue: Cleanup Not Running​

Symptoms:

  • Records older than retention period still exist
  • No cleanup log entries

Solutions:

  1. Check application is running: ps aux | grep mix
  2. Verify RetentionCleanupWorker started:
    • Check logs for [RetentionCleanupWorker] Starting
  3. Check InfluxDB connection:
    • Visit InfluxDB Status page: https://localhost:9443/nokia/influx
  4. Verify retention settings are configured:
    • Check config/config.exs for data_retention_hours

Issue: Manual Cleanup Failed​

Symptoms:

  • Error message when clicking "Clean Old Data"
  • Records not deleted

Solutions:

  1. Check InfluxDB is accessible:
    • Test connection in dashboard
  2. Verify record counts are accurate:
    • Click "Refresh" to update
  3. Check application logs for errors:
    • Look for [RetentionPolicy] error entries
  4. Verify eNodeB is registered:
    • Check BTS Status page

Issue: High Memory Usage After Cleanup​

Symptoms:

  • Application becomes slow after cleanup
  • Memory usage spikes

Solutions:

  1. This is normal for large deletes
  2. Allow 5-10 minutes for memory to normalize
  3. Consider reducing cleanup frequency:
    • Change @cleanup_interval_ms (default 1 hour)
  4. Or reduce retention period for affected eNodeBs

Issue: Incorrect Record Counts​

Symptoms:

  • Record counts don't match InfluxDB UI
  • "Clean Old Data" shows different numbers

Solutions:

  1. Click "Refresh" to force update
  2. Check InfluxDB query:
    • May take time to reflect recent deletes
  3. Wait a minute and try again:
    • InfluxDB may be processing delete operations
  4. Check eNodeB name matches exactly:
    • Case-sensitive comparison

📖 Related Documentation​

🔗 Access Points​

  • Data Retention Dashboard: https://localhost:9443/nokia/retention
  • BTS Status: https://localhost:9443/nokia/status
  • InfluxDB Status: https://localhost:9443/nokia/influx
  • Live Logs: https://localhost:9443/nokia/logs

💡 FAQ​

Q: Will cleanup delete active data?​

A: No. Only records older than the retention period are deleted. Currently being collected data is never affected.

Q: Can I set different retention for different eNodeBs?​

A: Yes! Each eNodeB can have its own retention setting. If not set, it uses the global default.

Q: How often does automatic cleanup run?​

A: Every hour by default. Adjust @cleanup_interval_ms in the worker if needed.

Q: What happens if I clear all data?​

A: All records (Performance Metrics, Configuration, Alarms) for that eNodeB are permanently deleted. This cannot be undone.

Q: Can cleanup affect data collection?​

A: No. Cleanup and data collection are independent. New data will continue being written while old data is deleted.

Q: How long does cleanup take?​

A: Depends on record count:

  • Small (< 100k): < 1 second
  • Medium (100k-1M): 1-10 seconds
  • Large (> 1M): 10-60+ seconds

Q: Can I manually delete specific records?​

A: Not via the UI. Only full cleanup or complete clear available. For granular deletions, use InfluxDB CLI or API directly.

Q: What if InfluxDB is unavailable?​

A: Cleanup will fail silently and retry next hour. Data collection continues unaffected.

Q: Does cleanup affect performance?​

A: Minor impact during cleanup (seconds to minutes depending on data size). Hourly interval chosen to minimize impact.

📝 Implementation Details​

Files Modified​

FileChanges
lib/ran_monitor/database/nokia/airscale.exAdded retention_hours field
lib/ran_monitor/config/config.exAdded data_retention_hours() getter
config/config.exsAdded global retention config and page route
lib/ran_monitor/application.exAdded cleanup worker to supervision tree

Files Created​

FilePurpose
lib/ran_monitor/data/retention_policy.exCore retention service
lib/ran_monitor/data/retention_cleanup_worker.exHourly cleanup GenServer
lib/ran_monitor/web/live/retention_policy_live.exManagement UI page
priv/repo/migrations/20251211065257_add_retention_hours_to_airscales.exsDatabase migration

Key Functions​

RetentionPolicy Module:

  • get_retention_hours(airscale) - Get effective retention
  • get_record_counts(airscale_name) - Fetch record counts
  • get_total_record_count(airscale_name) - Total count
  • delete_old_records(name, hours) - Clean old records
  • clear_all_records(name) - Complete wipe

RetentionCleanupWorker GenServer:

  • start_link(opts) - Start cleanup worker
  • init(:ok) - Initialize worker
  • handle_info(:cleanup, state) - Run cleanup cycle

🚀 Getting Started​

Setup (One-Time)​

  1. Run the migration:

    mix ecto.migrate

  2. Restart the application:

    mix phx.server

  3. Verify installation:

    • Navigate to https://localhost:9443/nokia/retention
    • Should see Data Retention dashboard

Data Retention Dashboard

First Use​

  1. Check current settings:

    • View global retention (default: 720 hours)
    • View per-eNodeB retention settings

  2. Customize if needed:

    • Update global retention in config/config.exs
    • Or set per-eNodeB via UI

  3. Monitor cleanup:

    • Watch logs for [RetentionCleanupWorker] entries
    • Verify record counts decrease over time

📞 Support​

Need Help?​

  1. Check logs: Look for [RetentionPolicy] or [RetentionCleanupWorker] entries
  2. Review this guide: Most issues covered in Troubleshooting section
  3. Check other docs: Refer to related documentation links above
  4. Verify setup: Ensure migration ran and worker started

Reporting Issues​

Include:

  • Error message from UI or logs
  • eNodeB name affected
  • Current retention settings
  • Record counts before/after
  • Steps to reproduce

🎓 Learning Resources​

  • InfluxDB v2.x: Time-series database with retention policies
  • Retention Policy: How long data is kept
  • Cleanup: Automated deletion of old data
  • Measurement Types: Performance Metrics, Configuration, Alarms

External Resources​

Last updated on