Alarm Management & Escalation
Fault Handling, Severity Levels, and Operational Response
Guide to managing alarms, investigating faults, and escalating issues
Table of Contents
- Overview
- Alarm Lifecycle
- Severity Levels
- Alarm Categories
- Investigation & Troubleshooting
- Escalation Procedures
- Resolution Tracking
- Best Practices
Overview
Alarms (also called "faults") represent detected problems or anomalies on Nokia AirScale base stations. RAN Monitor continuously monitors active alarms and tracks their lifecycle from generation through resolution.
Example Alarm Dashboard:
Example showing 4G Status with alarm overview table displaying alarm status (active/cleared), severity levels (critical/warning), timestamps, and alarm descriptions for optical interface failures.
Alarm Sources
Key Alarm Attributes
Every alarm contains:
| Attribute | Example | Purpose |
|---|---|---|
| Alarm ID | a1b2c3d4-e5f6-... | Unique identifier |
| Severity | Critical, Major, Minor | Priority level |
| Probable Cause | "Cell Unavailable" | Root cause category |
| Specific Problem | "S1 Connection Lost" | Detailed issue |
| Affected System | /BSC-1/BTS-23/Cell-A | What's impacted (DN) |
| Event Time | 2025-12-10 14:23:45 | When detected |
| Status | Active / Cleared | Current state |
Alarm Lifecycle
State Transitions
Example Alarm Timeline
14:23:45 UTC - Issue Occurs
└─> Base station detects connectivity loss
└─> Generates alarm: "S1 Connection Lost" (Critical)
14:23:47 UTC - Alarm Pushed to RAN Monitor
└─> NE3S webhook notification received
└─> Stored in InfluxDB
└─> Alert rule triggered
14:23:50 UTC - Notification Sent
└─> Grafana alert fires
└─> Slack message to NOC team
└─> PagerDuty incident created
14:24:15 UTC - Operator Acknowledges
└─> NOC team marks as acknowledged
└─> Duration tracking begins
14:28:35 UTC - Issue Auto-Resolves
└─> Connectivity restored
└─> Base station clears alarm
└─> RAN Monitor records "Cleared"
14:28:36 UTC - Alarm Closed
└─> Duration recorded: 5 minutes 51 seconds
└─> Tracked for SLA reporting
└─> Archived after 30 days
Severity Levels
RAN Monitor tracks five severity levels, each with different operational impact and escalation requirements:
Critical Severity
Definition: Service-impacting, requires immediate action
Examples:
- Device completely unreachable (connectivity loss)
- All cells down (baseband failure)
- Control plane interface down (S1 lost)
- Complete data forwarding failure
- Base station not responding to management
Escalation:
- Notify on-call engineer immediately (phone call)
- Create incident in incident management system
- Update status page
- Inform management if SLA affected
Response SLA: < 15 minutes
Major Severity
Definition: Degraded performance, requires urgent investigation
Examples:
- Cell availability < 95% for > 15 minutes
- Handover success rate < 95%
- DL/UL resource blocked (> 95% utilization sustained)
- RLC retransmission rate > 5%
- Multiple cells showing poor quality
- Link degradation (E1/T1 errors increasing)
Escalation:
- Notify NOC team + senior engineer
- Create incident in incident management
- Page engineering team if still open after 30 minutes
- Check for cascading issues to other cells/sites
Response SLA: < 30 minutes investigation
Minor Severity
Definition: Degradation, track and investigate
Examples:
- Cell availability 95-98% (trending down)
- High temperature warning on amplifier
- License capacity approaching limit
- Configuration consistency issues
- Slow performance on management interface
- Intermittent alarms (< 5 occurrences/hour)
Escalation:
- Log in dashboard for awareness
- Assign to engineering for investigation
- Schedule for next maintenance window if needed
- Create ticket for trend analysis
Response SLA: Same day review
Warning Severity
Definition: Informational, monitor for trends
Examples:
- Cell availability > 98% but trending down
- Temperature/power in normal range but elevated
- Resources at 60-70% utilization
- Configuration mismatch (non-critical parameters)
- First occurrence of new fault type
Escalation:
- Dashboard visibility only
- No automatic notifications
- Manual review on cadence
Cleared
Definition: Previously active alarm now resolved
Purpose:
- Documents that issue has been resolved
- Tracks mean time to repair (MTTR)
- Enables SLA compliance reporting
- Identifies repeat problems
Alarm Categories
Connectivity Alarms
Category: External Connectivity
Probable Causes:
- S1 Connection Lost → MME/SGW unreachable
- Backhaul Link Down → IP transport failure
- USIM Interface Error → HSS connectivity issue
- NTP Synchronization Lost → Time sync network issue
Impact: Service interruption, call setup failures
Investigation:
1. Check network connectivity between devices
2. Verify firewall rules allow required protocols
3. Check peer device status and errors
4. Review network interface statistics
Hardware & Environmental Alarms
Category: Physical Infrastructure
Probable Causes:
- High Temperature Warning → Cooling system degraded
- Power Supply Degraded → UPS/PSU issue
- Fan Failure → Cooling fan malfunction
- Disk Space Low → Storage approaching limit
- Memory Exhaustion → Process memory leak
Impact: Potential cascading failures, data loss
Investigation:
1. Check hardware status via management interface
2. Review temperature trends
3. Check cooling system operation
4. Monitor memory and CPU utilization
Software & Process Alarms
Category: Application Layer
Probable Causes:
- Process Crash → Software error or OOM
- High CPU Utilization → Performance bottleneck
- Queue Overload → Message processing backlog
- License Violation → Capacity exceeded
Impact: Service degradation or interruption
Investigation:
1. Check process status
2. Review logs for error messages
3. Monitor CPU/memory/queue depth
4. Verify license status
Radio Resource Alarms
Category: Air Interface
Probable Causes:
- Cell Unavailable → No radio coverage/power
- DL Resource Blocked → Capacity exhaustion
- Handover Failure Rate High → Neighbor configuration issue
- Poor Cell Quality → RF interference or path loss
Impact: User experience degradation
Investigation:
1. Check cell physical parameters
2. Review neighbor cell configuration
3. Analyze RF quality metrics
4. Check antenna alignment
Configuration Alarms
Category: System State
Probable Causes:
- Parameter Mismatch → Configuration inconsistency
- License Expired → Licensing issue
- Config Checksum Error → Corruption or conflict
- Feature Not Licensed → Feature usage exceeds license
Impact: Feature unavailability or degradation
Investigation:
1. Review configuration changes
2. Compare current vs. intended config
3. Check license file and expiry
4. Verify software version compatibility
Investigation & Troubleshooting
Investigation Workflow
Step 1: Review Alarm Details
When an alarm fires, start by gathering information:
What to collect:
- Alarm ID and unique identifier
- Severity and probable cause
- Affected system DN (device/cell/component)
- Event time (when it occurred)
- Duration (how long it's been active)
- Full alarm description and text
Tools:
- RAN Monitor Web UI → Alarms page
- Grafana → Active Alarms table
- InfluxDB → Query raw alarm record
Step 2: Research Probable Cause
Each alarm type has known causes and investigations:
Documented Knowledge:
- Nokia AirScale troubleshooting guides
- Previous ticket history (similar issues)
- RAN Monitor documented runbooks
- Team expertise (subject matter experts)
Step 3: Check Related Metrics
Cross-correlate alarms with performance metrics to understand impact:
Example: "DL Resource Blocked" Alarm
├─ Check DL Resource Utilization (should be > 95%)
├─ Check Traffic Throughput (trending up?)
├─ Check Call Setup Success (dropped?)
├─ Check Handover Success (affected?)
└─ Check Cell Availability (down?)
Tools:
- Grafana → Device-specific dashboard
- Web UI → Device detail page → Metrics section
- InfluxDB direct queries for correlation
Step 4: Correlate with Recent Changes
Many issues are caused by recent modifications:
Timeline:
├─ Configuration changes (last 4 hours)
├─ Software upgrades (last 24 hours)
├─ Feature parameter tuning (last 7 days)
├─ Maintenance activities (last 7 days)
├─ Network changes (last 7 days)
└─ Third-party changes (external network)
Tools:
- RAN Monitor → Configuration history
- Change management system
- Incident history (similar issues before)
- Cross-team notification logs
Step 5: Diagnose Root Cause
Based on investigation, identify the root cause:
Decision Tree Example: "Cell Unavailable" Alarm
Cell Unavailable Alarm
│
├─ Is device responding to management?
│ ├─ NO → Check device connectivity, reboot if needed
│ └─ YES → Continue
│
├─ Are all cells down or specific cell?
│ ├─ All cells → Check baseband/power supply hardware
│ └─ Specific cell → Continue
│
├─ Is cell transmitting power?
│ ├─ NO → Check Power Amplifier, antenna connection
│ └─ YES → Continue
│
├─ Are neighbor cells reporting this cell?
│ ├─ YES → Other devices see this cell as unavailable
│ │ → Check antenna alignment, cable connection
│ └─ NO → Cell is down for internal reason
│ → Check baseband module, DSP status
│
└─ Check logs for error messages
→ Software crash
→ License violation
→ Parameter error
Step 6: Implement Resolution
Once root cause is identified, implement fix:
Types of Resolutions:
| Type | Method | Example |
|---|---|---|
| Immediate | Restart/reboot | Device restart to clear hung process |
| Configuration | Adjust parameters | Change handover threshold |
| Hardware | Replace/repair | Swap failed power supply |
| Software | Upgrade/patch | Install software bug fix |
| Network | Fix connectivity | Restore BGP route, fix firewall |
Step 7: Verify Resolution
After implementing fix, verify:
Verification Checklist:
□ Alarm cleared in RAN Monitor
□ Related metrics have normalized
□ No secondary/cascading alarms
□ Performance metrics returned to baseline
□ Customer reports (if applicable) resolved
□ System is stable (> 30 minutes observation)
Step 8: Document Learning
Record findings for future reference:
Document:
- Root cause and contributing factors
- Steps taken to resolve
- Time spent (for SLA tracking)
- Preventive measures taken
- Knowledge shared with team
Escalation Procedures
Escalation Ladder
Escalation Triggers
Escalate to Level 2 if:
- Critical alarm not cleared after 15 minutes
- Major alarm not cleared after 30 minutes
- Issue is outside NOC team's expertise
- Requires device reboot/major change
- Affects > 1 site simultaneously
Escalate to Level 3 if:
- Level 2 unable to diagnose after 1 hour
- Critical issue persists > 30 minutes
- Hardware failure suspected
- Cascading issues detected
- Requires vendor involvement
Contact Vendor (Level 4) if:
- Hardware failure confirmed (PSU, CMON, etc.)
- Software bug suspected (unrecoverable crash)
- License/activation issue
- Issue documented in vendor known issues
- Multiple escalation levels unable to resolve
Escalation Communication
Template for Escalating to Level 2:
Subject: Escalation - [Severity] - [Device] - [Issue]
Alert Time: [2025-12-10 14:30 UTC]
Duration: [15 minutes]
Device: [SITE_A_BS1]
Issue: [Cell Unavailable]
Symptoms:
- Cell A1 not responding to management
- All cells reporting cell unavailable
- Device ping successful
Investigation Performed:
- Device connectivity verified
- Baseband module status checked
- Power supply status: OK
- Temperature: Normal
Metrics:
- Cell availability: 0%
- No traffic on cell
- Control plane: Connected
Initial Analysis:
- Possible baseband module failure
- Or power amplifier hardware issue
Next Steps Recommended:
- Hardware diagnostics
- Module swap if available
- Device restart as last resort
Escalation Link: [Dashboard Link]
Resolution Tracking
SLA Tracking
Track time-to-resolution for SLA compliance:
Alarm Timeline:
├─ Event Time: 14:23:45 ← When issue occurred
├─ Detection Time: 14:23:47 (2 sec) ← Management detected
├─ Alert Time: 14:23:50 (5 sec) ← Operations notified
├─ Ack Time: 14:24:15 (30 sec) ← Operator acknowledged
├─ Investigation: 14:24:15 → 14:28:00 (3.75 min)
├─ Resolution: 14:28:00 → 14:28:35 (35 sec fix + verification)
└─ Clear Time: 14:28:36 ← Alarm cleared
└─ Total Duration: 5 min 51 sec
SLA Metrics:
├─ Detection Latency: 2 seconds
├─ Alert-to-ACK: 30 seconds
├─ Time-to-Resolve: 5 min 51 sec
└─ SLA Status: PASS (< 15 min target)
Trend Analysis
Track patterns in alarm data:
Questions to ask:
- Are we seeing the same alarm repeatedly?
- Is alarm rate increasing/decreasing?
- Do alarms cluster at specific times?
- Are multiple sites affected simultaneously?
- Is MTTR improving over time?
Tools:
- Grafana → Alarm trends dashboard
- Top alarms report (weekly)
- MTTR tracking by device/type
Preventing Repeat Issues
Best Practices
Operational Excellence
1. Alarm Fatigue Prevention
- Set meaningful thresholds (not too sensitive)
- Use duration windows (not single spike)
- Aggregate related alarms
- Suppress known false positives
2. Rapid Response
- Keep runbooks up-to-date
- Train team on common issues
- Use automation for routine resets
- Have escalation contacts readily available
3. Quality Documentation
- Document every incident
- Share learnings with team
- Update runbooks based on incidents
- Cross-train team members
4. Proactive Monitoring
- Watch for warnings before critical
- Trend analysis for capacity planning
- Regular health checks
- Performance baseline establishment
Runbook Development
Every frequent alarm should have a runbook:
Template:
Alarm: [Cell Unavailable]
Probability: [High]
MTTR: [5-15 minutes]
SLA Target: [Resolved within 30 minutes]
Symptoms:
- Alarm: "Cell Unavailable"
- Users: Unable to connect
- Metrics: Cell availability 0%
Quick Diagnosis (< 5 minutes):
1. Is device responding to ping?
2. Are other cells working?
3. Is baseband running (check logs)?
Resolution Steps:
Step 1: Device Connectivity Check
- Ping device: ping 192.168.1.100
- If no response → Check network connectivity
Step 2: Hardware Status
- Check Power Amplifier status
- Check baseband module LEDs
- Check antenna connection
Step 3: Restart Cell
- Restart cell via management interface
- Wait 60 seconds for startup
- Verify metrics normalize
Step 4: If Still Down
- Escalate to Level 2
- Prepare for device reboot
- Notify customer
Escalation:
- If > 15 minutes → Escalate to [Engineer Name]
- If > 30 minutes → Escalate to [Senior Engineer]
- If hardware failure → Contact Nokia Support
Prevention:
- Regular baseband firmware updates
- Preventive power supply replacement
- Antenna connection inspection quarterly