Centralized Logging

Overview

OmniCore includes a centralized logging system that collects logs from all components and makes them searchable through Grafana. The system uses:

• Grafana Alloy — Log collection agent deployed on all VMs
• Grafana Loki — Log aggregation and storage server on monitoring hosts
• Grafana Dashboards — Pre-built dashboards for each component type

Architecture

How It Works

Log Collection Flow

Log Labels

Every log entry includes these labels for filtering and querying:

Label	Description	Example
job	Hostname of the source VM	customer-mme01
hostname	Same as job (for compatibility)	customer-mme01
component	Component type from inventory group	mme, pcscf, ocs
unit	systemd unit name	omnimme.service
level	Log severity	info, warning, error
service	Syslog identifier	omnimme

---

Configuration

Automatic Deployment

Logging is automatically configured when:

1. A monitoring group exists in your inventory
2. The common role runs on target hosts

No additional configuration is required for basic functionality.

Inventory Requirements

monitoring:
  hosts:
    customer-monitoring01:
      ansible_host: 10.10.2.200
      gateway: 10.10.2.1

When the monitoring group is defined:

• Monitoring hosts: Run Loki server (receives and stores logs)
• All other hosts: Run Alloy agent (collects and sends logs)

Retention Configuration

Loki is configured with time-based retention. The compactor deletes logs older than the retention period:

Setting	Default	Description
Time-based retention	72 hours (3 days)	Logs older than the retention period are deleted

To customize retention, override this variable in your inventory:

all:
  vars:
    loki_retention_period: "168h"  # e.g. 7 days in hours

Alloy Buffer Configuration

Alloy buffers logs locally in a write-ahead log (WAL) when Loki is unreachable. Buffered segments are aged out to prevent indefinite disk growth:

Setting	Default	Description
WAL max segment age	1 hour	Buffered log segments older than 1 hour are dropped

When Loki is unreachable for longer than the segment age, the oldest buffered logs are dropped.

---

Grafana Dashboards

Pre-built dashboards are automatically provisioned for each component type.

Available Dashboards

Dashboard	Location	Description
CSCF Logs	Logs → CSCF Logs	P-CSCF, S-CSCF, I-CSCF (OmniCSCF) logs
MME Logs	Logs → MME Logs	MME attach/detach events and errors
SGW Logs	Logs → SGW Logs	SGW session and bearer events
PGW Logs	Logs → PGW Logs	PGW PDN and session events
HSS Logs	Logs → HSS Logs	HSS Diameter messages and auth events
OmniMessage Logs	Logs → OmniMessage Logs	SMS delivery and SMPP events
OCS/CGrateS Logs	Logs → OCS/CGrateS Logs	Charging events and JSONRPC traffic
CGrateS RPC Calls	Logs → CGrateS RPC Calls	Search and correlate RPC requests/responses with latency

Dashboard Features

Each dashboard includes:

• Search box — Free-text search across all logs
• Log rate graphs — Volume over time by host and severity
• Component sections — Grouped views (e.g., P-CSCF, S-CSCF, I-CSCF)
• Error filtering — Pre-filtered views for errors and failures
• Live tail — Real-time log streaming (10-second refresh)

Using the Dashboards

1. Navigate to Grafana (typically http://<monitoring-host>:3000)
2. Go to Dashboards → Logs → select component
3. Use the Search variable to filter logs
4. Adjust time range as needed

---

Querying Logs

LogQL Basics

Loki uses LogQL for querying. Basic syntax:

{label="value"} |= "search string"

Common Queries

All logs from a specific host:

{hostname="customer-mme01"}

All MME component logs:

{component="mme"}

Search for errors across all CSCFs:

{component=~"pcscf|scscf|icscf"} |~ "(?i)error"

Filter by systemd unit:

{unit="omnicscf.service"}

Combine filters:

{component="hss", hostname="customer-hss01"} |= "ULR" |= "DIAMETER_SUCCESS"

Log Rate Queries

Log volume per component:

sum by (component) (rate({job=~".+"} [5m]))

Error rate for CSCF:

sum(rate({component=~"pcscf|scscf|icscf"} |~ "(?i)error" [5m]))

---

CGrateS JSONRPC Logging

For OCS/CGrateS deployments, JSONRPC traffic is captured and logged with request/response correlation and latency tracking for performance analysis and debugging.

The JSONRPC logger is defined in the cgrates role and its log file (/var/log/cgrates/jsonrpc.log) is collected by Alloy on both ocs and cgrates host groups.

Architecture

Request/Response Flow

What Gets Captured

Traffic on these CGrateS ports is captured:

Port	Service Name	Protocol	Description
2012	rpc_json	TCP	Raw JSONRPC over TCP
2080	http	HTTP	HTTP API (filters out /metrics and /health)

The logger automatically filters out:

• Prometheus metrics scrapes (GET /metrics)
• Health check requests (GET /health)
• Gzip-compressed responses (binary data)

Log Format

JSONRPC logs are written to /var/log/cgrates/jsonrpc.log. Each log entry represents a completed request/response pair with latency measurement:

2026-03-01T10:30:45.123456 port=2012 service=rpc_json request_id=1 method=SessionSv1.InitiateSession latency_ms=2.45 status=ok error= src=10.10.1.50:45678 dst=10.10.2.100:2012 request={"id":1,"method":"SessionSv1.InitiateSession",...} response={"id":1,"result":{...}}

Log Fields

Field	Description	Example
timestamp	ISO 8601 timestamp when response was received	2026-03-01T10:30:45.123456
port	CGrateS port the request was made to	2012, 2080
service	Service name for the port	rpc_json, http
request_id	JSONRPC request ID for correlation	1, 42, (empty for null)
method	JSONRPC method name	SessionSv1.InitiateSession
latency_ms	Round-trip latency in milliseconds	2.45
status	Result status	ok, error
error	Error message if status is error	INSUFFICIENT_CREDIT
src	Source IP:port (client)	10.10.1.50:45678
dst	Destination IP:port (CGrateS)	10.10.2.100:2012
request	Full JSONRPC request payload (compacted JSON)	{"id":1,"method":"..."}
response	Full JSONRPC response payload (compacted JSON)	{"id":1,"result":{...}}

Grafana Dashboard

The OCS / CGrateS Logs dashboard includes dedicated panels for JSONRPC analysis.

JSONRPC Latency by Method

A time series graph showing latency for each JSONRPC method over time. Useful for identifying:

• Slow methods that may need optimization
• Latency spikes indicating system issues
• Performance trends over time

The legend displays mean and max latency per method.

JSONRPC Traffic

A logs panel showing all JSONRPC request/response pairs with:

• Timestamp
• Method name
• Latency
• Full request and response payloads (prettified)

Method Drill-down

Filter JSONRPC traffic by a specific method using the Method variable at the top of the dashboard. Enter the method name (e.g., SessionSv1.InitiateSession) to see only traffic for that method.

CGrateS RPC Calls Dashboard

The dedicated CGrateS RPC Calls dashboard provides focused tools for troubleshooting API calls and correlating requests with responses.

Use Case: Finding a Failed Call

When a user reports "I tried to call 1234 and it failed":

1. Open Dashboards → Logs → CGrateS RPC Calls
2. In the Search field, enter the phone number or account ID: 1234
3. Set Status to Error to see only failed calls
4. The RPC Call Search panel shows all matching calls with full request/response

Each log entry shows the complete request payload (what was sent) and response payload (what CGrateS returned), making it easy to identify the exact error.

Dashboard Variables

Variable	Purpose	Example
Search	Free-text search in request/response payloads	1234, Account123, INSUFFICIENT_CREDIT
Method	Filter by RPC method name	SessionSv1.InitiateSession, CDRsV1.ProcessEvent
Status	Filter by result status	All, Success, Error

Dashboard Panels

Summary Stats (top row):

• Total RPC Calls — Count of all calls in the time range
• Errors — Count of failed calls (color-coded: green=0, yellow=1-9, red=10+)
• Avg Latency — Average round-trip time (color-coded thresholds)
• Max Latency — Highest latency call

RPC Latency by Method: Time series showing latency for each method. Hover to see specific values.

RPC Call Search: Main log panel showing matched calls. Click any entry to expand and see full request/response JSON.

Failed RPC Calls: Pre-filtered view showing only status=error calls.

Method Breakdown: Pie charts showing call distribution and error distribution by method.

Querying JSONRPC Logs

Basic Queries

All JSONRPC traffic:

{job="cgrates-jsonrpc"}

Filter by method:

{job="cgrates-jsonrpc"} |= "method=SessionSv1.InitiateSession"

Search request/response payloads:

{job="cgrates-jsonrpc"} |= "Account\":\"12345"

Errors only:

{job="cgrates-jsonrpc"} |= "status=error"

Latency Analysis

Extract latency as metric (for graphs):

max_over_time(
  {job="cgrates-jsonrpc"}
  |= "method="
  | regexp "method=(?P<method>[^ ]+) latency_ms=(?P<latency>[0-9.]+)"
  | __error__=""
  | unwrap latency [$__auto]
) by (method)

Find slow requests (latency > 100ms):

{job="cgrates-jsonrpc"}
| regexp "latency_ms=(?P<latency>[0-9.]+)"
| latency > 100

Method-Specific Queries

CDR processing:

{job="cgrates-jsonrpc"} |= "method=CDRsV1"

Session management:

{job="cgrates-jsonrpc"} |~ "method=SessionSv1"

Account operations:

{job="cgrates-jsonrpc"} |~ "method=ApierV"

Service Management

The JSONRPC logger runs as a systemd service on OCS/CGrateS hosts.

Check service status:

systemctl status cgrates-jsonrpc-logger

View service logs:

journalctl -u cgrates-jsonrpc-logger -f

Restart the service:

systemctl restart cgrates-jsonrpc-logger

Troubleshooting

No JSONRPC Logs Appearing

Symptoms: JSONRPC Traffic panel shows no data

Possible causes:

• Logger service not running
• ngrep not installed
• Network interface mismatch
• Alloy not collecting the log file

Resolution:

1. Check logger service status:

   systemctl status cgrates-jsonrpc-logger
   journalctl -u cgrates-jsonrpc-logger -n 50

2. Verify ngrep is installed:

   which ngrep

3. Check log file is being written:

   tail -f /var/log/cgrates/jsonrpc.log

4. Verify Alloy is collecting the file:

   journalctl -u alloy | grep jsonrpc

Latency Graph Shows No Data

Symptoms: JSONRPC Latency by Method panel shows "No data"

Possible causes:

• No JSONRPC traffic in the selected time range
• Log format mismatch (regex not matching)

Resolution:

1. Verify logs exist for the time range:

   {job="cgrates-jsonrpc"} |= "method="

2. Check that latency_ms field is present in logs
3. Expand the time range

High Pending Request Count

Symptoms: Debug logs show many stored requests but few logged completions

Possible causes:

• Requests without responses (client disconnected)
• Response captured before request (packet ordering)
• Request ID mismatch between request and response

Resolution:

1. The logger automatically cleans up pending requests older than 60 seconds
2. Check for network issues between client and CGrateS
3. Verify CGrateS is responding to requests

---

File Logs

In addition to systemd journal logs, Alloy collects specific log files:

FreeSWITCH Logs (Application Servers)

Path	Job Label
/var/log/freeswitch/freeswitch.log	freeswitch
/var/log/omnitas-freeswitch.log	freeswitch-structured

Nginx Logs (OmniCRM)

Path	Job Label	Type Label
/var/log/nginx/access.log	nginx	access
/var/log/nginx/error.log	nginx	error

CGrateS JSONRPC (OCS)

Path	Job Label
/var/log/cgrates/jsonrpc.log	cgrates-jsonrpc

---

Troubleshooting

Logs Not Appearing in Grafana

Symptoms: No logs visible in Loki dashboards

Possible causes:

• Alloy service not running on source host
• Loki service not running on monitoring host
• Network connectivity between hosts blocked
• Alloy cannot reach Loki on port 3100

Resolution:

1. Check Alloy status on source host:

   systemctl status alloy
   journalctl -u alloy -f

2. Check Loki status on monitoring host:

   systemctl status loki
   journalctl -u loki -f

3. Test connectivity from source to monitoring:

   curl http://<monitoring-ip>:3100/ready

4. Verify firewall allows TCP 3100 from all hosts to monitoring

Alloy High Disk Usage

Symptoms: /var/lib/alloy consuming significant disk space

Possible causes:

• Loki unreachable for extended period
• WAL buffer filling up

Resolution:

1. Check Loki connectivity (see above)
2. If Loki is down, logs buffer locally in the WAL
3. Once Loki is reachable, buffered logs are sent automatically
4. Buffered segments older than 1 hour are dropped (by design)

Loki Storage Full

Symptoms: Loki stops accepting logs, disk full on monitoring server

Possible causes:

• Log volume exceeds available disk before retention deletes old logs
• Retention compaction not running

Resolution:

1. Check Loki storage usage:

   du -sh /var/lib/loki/

2. Verify compactor is running (check Loki logs)
3. Manually trigger compaction if needed by restarting Loki
4. Consider increasing disk allocation or reducing retention period

Missing Component Labels

Symptoms: Logs appear but component label is infrastructure instead of expected value

Possible causes:

• Host not in expected inventory group
• Alloy config not regenerated after inventory change

Resolution:

1. Verify host is in correct inventory group
2. Re-run Ansible to regenerate Alloy config:

   ansible-playbook -i hosts/customer/hosts.yml services/all.yml --limit <hostname>

3. Restart Alloy:

   systemctl restart alloy

---

Service Ports

Service	Port	Protocol	Description
Loki	3100	HTTP	Log ingestion and query API
Loki	9096	gRPC	Internal gRPC (not used externally)
Alloy	12345	HTTP	Alloy metrics and UI
Grafana	3000	HTTP	Dashboard access

---

Related Documentation

• Monitoring & Observability — Grafana, Prometheus, dashboards, and alerting
• Deployment Architecture — Overall system architecture
• Hosts File Configuration — Inventory configuration
• Configuration Reference — Complete parameter reference