Utility Playbooks

Utility playbooks provide operational tools for managing deployed OmniCore infrastructure. These playbooks are located in the util_playbooks/ directory and can be run independently to perform common maintenance and troubleshooting tasks.

Quick Reference

Playbook	Purpose
proxmox.yml	Provision VMs on Proxmox
proxmox_delete.yml	Delete VMs/LXCs on Proxmox
proxmox_lxc.yml	Provision LXC containers on Proxmox
proxmox_restart_hung.yml	Force power-cycle VMs/LXCs via the Proxmox API when SSH is unresponsive
vmware.yml	Provision VMs on VMware vSphere
vmware_delete.yml	Delete VMs on VMware vSphere
health_check.yml	Generate comprehensive health report for all services
restore_hss.yml	Restore HSS database and/or configuration from backup
seed_hss_replication.yml	Seed an empty OmniHSS node from a populated peer (initial replication data load)
restore_ocs.yml	Restore OCS KeyDB, MySQL, and configuration from backup
ip_plan_generator.yml	Generate network documentation with Mermaid diagrams
get_ports.yml	Audit open ports and listening services across all hosts
getLocalCapture.yml	Retrieve packet capture files from hosts
delete_local_user.yml	Remove a local user account from all hosts
clear_mnesia.yml	Clear the Mnesia database for OmniSS7 / OmniMessage and restart (destructive)
delete_license.yml	Delete the license file on the license server and restart it

VM Provisioning

VM provisioning playbooks create and manage virtual machines on hypervisor platforms. All playbooks support --limit to target specific hosts or groups.

See Proxmox Deployment for detailed configuration.

Proxmox

Files: util_playbooks/proxmox.yml, util_playbooks/proxmox_lxc.yml, util_playbooks/proxmox_delete.yml

# Provision VMs
ansible-playbook -i hosts/customer/host_files/production.yml util_playbooks/proxmox.yml

# Provision specific group only
ansible-playbook -i hosts/customer/host_files/production.yml util_playbooks/proxmox.yml --limit mme

# Provision LXC containers
ansible-playbook -i hosts/customer/host_files/production.yml util_playbooks/proxmox_lxc.yml

# Delete VMs/LXCs
ansible-playbook -i hosts/customer/host_files/production.yml util_playbooks/proxmox_delete.yml --limit old-host

Proxmox Restart Hung Hosts

File: util_playbooks/proxmox_restart_hung.yml

Checks each host's SSH responsiveness and force power-cycles any unresponsive VMs or LXC containers through the Proxmox API. QEMU VMs are hard-stopped and started; LXC containers are started (or rebooted if already running). Hosts that respond to SSH are skipped.

ansible-playbook -i hosts/customer/host_files/production.yml util_playbooks/proxmox_restart_hung.yml

# Restrict to a single host or group
ansible-playbook -i hosts/customer/host_files/production.yml util_playbooks/proxmox_restart_hung.yml --limit hss01

VMware vSphere

Files: util_playbooks/vmware.yml, util_playbooks/vmware_delete.yml

Both playbooks include the per-VM task files under util_playbooks/vm/vmware/ (create_individual_vm.yml and delete_individual_vm.yml). vmware_delete.yml is a thin wrapper around the delete task; vmware.yml additionally waits for SSH after provisioning, then removes stale netplan files and applies netplan on the new VM.

Prerequisites: Requires dependencies from requirements.txt to be installed.

Usage:

# Provision VMs
ansible-playbook -i hosts/customer/host_files/production.yml util_playbooks/vmware.yml

# Provision specific group only
ansible-playbook -i hosts/customer/host_files/production.yml util_playbooks/vmware.yml --limit mme

# Delete VMs
ansible-playbook -i hosts/customer/host_files/production.yml util_playbooks/vmware_delete.yml --limit old-host

Required Variables:

all:
  vars:
    vcenter_ip: "vcenter.example.com"
    vcenter_username: "administrator@vsphere.local"
    vcenter_password: "password"
    vcenter_datacenter: "Datacenter"
    vcenter_folder: "OmniCore"
    vcenter_vm_template: "ubuntu-2404-template"
    vhosts:
      esxi-01:
        vcenter_cluster_ip: "192.168.1.10"
        vcenter_datastore: "datastore1"

Vultr

Files: util_playbooks/vm/vultr/vmProvisionVultr.yml, util_playbooks/vm/vultr/vmTeardownVultr.yml, util_playbooks/vm/vultr/vmCommonSetup.yml

Provisions and tears down instances on Vultr via the Vultr API. vmProvisionVultr.yml creates instances, vmTeardownVultr.yml removes them, and vmCommonSetup.yml performs post-provision base setup (DNS, reboot, etc.). All three are top-level entry-point playbooks rather than task includes. They read the API token from the VULTR_API_TOKEN environment variable, and the provisioning playbook references a cloud-config.yaml from the same directory, so run them from util_playbooks/vm/vultr/.

export VULTR_API_TOKEN=<token>
ansible-playbook -i hosts/customer/host_files/production.yml util_playbooks/vm/vultr/vmProvisionVultr.yml

Health Check

File: util_playbooks/health_check.yml

Generates a comprehensive HTML health report covering all deployed OmniCore and OmniCall services.

ansible-playbook -i hosts/customer/host_files/production.yml util_playbooks/health_check.yml

Output: /tmp/health_check_YYYY-MM-DD HH:MM:SS.html

Information Collected

Component	Data Collected
All services	Service status, version, uptime
OmniHSS	Database status, Diameter peer connections
OmniDRA	Diameter peer connections and status
OmniTAS	Active calls, sessions, registrations, CPU usage
OCS	KeyDB replication status

HSS Restore

File: util_playbooks/restore_hss.yml

Restores OmniHSS from backup files. Supports restoring database only, configuration only, or both.

ansible-playbook -i hosts/customer/host_files/production.yml util_playbooks/restore_hss.yml

Backup File Formats

Type	Filename Pattern	Contents
Database	hss_dump_<hostname>_<timestamp>.sql	PostgreSQL (pg_dump) dump of the OmniHSS database (default omnihss)
Config	hss_<hostname>_<timestamp>.tar.gz	Archive of /etc/omnihss directory

The current HSS state (database and/or config) is backed up before the restore runs.

Prompts

Prompt	Required	Description
HSS SQL dump path	No	Full path to the HSS SQL dump file (leave empty to skip database restore)
HSS config tar.gz path	No	Full path to the HSS config tar.gz file (leave empty to skip config restore)
Confirmation	Yes	Type yes to confirm the restore operation on the listed host(s)

HSS Replication Seed

File: util_playbooks/seed_hss_replication.yml

Performs the one-time initial data load for an OmniHSS node joining an already-populated active-active replication mesh. A new node comes up empty (subscriptions are created with copy_data = false); this playbook recreates the subscription to a single chosen source peer with copy_data = true, so PostgreSQL copies the existing data and then continues streaming with no gap.

It runs in one of two modes:

Interactive (default) — omit the seed_source_* variables. The playbook detects which limited target(s) are empty, probes the valid peers for ones that hold data, lists them with subscriber counts, and prompts you to choose a source. Only empty targets are seeded.

ansible-playbook -i hosts/customer/host_files/production.yml \
  util_playbooks/seed_hss_replication.yml \
  --limit site-hss01,site-hss02

Peers with data available to seed from:
  1) dev-hss01 (10.4.10.142) - 838 subscribers
Enter the NUMBER of the source to seed from

Explicit — pass the source directly to skip the menu. This forces a re-seed even if the target already holds data (use for recovery).

ansible-playbook -i hosts/customer/host_files/production.yml \
  util_playbooks/seed_hss_replication.yml \
  --limit site-hss01 \
  -e "seed_source_name=dev-hss01 seed_source_host=10.4.10.142"

Parameters

Parameter	Required	Default	Description
seed_source_name	No	(prompted)	Inventory hostname of the populated source peer. Used to name the subscription (sub_from_<name>) and resolve its publication (<name>_pub). Omit for interactive mode.
seed_source_host	No	(prompted)	IP address of the source peer, reachable on its PostgreSQL port. Omit for interactive mode.
seed_source_port	No	target's omnihss.database_port (5432)	PostgreSQL port on the source peer.

Both seed_source_name and seed_source_host must be supplied together to use explicit mode; supplying neither triggers interactive mode.

Behaviour

1. Counts subscribers on each limited target to determine which are empty.
2. Interactive mode: probes the valid peers (local hss group + connected_omnihss) for ones holding data, lists them, and prompts for a choice. Explicit mode: prompts for yes to confirm.
3. Ensures the source IP is allowed in the target's pg_hba.conf.
4. Drops and recreates the subscription to the source WITH (copy_data = false) (per-link slot_name of <target>_from_<source>) — it streams all tables but performs no bulk tablesync.
5. Backfills the existing master-table rows from the source with a conflict-safe load (pg_dump --data-only --inserts --on-conflict-do-nothing --disable-triggers). The transient state tables (subscriber_state, pdn_session, lte_call) are not backfilled.
6. Reports the subscriber count on the seeded node.

Why state tables are skipped: they are hot tables; a bulk COPY of them on a live system collides (duplicate-key) with concurrent writes and crash-loops the tablesync worker. They repopulate from streaming replication and live traffic instead.

Important: Always --limit to the target node(s). In interactive mode, non-empty targets are skipped automatically. See OmniHSS Replication & Node Operations for the full workflow.

OCS Restore

File: util_playbooks/restore_ocs.yml

Restores OCS (CGrateS) from backup files. Handles multi-master KeyDB replication by restoring to one node and allowing replication to sync to others.

ansible-playbook -i hosts/customer/host_files/production.yml util_playbooks/restore_ocs.yml

Restore Process

1. Stops CGrateS and KeyDB on all OCS nodes
2. Clears AOF files to prevent conflicts with restored data
3. Restores KeyDB RDB to first node, starts KeyDB, lets replication sync
4. Restores MySQL StoreDB (optional)
5. Restores /etc/cgrates configuration (optional)
6. Starts CGrateS on all nodes

Backup File Formats

Type	Filename Pattern	Contents
KeyDB DataDB	keydb_dump_<hostname>_<timestamp>.rdb	KeyDB RDB snapshot
MySQL StoreDB	cgrates_dump_<hostname>_<timestamp>.sql	MySQL dump of cgrates database
Config	cgrates_<hostname>_<timestamp>.tar.gz	Archive of /etc/cgrates directory

Prompts

Prompt	Required	Description
KeyDB RDB dump path	Yes	Full path to the KeyDB RDB backup file
MySQL SQL dump path	No	Full path to the CGrateS SQL dump (skip to preserve current StoreDB)
Config tar.gz path	No	Full path to the config backup (skip to preserve current config)

IP Plan Generator

File: util_playbooks/ip_plan_generator.yml

Generates network documentation from inventory, including:

• Host IP assignments (primary and secondary NICs)
• Network segment overview
• Interface connectivity diagrams (Diameter, GTP, PFCP, SIP, SS7)

ansible-playbook -i hosts/customer/host_files/production.yml util_playbooks/ip_plan_generator.yml

Output Files

File	Format	Description
/tmp/ip_plan_<customer>_<site>.md	Markdown	Text-based documentation
/tmp/ip_plan_<customer>_<site>.html	HTML	Interactive diagram with filterable layers

Port Audit

File: util_playbooks/get_ports.yml

Audits all listening ports across the deployment and generates documentation.

ansible-playbook -i hosts/customer/host_files/production.yml util_playbooks/get_ports.yml

Output Files

File	Description
/tmp/all_ports.csv	CSV with hostname, IP, protocol, port, service
./open_ports.rst	reStructuredText table for Sphinx documentation

Data Collected

Field	Description
Hostname	Inventory hostname
IP	Host's ansible_host IP address
IP Version	IPv4 or IPv6
Transport	TCP or UDP
Port	Listening port number
Service	Process name

Local Capture Retrieval

File: util_playbooks/getLocalCapture.yml

Retrieves the two most recent packet capture files from each host's /etc/localcapture directory.

ansible-playbook -i hosts/customer/host_files/production.yml util_playbooks/getLocalCapture.yml

Output: ./localCapturePcaps/<hostname>/*.pcap

User Management

File: util_playbooks/delete_local_user.yml

Removes a local user account from all hosts in the inventory.

ansible-playbook -i hosts/customer/host_files/production.yml util_playbooks/delete_local_user.yml

Prompt: Enter the username to delete when prompted.

Clear Mnesia Database

File: util_playbooks/clear_mnesia.yml

Stops the service, deletes its Mnesia database directory, and restarts it. Runs against the ss7 group (OmniSS7) and the omnimessage group (OmniMessage).

Warning: This is destructive. The Mnesia database is permanently deleted and rebuilt empty on restart.

ansible-playbook -i hosts/customer/host_files/production.yml util_playbooks/clear_mnesia.yml

# Limit to a single service or host
ansible-playbook -i hosts/customer/host_files/production.yml util_playbooks/clear_mnesia.yml --limit ss7

Delete License

File: util_playbooks/delete_license.yml

Deletes /etc/license_server/license.json on the license_server group and restarts the license server.

ansible-playbook -i hosts/customer/host_files/production.yml util_playbooks/delete_license.yml

Running Utility Playbooks

Basic Syntax

ansible-playbook -i <inventory_file> util_playbooks/<playbook>.yml

Common Options

Option	Description
-i <inventory>	Specify inventory file
--limit <hosts>	Limit to specific hosts or groups
-v / -vv / -vvv	Increase verbosity
--check	Dry run (show what would change)
--diff	Show file differences

Examples

# Run health check on production
ansible-playbook -i hosts/acme/host_files/production.yml util_playbooks/health_check.yml

# Restore HSS on a specific host
ansible-playbook -i hosts/acme/host_files/production.yml util_playbooks/restore_hss.yml --limit hss01

# Generate IP plan with verbose output
ansible-playbook -i hosts/acme/host_files/production.yml util_playbooks/ip_plan_generator.yml -v