Test Devices

Test Devices turns real Android handsets at remote sites into a remotely-controllable test farm. From OmniWeb you can watch a phone's screen live, drive it (tap, swipe, type, hardware keys), place calls and send texts, and run scheduled health checks — VoLTE registration, signal, connectivity — whose pass/fail results are exported to Prometheus for alerting and dashboards.

← Back to Operations Guide

It exists to answer the question a NOC keeps asking about a remote cell site: "is a real phone actually working on this network right now?" — registered for VoLTE, able to call, text, and reach the internet — without anyone driving to the site.

Architecture

Each host with phones plugged in via ADB runs a lightweight agent. The agent dials out to central OmniWeb, so sites behind NAT need no inbound access. Live video uses WebRTC (UDP) so it survives lossy, high-latency links far better than TCP screen-sharing; input and test results are relayed through central.

Network paths matter. The web UI, input and call setup ("signalling") ride the control plane over HTTPS to the OmniWeb server, but the live screen video streams directly between the operator's browser and the agent host over UDP — it does not pass through the server. If a browser can reach the server but not the agent host over UDP, the console will hang on "Connecting…". See Network Requirements for the firewall rule to request and how to recognise the problem.

Component	Role
Phone agent	Discovers ADB devices, captures the screen, injects input, runs host test scripts. One per site; installed by the device_tester Ansible role.
Central OmniWeb	Device registry, WebRTC signalling relay, script dispatch, results history, Prometheus metrics.
Prometheus	Scrapes scheduled-test results from /api/android/metrics.

Device types

The farm is no longer phones-only. Every connected device is one of three kinds, auto-detected by the agent and shown with its own badge (and the Android robot logo for phones) in the fleet:

Kind	What it is	Console
Android phone	A normal handset (Android 5.0+ / API 21+) over ADB	Live screen + full control
USB modem	A cellular module exposing AT ports (e.g. Quectel EP06)	AT-driven cellular console — status, calls, SMS, cell lock, VoLTE
Old-Android dongle	A cheap LTE dongle running old Android (API < 21, e.g. UZ801) too old for the live-screen view	Poll-based screencap view + control
RAN Simulator UE	A virtual UE from the RAN Simulator — no hardware	UE control panel — attach/detach, idle, out-of-coverage, IMS register, SMS, calls

Clicking a device opens the console appropriate to its kind. Inside any console the Swap phone button (top-right) switches to another device without going back to the fleet grid.

Simulated UEs from the RAN Simulator appear in the fleet tagged as simulators, so you can drive emulated subscribers alongside real handsets — and the Test Devices filter lets you narrow to just the RAN Simulator UEs. They use the same UE control panel shown in the RAN Simulator's own Network Map, so there is one place to attach, idle, drop out of coverage, register to IMS, and send SMS/calls. See RAN Simulator for the full UE lifecycle.

Adding a device to a site

A device joins the fleet automatically once the agent host at that site can see it over ADB. There are two steps: enable ADB on the handset, then — if the agent runs in a VM or container — pass the USB device through to it.

1. Enable ADB on the phone

1. On the handset, open Settings → About phone and tap Build number seven times to unlock Developer options.
2. In Settings → System → Developer options, turn on USB debugging.
3. Plug the phone into the agent host with a data-capable USB cable.
4. The phone shows an "Allow USB debugging?" prompt with the host's RSA key — tick Always allow from this computer and accept. (Without the tick, the prompt returns every time it re-connects and the device shows as unauthorized.)

A normal handset needs nothing more — the test scripts and live screen use only standard ADB. Keep the screen unlocked for the first connection so the prompt can be accepted.

2. Pass the USB device through to the agent host

Skip this if the agent runs on bare metal. If the agent host is a virtual machine or container, the hypervisor owns the USB port, so the device must be forwarded:

Platform	How to forward the device
Proxmox — VM	qm set <vmid> -usbN host=<bus>-<port> to bind a physical port (the mapping survives swapping phones on that port), or host=<vendor>:<product> to follow one specific device. Find the ids with lsusb on the Proxmox host.
Proxmox — LXC container	Add to /etc/pve/lxc/<ctid>.conf: a device-cgroup allow for USB (lxc.cgroup2.devices.allow: c 189:* rwm) and a bind-mount of the USB bus (lxc.mount.entry: /dev/bus/usb/<bus> dev/bus/usb/<bus> none bind,optional,create=dir), then restart the container. Re-plugging a device onto a different bus needs the matching bus mounted.
VMware	With the VM running, VM → Removable Devices → &lt;phone&gt; → Connect (or add a USB controller and attach the device in the VM settings).

USB modems and old-Android dongles forward the same way — they appear as the same USB device class, so a bus that's passed through carries phones, modems and dongles alike.

3. Confirm it appears

On the agent host, adb devices should list the handset as device (not unauthorized or offline). Within a few seconds it appears in the Fleet View for that site. If it doesn't, see Troubleshooting.

Fleet View

The Test Devices view lists every connected handset across all sites, grouped by site. Each card shows the model, manufacturer, Android version, screen resolution, network operator, battery, and build — at a glance you can see which phones are online and healthy.

The fleet groups devices by site (e.g. "Cell Site 12", "Lab North"). Each card shows live status, battery, and operator. Select phones and a script to fan a test out across them; click a card to open its console.

Labelling a device. The pencil on a fleet card opens Edit device metadata, where you can give a device a friendly name and record its operator, IMSI and MSISDN. These are operator-assigned (modern Android hides the IMSI/MSISDN from ADB) and stick to the device even when it's offline — the name is what shows in the console headers and device pickers.

The sidebar's Test Devices entry expands to show each site (with a device count), the Agents view, the script library, the scheduler and the device tools, so you can jump straight where you need.

Running tests across devices

Select one or more phones, choose a script, and Run on selected fans it out in parallel. Results stream into a table — one row per device with a pass/fail badge, exit code, duration, and expandable output.

Agents

The Agents view (sidebar → Test Devices → Agents) lists every bench connected to this server — one row per site host — with its online status, IP, the devices it carries, when it registered, and its software version. It answers "which sites are actually reporting in, and from where?" at a glance, and is where you confirm a newly-installed bench has dialed in.

Device Console

Clicking a phone opens its console: a live view of the screen with full remote control.

The console streams the phone's screen live. Click to tap, drag to swipe, type to enter text. A network-quality badge (top-right of the screen) shows green/amber/red at a glance — click it for the full stream stats (resolution, FPS, bitrate, 5-second packet-loss %, jitter, RTT). The hardware-key bar (back, home, recents, volume, power) and quick actions (Airplane, Dialer, Messages, Settings) sit below; a tabbed tool panel — Cellular validation, Scripts, Shell, Logcat, Files and Record — is alongside.

One viewer at a time. A device streams to one operator at a time. If someone else is already viewing it, the console shows "In use by <user> (<IP>)" — the IP lets you tell whether it's you on another machine or a different operator — with a Take over button to claim the stream. A dropped or reloaded console reconnects to its own session automatically rather than fighting for it.

Remote control

Action	How
Tap	Click anywhere on the screen
Swipe	Click and drag
Type	Click the screen, then type on your keyboard
Hardware keys	Back, Home, Recents, Volume ±, Power buttons
Quick actions	One-click Airplane toggle, or open the Dialer, Messages or Settings app. These fire in the background — they don't run a test or block the console.

The streamed screen is the live device — any app or web page renders and is fully drivable from the console. Here a test phone has a page open in the browser, controlled remotely:

Viewing more than one phone

To watch several handsets at once — both ends of a test call, say — open one phone's console and click View second phone (top-right). Pick another phone from the list and its live screen opens alongside the first, with its own full controls: tap/swipe/type, hardware keys, quick actions, listen-to-audio and stream stats. Add more the same way, up to four screens at once.

Each extra screen carries its own picker to swap which phone it shows and an ✕ to close it. The arrangement is captured in the page address (?with=…), so a specific multi-phone layout can be bookmarked or shared.

The tabbed tool panel (Cellular, Scripts, Shell, …) always acts on the primary phone — the one you opened — which is named in the panel header so it's clear where a script or command will run.

Console tools

Alongside the screen, a tabbed tool panel keeps the per-device tools together — switch tabs without losing each tool's state:

Tab	What it does
Cellular	The call / text / ping / speedtest validation tools (below)
Scripts	Run a library script on this device and see its full run history, with a one-click re-run on any past run
Shell	An interactive command/response console for the device
Logcat	Live device log stream with text + level filtering
Files	Browse, download, upload, rename and delete files on the device
Record	Capture a tap/swipe sequence as replayable script lines

Cellular Validation panel

Built-in tools for the checks an operator runs by hand, each taking the inputs you'd expect:

Tool	Inputs	What it does
Call	Number, hold time	Dials, confirms the call goes off-hook, holds, hangs up
Text	Number, message	Sends an SMS and confirms it left the device
Ping	Host, count	Pings a host over the device's data connection
Speedtest	Test URL	Measures download throughput over the device's data link

The Number box remembers numbers you've recently called or texted (kept in your browser only, not on the server) and offers them as a dropdown, so re-dialling a test number is a click.

Scripts

The Scripts tab runs any library script against this one device and keeps its full run history — filterable by source, status and test — without leaving the console. Every past run has a re-run button, so repeating a check (or a script that just failed) is one click; scripts that take inputs re-prompt for them.

Shell

The Shell tab is an interactive command/response console for the device — type a command, see its output and exit code, with ↑/↓ to recall history. It's the device analogue of the modem AT console, for the times a quick command is faster than writing a script.

Logcat

The Logcat tab streams the device's live system log. Filter by free text and minimum level (Verbose → Fatal); levels are colour-coded. Start/Stop controls the stream, and stopping keeps what you've captured on screen so you can scroll back through it.

Files

The Files tab is a file manager for the device. Browse directories (starting at /sdcard), download a file to your machine, upload a file to the device, and rename, delete or create folders. Useful for pulling a capture or diagnostic file off a device, or pushing a config onto it.

Gesture recording

Toggle Record, perform a tap/swipe sequence on the screen, and the console emits the equivalent adb input lines. Paste them into a folder script to turn a manual UI flow into a repeatable test.

USB Modem console

A USB cellular modem (Quectel EP06 and similar) appears in the fleet as a modem and opens a cellular console instead of a screen. AT commands are relayed agent → modem over its serial (AT) port; every command and response is logged to the Terminal tab as an audit trail.

Tab	What it does
Status	One-shot read of model / firmware / IMEI / IMSI / SIM, signal (CSQ + per-RAT QCSQ), operator, EPS registration, attach, IMS, IP, APN, DNS/PCO
Actions	Set APN (preset or custom, with PDP / IP type), attach/detach data, show IP + DNS, ping, send SMS, radio (airplane / reboot)
Networks	Operator scan (AT+COPS=?) and a serving/neighbour cell measurement (AT+QENG) → EARFCN / PCI / band / RSRP / RSRQ / SINR table
Cell lock	Lock to a specific EARFCN (+PCI) (AT+QNWLOCK), band lock (AT+QCFG="band"), RAT preference, or release back to auto
Calls	Dial / answer / hang up, an AT+CLCC call table, and an Enable / Disable VoLTE control (AT+QCFG="ims")
SMS	Read the inbox (AT+CMGL), delete, and send
Events	Enables the relevant URCs and polls them into a timestamped log (registration / call / signal / new-SMS changes)
Terminal	Raw AT entry plus the full request/response log

The full AT-command reference, with real EP06 responses, is in modem-at-commands.md.

VoLTE: the EP06 is VoLTE/IMS-only for voice. Enable VoLTE sets AT+QCFG="ims",1; an AT+CFUN=1,1 reboot may be needed to (de)register IMS, and it only registers if the network + SIM provision it and the correct carrier MBN is selected.

Old-Android dongles (screencap mode)

Cheap ~$10 LTE "MiFi" dongles (e.g. the UZ801, Qualcomm msm8916) run Android 4.4 (API 19), which is too old for the live-screen view. These devices fall back to a screencap console: the agent polls adb screencap for frames (a few per second — screencap itself is the bottleneck on this hardware) and injects adb input for control. You get a click-to-tap, type-to-text view — not smooth video, but enough to drive it.

The agent smooths over the rough edges of these dongles on the host side:

• ADB unlock — they ship in RNDIS mode; a udev-triggered service curls the dongle's usbdebug.html to switch it into ADB mode. See agent/host-setup/mifi-dongle.
• Auto-English — they ship in Chinese; the agent sets persist.sys.language to en-US the first time it sees an API-<21 device (disable with OMNIWEB_DONGLE_AUTO_ENGLISH=0).

Programmatic control (MCP)

The same device-control stack is exposed as an MCP server, so an MCP client (Claude, scripts) can drive the farm: list_devices, screenshot, tap / swipe / input_text / key, adb_shell, list_scripts / run_script, and modem at_command / send_sms. It wraps the /api/android/* API and is bearer-token gated. Setup and the claude mcp add … command are in mcp/README.md.

Scripts

Tests are bash scripts that run on the agent host (so host tools like grep/awk are available) and target a device via ADB. There are two kinds:

• Folder scripts — no-argument health checks, run with just the device. These appear in the library, the fan-out runner, and the scheduler. Examples: Device Connected, VoLTE Registration, Report Signal Strength, Toggle Airplane Mode.
• Built-in tools — the parameterised UI tools (Call, Text, Ping, Speedtest) driven from the Cellular Validation panel. They take inputs and are not part of the library or scheduler.

The script library lists the no-argument folder scripts. Each shows its description and timeout; click one to view its source (read-only). Recent runs appear below.

A script reports its result by exit code — 0 is a pass, anything else a fail — and its captured output is shown in the results table.

Scheduled Tests

Scheduled tests are defined declaratively in schedules.json in the scripts directory. The file is re-read on every cycle, so edits take effect without a restart. Each entry runs a folder script against its target devices on an interval.

Configuration

{
  "schedules": [
    {
      "name": "VoLTE Registration",
      "script": "volte_registration.sh",
      "interval_seconds": 120,
      "serials": [],
      "enabled": true
    }
  ]
}

Parameter	Type	Required	Default	Description
name	String	Yes	-	Unique label for the schedule. Appears as the schedule label on the exported metric.
script	String	Yes	-	Filename of a folder script in the scripts directory to run.
interval_seconds	Integer	No	300	How often to run the script, in seconds. Minimum 30.
serials	List	No	[]	Device serials to target. An empty list means all currently-connected devices, so the schedule auto-follows whatever phones are plugged in.
enabled	Boolean	No	true	Whether the schedule is active. Set false to pause it without removing the entry.

Example: VoLTE check across a site

{
  "name": "VoLTE Registration",
  "script": "volte_registration.sh",
  "interval_seconds": 120,
  "serials": [],
  "enabled": true
}

How it works: Every 120 seconds the scheduler runs volte_registration.sh against every connected phone. The script checks the device is voice-registered (IN_SERVICE) on LTE with VoPS (Voice-over-PS) support, and exits 0 if VoLTE is registered. The result becomes a Prometheus gauge per device.

Use case: Continuously prove that real handsets at a site can register for VoLTE, and alert the moment one can't.

Metrics

Central OmniWeb exposes the latest result of each scheduled test, per device, in Prometheus format at /api/android/metrics. Prometheus scrapes it as the android_device_tests job (configured automatically by the monitoring role).

Metric: omniweb_android_test_pass Type: Gauge Description: Result of the most recent scheduled run — 1 for pass, 0 for fail Labels:

• phone — device model (e.g. SM-A536E)
• serial — device serial number
• site — the agent's site label (e.g. Cell Site 12)
• agent — the agent host the device is attached to
• script — the script that ran
• schedule — the schedule name

Companion gauges share the same labels:

Metric	Description
omniweb_android_test_return_code	Exit code of the last run
omniweb_android_test_duration_ms	Duration of the last run, in milliseconds
omniweb_android_test_last_run_timestamp_seconds	Unix time of the last run

Example queries:

# Phones currently failing VoLTE registration
omniweb_android_test_pass{script="VoLTE Registration Status"} == 0

# Failing tests grouped by site
count by (site) (omniweb_android_test_pass == 0)

# A test that hasn't run recently (stale agent or device offline)
time() - omniweb_android_test_last_run_timestamp_seconds > 600

Troubleshooting

A phone doesn't appear in the fleet

Symptoms: A device is plugged in at a site but isn't listed under Test Devices.

Possible causes:

• The phone isn't authorised for ADB on the agent host
• The agent isn't running or can't reach central
• The phone is in an offline/unauthorised ADB state
• The central Test Devices service isn't running (Test Devices is served by its own dedicated backend service, separate from the main request pool)

Resolution:

1. Confirm the phone shows as device (not unauthorized/offline) in ADB on the agent host, accepting the RSA prompt on the handset if needed.
2. Check the agent service is running and that its configured central URL and token are correct.
3. On central, confirm the Test Devices service is running and that the gateway routes /api/android/* to it. If the agent registered but the fleet is empty, this service is the usual cause.
4. Verify the phone's USB connection; re-seat the cable if it shows as offline.

The console screen stays on "Connecting"

Symptoms: The device console opens but the screen never appears, while the rest of the UI — fleet list, device details, input — works normally and the device shows online.

Possible causes:

• UDP is blocked between the browser's network and the agent host (the live video streams directly browser → agent host, not via the server — by far the most common cause)
• The agent can't start screen capture on that handset

This is not an SSL or MTU problem: TLS is proven working (the UI loaded), and with MTU issues the video would connect and then tear rather than never starting. The tell-tale sign is a device that shows online in the fleet yet whose live view alone won't start.

Resolution:

1. Confirm the browser's network can reach the agent host over UDP (the video path). This usually means a firewall/VLAN rule permitting the operator's subnet to the bench hosts over UDP — see Network Requirements for the exact rule to request and how to confirm the fix.
2. Check the agent log for screen-capture errors; some handsets can't have their screen captured over ADB.

VoLTE / signal checks report "not readable"

Symptoms: VoLTE or signal scripts fail with the telephony state unreadable.

Possible causes:

• The handset restricts telephony status to privileged access

Resolution:

1. Use test handsets that expose telephony status to ADB. Vendor-locked consumer devices may hide it without privileged access.