Being yelled at because a household member couldn’t raid in their MMO made me take a look at how I should be monitoring.

The last post ended on a table of limits, all of them set against a gigabit link. This post is about the morning that stopped being true. The torrent VM post covers why those limits exist and which ones bind. Here I only need two of them: Smart Queues on the UDM Pro’s WAN, shaped at 260 Mbps down, and a 102.4 Mbps download cap inside qBittorrent.

At about 04:10 on 10 September the UDM’s WAN port renegotiated from 1000 Mbps to 100 Mbps and stayed up. Cox scored 99.94% uptime for the day, because to every up/down check the link was healthy. It was a tenth as fast, and that’s a different failure from being down.

What the house noticed was latency. Every device on the network hit 178 ms and about 16% packet loss at once, and it held there for hours.

It started while most everyone was asleep. About four hours in, people started waking up, and it took another two hours for it to get reported back to me and for me to start digging. Yes, my SLA is horrible, but they aren’t paying me jack either.

The missing bandwidth wasn’t the whole story. Smart Queues and the qBittorrent cap were both correct for a gigabit link, and both stopped doing anything at the same moment:

SettingValueAgainst a 100 Mbps wire
Smart Queues, download260 MbpsThe ceiling sits above the real bottleneck, so the shaper never engages and the queue builds in the port buffer
qBittorrent download cap102.4 MbpsAbove line rate, so the one VM can fill the pipe by itself

The torrent post explains why a shaper does nothing once its rate sits above the real bottleneck. When the bottleneck moved under it to a slower wire, the shaper at 260 was shaping nothing, and the torrent VM was allowed to use more than the whole link. Then I replaced the cable and gigabit came back. Both settings started working again with no edits.

The issue was a Cat5 cable that had lost its clip. With nothing holding the connector in, someone walking over to the printer to get something must’ve shaken it in its seat.

Two numbers that pointed at the wire

I didn’t start from “the link negotiated down.” I started from a house with terrible latency, and the controller’s numbers turned it into a wire problem.

The WAN port’s lifetime average throughput was 116.9 Mbps. A 100 Mbps link can’t carry that. Whatever had changed was recent, because most of the port’s history had happened at a speed it was no longer running at.

Its peak after the event was 97.3 Mbps. qBittorrent’s own cap was 102.4 Mbps, and a total lower than one client’s individual cap has to come from the wire. Nothing on the software side was holding the traffic at 97.

Receive errors on the port stayed flat the whole time. That ruled out a noisy link and pointed at a clean down-negotiation: two working pairs in the cable instead of four. The same signature had already hit ms01a once.

The obvious follow-up was why I’d found out from latency. UniFi’s notifications cover a device going offline, a WAN failover and an ISP outage, and the only alert settings in the Network application are for IPS. A port that stays up at a lower speed fits none of those. That’s what the controller showed me in September, so check yours before you rely on it.

Uptime Kuma could watch for it, but Kuma has no UniFi monitor type, and the controller won’t answer anything without a login. Kuma’s Push monitor fits that gap. Something that can reach the controller reads the facts, decides, and pushes a verdict to a URL Kuma gave it. If the verdicts stop arriving, Kuma notices that too.

That something is wan-health-to-kuma.py, a stdlib-only Python script on my Raspberry Pi, because the Pi has no jq and I didn’t want a package install for one script. It logs in with a local admin that has the Viewer role (the same account the torrent post uses to read Smart Queues), pulls two endpoints, and emits two signals.

The script reads stat/device and walks every port on every device:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

slow, checked, wan_speed = [], 0, None
for dev in devices.get("data", []):
    dname = dev.get("name") or dev.get("model") or "?"
    for port in dev.get("port_table") or []:
        if not port.get("up"):
            continue
        pname = (port.get("name") or "port %s" % port.get("port_idx"))
        speed = port.get("speed") or 0
        if "wan" in pname.lower():
            wan_speed = speed

Only ports that are up count. A port that’s down is the controller’s problem, and a port that’s up and slow is mine. After the exemption checks (more on those below), anything between 0 and EXPECT_SPEED goes on the list:

1
2
3

            checked += 1
            if 0 < speed < expect:
                slow.append("%s/%s=%dM" % (dname, pname.strip(), speed))

The threshold is 1000 for every port, not only the WAN. That was a choice. Since ms01a had already shown that fault, a WAN-only check would have watched the one place it had just happened and ignored the rest. Any port on any device that’s up and below gigabit goes red. If you shape your own WAN, this is the check I’d want you to have, because the shaper’s rate and the port’s speed are two numbers that only work together.

Exemptions come from the ALLOW_SLOW setting, parsed into two forms. Device/3 pins to a physical port, and a bare string matches against the port’s name:

1
2
3
4
5
6
7

allow_names, allow_ports = [], set()
for entry in (e.strip().lower() for e in cfg.get("ALLOW_SLOW", "").split(",") if e.strip()):
    dev, sep, idx = entry.rpartition("/")
    if sep and idx.strip().isdigit():
        allow_ports.add((dev.strip(), int(idx)))
    else:
        allow_names.append(entry)

The loop checks both before counting a port:

1
2
3
4

            if (dname.lower(), idx) in allow_ports:
                continue
            if any(a in pname.lower() for a in allow_names):
                continue

I prefer the pinned form. A name match follows a rename and can catch ports I never meant to exempt, while the device and port index stay put.

A latency check that waits for three samples

The second signal is WAN latency, the symptom the house feels. The controller reports one ping per read, and Cox produces spikes of 100 to 300 ms several times a day. On 17 September there were 33 of them while the average sat near 27 ms. My first version flipped the monitor down on any sample over 90 ms, then back up a minute later, so every spike alerted twice.

The event this check exists for was 178 ms sustained for hours, so the breach has to persist:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

streak = read_streak()
if latency is None:
    lat_status, lat_msg, ping = "down", "no latency reported by controller", ""
    write_streak(0)
elif latency > lat_max:
    streak += 1
    write_streak(streak)
    ping = str(latency)
    if streak >= sustained:
        lat_status = "down"

Three consecutive samples over 90 ms marks the monitor down. At one sample a minute, that’s three minutes of sustained trouble. A single spike stays up, but it still carries its real ping, so the graph shows it and the alert doesn’t fire. The streak counter lives in /run, a tmpfs, so a reboot resets the count to zero. That errs toward not alerting, which is the direction I want from a bookkeeping file.

The timer fires on the wall clock, on the minute, and that was a fix:

1
2
3
4

OnCalendar=*-*-* *:*:00
AccuracySec=1s
RandomizedDelaySec=0
Persistent=false

My first timer used OnUnitActiveSec. It counts from the end of the previous run. Each push landed about a second later than the one before it, until one crossed Kuma’s own check. On 17 September that produced somewhere between 25 and 80 “No heartbeat in the time window” alerts a day while the script ran perfectly every minute. The 17:40:20 check had fired one second before the 17:40:21 push. The timer runs every 60 seconds against a 120 second heartbeat, so a push has half an interval of slack.

Annoying to debug the script and find it working fine, only for the problem to be how I was reporting on it.

When the script can’t reach the controller, it pushes nothing. Kuma expires the heartbeat and shows no data, and that reads as “I don’t know.” Pushing “down” would claim a verdict the script couldn’t support.

Pushing verdicts into Kuma

Both signals go out through one small function. Kuma’s Push monitor takes a status, a message and a ping value as query parameters on a URL that contains its token:

1
2
3
4
5
6

qs = urllib.parse.urlencode({"status": status, "msg": msg[:250], "ping": ping})
url = "%s/api/push/%s?%s" % (kuma_url.rstrip("/"), token, qs)
try:
    urllib.request.urlopen(url, timeout=8).read()
except Exception as exc:  # noqa: BLE001 -- deliberately total
    print("wan-health: push failed: %s" % exc, file=sys.stderr)

It never raises. A failed push on the link signal mustn’t stop the latency push after it, and a push that can’t be delivered shows up as silence anyway. The two monitors in Kuma are “WAN link speed”, with a 120 second heartbeat and one retry, and “WAN latency”, with the same heartbeat and three retries.

The PS5, the Arlo and Luna

A check that treats every port the same turns up the ports that aren’t the same. Mine has two permanent exemptions, and the config file explains each in comments:

1
2
3
4
5
6
7

# Ports exempt from EXPECT_SPEED. Two forms, comma separated:
#   Device/3          pins to a physical port, survives the port being renamed
#   some text         case-insensitive substring of the UniFi port NAME
# Prefer the first. Keep this list SHORT -- every entry is a port that can
# silently degrade with nobody watching, which is the exact fault this exists
# to catch.
ALLOW_SLOW=Arlo basestation, YunYun/1, Vanir/1

The Arlo base station is a 100 Mbps device. YunYun port 1 is the PS5, and it’s the better story. In rest mode the console keeps the link up so wake-on-LAN works, and the NIC downshifts to 100 Mbps to save power. It also transmits nothing, so the switch never learns its MAC address and floods broadcast and multicast at it forever. When I first looked, that port showed 233 MB sent and zero received, and that looks like a cable with two dead pairs. Dividing it out settled it: 11.6 packets a second at 159 bytes each is ARP and mDNS background traffic. Receive errors sat at zero the whole time. That’s what tells a sleeping PS5 apart from a failing cable, and I underweighted it on an evening when a real two-pair fault was in progress.

Writing this post turned up a third one. The dry run on the Pi came back red:

1
2
3

link     : down negotiated below 1000M: Vanir/Port 1=100M
latency  : up   22ms (down 4 / up 6 Mbps)
tokens   : link=True latency=True

--dry-run prints the verdicts without pushing them or touching the latency streak. Vanir is the USL8LP switch, and port 1 is a PoE port feeding a UniFi Protect device named Luna, which the controller shows at 100 Mbps. Port 1 was up, below 1000, and not in ALLOW_SLOW, so the link signal read down. I’d already paused the “WAN link speed” monitor in Kuma because Luna kept it red. That was a bad trade, because a paused monitor stops watching every other port too, the WAN included.

Pinning the exemption to the port was the better fix. I added Vanir/1 to ALLOW_SLOW, which is the line in the config block above, and ran the dry run again:

1
2
3

link     : up   WAN 1000M, 18 ports all >=1000M
latency  : up   16ms (down 19 / up 7 Mbps)
tokens   : link=True latency=True

Eighteen ports checked, all at gigabit, and the monitor is running again. The list stays at three: the Arlo base station, the PS5 and Luna.

Where it lives

The script, its two systemd units and the example config are in my homelab repo under pi/, and the live config with the password and push tokens stays untracked at /etc/homelab/wan-health.conf, mode 600. Kuma has two Push monitors, “WAN link speed” and “WAN latency”, each with a 120 second heartbeat. The torrent post says I didn’t set up the WAN latency monitor in Kuma until 10 September. That’s the same day the port dropped.

The qBittorrent cap and the Smart Queues rate haven’t changed since the incident. Neither of them can see the wire under it, so the monitor reads the port speed they depend on every minute and tells me when it isn’t what I sized them for.