I have an informal arrangement for keeping the local network of one particular office in order. I spawned a PFSense box there many years ago, replacing some previous whatever router, and just reused the sprawling LAN setup that was there at the time.
It worked fine, for a long time. I've not heard real systematic complaints about the LAN experience, barring some rare, seemingly unrelated problems.
Recently, I was asked to set up a site-to-site IPSec VPN. I decided the easiest way was to spawn a tiny gateway server instead of messing around with PFSense config, so I did that - a simple Ubuntu setup, PFSense only needing a static route towards my new gateway for the subnet served by the VPN. It tested and worked fine, so I handed it over and expected no trouble.
A few weeks later, I sort of overheard that the software that ran over the VPN felt "annoyingly slow". I got a bit anxious - did I mess up the VPN somehow? I got onto the gateway and set up Netdata, along with some ping probes. Soon enough, I figured out that the VPN wasn't even being used for the software that supposedly ran over it. Not really relevant to the story, I guess - the traffic just went over the Internet, probably the supplier forgot that we got their prescribed VPN set up. Oh well. By then, I also heard a few more mentions of "internet being slow".
So, what causes the "annoyingly slow"? Claude and I started investigating, and we noticed a few things: the LAN port of PFSense had over 9 million flow control pauses accrued. Something on the LAN was constantly asking things around it to slow down transmission. Our theories were a lame spliced Ethernet run or a faulty switch.
We tried to stop the damage: we disabled flow control on pfSense's LAN port, so a potentially faulty switch could no longer pause it. That eliminated the LAN-wide latency storms - a win. But we still suspected the faulty hop to degrade the network experience for anyone downstream of it - Access Points included.
Claude and I then developed a simple path testing speedtest app: users of the LAN were given a simple web interface, asking them to identify themselves, their device and their connection method. We hosted that on the new gateway server I provisioned (I was really happy about my decision NOT to ride the IPSec on PFSense by then!) A rudimentary 20 second download and upload speed test would have the gateway server ping the computer doing the test, and track the Flow Control Pause delta on the gateway server network interface.
What it told was interesting, and something Claude and I expected: some paths would go close to gigabit and produce very few or no Flow Control Pause requests. And yet, others would stick suspiciously close to 100mbps speeds, and produce a ton of Flow Control Pause requests, and latency toward the tested machine spiking hard.
By then, it was clear - we have a faulty cable or a switch, and a way to trace it.
So, I went on site and started running the test on every switch I could plug into. That led me to one particular switch. I probed around, and eventually just decided to wing it and replace it with a different, known good switch. And that was that - all previously failing ports were doing great. I concluded that the switch just got lame over the years and that was that. But then I started measuring WiFi speeds.
WiFi was really slow. Like, across the board - no more than 120mbps on a LAN that now provably carries gigabit, and a symmetrical gigabit internet connection. I tried switching off all access points and testing one by one - all of them were capped at around 120mbps, even on 5GHz! I then plugged in one of my testing access points, and it was able to push 800Mbps both ways with low latency and jitter - totally what I expected. The access points on that location are all old TPLink Archer C7's (Version 4), all flashed with somethng OpenWRT-esque, years ago. I wasn't in the mood to try to figure out what was wrong with them, but 120mbps was definitely below their expected throughput on 5GHz.
In the end, I replaced one of those APs with one of my own, in the room where people were suffering the most. I speedtested it - it was able to saturate the WAN. The switch was replaced, and everything was working right. Next morning, workers came in, and I got zero feedback, except for one message about wired LAN getting a lot faster. So, success, I suppose?
I'm totally into networking, but I'm not what you'd call a network engineer. This was a fun experience, definitely what I'd call a good time. And the results were encouraging. The site will get fully redone on Ubiquiti just months from now, so maybe it wasn't too necessary, but just a few months of happier employees can indeed make a difference, and it didn't even take that long in the end - just one evening.
Next morning I reflected on the experience and realized that the local network there probably sucked for years. I used to hang out on that location, and I don't remember having good LAN or WiFi experience myself there most of the time. But what if the WiFi AP and the switch problems were causing things to work just well enough to not really be a bother?
The 100Mbps hop was never a problem, because the APs were always slow - seldom fast enough to even trigger Flow Control. But people were just sort of dealing with somewhat slow WiFi, and it was fine.
The thing is - recently, the location bred a startup, and my theory is that somewhat suddenly a lot more demand landed on that LAN, and it buckled: WiFi got congested due to slow APs, people started using ethernet - which facilitated much more frequent Flow Control storms, which caused everyone else's experience to degrade even further.
So, what's the learning here? Slow networks become a problem when more people start using them? A little obvious, really. But, a fun issue to resolve nonetheless! And we tracked and marked a bunch of cables around the switch we replaced, so the Ubiquiti rework will be just that little bit more manageable.