+1000 points for the PiKVM V4 Plus. We (Revise Robotics - a YC company!) refurbish laptops with robots and AI - as part of this, we (or rather, the AI) send(s) keyboard commands in software to the computers we're refurbishing.
How/why? The AI needs to navigate the BIOS among other tasks - so we need a KVM to send arrow down and enter, roughly speaking.
We were a GL.iNet KVM shop until we ran into a nasty issue with a specific ThinkPad - the GL.iNet would send an incorrect USB 0 byte which most laptops ignored, except this ThinkPad which was freaked out by it / beeped / wouldn't accept any key command.
I couldn't let this problem go, so I got a low level USB debugger [0] (which I extremely recommend) and wire-debugged the USB signal, A/B comparing the GL.iNet and the PiKVM. The PiKVM was doing things properly (usb-wise), so we swapped all (~10) of our KVMs for it.
I also remember that the GL.iNet was stranger/more difficult to customize (it's just running pikvm the software but doesn't let you customize it as much). The GL offers a nicer UI, but it doesn't matter that much (we drive it via API) and we're happy to support the actual PiKVM authors/company. It's a fantastic product. Not cheap, but truly truly great.
P.S. If someone from GL wants to reach out, I can offer you a lot of low-level debugging info -- fixing this issue would be great.
I think if you can afford it, the PiKVM is still the gold standard... and also one of the best engineered devices, with the most flexibility. It costs a lot because it's worth it.
They also have a nice (if messy) solution for multiple device management with an external box (TechnoTim had some good coverage on his channel/website[1]).
Most other Pi-based KVMs use PiKVM's software anyway, and I'm not sure if any directly support PiKVM upstream (they should, IMO).
The other class is the JetKVM and its derivatives (many forked JetKVM's snappier Go software), and that's another reason I've stuck with JetKVM itself. It seems to have a nice community around it for mounts in almost any situation, and hacks to get weird things working correctly with it. They're also going to have a PoE + full-size HDMI version soon, I think, with microSD for expanded ISO storage.
I'm not from GL. I write my own USB HID handlers and I'm being nosy about bad protocol that exists in the wild. Was the 0 byte in some input report? In a descriptor? Somewhere else?
TLDR - GLKVM's kernel emits a trailing 0-byte DATA packet (ZLP) (which PiKVM does not)
Supposedly, this is due to the `req->zero = ((count % maxpacket) == 0)` line in the kernel's f_hid.c - which, for 8-byte HID keyboard reports, makes it append a trailer packet. Strict BIOS HID stacks treat this 0-byte packet as malformed and beep.
It's one of those things where $400 is WAY too much for a homelabber, but most companies (even small ones) can barely count that low when speccing out hardware, ESPECIALLY these days. $400 is like one hard drive in a machine that will potentially have 8-24 of them.
......Unsettlingly that's also like 16GB of dram in a machine that might be measuring memory in TBs
I kinda wanted to get a KVM recently, but decided to save my money to afford RAM for my planned server build instead. Might even get a 32 GB kit for $400.
Surprised nobody mentioned Intel vPro AMT so far. It is basically an always-on KVM that's part of CPU firmware, powered by an always-on 5V PSU rail. There is a scary amount of options, including unattended periodic (or alarm based) phone home, user acceptance or full user override, boot media spoofing, Serial over WiFi... All built-in into consumer(-ish) CPUs.
I'm not surprised that nobody noticed it at all. That's because it's not a KVM that you can purchase and test, the subject to the article.
Instead, you're just talking about IMPIjr. These specific features are not in consumer grade CPUs, but in CPUs marked for workstations. Ain't no consumer buying fuckin vPro machines. These are enterprise IT management features, not user features. They're also subject to frequent insane rants by people obsessed with them as possible privacy issues.
Hey Jeff, I did some research on the jetkvm after reading this as I was very impressed but wanted full scale hdmi + Poe and was going to pull the trigger on the clone you mentioned later, ArkKVM but felt like I’d rather support the main project if I could…
What I found seems to indicate that Jet fixed those two issues in a hardware revision but it’s really difficult to distinguish the new on from the old as they’ve seemingly kept the same name and not added a v2 or something like that to the naming. One of their vendors has a Poe va non poe sku, the other has a an emmc vs tf card sku. All seemingly without a name distinguishing them.
There’s also just chaos on Amazon as they are being sold in at least 4 separate listings with no name distinguishing which model is which, non of them mention poe and all claim full size hdmi.
In any case I thought you should know that your write up is out of date here but you probably need to do some digging to figure it out.
Not affiliated but I had good experience with GL.inet’s comet line [1]. They have one on kickstarter that’s the size of a google cast puck that uses purely usb-C. Though all my KVM do not have internet access (blocked at my gateway). I can only access it via tailscale externally.
There are certainly edge cases where you want native USB and display, but after initial bring up, the device is on the network, and can be managed over the network.
So, sure, nerd out and add more hardware to your rack, but I need a physical keyboard and mouse attached to a machine in my rack like once per year.
Okay, there were two things that bothered me with these KVM switches: the power adapters are massive so there's too many cables, and the cables don't go always in the same side of the thing. Your post covers both and I'm thrilled. The final thing I recognize is a bit of a nice to have but the only thing that I want to rack that doesn't have a BMC is my Mac Mini which I've hesitated to put in the Sonnet RackMac and run because I don't know how to KVM power it on/off. My cabinet is an hour away from me and the Mac Mini runs the family AI agent so I need it to be available all the time. So far, it hasn't ever needed any attention (it comes back on after power outages at home) but I'd prefer to be able to turn it on/off remotely ideally.
Do you know which one of these works well with that?
Could you get a kvm connected to a networked PDU with per-outlet control? Then a power cycle on the plug for the mac would accomplish the same thing. Or just use the network port on the PDU directly w/o kvm.
The annoying thing with Macs is they don't have the ability to 'Power on after clean shutdown and power cycle'. They can power on after a power fail, but that's not the same thing.
I want to be able to have power toggle on my Mac remotely, but without soldering in a jumper on the power button, AFAIK there's no way to do it :(
You're right. I just need to upgrade my PDU. But as you can imagine, I'm hesitating because I don't want to power down the rest of the hardware either.
I have a mild distrust of some of the cheap IP KVMs. I don't think vendors are malicious, but I don't expect they get it right every time either.
Admittedly, I haven't looked at any open-sourced firmwares either which could have improved things.
I have found the Sispeed USB KVM very useful, the convenience is well worth the $50 it cost me. The UX isn't great but you don't really need it to be. It works (most of the time) via WebUSB for the keyboard mouse.
I went with Sipeed NanoKVM (PCIe) units across my homelab as well and I've also been happy with them. For a while it's been the best value option (not to mention the most consistently available option) cf. GL.iNet, PiKVM, and JetKVM. The PoE versions are great in a rack and the integrated ATX control is fully-featured (including the little power switch icon in the web UI turning green when the system is powered on). I set up an isolated OoBM VLAN with no Internet access and any switch ports assigned to it are isolated by default as well.
Always a fun adventure when you buy an old server motherboard to see what state of disarray the BMC is in. Or if you have to patch the license on older boards just to get into it. Usually I just ignore the BMC on any hardware more than 5 years old.
I've been interested but the idea of SPI flasher recovery has always meant I've not bothered actually trying. Maybe I will finally on one of these old boards I have lying around..
Same here; trying to get OpenBMC flashed onto older motherboards can be a rabbit-hole. I generally throw an IP KVM on top if I really need remote control and can't settle on just SSH and a remote power cycle if that fails.
It will never make sense to me why KVMs are such a hard problem to solve. It seems like something we should have a good answer for by now but we still really just don't without dropping hundreds of dollars, and even then it still feels like a crap shoot.
It’s bizarre that one needs to do all this via a keyboard/monitor interface.
Nearly thirty years ago when I used to work on Silicon Graphics kit everything from powering on and off a server to installing the OS from scratch over the network could be done out of band via serial - and automatically, using expect(1)
server ipmi includes serial over lan, which can work ... it's helpful if the console settings for early boot match the console settings for OS boot, and there's no standard for sharing them, so that's a bit of a mess... and then you have some where serial over lan drops out during boot, which is kind of terrible. :(
But, the devices here are geared towards people using consumer products as servers (which I do at home, not critizing), and serial console during early boot is not an option on those, and most boards don't even have a serial port anymore.
I will say, if you're presenting a usb keyboard, you might also be able to present usb storage which could be nice for booting off of.
> serial console during early boot is not an option on those
But it should be. There's really no excuse for the current state of affairs. There ought to be a standardized low level API for all this stuff. One should never need to directly attach a keyboard, mouse, and monitor in order to provision a piece of hardware.
As other comments have indicated, this is basically solved for all servers. All Poweredges and supermicros have come with increasingly sophisticated integrated remote management systems for the past 20+ years. I have some R610s and their drac is old but still quite good all things considered.
They aren't a hard problem to solve. In the server market it's completely solved with a BMC. The problem being solved here is someone wants to make a product using some commodity product like a raspberry pi, perform video capture on a VGA/HMDI/DP port. This is not actually a problem end users have.
If you want to plug into a system that isn't server class, then they should be producing a video card that hooks into the USB bus, the always on rail of the power supply, the power switch pin of the power supply, and has an RJ45 jack. The contents of the card should be an off the shelf BMC chip.
But realistically if you want this kind of functionality, just buy server class systems that come with it.
All of these support a single computer only, so if your home lab has multiple PCs you need a remote control KVM switch too. And separately a remote control power strip/outlet if you need that also.
KVM switches are relatively cheap[^1] so I'm surprised there isn't an integrated solution.
There are a few, like this add-on for PiKVM[1], a fully integrated version from Geekworm[2], and an as-yet unreleased 4-port IP KVM switcher from GL-iNet[3].
Keep in mind you absolutely can DIY the PiKVM, it's supported in their docs. Buy the Pi, buy the extra hardware you may need for power splitting & video in, install PiKVM software (unfortunately it's a rather massive package of custom software), and it's probably around a hundred bucks cheaper than their cheapest product version.
The JetKVM picture is aggravating to look at. It shows IPv4 address in large print and the IPv6 address in small print. That’s fine, but the IPv6 address is truncated with an ellipsis. How is that useful? A truncated IP address doesn’t work. Do the developers expect autocomplete for IP addresses? (And the IPv6 address is a ULA, so it could have been much shorter if SLAAC was involved in creating the address. Otherwise blame the DHCPv6 server I guess.)
I have a CSE847 and HP DL380 G10 that have gone down for me due to power outages. Many of these look complex, and I basically just need remote power-on/toggle capability. Should I be looking at something else?
You could configure BIOS/UEFI to power-on when power is restored (or return to last state). Or use wake-on-LAN, which quite a few consumer routers can send through their web interface.
Don't your servers already have BMCs supporting IPMI to provide full remote management? Often features like full KVM will require extra licensing, but remote power on is one of the most basic features and I've never encountered a BMC that didn't provide at least that much remote control.
Sounds like you might need a UPS/Battery backup for outages.
Some of those servers can have multiple power supplies for failover too. They also can have cards in them to power them on/off remotely as well as long as they have power.
How's the video quality/latency on all of these? RDP or parsec are probably the gold standard, but I doubt cheap arm SOCs can implement either properly.
On some the latency is within 45-60ms (that's the best I've tested, under pretty strict LAN conditions), but average is more like 100-200ms. Not good enough for gaming, but fine for things like watching video, if you really want to do that.
Most employ heavy compression by default, and it looks a little more pronounced with motion. A few have 'high quality' mode, but especially on the Pi-based KVMs this eats up more CPU, so I don't use that.
It's on the whole best for use cases where you just need to log into remote servers and check on things, or reinstall/re-image something. I don't think I'd like using any of these for a constant remote session for daily work. I do use Screen Sharing on macOS for that sometimes, and it works great in its low-latency mode. None of the KVMs I've tested are quite to that level.
I was pretty disappointed to discover DGX spark has no BMC. I was able to hook it up to pikvm by connecting leads to the power switch wires. It honestly works really well and I can provision it with MAAS
Shout-out to pikvm for their redfish support with the pikvm switch
Great writeup there are quite a few here that are new to me
I'm running a PiKVM DIY on a pi02w. Adequate, but I'd like more functionality and performance.
I bought a SiPeed NanoKVM. It caught fire 15 minutes after being plugged in. Despite providing pictures of the charred PCB, they insisted I ship it back, costing me €20, and then tried 3 times during the transit to get AliExpress to void my return as fraudulent. I eventually provided proof of signed delivery to their people on the last possible day of my final appeal, and AliExpress ruled in my favor, refunding my purchase price but not the return shipping cost. Better some money back than none at all!
Another vote for JetKVM. Tailscale support is great. I'm glad to see audio is in the works because a Mac mini screaming from separate room during a remote session is disappointing.
How/why? The AI needs to navigate the BIOS among other tasks - so we need a KVM to send arrow down and enter, roughly speaking.
We were a GL.iNet KVM shop until we ran into a nasty issue with a specific ThinkPad - the GL.iNet would send an incorrect USB 0 byte which most laptops ignored, except this ThinkPad which was freaked out by it / beeped / wouldn't accept any key command.
I couldn't let this problem go, so I got a low level USB debugger [0] (which I extremely recommend) and wire-debugged the USB signal, A/B comparing the GL.iNet and the PiKVM. The PiKVM was doing things properly (usb-wise), so we swapped all (~10) of our KVMs for it.
I also remember that the GL.iNet was stranger/more difficult to customize (it's just running pikvm the software but doesn't let you customize it as much). The GL offers a nicer UI, but it doesn't matter that much (we drive it via API) and we're happy to support the actual PiKVM authors/company. It's a fantastic product. Not cheap, but truly truly great.
P.S. If someone from GL wants to reach out, I can offer you a lot of low-level debugging info -- fixing this issue would be great.
[0] https://greatscottgadgets.com/cynthion/
reply