I'm running OPNsense on a Dell PowerEdge R210 II that I bought used. It's been running great for 2 years or so. I've really liked OPNSense.
homelab
You can get Mini-ITX 19" rack mounts.
IMHO, it'll use less power than a server, plus, it's about the right amount of processing power for a router / firewall (I use pfSense, but there are other router solutions out there)
You won't want to virtualise / containerise that function, so anything more powerfull can be kept in a separate device.
I really like my UDM Pro, but you can also run OPNsense on off the shelf hardware.
Are you looking for something with a nice UI or just hardware you can install your OS on and roll everything yourself? What's your budget?
Ubiquiti makes a pretty user-friendly rack-mounted router: https://store.ui.com/us/en/products/udm-pro
You can also check out pfSense, which is an OS that does routing and more. You can install it on your own hardware or buy a machine through them: https://www.pfsense.org/products/
If you're just looking for a compact server that you can install your own OS on, then SuperMicro has a sale going on right now: https://store.supermicro.com/us_en/catalogsearch/result/?q=SuperServer%201u&categories=Deals
Just know that whatever you pick to replace the router-portion of your current Netgear device, you'll also have to invest in a separate WiFi access point to provide your wireless network.
I'd recommend OPNsense over PFsense due to multiple shady moves by netgate (the parent company of pfsense), including moving to closed-source:
- pfsense is falsely open-source: https://news.ycombinator.com/item?id=26476030
- pfsense botched/rushed their wireguard implementation: https://forum.endeavouros.com/t/migration-from-pfsense-to-opnsense-drama-about-wireguard/12798
- pfsense squatted on competitor domain and used underhanded/defamatory practices: https://opnsense.org/opnsense-com/
If you don't mind the drama, both PFsense and OPNsense are perfectly competent router OSes.
Regarding hardware:
- OPNsense also sells rack-mountable server hosts.
- OP may not actually need a rack-mounted server -- I have several machines just sitting on a 2u rack-mounted shelf. My opnsense install runs on a cheap protectli box, and there's enough room for a handful of raspberry pis and their power bricks on the shelf next to it.
I watched the opnsense.com on wayback machine in 2016 and... Omg that's so shitty. How old are them, 8?