I don’t think you’re aware of how independent audits, open source, good cryptography, a non-profit, government data subpoenas, and a lack of data collection works.

I think that you maybe the one who doesn't understand how any of this works. Security and privacy are guaranteed by design, and any information that is collected has to be assumed to be available to bad actors. Period. The same reason logic about trusting the server to do the encryption applies to letting the server handle metadata. No amount of audits can guarantee that people operating the server are doing it in good faith.

Meanwhile, the concern isn't just about somebody having your phone number it's about Signal server having the ability to map out relationships between these numbers. It's perfectly fine for people to reason that this is not something they're worried about, and make an informed choice to use Signal. However, it's incredibly disingenuous to pretend this problem doesn't exist.