An open network for secure, decentralized communication
Homeserver admins (of all homeservers that federate given Matrix room) can see who (meaning: Matrix user id) you've talked to and when it happened. They cannot access the content of the messages itself, as it is protected by E2EE.
Matrix P2P has been proposed to circumvent this issue (by embedding a local Matrix server within the client), but I have not yet seen it "in the wild".
Homeserver admins (...) can see who (...) you’ve talked to and when it happened.
I'm assuming that since it is federeated, even if one server was very diligent with not keeping such metatadata, the issue would still remain with the other server if it did not follow the same practices?
That's about right. The matrix protocol, while quite big on protecting messages, is not quite so worried about the metadata. This can be minimized if both users are from the same server.
Whether the metadata leakage is important depends largely on your threat environment.
As for Signal not having anything to be able to hand over... I'm not sure I take them at their word. That may just be me though, I'm a distrusting bastard.
Because of its federated nature, Matrix shares tremendously large amounts of metadata between instances, even though the messages (and I believe the usernames you send to, but someone correct me on this) themselves are encrypted.
On the flip side, Signal requires you to sign up using your phone number. Some Matrix instances only require you to choose a username and be done with it.