What it ultimately comes down to is that truly secure systems cannot be based on trust. The article does a good job outlining all the ways the users have to trust Whisper Systems without any ability to do independent external verification.

Even if we assumed that Signal works as advertised the fact that it's tied to your phone number is incredibly dangerous. Obviously if this information was shared with the government it will disclose your identity as the article notes. This information can then be trivially correlated with all the other information the government has on you and your social network. Given that Signal is advertised as a tool for activists, that means it creates a way to do mass tracking of activists.

Being centralized is another huge problem given that the service could simply be shut down at any time on government order. If you're at a protest and rely on Signal it could just stop working.

edit: as people have pointed out, it turns out you can use third party clients

~~Finally, since the client is a binary distributed by Whisper, it's not possible to verify that the client and server use the published protocol independently. Since alternative clients aren't allowed to connect to the server, we can't test the protocol and have to rely on trust.~~