There definitely are rules to language, which are determined by how the language is actually used. The issue with prescriptivists is that they invent their own rules which often go against how the language is used, i.e. the rules are nonsense.

Take the "less vs fewer" argument. Everyone happily uses 'less' in pretty much the same way for nearly a millennium, then some prescriptivist asswipe comes along and says they don't like it so now there's a rule. Prescriptivists spend the next couple of centuries yelling about their new rule and creaming themselves over how they're now 'better' at the language than other people while everyone else just doesn't give a fuck and continue to speak normally.

In the end language is just a tool to communicate ideas. If you clearly understood someone but whine because they ignored your made up rules you're the asshole.