It's already being done as much as possible. Oli London did accuse a trans women for being a pedophile for liking "lolita anime", which turned out to be just Gundam Seed or something along that line.

Other issues:

It's illegal in a lot of countries, and the places where it's legal it's still being called "virtual CP". On some level, I can understand this sentiment, it can risk the legality of your instance too, but they're still essentially just drawings for the most part, unless CSEM was used for reference, features a drawing of a real child, can't be distinguished from photography easily, etc.
Pedojacketing happens with a lot of kinks, even in supposedly "kink-positive" circles. I got called a pedo for liking navels, hairless genitalia, etc., so people will try to connect kinks to pedophilia, especially those pesky puriteens, who also think the age of consent should be 25.

I personally don't like such material (although I don't strongly age-check every character I see online), but I'm also against any legislation targeting such things, especially within the current anti-LGBT climate. This would easily lead to LGBT relationships also being labeled as "virtual CSEM", while obviously giving the straight stuff the green light.