I think this especially stems from how those games are seen by some more asshole-ish people. Few would disagree that strategy RPGs are not gaming, but I've had a clerk at gamestop lecture me about how my then-favorite title wasn't a "real" part of the series. So it's still my favorite, but I don't bring it up anymore.

For much the same reason, there's an internalized mental block concerning whether or not something like The Sims, Stardew, Animal Crossing, etc. are "real" games because they don't really have a goal or require the same level of skill. If someone asks me what I play, I'm going to mention Hades before I ever breathe a word about Slime Rancher because one of those is going to get me stereotyped and insulted.

If all I played were really chill, micromanaging "girl games," I just wouldn't say anything and would have doubts about describing myself as being on the level "gamer" normally entails.