It's not a group you sign up for. If you're not a racist, then yes, your most natural stance is being against fascism: antifa.
I don't understand how being anti-racist can possibly be controversial in any way... and if it is for you, then I'm not sure what that says about you.
Here is the actual difference: