Those are valid points and make some practical sense, but I've talked too much with mathematicians about this so let me give you another point of view.

First of all, we do modular arithmetic with integers, not natural numbers, same with all those objects you listed.

On the first point, we are not talking about 0 as a digit but as a number. The main argument against 0 being in N is more a philosophical one. What are we looking at when we study N? What is this set? "The integers starting from 0" seems a bit of a weird definition. Historically, the natural numbers always were the counting numbers, and that doesn't include 0 because you can't have 0 apples, so when we talk about N we're talking about the counting numbers. That's just the consensus where I'm from, if it's more practical to include 0 in whatever you are doing, you use N~0~. Also the axiomatization of N is more natural that way IMO.