This seems like a great system.
I really hate all the reddit awards. I didn't even know they exist until I opened the steve huffman ama in new reddit, and it had about a million awards that were all a different (moving/sparkling) emoji. Facebook has those too, all the little icons for like, haha, sad, heart etc. I find that stuff really distracting to look at and it's one of many reasons i refuse to use facebook, or reddit's official website & app.
Something like you're describing sounds like it would work really well though, especially if there were just maybe different colours or something for the upvote/downvote type, instead of space-wasting icons and images.