I don't have great suggestions for improvement, but I can verify that I sometimes have to flip around between players to get a particular video to work properly. It's rare that I have to go all the way to VLC, but your report doesn't flat out shock me. Some ideas:
- Try https://github.com/jarnedemeulemeester/findroid as an alternative. I have both installed and flip back and forth between the official player and findroid. They're both nice. Findroid may have different nuances in codec support though and maybe they'll play in your favor.
- Fiddle with any hw/sw decoding settings you can find.
- Fiddle with the transcoding settings (sounds like you have... but try all the options)
- If necessary, reencode the file on disk on the Jellyfin server... at least as a test to see if you can get ANY codecs to work reliably.
I use both apps all the time, and I'd generally say they work great. But occasionally I get a weirdly encoded file I have to mess around to get to play on mobile. I haven't tracked which codecs cause trouble though.