First, what do you mean by difficulty? Like, you can't select it or it won't play back at that quality once selected? Something else?
Videos
Neat vids from youtube or wherever. Rules later
Loads much slower
Difficult to diagnose since the issue could be on either side of the wall. I'd guess YouTube's CDN's and/or differences in requested formats (WebM/AV1 vs. mp4/h264).
It could be YouTube using a global Content Delivery Network (CDN), which are servers all over the world caching (having a copy) of the videos. They've got limited space, and thus only room for the most popular videos. Every video is available in various qualities (360p, 480p, etc.), and I can imagine a scenario where the 360p video is stored "centrally" while the 720p/1080p versions are spread around the difference caches for easy access.
I've experienced up to a minute of load-time on some of my older, obscure, never-watched decade old uploads.