For me, it's not stuttering but rather YouTube streams perform better given the same quality setting. For example, my connection sometimes fails to smoothly serve 1080p60 on Twitch yet handles YouTube's 1080p60 pretty well.
I wonder if that's because YouTube uses adaptive streaming, so it may continue to show you "1080p" content, but with slightly fewer bits of data to accommodate the decline in internet speed.