I thought PeerTube’s problem was largely federation (need to know which servers to use), which results in making it hard to find content to watch and probably has something to do with how load balancing works (i.e. are you mostly streaming from your instance?). I think Lemmy has a similar problem, but it’s at least pretty fast because text and images are a lot easier to manage than video.