This is a long post with a lot of detail so read carefully. Many have commented on the stutter and from what I’ve seen there are two types:
- A stutter which occurs occasionally for a short period of time with long random intervals in between
- A stutter that occurs continually for a playback but a pause/resume tends to resolve.
Both of these are particularly noticeable when refresh-rate matching is enabled. To explain what is going on, I first have to explain some architecture. This is very similar to PMP so anyone who has studied that can very well guess everything I’m going to describe here.
The Current Render Process:
MPV is currently using the libmpv renderer which renders into an OpenGL frame buffer object. This allows it to be rendered within a scene with other elements. This is how controls are easily rendered on top of the video or the video is rendered on top of the controls for instance (both modes are currently used in different circumstances). This process goes through elements starting with the back-most element and rendering them till it has finished all elements and that constitutes a frame which is displayed on the screen. Since Qt enables us to easily integrate Chromium (the basis of the Chrome browser) into this view, we used its rendering architecture for the entire render process.
The Cause of The Stutter:
Qt has two modes of rendering where one is threaded and the others are not. If you read through Qt Quick Scene Graph | Qt Quick 5.15.6 you can see that MacOS lost the ability to use the threaded render pipeline when apps were built against the 10.14 or higher SDK (which we need for other things). Windows cannot use the threaded render pipeline when ANGLE is in use because the library isn’t thread-safe. I believe that in the old PMP on Windows we were able to use the threaded render pipeline but I’m not 100% certain of that.
So we have a render pipeline that uses the main thread for the process effectively blocking it for that time. This means that events which run on the main thread (which is going to be just about everything) have to wait for the render pipeline to finish before they get processed. These queued events get processed between frames which can change the timings of the start of the next frame render. This results in jitter in the timings of when MPV’s render function is called. In certain circumstances this can cause the timing of this function call to be on the edge of a frame time and thus randomly fall on one side or the other of a frame’s time (type (2) above). Additionally one these event that gets stuffed into this queue waiting for the main thread is the request by MPV to render the next frame. If this event isn’t processed in time for the next render run start, the video frame just won’t get rendered in the next frame (type (1) above).
In the threaded render pipeline, the main thread is only blocked for a short period of time and so events that run there don’t build up as much. This means that the aforementioned event for MPV requesting a render is much more reliably processed before the render pipeline starts the next frame.
What’s To Be Done?
We’ve attempted to introduce our own threading into the pipeline using two frame buffers which are rendered in a separate thread for just MPV. This involved creating a new GL context which is assigned to the thread and allowed to share frame buffers with the main context. It worked well on MacOS. Windows was … a different story. There it produced a stochastic flash pattern between the correct frame and black which it proceeded to do for a few seconds before it crashed in ANGLE. Apparently even this very limited threading was beyond what ANGLE could handle.
Recently Qt 6.2 was released which is the first of the Qt 6 line that could potentially be used because it’s the first that contains Chromium. Though it’s worth noting that switching to it is not just a simple drop-in replacement so don’t expect that soon. It does away with ANGLE completely on Windows and ceases using OpenGL on Mac. Instead it uses D3D and Metal respectively. Since libmpv still requires rendering into a GL context, we would have to layer our own ANGLE on top of D3D. In this case ANGLE would only ever be on a single thread so hopefully it will work fine. On MacOS we can make a texture that’s compatible with both Metal and OpenGL so we can have libmpv render to it using OpenGL and put that in a Metal command buffer to render.
Linux?
I’ve not mentioned Linux above because it’s the platform where things don’t change. It can still use the threaded render pipeline and in Qt 6.2 it still uses OpenGL. So there’s just not much to discuss there and that’s a good thing.
Can We Eliminate Qt’s Renderer For Video?
Another avenue which we are examining is the possibility of pulling MPV out of Qt’s render pipeline. I mentioned earlier the contention of the main thread and even with the threaded renderer that problem won’t be fully eliminated. The fact is that way too much of Qt’s design is reliant on running on the main thread, likely due to the fact that Qt does make threading harder than it should be. We previously mentioned this in the context of saving settings taking so long on the main thread and I recently discovered another example of this which will be fixed in the next version. So we are investigating the possibility of separating Qt and MPV’s render process entirely from each other. Considering that every example of stutter I’ve seen has been entirely the fault of something in Qt, this has potential to completely and permanently eliminating the problem. While there is hope that this will work, this is a large architectural shift which could bring some nasty surprises so keep your fingers crossed.
