How I fixed choppy playback in Plex HTPC for Windows (Nvidia GPU)

It is likely the Vertical sync which is the key setting among those you customized. Is this actually the case for you?

In my case I had to change the Power management mode to Prefer Maximum Performance or it would occasionally drop audio whether passthrough was used or not (HTPC Tips and Tricks - #5 by gbooker02). Usually it occurred around black or nearly all black frames. I have a video file where it occurred 3-5 times in the first minute each around an all black screen with some white text to indicate the actors’ names.