Ok, Tried the transcoder quality setting set to fast and it seemed to make a difference for my 480p content. But my 720p content is being transcoded and looks like crap (blurry). Again, viewing from xbox built-in DNLA client it looks great.
I also tried the “disable video stream transcoding” setting and it didn’t play at all (threw an error). It’s like the Plex client for xbox refuses to actually use Direct Play or Direct Stream.
Note that on the xbox client, there are settings for whether to use direct play or direct stream, and you can also force this per-video by going to the info pane about the movie you’re playing.
For now I’m giving up and just using DNLA. 
