[Bug?] Transcoder quality overriding TranscodeH264Preset and TranscodeH264BackgroundPreset

[MrFrizzy]

Server Version#: 1.32.5.7349

In my preferences.xml file, I have TranscoderQuality="3" TranscoderH264Preset="medium" TranscoderH264BackgroundPreset="veryslow". I can see in the debug logs that those presets are getting passed to the transcoder when a file is either on-the-fly transcoded or part of the Media Optimizer. However, the -x264opts from the Transcoder quality setting are still getting passed to the transcoder effectively overriding some options of the selected preset.

For example, when doing a “Media Optimizer” job, I see the below in the debug logs:

Aug 08, 2023 12:31:31.853 [23009161046840] DEBUG - [...] "/usr/lib/plexmediaserver/Plex Transcoder" -codec:0 h264 -codec:1 eac3_eae -eae_prefix:1 ff559808-0a6e-4559-8872-10079be1b07a_ -analyzeduration 20000000 -probesize 20000000 -i /data/TestLib/Movies/test_movie.mkv -filter_complex "[0:0]scale=w=1280:h=536:force_divisible_by=4[0];[0]format=pix_fmts=yuv420p|nv12[1]" -map "[1]" -codec:0 libx264 -crf:0 18 -maxrate:0 4000k -bufsize:0 8000k -r:0 23.975999999999999 -preset:0 veryslow -level:0 4.0 -x264opts:0 subme=2:me_range=4:rc_lookahead=10:me=hex:8x8dct=1 -filter_complex "[0:1] aresample=async=1:ochl='stereo':rematrix_maxval=0.000000dB:osr=48000[2]" -map "[2]" -metadata:s:1 language=eng -codec:1 aac -b:1 216k -f mp4 -map_metadata -1 -map_chapters -1 -movflags [...]

As you can see, -preset:0 veryslow gets passed in, but then -x264opts:0 subme=2:me_range=4:rc_lookahead=10:me=hex are overriding the veryslow preset options of subme=10:me_range=24:rc_lookahead=60:me=umh.

Same goes for any on-the-fly transcode:

Aug 09, 2023 11:47:57.114 [23403113483064] DEBUG - [...] "/usr/lib/plexmediaserver/Plex Transcoder" -codec:0 h264 -codec:1 aac -analyzeduration 20000000 -probesize 20000000 -i /data/TestLib/Movies/test_movie.mkv -filter_complex "[0:0]scale=w=1280:h=536:force_divisible_by=4[0];[0]format=pix_fmts=yuv420p|nv12[1]" -map "[1]" -codec:0 libx264 -crf:0 22 -maxrate:0 3368k -bufsize:0 6736k -r:0 23.975999999999999 -preset:0 medium -x264opts:0 subme=2:me_range=4:rc_lookahead=10:me=hex:8x8dct=1 -force_key_frames:0 "expr:gte(t,n_forced*5)" -filter_complex "[0:1] aresample=async=1:ochl='5.1':rematrix_maxval=0.000000dB:osr=48000[2]" -map "[2]" -metadata:s:1 language=eng -codec:1 libopus -b:1 420k -segment_format matroska -f ssegment -individual_header_trailer 0 -flags +global_header -segment_header_filename header -segment_time 5 -segment_start_number 0 -segment_copyts 1 -segment_time_delta 0.0625 -segment_list [...]

-preset:0 medium gets passed in, but then -x264opts:0 subme=2:me_range=4:rc_lookahead=10:me=hex are overriding the medium preset options of subme=7:me_range=16:rc_lookahead=40.

My current workaround is to add TranscoderH264OptionsOverride="subme=7:me_range=16:rc_lookahead=40 to the preferences.xml in order to force the transcoder to use my -x264opts instead of what is passed in by the TranscoderQuality setting. The problem is, the Media Optimizer is still using the TranscoderH264OptionsOverride setting meaning the preset="veryslow" is still getting overridden by some lesser options.

I would request two things:

1. Add a 4th Transcode quality level to either no longer pass in -x264opts to the transcoder (use only those from the hidden TranscoderH264OptionsOverride) or to pass in a custom list of options (perhaps a new GUI link to the hidden TranscoderH264OptionsOverride).
2. Have the Media Optimizer completely ignore the TranscoderQuality setting (only follow the selected background preset) or add a new option somewhere such as TranscoderH264BackgroundOptionsOverride to specify specific -x264opts flags for background transcodes.

I can provide logs in a DM if any Devs or Mods need them. I don’t wish to post them publicly.

[Volts]

Are you chasing a specific quality problem? If you can demonstrate better settings, you might have more luck encouraging Plex to adjust the default settings.

If you search the forum for TranscoderH264OptionsOverride you’ll find some ancient comments. There are some additional reasons the behavior may be different, too.

There have been a few interesting hacks where people swap out the Plex Transcoder binary with a script, and use that script to match-and-edit the arguments. A couple links here -

Script to regenerate video previews multi threaded? - #9 by Volts

[MrFrizzy]

Are you chasing a specific quality problem?

Yes and no.

I find that the “Make my CPU hurt” option provides pretty terrible quality for on-the-fly transcodes. I had setup the TranscoderH264Preset="medium" option months ago in attempt to significantly increase the quality of those transcodes. While the quality is better, it is still not good. My other attempt at improving the quality of transcodes was to use the Media Optimizer to use the “veryslow” preset to get essentially the best quality I can out of the 720p 4Mbps profile. It was when I noticed that the process was hitting 2-6x speeds that I knew that something was not right. Upon checking the output in the “Plex Versions” folder, I saw that the encoder settings used were not what I would normally see in a “veryslow” preset encode. More specifically, the -x264opts:0 subme=2:me_range=4:rc_lookahead=10:me=hex options being closer to the “veryfast” preset.

So, that is to say that the problem is Plex not allowing me to select or otherwise configure the transcodes to be at a much higher quality. I have plenty of CPU to handle multiple “medium” preset encodes simultaneously and I want Plex to take advantage of that. If the on-the-fly transcoder would just follow the “medium” preset, I would not even bother with the Media Optimizer. On the other hand, if the Media Optimizer followed the TranscodeH264BackgroundPreset completely without the “Make my CPU hurt” (or any other TranscoderQuality setting) overriding some of the preset options with much lower quality ones, I would at least have another option to actually optimize my media.

At this time, the Media Optimizer using the “veryslow” preset offers very little quality improvement over the on-the-fly transcodes because both of them are using the same values for the subme, me_range, rc_lookahead and me options that are simply too low for a quality output. It feels like I am stuck with settings appropriate for a processor from 2012.

Thanks for the link. I’ll have to dig through that and see what I can gather from it.