[How To] Pre-encode to avoid Transcoding (using VidCoder Presets)

General Discussions Tips, Tricks & How-Tos
1

Hi all,

Downmix your Surround Sound to Stereo before using this “Pre-encoding” guide

I have decided to make available my x264 presets for VidCoder that have taken me so long to come up with, but which will 99.9% do away with all your transcoding issues :slight_smile:

There are many Handbrake/VidCoder presets and settings sprinkled throughout the Plex forums and the internet for that matter.
What makes these presets different?
Why would I not just use the normal encoder “Preset” and “Tune” options for x264?

Although these presets ‘mimic’ the Fast, Medium, Slow, Slower & Very Slow and Film & Animation settings of x264, they have been customized specifically for two very distinct reasons:

These presets result in an MP4 file that has a variable bitrate that is capped to stay within the bounds of Plex’s Quality settings, 3Mbps, 4Mbps, 8Mbps, 10Mbps, 12Mbps, 15Mbps & 20Mbps

These presets result in an MP4 file that is extremely device compatible

Both of the above reasons result in an MP4 file that can Direct Play - no more transcoding!

From my extensive testing, these presets easily beat Plex’s own “Optimized Versions” in both respects.

The idea is that you take your original Blu-ray file, MKV file, MP4 file, etc and run it through VidCoder to give you the most “bandwidth ready” and “device compatible” MP4 file.
Sit this file next to your original and let Plex choose it when necessary, or use it as an archive file and delete the original - up to you.

These presets are designed to run on 8-bit 720p/1080p/2160p & 23.976/24/25/29.97/30 framerate source files.

In order of highest to lowest quality / slowest to fastest encoding speed:

  • x264 Custom - Archive quality / Takes forever (use highest bitrate needed to archive)
  • x264 Slower - Extra high quality / Extra slow speed
  • x264 Slow - High quality / Slow speed
  • x264 Medium - Medium quality / Medium speed
  • x264 Fast - Low quality / Fast speed

Why VidCoder and not Handbrake?
In my opinion, VidCoder handles audio passthrough far better, has a nicer user interface, and makes subtitles and chapters easier to handle.

Why H.264 and not H.265?
Our aim is to prevent transcoding, so device compatibility is a high priority, hence H.264.

Why AAC stereo at 160Kbps?
Again, our aim is to prevent transcoding, so device compatibility is a high priority. AAC stereo is the most compatible, and 160Kbps is the highest bitrate with the highest device compatibility.

VidCoder is a front-end for Handbrake, both popular pieces of encoding software.
(If you’re not sure what VidCoder or Handbrake are, this How-To may be a little over your head).

But… Before starting, determine your “Bitrate Requirement”.
Do I have low bandwidth wifi in my local network?
Do I have ‘remote users’ now?
Might I have ‘remote users’ in the future?
Might I be my own ‘remote user’ in the future (on holiday)?
What internet download speed (remote user) is likely to be the limit? 3Mbps, 8Mbps, 20Mbps?
What internet upload speed (Plex server) is likely to be the limit? 3Mbps, 8Mbps, 20Mbps?
How many concurrent streams do I anticipate? 1, 2, 10?
Can my upload speed handle that many concurrent streams?

The answers to these questions will help you determine which preset you decide to use.

Also, before encoding your entire library, just try one or two test files first and confirm the results.

How To:

  1. Download, install and run VidCoder
  2. Download presets:
  1. Import presets into VidCoder
  2. Choose desired preset:
  • 720p + AAC - 3Mbps, 4Mbps
  • 1080p + AAC - 6Mbps, 8Mbps
  • 1080p + AAC/AC3 - 8Mbps, 10Mbps, 12Mbps, 15Mbps or 20 Mbps
  • Choose either “Cartoon” animated or “Film/CGI” style
  1. Select source file (remember, only normal source files will work with these presets, 8-bit 720p/1080p/2160p & 23.976/24/25/29.97/30 framerates)
  2. Select surround sound as second audio track if required (if “AAC/AC3” preset is selected)
  3. “Burn In” a subtitle track if required (Foreign Audio, etc)
  4. Check “Cropping” (Settings > Sizing) if necessary
  5. Adjust "Constant Quality (Settings > Video Encoding) if necessary
  6. Encode

Remember to set your Plex Server’s “Upload Bandwidth” settings correctly, as well as all of your (and your remote users) device’s “Plex Quality Settings” (both “Home” and “Remote”).
These settings will allow Plex to correctly choose your newly encoded MP4 file.

Also, allow Plex to “Perform extensive media analysis during maintenance” in your Plex Server settings (Scheduled Tasks).
This will allow Plex to perform a “Deep Analysis” of the file and determine its “Required Bandwidth”, which will prevent transcoding due to unnecessary bandwidth assumptions.

A few notes on these presets:

“(x264 Custom)” presets sit somewhere between “Very Slow” and “Placebo”.
These were my original and only settings. I have since added the others :slight_smile:

Personally, I like to set the “Constant Quality” (Settings > Video Encoding) to 10.
This might seem like overkill, but as these presets are “bitrate capped”, I like to squeeze every bit into the file as possible.
Feel free to move this slider to your liking, which will potentially reduce the file size - say somewhere between 16 - 20.

The 3Mbps - 4Mbps 720p presets results in the most compatible MP4 file (Profile: High / Level: 3.1) - able to be played on almost all devices since 2010.
Although this is ‘only’ 720p, assuming the source file is original Blu-ray, these will still look pretty darn good upscaled on a 4K HDR TV, especially the 4Mbps 720p Custom preset, considering the limited bitrate. I don’t think any ‘remote users’ would complain with this quality (YMMV).

The 6Mbps -20Mbps 1080p presets will break a small amount of device compatibility (Profile: High / Level 4.0) , but will allow for higher quality encodes.
The resulting MP4 file requires a faster internet connection if you are remote streaming obviously (Server: Upload / Device: Download) .

I hope this helps someone or two :slight_smile:

Further information on requiredBandwidths:
https://forums.plex.tv/t/calculated-bandwith-vs-required-bandwidth/403057

References:
https://en.wikibooks.org/wiki/MeGUI/x264_Settings
https://trac.ffmpeg.org/wiki/EncodingForStreamingSites
http://www.lighterra.com/papers/videoencodingh264/
http://blog.mediacoderhq.com/h264-profiles-and-levels/
https://developer.apple.com/documentation/http_live_streaming/hls_authoring_specification_for_apple_devices
https://everymac.com/systems/apple/ipad/specs/apple-ipad-original-specs.html

9 Likes
2

Have you tried using remote devices with low download speeds, to see if you still transcode?
I had a similar test with my system and still got transcoding on the remote side. My family and friends still get good quality and speed, but still transcoding. This is the test I did You will never avoid remote transcoding at the end of the day

3

How low is “low download speeds”?

I can confirm that if I have 4Mbps upload speed (server) and my remote user has 4Mbps download speed (client) and I have encoded using one of my 4Mbps presets, and I have the Quality Settings correct at both ends, then it Direct Plays.

I could make a preset for 3Mbps or 2Mbps for you to try, at 720p, or even lower and can almost guarantee it will work (it’s not just the bitrate that will force a transcode, there’s many other factors).

How low are you talking?

4

I been able to see why my media transcodes. It’s because I have downloaded all my media to keep it at low presets and at 1080p with about 8k bitrate. When old movies of 720p or 480p it does direct play. But since with my setup and my family with 4k tvs and what not. It best to play at least at 1080p. Would be kinda of a bummer to have a 4k tv to play 720p or 480p movies lol. and I seen them download as low as 3mbps. Depending on where they are at.

5

If Plex determines that it needs to transcode, it will do this on-the-fly and destroy a lot of quality while doing it.
Yes it will be 1080p, but it is being encoded at ‘Very Fast’ preset settings. So your 1080p 8Mbps will look rubbish, even though it’s still “1080p”.

A Pre-encoded 720p at 3Mbps will look far superior to an on-the-fly transcoded 1080p at 3Mbps any day of the week.

The 4K TV will upscale both, but I guarantee the Pre-encoded 720p will look far better.

6

I have original Blu-ray remuxed files (30-50Mbps 1080p) that sit next to two pre-encoded MP4 files, one at 12Mbps 1080p and the other a 4Mbps 720p.

Plex simply chooses the correct file based on the remote users Quality settings - Direct Plays every time.

Of course you have to take into consideration fluctuations on internet bandwidth, that’s a given.

But I can almost guarantee it will be the fluctuation of the file’s bitrate.
Check Plex > Get Info > View XML > requiredBandwidths.

7

Yea, I’ve had my family do these test with me when they are at home with there wifi and at work on cell phone service. Original playback looks awesome and direct plays on their wifi. But when at work and Plex changes the settings from 1080p to 720 or 480, you can tell a different. Not really in a bad way difference but difference, still watchable and good. What I’m trying now is I upgraded my gpu from Gtx 1080 ti to Rtx 2080 ti. Wanna see if this new nvenc and Tensor cores help in any way since they are really good for streaming on twitch etc…

8

For me, I much prefer to Pre-encode once, rather than transcode on-the-fly every time.

I prefer my presets (which granted, take a long time to encode) over the “Optimized Versions” that Plex presents as a Pre-encoded solution.

I’m simply sharing these :slight_smile:

Feel free to actually try these presets on one of your Blu-ray files, you might be surprised.

10
  • x264 3Mbps 720p and 15Mbps 1080p presets added to original post
  • Intel QSV 3Mbps 720p - 20Mbps 1080p presets added to original post
11
  • Updated all presets to work with VidCoder version 5.13 and above
12
  • x264 “Slow” presets added to original post
  • x264 original presets renamed to “Very Slow”
  • Removed Encoder Settings details from original post, as there are now too many options to detail
13
14
  • x264 “Slower” presets added to original post
  • x264 “Medium” presets added to original post
  • x264 original presets renamed to “Custom”
15
  • Added 6Mbps 1080p presets to each package (use Plex Quality setting of 8Mbps)
16

The 8 Mbit preset uses vbv-bufsize=7400:vbv-maxrate=7400.
Does this actually prevent massive spikes in bitrate?
I usually use nvenc (StaxRip) and encode my videos at
~ 5 Mbit/s with vbv-maxbitrate=7500 and vbv-bufsize=5000
This restricts bitrate peaks to ~ 10 Mbit/s and the results are good, more than adequate for home streaming purposes. (1080p)
When I set vbv-maxbitrate=7500 and vbv-bufsize=7500, bitrate spikes up to 15 Mbit/s…
I read somewhere that the max bitrate should be limited to 1.5x avg bitrate because high bitrate spikes are bad for streaming.
But why is the max bitrate limit of 7,5 Mbit/s not maintained?
When I use bitrate=5000, vbv-maxbitrate=5000 and vbv-bufsize=5000,
bitrate peaks to a max of 7,5 Mbit/s.
Does x264 behave the same?

And isn’t it better to have a gop length of 1-2 sec (seeking)?
e.g. keyint=48:min-keyint=24 (preset uses keyint=240)

Most people will say nvenc produces bad quality.
But I can’t agree with that. And the encoding speed is insane.
Like 400 fps ?

When hardware encoding support is checked in plex server settings, does it use ffmpeg nvenc backend?

17

The 8 Mbit preset uses vbv-bufsize=7400:vbv-maxrate=7400.

All of the posted presets use a vbv-bufsize that is equal to the vbv-maxrate, this means a buffer of 1 second.

Remember, we are targeting two buffers here, the ‘device buffer’ which is the hardware chip that is carrying out the decoding and the ‘network buffer’ which requiredBandwidths is calculated for.

By targeting a buffer of 1 second, we are aiming for the best device compatibility/bandwidth ready MP4, but with enough room to allow the bitrate to move as Constant Quality determines.

Does this actually prevent massive spikes in bitrate?

Yes it does :slight_smile: To a buffer of 1 second at 7400Kbps.

I read somewhere that the max bitrate should be limited to 1.5x avg bitrate because high bitrate spikes are bad for streaming.

Firstly, are you using Constant Quality or Average Bitrate to encode?
Secondly, this is most probably someone’s own rule-of-thumb.
If you have done the calculations for vbv correctly (as per the device/network you are targeting), then the peaks will not matter when it comes to playback (and are in fact desired in order to ‘capture’ quality), as the buffer will not underflow or overflow and therefore your playback will not suffer due to buffering.

But why is the max bitrate limit of 7,5 Mbit/s not maintained?

This thread gives a better explanation of vbv than I could ever give, it’s a good start to understanding it.

From my understanding, the vbv-maxrate is not the “peak bitrate” that is allowed, it’s the max bitrate that can enter the buffer.

Does x264 behave the same?

x264 does have peaks in the bitrate, but whenvbv is defined, it’s constrained to those limits. Again, vbv-maxrate is not the “peak bitrate” limit.

When I was testing my encodes using Intel QSV, I realized that vbv was not compatible with Constant QP, Intelligent Constant Quality or LookAhead rate control methods.
Is this the same with NVENC? Might NVENC be ignoring your vbv settings due to another parameter?

And isn’t it better to have a gop length of 1-2 sec (seeking)?
e.g. keyint=48:min-keyint=24 (preset uses keyint=240)

x264’s (or maybe it’s Handbrake’s?) default for keyint is 250 (10 x the framerate @ 25FPS) and for min-keyint is 25 (1 x the framerate @ 25FPS).

This thread defaults to 10 seconds, but suggests 2 to 5 seconds for HD, this suggests 2 seconds, but this thread suggests 12 seconds and gives a very good explanation as to why - give it a read.

keyint has a very big impact on quality. The lower you make your keyint value, the more you reduce your quality. Yes, you have slightly better seeking, but who cares too much about the performance of the seeking when it’s at the cost of quality? If you do, feel free to keep it to your liking.

I prefer quality over better seeking, especially when we are capping the bitrate, so I’ve stuck with the 10 second / 1 second rule, but assumed most source files will be at 23.976/24 framerate.

Most people will say nvenc produces bad quality.

I can’t say for NVENC (I’ve never used it), but Intel QSV does look ‘bad’ compared to CPU x264. Using any matching preset, with a clean source file, I can easily see the difference between Intel QSV and x264 Medium, and this is with the Intel QSV settings at the highest quality too.
Intel QSV is approximately twice as fast (on my PC) as Medium though, so you are saving time.

When hardware encoding support is checked in plex server settings, does it use ffmpeg nvenc backend?

This I do not know, I’m sorry.

18

And a side note, the reason I chose 7400 for the 8Mbps 720p + AAC preset:

8Mbps = 8000Kbps
8000 - 1 = 7999 (the limit to which Plex will not transcode when set at 8Mbps Quality Setting)
7999 / 1.05 (Plex’s own 5% ‘safety margin’) = 7618
7618 - 161 (AAC aduio track) = 7457
7457 rounded down to nearest 100 = 7400 (allows for header and chapter markers, etc)

19

@PlexyChip
I use average bitrate for my encodings.

Actually, I’m not sure about this one.
Let’ assume, two network connections.
The first has a limitation of 10 Mbit/s and the other has a limitation of 20 Mbit/s.
When a bitrate preset of 8 Mbit with a VBV buffer setting of 1 sec is chosen,
And the 1sec vbv buffer allows the bitrate to peak to 15 Mbit/s,
Can the 10 Mbit/s connection keep up with that?

https://learn.akamai.com/en-us/webhelp/media-services-on-demand/media-services-on-demand-encoder-best-practices/GUID-FE23108A-7691-439E-B019-FAA006410645.html

20

But vbv doesn’t allow the bitrate to peak to 15Mbps for 1 entire second (as in Mega-bits-per-second), it is constrained to 8Mbps.
The peak is only a frame or several frames, but not all frames for an entire second or longer. The troughs then allow the buffer (both device and network) to recover from the peaks - averaging.

So vbv constrains the 1 second to 8Mbps average, with frames allowed to go above (15Mbps) an below (4Mbps).

Yes :slight_smile: Thanks to the Network Buffer.
7400/7400 keeps my encodes within the 8Mbps threshold, even with peaks and troughs in each frame.

Maybe someone with a better understanding can chime in here, as I am only new to this stuff and may not fully understand it.

21

Yes, as you said it may allow the bitrate to peak to 15 Mbit/s (or even higher) for a short period of time.
This high peak in bitrate needs a longer time on to transmit on the 10Mbit/s connection vs on the 20Mbit/s connection, isn’t it?