Generate Thumbnails in parallel - Inefficient implementation does not scale to 200 TB MKVs

Server Version#: Plex Media Server 1.19.4.2935
Player Version#:

To speed up thumbnail generation it would be optimal to run several thumbnail generations at once since there is headroom for CPU (32 core), 2x10Gib NIC and 2x rtx 2060 KO with unlocked nVidia drivers supporting nvenc and nvdec.

With the current speed it will take 4 plus months to generate thumbsnails.
I can see that only one process is running:

“C:\Program Files (x86)\Plex\Plex Media Server\Plex Transcoder.exe” -codec:v hevc -skip_frame:v nokey -i “\BARNEY\VIDEO_20\MKV\UHD\The Amazing Spider-Man (2012)\The Amazing Spider-Man (2012)-UHD.mkv” -threads 0 -nostats -q 3 -loglevel quiet -filter_complex “[0:V:0] fps=fps=0.500000:round=up,scale=w=320:h=240:force_original_aspect_ratio=decrease [out]” -map [out] “C:\Users\Admin\AppData\Local\Plex Media Server\Media\localhost\0\a8b278c1407c5c359f1583ec8887a931941c1e9.bundle\Contents\Indexes\tmp\img-%06d.jpg” -progressurl http://127.0.0.1:32400/video/:/transcode/session/bif/1cc6f000-7626-4630-aa0b-e45aa492993c/progress

1 Like

Hi @rithram,
wait - you are talking about 200TB of data?! Why on earth would anyone need so many movies??
Well, apart from that, you have (I believe) two options:

  1. Just disable video thumbnail generation. Go to your library, three dots, manage, edit, advanced, disable video thumbnail generation.
  2. Create / download your own thumbnails and calling them like the original movie but replace .mkv with .jpg or .png or anything you like really (“Movie_ABC.mkv” <-> “Movie_ABC.jpg” automatically recognizes it as thumbnail) and throw them in the same folder.
    2.5. If you have very few but huge files (e.g. surveillance) you can manually select a thumbnail by going to each video and pressing the edit icon, though I don’t know if that automatically opens up another thread or if this just prioritizes this thumbnail (cancels previous one and cares about the one you’re currently editing).
    Hope this helps

Hi @Super4Jet,

Thank you very much for making me aware of the possibility of generating custom thumbnails. I will try this out and share my findings.

If it works I will automate and optimize the thumbnail generation and hopefully my 10 GiB NIC will be the next bottelneck :wink: (And not a poor design)

The method described above, applies to a different sort of thumbnail. It has nothing to do with video preview thumbnails.

1 Like

@OttoKerner

Yes, I just discovered that :frowning:
Video Thumbnails are in the BIF format.

Planning to reverse the Plex Media Server database and add BIF indexes myself.

thumbnail generation is intended to be a background task, not on demand.

you can adjust the schedule hours so that the background tasks can run longer (or all the time).

Plex Web > settings > scheduled tasks > start/stop times

or, you can just be patience and wait like everyone else :slight_smile:

Video preview thumbnail generation basically plays the entire movie, extracting and saving a few still frames to .jpg. Those get bundled into an index-sd.bif file.

On my machine with a slow CPU, video preview thumbnail generation is completely limited by disk speed. If you are able to pull 1GB/s, you should be able to make previews for 200TB in … what, a few days?

How long does it take to create video preview thumbnails for each movie? Is your network as utilized as you would expect? If not, maybe there’s something else going on.

Video preview thumbnail generation can be throttled for a number of reasons. If TV episode intro detection is active, it will be throttled. Depending on other settings, active optimization jobs or active transcoding sessions can throttle it too. Plex generally (but not always) logs when transcoder processes are being throttled. Is your system doing other things, too? Are other transcoder processes present?

@OttoKerner
Yeah, I wasn’t sure, but I assumed he meant the preview-of-the-movie-in-collections-view-thumbnail, not the video-loading-before-first-frame-thumbnail, as this was the only variant I knew a solution to. But do you know if it is possible to set the preview thumbnail = in-video thumbnail? That would just make the whole movie selecting process more uniform and easier to understand.

I don’t understand your question about preview thumbnail = in-video thumbnail. Can you share more to explain what you mean?

When you’re looking at a list of movies, or a collection, do you mean what Plex calls a Poster?

The image-before-playback-starts … do you mean what Plex calls a Background?

Not disk limited, at least not for REMUX 1080p/2160p content, I have a Xeon E3-1245v6 and a Intel SSD DC P3700 (NVME SSD, capable of 2800 MB/s reads). Using iotop while the thumbnail process is running shows it running between 50-80 MB/s. Total CPU usage equivalent to 1 core in use for the process.

A setting, even hidden in the Preferences.xml that lets you select the number of active video thumbnail generators at once would be much appreciated.

1 Like

Yes
There is a Background image, shown when your movie is loading…
image
… and there is your “real” thumbnail, shown in your list of movies…
image
… and this is literally like the only difference. I presume this is because you want to see the Blu-Ray cover as thumbnail, which is in portrait, but most TVs and movies are in landscape orientation, and portrait doesn’t fit into landscape quite as well as a native landscape image (BTW portrait video is the worst thing you can ever do). AFAIK it’s a lot easier to mass import the “real” thumbnails because you can just throw in the pictures (mentioned above), while I don’t really care about the background image too much because I rarely see it.

I think you’re talking about something different from what rithram started this post about.

He’s talking about video preview thumbnails, which are shown by the player when you pause and fast-forward media. They give you a preview of the scene that you’re seeking to.

There’s a job in Plex that scans the entire video and extracts these. It doesn’t really make sense to provide a custom version of this - you want it to match the specific video file.

Oh sorry, my fault, I forgot about the third kind of preview :sweat_smile:
The seek bar is made of a lot more pictures and it absolutely makes sense that it takes up so much performance.

Yo dawg, I heard you like previews, so I put a preview of your preview in the preview, so you can …

1 Like

@rithram,

You might check out these other threads.

A significant fix for those issues was included in 1.19.4.2935. 1.19.5.3006 even seemed to include more changes, but it was pulled back for other problems. Maybe the next version will help you; maybe it’s worth waiting for that.

You said this was the only process running - there aren’t any other Transcoder processes at all - playback, media optimizer, intro detection?

You could test with media you have stored locally, to rule out any network/NAS interaction. Not my first guess, but an easy thing to rule out.

Your system is fast where my system is slow. I noticed that running the Transcoder to extract these images, the same way Plex does, but excluding the “-progressurl” argument, was MUCH faster for me.

You could try something similar.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.