Preview Thumbnails Analyzer - Multithread?

Plex Media Server
1

Server Version#: 1.23.2.4656
Player Version#: Online

I have my Plex Server running on my desktop PC with a Ryzen 5900x, 32 GB RAM and an Nvidia GTX 1080 TI. The files are stored on three 8 TB SATA HDDs combined with mergerfs. The OS is Manjaro on the current rolling release.
Currently I have the Plex Server analyzing the previews of my 4k Video Collection and it’s taking weeks. Which would not be a problem if I did not have the feeling that it isn’t using my computing ressource to their full potential. When taking a look at top, the plex transcoder is only using up to 200% CPU usage (meaning two threads on a 24 thread cpu)

~ >>> top                                                                                                                                                                                   

top - 12:00:43 up 12:41,  1 user,  load average: 3,24, 2,83, 2,58
Tasks: 529 total,   1 running, 528 sleeping,   0 stopped,   0 zombie
%CPU(s):  2,0 us,  1,8 sy,  6,2 ni, 89,0 id,  0,8 wa,  0,1 hi,  0,1 si,  0,0 st
MiB Spch:  32117,2 total,    261,3 free,   7738,7 used,  24117,3 buff/cache
MiB Swap:  35332,3 total,  33023,3 free,   2309,0 used.  23752,2 avail Spch

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     ZEIT+ BEFEHL                                                                                                                
 192704 plex      35  15 1086492 987,4m   9600 D 151,0   3,1   0:56.52 Plex Transcoder                                                                                                       
   3222 jan       20   0 4723012 761924 209416 S  77,8   2,3  20:01.38 firefox                                                                                                               
    826 root      10 -10 1806536  23532   1016 S   3,6   0,1  63:06.30 mergerfs                                                                                                              
   1416 root     -51   0       0      0      0 S   3,3   0,0   4:44.32 irq/129-nvidia                                                                                                        
   1418 root      20   0       0      0      0 S   2,3   0,0   5:16.50 nv_queue

and the gpu not at all:

~ >>> nvidia-smi                                                                                                                                                                            
Wed Jun 16 12:01:20 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 465.31       Driver Version: 465.31       CUDA Version: 11.3     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:09:00.0  On |                  N/A |
|  0%   32C    P8    21W / 300W |    890MiB / 11175MiB |      9%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      2327      G   /usr/lib/Xorg                     440MiB |
|    0   N/A  N/A      2395      G   /usr/bin/gnome-shell               92MiB |
|    0   N/A  N/A      2792      G   ...images/nextcloud.AppImage        2MiB |
|    0   N/A  N/A      3090      G   ...AAAAAAAAA= --shared-files       17MiB |
|    0   N/A  N/A      3222      G   /usr/lib/firefox/firefox          284MiB |
|    0   N/A  N/A      3308      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A      6074      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A    147680      G   /opt/zoom/zoom                     37MiB |
|    0   N/A  N/A    156202      G   gnome-control-center                2MiB |
+-----------------------------------------------------------------------------+

Taking a look at iotop I can see a total disk read around 60 M/s, which is far slower than the capabilities of the disk. (When copying one of the files which is to bee analyzed it tops out at around 250-260 M/s)

Is there any way to make this faster? To make the transcoder use the GPU (which it does use when transcoding for a client) or to have it use more threads? It is currently analyzing an HEVC Codec File.

2

There’s no practical way to accelerate this today, and the current process is sequential, CPU-bound, and limited to a couple of threads.

You can speed it up - slightly - by making fewer thumbnails:

Big Media folder? Make smaller video preview thumbnails!

And usually only video codec keyframes are extracted; I’ve seen reports that changing this can make it faster. That’s not intuitive to me, but you could test it. Maybe big CPUs like yours can muscle through it!

See this for instructions, and the next message about it being faster for another user:

Video Preview Thumbnail Generation Fails for Some Items - #48 by Volts

If you felt absolutely crazy, it would be possible to generate the VPT thumbnail files using external scripts. :smiley:

1 Like
3

I just tried changing it to not only extract video codec keyframes, but that did make it slower. No surprise there, I download my files to spinning rust before analyzing them so I could not replicate Hitsvilles result.

When generating the Thumbnails via an external tool, where/how should I store them and embed them in the plex database? And do you have a suggestion for what to use? Or are you talking about the plex command line tools and scripting them?

4

That’s no surprise, but thank you for confirming it.

I meant that as a flippant suggestion … it’s definitely possible, but is it worth it? :slight_smile: :smiley:

So YMMV, don’t try this at home, blah blah. :slight_smile:

I think you’d completely reproduce what Plex is doing - using ffmpeg to extract the thumbnails, and then concatenate them together to produce a BIF file. You’d just do multiple jobs in parallel.

There’s some more information on the BIF files here. Roku

They’re stored in the Plex data dir, in Media/localhost/h/ash.bundle/Contents/Indexes/index-sd.bif.

You can get that hash value from the database, in the media_parts table.

I haven’t ever tested this tool and I don’t know if it’s 1) any good or 2) produces equivalent BIF files.

https://burningbushsoft.com/index-1.html

There was also this (ancient) script over in the Emby universe. It includes parallelization.

Recursive Batch Create Roku BIF Files - Tools and Utilities - Emby Community

But obviously doesn’t include checking if the file exists where Plex wants it, or putting files in the appropriate locations.

You could just … you know … wait a while. :slight_smile:

5

This is, unfortunately, true but it is not impossible to change. Due to the fact that plex uses FFmpeg under the hood even creating those images can be accelerated with a gpu (ffmpeg command “-hwaccel cuda” or “-hwaccel d3d11va” which the devs would need to implement. But it wouldn’t be that much faster. I have tried it with plain ffmpeg on 1080p content on windows and it was even slower, but with only 1/3 percent cpu usage)

1 Like
6

Yes agreed. For simple decoding of 1080p - especially if skip_frame nokey is used - hwaccel doesn’t seem helpful.

7

I´ve just tried it with a 4k file and to my surprise, it´s exactly the same as with the 1080p file. Using hwaccel is not any faster - again even slower. The only exception was-skip_frame nokey where I got a slight improvement but the gpu decoding engine was only used by about 3% and the cpu usage was identical to the test withoout hwaccel.

1 Like
8

I think the work of scaling and compressing the preview frames for the .bif dwarfs the work to decode the video stream anyway.

9

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.