Plex Server metadate scrapper slows (curl timeout on local) down and eventually the server crashes

Hi plex users and team,

im using plex since at least 10 years (but stopped upgrading on my old windows 2008 R2 server at version 1.18.x). Now the hardware and also the windows came into its years and i decided to move complete to linux.

So i setup a proxmox server and migrated one service after another. The last one to migrate was plex. So i setup a VM wint 16 GB Ram and 256 GB disk on a SSD thin-lvm.
Installed plex by adding the apt sources and installed the plex server. After that i moved the sql-lite dbs (not the metadata directories because of its huge size and that different agents wrote over a decade data to it) and added the directories to the libs, and did a rescan. Most of the files got there path replaced, the rest i removed by empty trash for that lib.

So now for the metadata.
I setup following agents:
Sub-Zero.bundle
XBMCnfoMoviesImporter.bundle
XBMCnfoTVImporter.bundle

Agent config for the TV shows:

Then i started “Refresh all Metadata” and in the beginning it worked like expected. Three “worker / importer” are scanning in parallel, adding the files.

But after a while i noticed that in the Alert, a show was not marked as “finished processing” in a short burst of messages any more, further a episode showed up and after a minute or so the next showed up as processed. Let it run a while and i noticed the other two “worker / importer” starting behaving the same. To trigger this the scanning runs about 1 - 2 hours. In the logs there is the follwing error if that happens at that snippet all 3 where having the problem, so the “HTTP simulating 408 after curl timeout” appears 3 times and then it stalles for half a minute or so …

The first time i let it run over night and in the moring the whole vm was crashed …
In the second attempt the messages started and stayed and the server was not reachable any more at port 32400 via web browser …

open file handles:

lsof | wc -l
12657

sysctl fs.file-nr
fs.file-nr = 1152       0       9223372036854775807


 ss -ptln
State                             Recv-Q                            Send-Q                                                       Local Address:Port                                                        Peer Address:Port                           Process
LISTEN                            0                                 128                                                              127.0.0.1:33483                                                            0.0.0.0:*                               users:(("Plex Script Hos",pid=9207,fd=4))
LISTEN                            0                                 128                                                              127.0.0.1:39949                                                            0.0.0.0:*                               users:(("Plex Script Hos",pid=9116,fd=4))
LISTEN                            0                                 1024                                                               0.0.0.0:32400                                                            0.0.0.0:*                               users:(("Plex Media Serv",pid=9090,fd=10))
LISTEN                            0                                 128                                                              127.0.0.1:37969                                                            0.0.0.0:*                               users:(("Plex Script Hos",pid=9434,fd=4))
LISTEN                            0                                 128                                                              127.0.0.1:35857                                                            0.0.0.0:*                               users:(("Plex Script Hos",pid=9405,fd=4))
LISTEN                            0                                 1024                                                             127.0.0.1:32401                                                            0.0.0.0:*                               users:(("Plex Media Serv",pid=9090,fd=11))
LISTEN                            0                                 128                                                                0.0.0.0:22                                                               0.0.0.0:*                               users:(("sshd",pid=477,fd=3))
LISTEN                            0                                 128                                                              127.0.0.1:32600                                                            0.0.0.0:*                               users:(("Plex Tuner Serv",pid=9164,fd=10))
LISTEN                            0                                 128                                                                   [::]:22                                                                  [::]:*                               users:(("sshd",pid=477,fd=4))



 ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 71797
max locked memory           (kbytes, -l) 2302334
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1024
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 71797
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited


ss |  wc -l
140

Also if go to an episode which has this behavior not metadata is shown (no pictures).

Server Version#: 1.32.3.7192-7aa441827
Player Version#: -

Jun 16, 2023 11:42:13.567 [139928054500152] DEBUG - [HttpClient/HCl#4f5b] HTTP simulating 408 after curl timeout
Jun 16, 2023 11:42:13.568 [139927971597112] ERROR - [Req#5d557] Error parsing content.
Jun 16, 2023 11:42:13.568 [139927971597112] ERROR - [Req#5d557] Error parsing XML response for update.
Jun 16, 2023 11:42:13.568 [139927971597112] DEBUG - [Req#5d557] We're going to try to auto-select an audio stream for account 1.
Jun 16, 2023 11:42:13.568 [139927971597112] DEBUG - [Req#5d557] Selecting best audio stream for part ID 150370 (language: )
Jun 16, 2023 11:42:13.569 [139927971597112] DEBUG - [Req#5d557] Audio Stream: 246774, Subtitle Stream: -1
Jun 16, 2023 11:42:13.569 [139928088279864] DEBUG - Completed: [127.0.0.1:35854] 200 GET /library/metadata/31820 (20 live) #5d557 GZIP 20008ms 1558 bytes
Jun 16, 2023 11:42:13.686 [139928054500152] DEBUG - [HttpClient/HCl#4f5f] HTTP simulating 408 after curl timeout
Jun 16, 2023 11:42:13.686 [139927957072696] ERROR - [Req#5d58d] Error parsing content.
Jun 16, 2023 11:42:13.686 [139927957072696] ERROR - [Req#5d58d] Error parsing XML response for update.
Jun 16, 2023 11:42:13.687 [139927957072696] DEBUG - [Req#5d58d] We're going to try to auto-select an audio stream for account 1.
Jun 16, 2023 11:42:13.687 [139927957072696] DEBUG - [Req#5d58d] Selecting best audio stream for part ID 166403 (language: )
Jun 16, 2023 11:42:13.687 [139927957072696] DEBUG - [Req#5d58d] Audio Stream: 306531, Subtitle Stream: -1
Jun 16, 2023 11:42:13.688 [139928090389304] DEBUG - Completed: [127.0.0.1:35934] 200 GET /library/metadata/32597 (19 live) #5d58d GZIP 20004ms 1696 bytes
Jun 16, 2023 11:42:13.731 [139928054500152] DEBUG - [HttpClient/HCl#4f62] HTTP simulating 408 after curl timeout
Jun 16, 2023 11:42:13.731 [139927954963256] ERROR - [Req#5d5c3] Error parsing content.
Jun 16, 2023 11:42:13.731 [139927954963256] ERROR - [Req#5d5c3] Error parsing XML response for update.
Jun 16, 2023 11:42:13.732 [139927954963256] DEBUG - [Req#5d5c3] We're going to try to auto-select an audio stream for account 1.
Jun 16, 2023 11:42:13.732 [139927954963256] DEBUG - [Req#5d5c3] Selecting best audio stream for part ID 151163 (language: )
Jun 16, 2023 11:42:13.732 [139927954963256] DEBUG - [Req#5d5c3] Audio Stream: 249088, Subtitle Stream: -1

The linux is an actual debian 11. with 5.10 kernel.

Anybody any idea ?

Im also attaching the log of the currently scanning plex server with the errors.

I hope somebody can help me or direct me to the right direction!

Nice Regards,
Martin
Plex Media Server Logs_2023-06-16_12-12-06.zip (7.6 MB)

PS.: Also the ram is ok (had it run memtextx86+ for 24 hours) and the drives seem also ok (raid6, did extensive copy data to drive and checksum for about 6 hours). All other vms/lcxs also dont have any problem whatsoever.

Update:

I started over with a new database and let it rescan everything, that worked, besides some deep analysis error (Deep analysis not working, transcoder failture (1.32.4.7195)) but no crash or timeouts so far.

So it seems that the database is somehow defect/corrupted and yes i tried to repair it but with the same results.

Nice Regards,
Martin

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.