Server randomly freezes at night but not crashing

Plex Media Server
1

Server Version#: 1.32.0.6918
Player Version#: 4.104.2
Logs here: Plex Media Server Logs_2023-04-14_22-24-11.zip - Google Drive

Hi,
I’m running a Plex Media Server on a Ubuntu Server (headless) with Xeon E5-2698 v4, and a Nvidia Quadro P400 for transcoding. For the last week, I’ve been facing some strange problems at night between 21h - 23h when most of the users use it (around 8-10, max 14). The problem is that the server suddenly seems to be down or extremely slow, I can not even download the Plex Logs when that happens, I use varkeen to graph some statistics and those are empty as well (I believe it does not respond to the calls varkeen is sending), is like the service freezes from outside but the service is on if I check it with systemctl, I also think that the users that are watching something on that moment keep watching it (since when it goes back “online” I see 7 or 8 users using it.

I’ve optimized the database and also performed a repair with the Plex SQLite Tool (it showed that nothing was wrong). So I’m out of ideas…
If you can check the logs, you can see many errors around 21h, which is when the server was unavailable.

Let’s see if someone can help me out.
Many thanks!

2

I have disabled hardware transcoding (to discard a GPU problem) and there were just 8 users (Direct playing), and happened again at 22:40h, server got really unstable/unresponsive and I could not even go to settings to enable transcode again.

I attach here the logs:

I really have no clue what is going on, I’ve been checking the logs but found nothing…

Many thanks

3

I’m adding more information…
Now the server is responsive (I can play and check options) but Tautulli is getting timeouts when trying to reach the endpoint which I believe it’s a symptom of my problem described above.

Thanks

4

I have changed the P400 for a 1660 Super, which is way more capable to check if that was the problem… and guess what it happened again just now from 20:00 to 20:30 with only 4 users, some problems, timeouts when trying to reach Plex API and one of the users suffered constant loading on the film.

All the stats on the server are nominal, CPU and GPU is good and, bandwidth, RAM… all good… it must be something inside the Plex service?

I’m attaching here the logs so if someone can check what was going on at 20:00h

Many thanks!

5

Right now the service is unavailable on the Web Player (any user device) but my service is up and running and PMS is generating logs, but as I said it appears as offline on the viewer.

Adding log:

6

Took a quick peek – only two things jumped out to me …

Apr 18, 2023 23:31:17.295 [0x7f2891231b38] WARN - [Req#acb3b] SLOW QUERY: It took 2020.000000 ms to retrieve 30 items.

Have you ever tried → GitHub - ChuckPa/PlexDBRepair: Database repair utility for Plex Media Server databases. The DBRepair.sh stop automatic status start exit options might help and could be worth a shot.

Second, your TV naming is not what plex likes. Example …

Apr 18, 2023 23:11:10.517 [0x7fd5717a6b40] DEBUG - [FFMPEG] - Opening '/home/pepllu/Plex/Series/Series_HDD13_18/The Ark (2023)/Season 1/[.HDG] The Ark 1x01 Todos querían ir en esta nave.mkv' for reading

The 1x01 for the episode names is not recommended and while it might be working it certainly will cause additional performance on your server as the scanner needs to work alot harder to. See https://support.plex.tv/articles/naming-and-organizing-your-tv-show-files for additional information on how to name as per their recommendations.

1 Like
7

I have been experiencing something similar. For me, it happens between 19h and 22h. I’m using Unraid and have switched containers from Linuxserver’s to the official Plex one and the problem persists. Nothing I do between those hours fixes it. Outside of those hours, it functions normally

8

Many thanks dokuro for your answer I really appreciate it!.

I did not know the DBRepair tool you mentioned, I will certainly try it since database gives some warnings as you pointed out.
About the naming I will take my time and rename all properly.

But unfortunately I do not think those are related to my problems,let me explain, in those last days I have been doing many tests and looking at the logs and I found out that every time a certain user connected (in my case “manum45” if you want to check the logs) with a Samsung (TV UE55JU7500) the server became unstable and not responsive (the symptoms that I have been explaining on the previous posts), so I asked the user to not use the TV to access Plex (luckily he has a PS4…) and it’s been three days with no problems at all… I think there is something wrong in the PMS that causes that behavior, but I’m still testing this out since I may have fount the cause but not the solution…
Hope we can find a better understanding.

9

@Pepllu

I took a look at your logs.

The problem is clear.

You have an E5-2698 v4

The Xeon does not have hardware transcoding.
The CPU will have no choice but to do it in software.

It may have 20 cores but only 23,000 total passmarks
https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E5-2698+v4+%40+2.20GHz&id=2806

Do the math and you have just over 1100 Passmarks / core

I have the same problem with my E5-2690 v4.
I can’t transcode HEVC without a GPU.

Burning subtitles, which is also happening here (the PS4 needs the server to burn them) isn’t going to work well because the single-core speed of these Xeons is very low.

Hate to say it but: - Wrong CPU for the media being used.

A GPU (Nvidia) would help offload some of the load (HEVC decode and H.264 encode) but you’ll still have subtitle burning. It might work with 1080p

You would actually be better off with an i9 or better CPU here which has the clock speed and efficiency to get the job done

-OR-

Converting ASS subtitles to SRT;
Pre-burning in PGS, VOBSUB, and DVDRIP subtitles into the media permanently

10

Many thanks for the answer @ChuckPa I really appreciate your comment, I know that CPU is not the best for transcoding, but I do have an GPU it’s a Nvidia 1660 Super, I have the patch applied so it can transcode more than 3 sessions at a time.
I have been using this setup for almost three years without issues (before I had a 1050Ti), but the such problems have been happening only in the last month.

Yesterday night I had the same problem (so my theory of the Samsung TV is not completely right, or maybe there is another user using problematic device, I can not check the logs since they are only saved for the last hours).

@ChuckPa if that is a transcoding problem why is that there is a timeout when the API is trying to access the PMS service? Does that make sense?

Many thanks for your input, lets see if we can get closer to a solution.

PD: I will do a cronjob to save the logs every couple of hour so I’ll be able to see them even a day after.

11

Still same issue, not it was unreachable so I decided to stop and perform a DBRepair as @anon5074910 suggested.

Here is the output:

pepllu@family:/opt/PlexDBRepair$ sudo ./DBRepair.sh stop automatic status start exit



      Plex Media Server Database Repair Utility (Ubuntu 22.04.2 LTS)
                       Version v1.0.3


[2023-04-22 22.18.38] Stopping PMS.
[2023-04-22 22.18.58] Stopped PMS.

[2023-04-22 22.18.58] Automatic Check,Repair,Index started.
[2023-04-22 22.18.58]
[2023-04-22 22.18.58] Checking the PMS databases
[2023-04-22 22.19.31] Check complete.  PMS main database is OK.
[2023-04-22 22.20.05] Check complete.  PMS blobs database is OK.
[2023-04-22 22.20.05]
[2023-04-22 22.20.05] Exporting current databases using timestamp: 2023-04-22_22.18.58
[2023-04-22 22.20.05] Exporting Main DB
[2023-04-22 22.20.32] Exporting Blobs DB

[2023-04-22 22.28.40] Successfully exported the main and blobs databases.  Proceeding to import into new databases.
[2023-04-22 22.28.40] Importing Main DB.
[2023-04-22 22.30.03] Importing Blobs DB.
[2023-04-22 22.31.24] Successfully imported databases.
[2023-04-22 22.31.24] Verifying databases integrity after importing.
[2023-04-22 22.31.46] Verification complete.  PMS main database is OK.
[2023-04-22 22.31.50] Verification complete.  PMS blobs database is OK.
[2023-04-22 22.31.50] Saving current databases with '-BACKUP-2023-04-22_22.18.58'
[2023-04-22 22.31.50] Making repaired databases active
[2023-04-22 22.31.50] Repair complete. Please check your library settings and contents for completeness.
[2023-04-22 22.31.50] Recommend:  Scan Files and Refresh all metadata for each library section.
[2023-04-22 22.31.50]
[2023-04-22 22.31.50] Backing up of databases
[2023-04-22 22.31.50] Backup current databases with '-BACKUP-2023-04-22_22.31.50' timestamp.
[2023-04-22 22.31.54] Reindexing main database
[2023-04-22 22.32.43] Reindexing main database successful.
[2023-04-22 22.32.43] Reindexing blobs database
[2023-04-22 22.32.50] Reindexing blobs database successful.
[2023-04-22 22.32.50] Reindex complete.
[2023-04-22 22.32.50] Automatic Check, Repair/optimize, & Index successful.

[2023-04-22 22.32.50]
[2023-04-22 22.32.50] Status report: Sat Apr 22 22:32:50 CEST 2023
[2023-04-22 22.32.50]   PMS is stopped.
[2023-04-22 22.32.50]   Databases are OK.
[2023-04-22 22.32.50]

[2023-04-22 22.32.50] Starting PMS.
[2023-04-22 22.32.50] Started PMS

Unfortunately everything looks okay…

12

Just to be clear. I suggested the db tool because of the slow queries in your logs and what that tool does is significant improve db responsiveness (among other things).

Will let you and chuck workout from here. I’d be looking at the CPU metrics when things go bad considering where the thread is going :blush:

13

Thanks!.

I’m monitoring all the parameters on the server, CPU never gets passed 9% all other serives work fine, only thing that goes crazy is PMS…

14

Hi.
I’m still struggling with that problem, sometimes the service is working fine, and somedays it just freezes randomly… the procstat sees an increaed usage of the process “Plex Media Server”, so something is definitly going on.
Last night the service freezed at arround 21:30h, and this morning is still not working, I can see that the logs are being filled with many of this messages.

May 03, 2023 04:46:35.269 [139736479902520] DEBUG - Activity: updated activity 87ba15d9-7171-4a81-a7d8-4492c1ef17bb - completed 99.9% - Butler tasks

May 03, 2023 05:06:20.897 [139736157956920] INFO - Completed forward migration 20200224131300.
May 03, 2023 05:06:20.897 [139736157956920] INFO - Running forward migration 20200327131300.
May 03, 2023 05:06:20.897 [139736157956920] DEBUG - Captured session 0.
May 03, 2023 05:06:20.897 [139736157956920] DEBUG - Captured session 1.
May 03, 2023 05:06:20.897 [139736157956920] DEBUG - Captured session 2.
May 03, 2023 05:06:20.897 [139736157956920] DEBUG - Captured session 3.
May 03, 2023 05:06:20.897 [139736157956920] DEBUG - Captured session 4.
May 03, 2023 05:06:20.902 [139736157956920] INFO - SQLITE3:0x80000001, 17, statement aborts at 57: [select * from metadata_items limit 1] database schema has changed
May 03, 2023 05:06:20.904 [139736157956920] INFO - SQLITE3:0x80000001, 17, statement aborts at 57: [select * from metadata_items limit 1] database schema has changed
May 03, 2023 05:06:20.906 [139736157956920] INFO - SQLITE3:0x80000001, 17, statement aborts at 57: [select * from metadata_items limit 1] database schema has changed
May 03, 2023 05:06:20.908 [139736157956920] INFO - SQLITE3:0x80000001, 17, statement aborts at 57: [select * from metadata_items limit 1] database schema has changed
May 03, 2023 05:06:20.910 [139736157956920] INFO - Completed forward migration 20200327131300.
May 03, 2023 05:06:20.910 [139736157956920] INFO - Running forward migration 20200401131300.
May 03, 2023 05:06:20.910 [139736157956920] DEBUG - Captured session 0.
May 03, 2023 05:06:20.910 [139736157956920] DEBUG - Captured session 1.
May 03, 2023 05:06:20.910 [139736157956920] DEBUG - Captured session 2.
May 03, 2023 05:06:20.910 [139736157956920] DEBUG - Captured session 3.
May 03, 2023 05:06:20.910 [139736157956920] DEBUG - Captured session 4.

May 03, 2023 05:17:09.777 [139736479902520] ERROR - [CreditsDetectionManager] BufferingLineReader: failed to read line (error: -1)
May 03, 2023 05:17:09.777 [139737611651896] DEBUG - Jobs: '/usr/lib/plexmediaserver/Plex Media Scanner' exit code for process 1107706 is 0 (success)
May 03, 2023 05:17:09.777 [139736479902520] DEBUG - [CreditsDetectionManager] Activity: updated activity faa39712-a1bf-4c63-a199-b5dd20ab1ca0 - completed 75.0% - Detecting Credits
May 03, 2023 05:17:09.777 [139736479902520] ERROR - [CreditsDetectionManager] Job failed: Failed to generate any thumbnails
May 03, 2023 05:17:09.778 [139736479902520] DEBUG - [CreditsDetectionManager] Activity: Ended activity faa39712-a1bf-4c63-a199-b5dd20ab1ca0.
May 03, 2023 05:17:09.778 [139736479902520] DEBUG - Activity: ended sub-activity 5538774c-91e8-440f-8a69-55dd4ab728cc parent: 87ba15d9-7171-4a81-a7d8-4492c1ef17bb overall progress: 100.0% (1487/1487)
May 03, 2023 05:17:09.778 [139736479902520] DEBUG - Activity: updated activity 87ba15d9-7171-4a81-a7d8-4492c1ef17bb - completed 100.0% - Butler tasks

I’m attaching here the full logs.

@ChuckPa could you please have a look when you have a moment?

Many thanks!

15

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.