Plex crashing every night on TrueNAS Core TrueNAS-12.0-U7

Server Version#: Version 4.63.0 and Version 1.24.5.5173
Player Version#: -

Plex Media Server Logs_2021-12-16_16-30-43.zip (3.2 MB)

So I have no idea what why where…

My Plex server crashes every night, All I know at the moment (due to TrueNAS guys claiming it was due to SWAP and I should increase RAM, so increased from 16GB → 32GB now), but still.
I can see there is a spike in Swap usage just after 3am, but can’t say if thats the cause of the crash, or time, I’ve attached my Plex logs…
Diagram attached.

My TrueNAS /var/log/messages contain…

Dec 15 03:13:05 vaultx kernel: pid 2010 (Plex Media Server), jid 1, uid 972, was killed: out of swap space
Dec 15 03:13:05 vaultx kernel[1095]: Last message 'pid 2010 (Plex Media' repeated 1 times, suppressed by syslog-ng on vaultx.xxxxxxx.com
Dec 15 09:25:59 vaultx kernel: pid 32784 (Plex Media Server), jid 5, uid 972: exited on signal 6
Dec 16 03:15:06 vaultx kernel: pid 44663 (Plex Media Server), jid 7, uid 972, was killed: out of swap space
Dec 16 03:15:06 vaultx kernel[1095]: Last message 'pid 44663 (Plex Medi' repeated 1 times, suppressed by syslog-ng on vaultx.xxxxxxx.com

Looks like you are out of swap space :slight_smile:

How large is your TrueNAS pool, etc.? I am at 96GB RAM for my 100TB+ pool. Running on 12.0-U6 and don’t have this issue.

So all the media is on TANK, apps are on Apps.
I\m also attaching a Swap usage diagram, you will see swap usage was always zero, up to a couple of weeks ago when a Plex upgrade was done.
RAM use to be 16GB, and stable, upgraded on Tuesday to 32GB after everyone implied this is a lack of RAM leading to SWAP leading to crash… my issue, my media volume have not grown significantly…

The problem of out of Swap, I think it is a symptom, it’s not the cause.

G

See attached.


Do you have a snapshot you can roll back to to verify that the older PMS version worked?

I don’t have access to my TrueNAS install currently (power outage) but is it possible that the jail is running out of swap space, and not TrueNAS itself?

Also, seems like the swap issue happens around 3AM, which I think might be the default time for the daily housekeeping tasks, etc… You could try disabling/re-scheduling this to see if that is having an impact as well.

I can change my schedule instead of 2-5 to be say 4-6 and see if it moves the problem.

it’s Def TrueNAS thats running out of SWAP as it’s int he TrueNAS /var/log/messages

haven’t considered rolling back, never even tried that… semi scared of doing that.

G

So I moved the the normal house keeping schedule from 2-5 → 4-7 and likewise my Plex crash/kill moved…

I’m a full Plex Pass pain up member, surely there is some form of official ticket system ?

root@vaultx[~]# cat /var/log/messages |grep Plex
Dec 17 05:14:35 vaultx kernel: pid 2020 (Plex Media Server), jid 1, uid 972, was killed: out of swap space
Dec 17 05:15:34 vaultx kernel[1100]: Last message 'pid 2020 (Plex Media' repeated 1 times, suppressed by syslog-ng on vaultx.xxxx.com
root@vaultx[~]#

You could check if this is the problem that is caused by having loads of extras (trailers/clips) in the database for movies - that issue was leading to memory allocation failures and crashes - there is no evidence of memory allocation failures in the Plex Media Server logs but still worth checking the database - running a query like this on the database

select count(metadata_item_id),metadata_item_id from metadata_relations group by metadata_item_id order by count(metadata_item_id) desc limit 20

Unfortunately we do not have crash reporting and mini dump creation on FreeBSD

hi there
how where do I run that command, see it’s SQL, so assume there is a database hanging around.

G

Do it after you shut down the Plex Media Server

The db file on your system is here
/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db

Follow the steps for running Plex Sqlite as outlined for database repair here
Repair a Corrupted Database | Plex Support

But instead of repair - run this sql
select count(metadata_item_id),metadata_item_id from metadata_relations group by metadata_item_id order by count(metadata_item_id) desc limit 20

missing something, that directory looks like it will be inside my jail
so… I can’t access it if I’ve shut the rail down, which in my case imply shutting Plex down.

Will keep it running as I’m only going to run a select

G

not finding the sqlite binary…
examples does not say where the binary would be in a TrueNAS build.
G

You can navigate into the jail and stop the plexmediaserver service. Assuming iocage jail setup:
sudo iocage console [jail_name]

Should be inside the jail now.

service plexmediaserver stop

SQL lite binary for FreeBSD can be found here:
/usr/local/share/plexmediaserver/Plex SQLite

While still present in the running jail, bring up SQLite and run the provided SQL.

can you navigate to the directory where the db file is before launching “Plex SQLite”

Then you .open this file com.plexapp.plugins.library.db and run the sql on that file

how do I open that file ?
work on assumption I’ve not done any of this before.
G

OK - it is not that straightforward - the db filename needs to be a parameter to the plex sqlite binary

So you navigate current directory ito
/Plex Media Server/Plug-in Support/Databases

then you launch “Plex SQLite” binary giving it this as first parameter com.plexapp.plugins.library.db

and that goes into sqlite command line where you can type
select count(metadata_item_id),metadata_item_id from metadata_relations group by metadata_item_id order by count(metadata_item_id) desc limit 20

1 Like

sqlite> select count(metadata_item_id),metadata_item_id from metadata_relationsgroup by metadata_item_id order by count(metadata_item_id) desc limit 20;
62|15200
60|11746
59|11463
55|12716
52|13884
51|14288
48|15284
44|13380
44|14783
43|8866
40|9166
40|11875
39|13394
39|13870
38|48801

That is good. Highest is 62. So not this issue

Without mini dump of the crashing plex media server process, it will be difficult to establish what the crash is

You can try out new beta releases as they come out to see if fixes for crashes investigated for other platforms fix your issue

Do you have to run on this Linux platform ?

I have no other host to host the volume of media…

It’s running on TrueNAS CORE atm (which is BSD based of course)… been thinking about seeing whats involved to move to TrueNAS SCALE (which is linux based)…

bit nervous of the move, but might end being the better option in the absence of any other.

G