PMS Shutting down/crashing randomly

[BertDude]

Server Version#:1.21.4.4079
Player Version#:

After installing the latest version of PMS on my Synology NAS, I have been experiencing random shutdowns/crashes. I would get one at least once a day. Any idea where I can begin looking to diagnose this issue? ANy other experiences?

Thanks for any support…

[ChuckPa]

Need the DEBUG (not VERBOSE) logs ZIP file which you download after restarting .

That will let us see what’s happening at the first level.
If more is involved, we’ll dig deeper.

[BertDude]

After the last shutdown, I turned on the logging, so should I wait until it shuts down again?

[ChuckPa]

Yes, unless you can cause it to crash? If so, please write down the steps. I’ll try to recreate here.

[BertDude]

Ok, I will have to wait, as the shutdowns were random and I could not tie them to any specific action.

[BertDude]

Plex Media Server Logs_2021-03-02_11-10-48.zip (5.0 MB)

The PMS shutdown again and here are the logs, please advise, thanks…

[BertDude]

Hi ChuckPa, has my logs revealed anything about the problem?

[ChuckPa]

Sorrry for the delay responding.

Your logs have provided a lot of information.

First, because you curate your music so well, you ran out of kernel resources

I’d like to fix that first.

This is a two step process. I’ve written it out in a How-To in the FAQ.

The main steps are:

1. Use File Station to count all the folders in use (grand total)
2. Do a little math then write that result to the Linux config file.
3. Restart and let PMS breathe better.

Here’s the How-To. Please don’t hesitate if you have any questions.

Synology FAQ - Questions, Answers, and a few How-To's

FAQ 16: Increase the number of folders Plex can monitor Applicable to all supported versions of DSM. When the Linux ‘inotify’ table fills up, this will cause when Plex to stop automatically detecting new media you add to the server. It will also manifest when there are transcoder problems with Dolby audio codecs. Linux has a default table size which is static. We can increase this at will. On some versions it’s 8192. On newer kernels, the default is 65536 folders. This is easy to in…

Logs showing where it stops:

Mar 02, 2021 11:10:28.362 [0x7f4a2288f700] DEBUG - [Notify] Now watching "/volume2/Music/Vernon's Collection/Vernon's Collection - U Thru Z/Woody Shaw/Jazz For a Rainy Afternoon"
Mar 02, 2021 11:10:28.362 [0x7f4a2288f700] DEBUG - [Notify] Now watching "/volume2/Music/Vernon's Collection/Vernon's Collection - U Thru Z/World Class Wreckin Crew"
Mar 02, 2021 11:10:28.362 [0x7f4a2288f700] DEBUG - [Notify] Now watching "/volume2/Music/Vernon's Collection/Vernon's Collection - U Thru Z/World Class Wreckin Crew/80'S && 90'S SLOW JAM && R&&B CLASSICS 4"
Mar 02, 2021 11:10:28.362 [0x7f4a2288f700] DEBUG - [Notify] Now watching "/volume2/Music/Vernon's Collection/Vernon's Collection - U Thru Z/World Famous Supreme Team"
Mar 02, 2021 11:10:28.362 [0x7f4a2288f700] DEBUG - [Notify] Now watching "/volume2/Music/Vernon's Collection/Vernon's Collection - U Thru Z/World Famous Supreme Team/Tommy Boy Presents Hip Hop Ess"
Mar 02, 2021 11:10:28.362 [0x7f4a2288f700] DEBUG - [Notify] Now watching "/volume2/Music/Vernon's Collection/Vernon's Collection - U Thru Z/World's Famous Supreme Team"
Mar 02, 2021 11:10:28.362 [0x7f4a2288f700] DEBUG - [Notify] Now watching "/volume2/Music/Vernon's Collection/Vernon's Collection - U Thru Z/World's Famous Supreme Team/Old School Jams 4 [Disc 1]"
Mar 02, 2021 11:10:28.362 [0x7f4a2288f700] ERROR - [Notify] Failed to add watch for "/volume2/Music/Vernon's Collection/Vernon's Collection - U Thru Z/Worthy of a Beating" (28: No space left on device)
Mar 02, 2021 11:10:28.362 [0x7f4a2288f700] ERROR - [Notify] Failed to add watch for "/volume2/Music/Vernon's Collection/Vernon's Collection - U Thru Z/Worthy of a Beating/Unknown Album" (28: No space left on device)
Mar 02, 2021 11:10:28.362 [0x7f4a2288f700] ERROR - [Notify] Failed to add watch for "/volume2/Music/Vernon's Collection/Vernon's Collection - U Thru Z/Wosp" (28: No space left on device)
Mar 02, 2021 11:10:28.362 [0x7f4a2288f700] ERROR - [Notify] Failed to add watch for "/volume2/Music/Vernon's Collection/Vernon's Collection - U Thru Z/Wosp/Power" (28: No space left on device)
Mar 02, 2021 11:10:28.362 [0x7f4a2288f700] ERROR - [Notify] Failed to add watch for "/volume2/Music/Vernon's Collection/Vernon's Collection - U Thru Z/Wrecks In Effects" (28: No space left on device)
Mar 02, 2021 11:10:28.363 [0x7f4a2288f700] ERROR - [Notify] Failed to add watch for "/volume2/Music/Vernon's Collection/Vernon's Collection - U Thru Z/Wrecks In Effects/Unknown Album" (28: No space left on device)
Mar 02, 2021 11:10:28.363 [0x7f4a2288f700] ERROR - [Notify] Failed to add watch for "/volume2/Music/Vernon's Collection/Vernon's Collection - U Thru Z/Wrecks In Effects/Wrecks In Effects" (28: No space left on device)
Mar 02, 2021 11:10:28.363 [0x7f4a2288f700] ERROR - [Notify] Failed to add watch for "/volume2/Music/Vernon's Collection/Vernon's Collection - U Thru Z/Wrecks 'n' Effect" (28: No space left on device)
Mar 02, 2021 11:10:28.363 [0x7f4a2288f700] ERROR - [Notify] Failed to add watch for "/volume2/Music/Vernon's Collection/Vernon's Collection - U Thru Z/Wrecks 'n' Effect/Wrecks 'N' Effect" (28: No space left on device)
Mar 02, 2021 11:10:28.363 [0x7f4a2288f700] ERROR - [Notify] Failed to add watch for "/volume2/Music/Vernon's Collection/Vernon's Collection - U Thru Z/Wreckx N Effect" (28: No space left on device)
Mar 02, 2021 11:10:28.363 [0x7f4a2288f700] ERROR - [Notify] Failed to add watch for "/volume2/Music/Vernon's Collection/Vernon's Collection - U Thru Z/Wreckx N Effect/Hard Or Smooth" (28: No space left on device)
Mar 02, 2021 11:10:28.363 [0x7f4a2288f700] ERROR - [Notify] Failed to add watch for "/volume2/Music/Vernon's Collection/Vernon's Collection - U Thru Z/Wreckx N Effect/Wrecks 'N' Effect" (28: No space left on device)```

[BertDude]

Thank you, I will review the proposal and implement. I will let you know if i have any questions.

[BertDude]

One question I have regarding the FAQ;

THe FAQ Starts (a) by adding 32768 to your number you obtain (If I understand this properly). However when you use your example number of 241,012 you omit step (a) and jump straight to step (b) where you divide… Is that an error? Is step (a) actually required? Or you forgot to apply it in your example?

F.A.Q. snippet below;
When done collecting the number of folders in use:
a. Add 32768 to the value
b. Divide by 32768
c. Add 1 to the the Integer result (No fractions here)
e. Multiply by 32768
f. Use this final result. (integer part only)
g. If the number of folders you are using is less than 8192, stop here unless you are about to add a lot more media.

Example: I have 241,012 directories (because I curate a lot of music and photos)

Copy to clipboard

241012 / 32768 = 7.35  ->   7
7 + 1 = 8
8 * 32768 = 262144  (This is the value I use - room to grow)

[ChuckPa]

I forgot to apply it in my example OOOPS !

What I’m trying to avoid is,

1. Adding 32768 to the total obtained guarantees N % 32768 (mod) is a non-zero number.
2. ( Non-zero result + 1 ) * 32768 = new value to assply (mod 32768)

NOTE: Math corrected in the FAQ … Thank you

[BertDude]

Hi ChuckPA, I have gotten the issue again three times after applying the changes suggested… (see attached logs)…Plex Media Server Logs_2021-03-20_19-46-24.zip (3.4 MB)

[BertDude]

Hi ChuckPA, I sent you some updated logs 6 days ago on an issue that is still occurring on a course of action that was suggested. Can you review and provide some feedback, thanks…

Bert

[ChuckPa]

@BertDude

I’m sorry. Somehow I missed notification of your logs posting.

I see the errors. Linux is completely out of file handles!

This is very rare and indicative of a major problem.

What did you increase your inotify table size to be ? I fear it’s too high for the memory installed and denying PMS of file handles now.

Mar 20, 2021 19:40:54.475 [0x7f18e3d11700] DEBUG - Request: [192.168.181.113:62537 (Subnet)] GET /library/parts/279072/1494869842/file.mp3 (2014 live) TLS Signed-in Token (BertDude) (range: bytes=0-1) 
Mar 20, 2021 19:40:54.477 [0x7f18e3d11700] DEBUG - Content-Length of /volume2/Audiobooks/Test Folder for Prologue/Audiobook Series - Cmd - James Patterson - Woman's Murder Club - (16 Books)/James Patterson - Women's Murder Club 08 - 8th Confession/James Patterson - Women's Murder Club 08 8th Confession 2009 004-End.mp3 is 2 (of total: 44603645).
Mar 20, 2021 19:40:54.477 [0x7f18e3d11700] ERROR - Error opening file '"/volume2/Audiobooks/Test Folder for Prologue/Audiobook Series - Cmd - James Patterson - Woman's Murder Club - (16 Books)/James Patterson - Women's Murder Club 08 - 8th Confession/James Patterson - Women's Murder Club 08 8th Confession 2009 004-End.mp3"' - Too many open files (24)
Mar 20, 2021 19:40:54.477 [0x7f18e3d11700] ERROR - Failed to open file for streaming: /volume2/Audiobooks/Test Folder for Prologue/Audiobook Series - Cmd - James Patterson - Woman's Murder Club - (16 Books)/James Patterson - Women's Murder Club 08 - 8th Confession/James Patterson - Women's Murder Club 08 8th Confession 2009 004-End.mp3
Mar 20, 2021 19:40:54.478 [0x7f1963fff700] DEBUG - Completed: [192.168.181.113:62537] 404 GET /library/parts/279072/1494869842/file.mp3 (2014 live) TLS 3ms 240 bytes (pipelined: 3) (range: bytes=0-1) 
Mar 20, 2021 19:40:54.498 [0x7f1968585700] DEBUG - Auth: authenticated user 1 as BertDude
Mar 20, 2021 19:40:54.498 [0x7f1912b7d700] DEBUG - Request: [192.168.181.113:62537 (Subnet)] GET /library/parts/279069/1494870466/file.mp3 (2013 live) TLS Signed-in Token (BertDude) (range: bytes=0-1) 
Mar 20, 2021 19:40:54.500 [0x7f1912b7d700] DEBUG - Content-Length of /volume2/Audiobooks/Test Folder for Prologue/Audiobook Series - Cmd - James Patterson - Woman's Murder Club - (16 Books)/James Patterson - Women's Murder Club 09 - 9th Judgement/James Patterson - Women's Murder Club 09 9th Judgement 2010-001-060.mp3 is 2 (of total: 48560256).
Mar 20, 2021 19:40:54.501 [0x7f1968585700] DEBUG - Completed: [192.168.181.113:62537] 206 GET /library/parts/279069/1494870466/file.mp3 (2013 live) TLS 2ms 2 bytes (pipelined: 4) (range: bytes=0-1) 
Mar 20, 2021 19:40:54.513 [0x7f1963fff700] DEBUG - Auth: authenticated user 1 as BertDude
Mar 20, 2021 19:40:54.513 [0x7f18e0e54700] DEBUG - Request: [192.168.181.113:62537 (Subnet)] GET /library/parts/279069/1494870466/file.mp3 (2013 live) TLS Signed-in Token (BertDude) (range: bytes=0-48560255) 
Mar 20, 2021 19:40:54.515 [0x7f1963fff700] DEBUG - Auth: authenticated user 1 as BertDude
Mar 20, 2021 19:40:54.515 [0x7f190940d700] DEBUG - Request: [192.168.181.113:59972 (Subnet)] GET /statistics/bandwidth?timespan=6 (2013 live) TLS GZIP Signed-in Token (BertDude)
Mar 20, 2021 19:40:54.522 [0x7f18e0e54700] DEBUG - Content-Length of /volume2/Audiobooks/Test Folder for Prologue/Audiobook Series - Cmd - James Patterson - Woman's Murder Club - (16 Books)/James Patterson - Women's Murder Club 09 - 9th Judgement/James Patterson - Women's Murder Club 09 9th Judgement 2010-001-060.mp3 is 48560256 (of total: 48560256).
Mar 20, 2021 19:40:54.525 [0x7f1968585700] DEBUG - Completed: [192.168.181.113:59972] 200 GET /statistics/bandwidth?timespan=6 (2013 live) TLS GZIP 10ms 4691 bytes (pipelined: 1614)
Mar 20, 2021 19:40:54.553 [0x7f1963fff700] ERROR - Couldn't accept new TLS connection: Too many open files
Mar 20, 2021 19:40:54.553 [0x7f1963fff700] ERROR - Couldn't accept new TLS connection: Too many open files
Mar 20, 2021 19:40:54.554 [0x7f1963fff700] ERROR - Couldn't accept new TLS connection: Too many open files
Mar 20, 2021 19:40:54.554 [0x7f1963fff700] ERROR - Couldn't accept new TLS connection: Too many open files
Mar 20, 2021 19:40:54.554 [0x7f1963fff700] ERROR - Couldn't accept new TLS connection: Too many open files
Mar 20, 2021 19:40:54.554 [0x7f1963fff700] ERROR - Couldn't accept new TLS connection: Too many open files
Mar 20, 2021 19:40:54.554 [0x7f1963fff700] ERROR - Couldn't accept new TLS connection: Too many open files

[BertDude]

Ok, so lets go over the process based on the directions previously presented…

1. Using Windows, I went to the top level folder of each category of media managed by Plex (Movies, Music, TV, Videos, Audiobooks)
2. From the top level share for each share category, I performed a “properties” on that folder in which windows presented me with information on all sub folders, files, etc… (Note that would include sub folders that may be more then 2 levels deep)
3. I then took the folder totals from each share category and summed them together I get 44091 (Note… If I do not include the one of the categories (you referenced it when I originally submitted this issue), I have 3525 folders between everything else.
4. ANyway taking that number and applying your process…
   4a) 44091 + 32768 = 76859
   4b) Divide 76859 / 32768 = 2.3455
   4c) I believe that you meant to say here was to make the fractional portion 1… So that would then give me a result of 3.
   4d) 32768 * 3 = 98304
   4e) I am assuming here that when you say use the final result (integer part only) you are referring to the result from step (4d) which is 98304. (Since we eliminated any possibility of getting a fraction from the step before that one)
   4f) Since my number is more than 8192… (again this is confusing because there is no way to multiply 32768 (from step e. In your directions by any integer greater than 1 and get a number less than 8192… maybe something else is missing)
5. So my final number is 98304.
6. From this point is is using the value I calculated (which is much less than your example) to apply to the DSM update.

My Synology NAS has 32GB which is the max memory that can be installed.

So reviewing that please let me know where I went wrong as you indicated that the number I used was huge, which is not possible if I compare it to the hypothetical number from your example.

[BertDude]

I just posted a response to you based on my calculations to your directions…

[ChuckPa]

This might be easier for a desktop calculator.

New_Watches_count = ( int( (Total_count + 32768 ) / 32768 ) +1 ) * 32768

However you want to slice that is fine.

[BertDude]

Ok, your condensed formula yields the same results… 98304…

(Int((44091+32768)/32768)+1)*32768

Now what? That’s what I updated PLex with and the results of that is what I last sent to you… please advise

[ChuckPa]

Let’s take this up another notch.
I think what’s happening is the known (and fixed) Hubs problem.

Choices are:

1. Disable crash uploading, wait for it to fail, grab the .DMP file and logs, give them to me for post morten
   -or-
2. Upgrade PMS now to 1.22.1.4262 and re-evaluate

What’s your preference ?

[BertDude]

I am currently running Version 1.22.1.4228. There currently isn’t a newer version being picked up my PMS at this time, is it available?

I would prefer to upgrade PMS and evaluate, when will it be published?