And shortly after that restart I have a few of people watching things, one of them an AppleTV user doing a transcode.
VMM shows that the committed memory jumps to 1.5G, Private Bytes 1.2G, 1G of Private Data.
Pulling the memory usage numbers from the media server log (Plex Media Server.3.log in the bundle) we see the memory ramp up starting at 19:01:41 at 87MB to 505MB by 19:02:50.
I’ve uploaded the mmp and the log bundle so you can see this growth from the start.
I haven’t updated in a while, but the behavior continues to be the same. It looks like it’s having the most impact when I have to transcode for my AppleTV users. It seems the memory always jumps pretty high for them and isn’t necessarily released.
I’ve added more log zips, mmps, and a couple PMLs from procmon when I’ve noticed an appletv user streaming and the memory has grown.
I realize this is a lot to sort through, let me know if there’s anything I can do to help streamline the gathering of the data. Thank you!
I ran into another crash last night, and it was a really unexpected quick one. It does look like one of my appletv users was involved though.
I see log messages about memory getting low after Feb 7 19:00, and it eventually crashed at 20:41. Looks like the crashdump wasn’t uploaded, and I also don’t see anything new in appdata/local/temp. I’ve uploaded the log dump to my google drive link.
I’m going to disable HW encoding to see if that changes anything with this memory growth until crash issue.
would like to see if this request crashes the server for you / spikes memory use
http://127.0.0.1:32400/library/metadata/116786/similar
Sent you private message for getting me the diagnostics
thank you
Got it done and uploaded everything you asked, along with a mmp before and after in case that helps. Mem usage is currently low, not sure if turning off HW encoding after the last crash has anything to do with that.
I’ll also keep an eye on my server for a busy time where mem usage is already high and try to run it then.
Thanks!
I’ve been following this thread and others related to the same “thread_resource_error” that I’ve been experiencing, and mine also seems to be a memory problem related to transcoding activities ever since migrating to brand new server hardware a couple months ago. Just wanted to share my common experience that I had routine crashes daily or every other day, always with the same pattern and messages in the logs. I turned off hardware transcoding to test that a couple of weeks ago and haven’t experienced another crash in all that time now for the first time on this server. I would like to have hardware transcoding back on, so looking forward to more info on the results on this research, but +1 on the thread resource error and memory leak with HW transcoding on.
This is an important point. I just turned HW enc back on to keep things the same.
I just ran it without the json param and have uploaded the mmps, logs, and DB again.
One thing that’s extremely bizarre is that ever since I disabled HW encoding, and have now re-enabled it, the server looks like it’s behaving as expected. Committed Private Data isn’t growing in leaps the way it had been. Of course, the one user that I could count on to cause the main issue hasn’t watched anything transcoded yet, so I want to wait and see if that happens again.
What happens within Plex and the OS when you disable/enable HW encoding?
Interesting
What day was hardware transcoding changed ?
I have logs from the 7th Feb showing the problem and logs from the 8th with similar requests not causing memory issues
For whole period of Feb 08, 2022 14:04 to Feb 15, 2022 07:09 there were no memory failures and usage max was 327Mb
I changed it right before I posted that message on Feb 11 12:46pm. There are uploads from the 15th on my google drive.
I changed nothing else on the server except for turning HW enc off on Feb 8, and then back on on Feb 11. I didn’t restart the service between those two changes, and just restarted it the other day (for Windows patching) after running stable since the change. And it’s been running stable since that restart as well.
Even though the issue would persist between service restarts and reboots, maybe specifically disabling it and re-enabling it changed some part of the way the back-end handles HW transcoding? Or transcoding in general?
I’ve had the same types of crashes over the last several weeks. Happening almost daily with the same memory increase eventually leading to thumbnails not showing up before it crashes entirely. I also get the same error message in the logs- “…Error resizing an image, we don’t trust what we cached…”
I also have hardware acceleration enabled with a nvidia quadro p2000. I’d love to find a solution as it has been driving me crazy. Prior to this, I’ve basically had 10 straight years of stability.
Thanks
We have logs covering whole period from Feb 01, 2022 12:16 to Feb 15, 2022 07:09
I can see that Use hardware-accelerated video encoding transcoder setting was disabled Feb 08, 2022 08:13:38 and Enabled Feb 11, 2022 12:45:55
Through out the Feb 1 to Feb 15 period there were memory allocation failures only on the 7th Feb - with first at Feb 07, 2022 19:48:14 and last at Feb 07, 2022 20:41:36
For the last period from PMS launch Feb 08, 2022 14:04:09 to Feb 15, 2022 07:09:19 - the logged memory used was between 43 Mb (min) and 327 Mb (max) through this period
The Use hardware acceleration when available setting was not altered during the period
I do not know if it is relevant - because there were also no failures between 1st Feb and first fail on 7th Feb.
Yeah, this is very strange. Things have been completely stable ever since making that one little change with no other changes happening on the system. I’ll let this topic close out automatically, just in case it starts happening again.
Thank you so much for all your help with trying to track this down.
This might be something to think about for other users that might be running into odd situations, maybe disabling HW enc and then re-enabling it changes/updates something in the back-end? Especially if that setting hadn’t been touched over the course of a couple years through many updates.
Things had been running fine, so on Mar 1 I decided to update to the new version, 1.25.7.5604-980a13e02. Ever since that upgrade, memory started growing the same way it had been before, although this time it wasn’t limited to users only on iOS. It ended up crashing today at 11:52.
I’ve uploaded an mmp and the logs from this most recent crash to my google drive link.
I’ll wait to hear from you before I make any changes to the system, although if it continues to happen I may disable/enable HW encoding again to see what happens. I may also look to downgrade to the previous version.
Spent today looking at your diagnostics -
The problem was because of the large movies library and the number of users you shared the server with who also have their own servers !
Plex Media Server collects metadata from all shared servers and so the servers owned by the users you shared your server with are collecting metadata from your server and the requeste are coming in unpaged needing to return info for 2500 movies for one library and 500 for the other
I have asked the Plex Media Server team to look into this
We also have a Plex Web timing issue where sometimes requests are unpaged - this has been reproduced and will be looked into
We also have the Plex for tvOS issue with similar movies lists and that is now being corrected and will be in the next release 8.0
That makes a lot of sense that it was related to stuff going on between the servers and not necessarily user activity. I’ve only had a few watches since the crash, but the server is behaving like normal, memory usage under 300M. I’ll keep watching things and keep my eyes out.
Let me know if you need any more info to help with the issues you mentioned.
Thanks again for all the help you give us!!
I just upgraded to 1.25.7.5604 and it’s crashed twice in just a few hours. Plex hasn’t crashed in ages on my server so have to roll back.
Also multiple PMS crashes v 1.25.7.5604 - though on a Win7 machine. Previously stable
I cannot find any uploaded crash reports for your accounts.
Could you please see if there are any files still in %TEMP% for the user account Plex Media Server runs in - files that were created around the time of the crash - the file names would ne a string of characters with dashes and no filename extension
(and if you have saved any server logs - please attach here)
Hello sa2000 - I hadn’t uploaded server logs. I’ll do what I can to assist.
