After having pounded away at a solution on my own, I’ve reached the stage of needing help from more knowledgeable folks.
Background: I’ve been using PMS on a Raspberry Pi since 2017. It has been a near flawless experience until, maybe, a month or two ago when regular - near daily - crashes have been occurring.
Current status: I’ve installed a fresh version of Raspian Lite and reinstalled PMS. I’ve done the corrupted database check (it was deemed “ok”) and did the suggested rebuild.
The crash kills the whole system and causes it to disappear from the network. It has taken awhile (my novice/intermediate skill set) to find anything in the system’s logs - both PMS or the OS’.
I’ve finally noted this set of 4 messages in /var/log/syslog as a pattern when it crashes:
Plex Media Server[599]: ERROR: Error decoding audio frame (Invalid data found when processing input)
Plex Media Server[599]: ERROR: Error decoding audio frame (Invalid data found when processing input)
Plex Media Server[599]: code:181: FutureWarning: The behavior of this method will change in future versions. Use specific ‘len(elem)’ or ‘elem is not None’ test instead.
Plex Media Server[599]: ERROR: The argument for -length must be a positive number
Is there an associate PMS log that would shed further light on what is happening here? “The Plex Media Server.log” shows no indication of the issue.
Incidentally, the last couple of crashes have occurred during the early AM maintenance procedures.
Server Version#: 1.19.5.3112
Player Version#: 4.34.3
I’ve seen this type of failure before. It was very common with ESXi and the old VMNET3 ethernet adapter emulation and Ubuntu 16.
LRO (Large Receive Offload) option caused the kernel to panic.
Your logs show the same characteristic line of NULL bytes which I saw there.
While this isn’t definitive in any way, I would suggest starting with the ethernet adapter settings – just to review.
Aug 22, 2020 02:25:26.498 [0x617fe440] DEBUG - MyPlex: updating with 26 access tokens
Aug 22, 2020 02:25:26.506 [0x5eff9440] DEBUG - HTTP requesting GET https://plex.tv/media/providers?X-Plex-Token=xxxxxxxxxxxxxxxxxxxx
Aug 22, 2020 02:25:27.302 [0x5eff9440] DEBUG - HTTP 200 response from GET https://plex.tv/media/providers?X-Plex-Token=xxxxxxxxxxxxxxxxxxxx
Aug 22, 2020 02:25:31.929 [0x60ffd440] DEBUG - Request: [127.0.0.1:56442 (Loopback)] GET /user/services/acoustid (2 live) GZIP Signed-in Token (derMechaniker)
Aug 22, 2020 02:25:31.931 [0x70dfe440] DEBUG - Completed: [127.0.0.1:56442] 200 GET /user/services/acoustid (2 live) GZIP 2ms 469 bytes
\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00
I would like to request you turn off the Crash Uploader in Settings - Server - General - SHOW ADVANCED
By doing so, the next crash will keep the DMP file local to your system.
When it does crash, would you please put that in a ZIP file and attach it here for me?
I will open it in the debugger and attempt to backtrack the process stacks to find exactly where the failure occurred.
It’s my hope that we can get a more focused starting point to figure out where this is coming from.
Thank you for your investigative work on my behalf.
I have made the change to the Crash Uploader setting, as requested.
For the record, I’ve even swapped the Raspberry Pi, itself with an identical model (3B) thinking there was potentially a hardware issue. The issue has persisted with the swap.
I feel like this whole issue began after I did a routine “apt update && apt upgrade” to my system earlier this summer.
Regarding ethernet adapter settings. What, specifically, are we talking about here? As stated, I’ve even done a complete reinstall/reconfiguration of the OS from scratch. I’ve done nothing special network settings-wise beyond plugging an ethernet cable into the Pi.
It crashed again early this morning, however it appears to have occurred at a later time with less messaging from Plex in /var/log/syslog.
Additionally, no .DMP file was generated in “Crash Logs.”
Most of what you’ve suggested as a potential cause is, frankly, over my head. I’ve attempted to run the same commands you’ve listed in your last post, substituting eth0 for bond10.
The first command’s result is: “large-receive-offload: off [fixed]”
The second command’s result is: “cannot change large-receive-offload”
I stopped there.
What I don’t understand is why this would suddenly come about and why it has become a regular early AM crash within the last few days. The crash originally had been happening after playing many tracks of music during the day (99% of what I use PMS for is serving music). After playing songs for awhile, after a song ended PlexAmp would be unable to load the next track, because the server had crashed. When this would occur the last played track was played in full giving no indication that the server had crashed. This would happen daily, but I don’t believe more than once per day.
After doing the suggested corrupt database fix, I thought I’d fixed the issue. However, it morphed to an issue where I can seemingly play music for hours, but now the server crashes every morning during the running of its Scheduled Tasks.
Thank you for also confirming the status of LRO with the adapter. It was worth checking and now can be eliminated from the cause of the crash.
May I see some of what you find in syslog about Plex? sudo journalctl -xe | grep -i plex is a good starting point unless you know better when to target (which I presume you do)
This is a strange problem and not the norm by any sense.
I’m leaning toward OS installation but can’t pin it down.
If it were systemic, everyone would be reporting it.
I’m not well versed with Pi CPUs but is this a 3 or better?
I guess I’m feeling this issue is related to my Library.
The issue persisted across an SD card format and reinstallation of Raspberry Pi OS and PMS. I’ve done no special configuring other than adding two rsync commands to crontab and the override.conf file for PMS that points to the Library’s location on an external drive. I didn’t even reinstall Samba.
The only constant here that existed pre & post OS/PMS refresh is my Library.
Since we are not seeing other Pi users having similar issues it points to my data.
I don’t want to blotto my Library and start over. Too much work has been performed fixing metatdata inconsistencies, Artist/Album/Title inconsistencies, etc. I really, really do not want to lose that work - my music collection is very large.
I am, however, tempted to temporarily generate a new alternate Library and add some content to it just to prove to myself that my Library is or is not the cause of the issue.
Yes, this is a Pi 3B. As stated, I’ve swapped my original Pi 3B with another 3B I have to ensure the hardware wasn’t the problem. The issue remains with either Pi running the system.
The only PMS messages (other than standard startup messages during a reboot) in syslog are the aforementioned ones in my original post. Those messages appeared, again, this AM when it crashed. However, the crash occurred about an hour later than it had Sat/Sun.
Aug 24 02:07:28 Plex-on-Pi Plex Media Server[1923]: ERROR: The argument for -length must be a positive number
Aug 24 03:25:31 Plex-on-Pi Plex Media Server[1923]: ERROR: Error decoding audio frame (Invalid data found when processing input)
Aug 24 03:27:27 Plex-on-Pi Plex Media Server[1923]: __code__:181: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead.
Aug 24 03:28:06 Plex-on-Pi Plex Media Server[1923]: ERROR: Error decoding audio frame (Invalid data found when processing input)
Aug 24 03:29:23 Plex-on-Pi Plex Media Server[1923]: ERROR: Error decoding audio frame (Invalid data found when processing input)
Aug 24 03:17:13 Plex-on-Pi systemd[1]: Starting Plex Media Server...
Aug 24 03:17:13 Plex-on-Pi Plex Media Server[595]: /bin/mkdir: cannot create directory ‘/mnt/music/plexmediaserver’: Permission denied
Aug 24 03:17:13 Plex-on-Pi systemd[1]: Failed to start Plex Media Server.
Aug 24 03:17:18 Plex-on-Pi systemd[1]: Stopped Plex Media Server.
Aug 24 03:17:18 Plex-on-Pi systemd[1]: Starting Plex Media Server...
Aug 24 03:17:18 Plex-on-Pi Plex Media Server[606]: /bin/mkdir: cannot create directory ‘/mnt/music/plexmediaserver’: Permission denied
Aug 24 03:17:18 Plex-on-Pi systemd[1]: Failed to start Plex Media Server.
Aug 24 03:17:24 Plex-on-Pi systemd[1]: Stopped Plex Media Server.
Aug 24 03:17:24 Plex-on-Pi systemd[1]: Starting Plex Media Server...
Aug 24 03:17:24 Plex-on-Pi systemd[1]: Started Plex Media Server.
Aug 24 08:51:01 Plex-on-Pi systemd[1]: Starting Plex Media Server...
Aug 24 08:51:01 Plex-on-Pi systemd[1]: Started Plex Media Server.
Aug 24 08:56:37 Plex-on-Pi Plex Media Server[602]: Connection to 184.105.148.89 closed by remote host.
I have a thought. I’ve seen where some users have had SD cards wear out.
Might that be happening here and, if so, do you have another so you can dd the image from one to another ?
/mnt/music/plexmediaserver
Is that a directory you’re supposed to be writing to?
Is it normal for PMS to be trying to mkdir its directory when it tries to start up?
I’m also seeing “Current command vanished from the unit file, execution of the command list won’t be resumed.”
When I run systemctl status plexmediaserver.service I get the “The unit file, source configuration file or drop-ins of plexmediaserver.service changed on disk. Run ‘systemctl daemon-reload’ to reload units.” error every time. I don’t understand why that isn’t persisting through subsequent reboots after a systemctl daemon-reload.
It’s not normal for PMS to be making anything except subdirectories under the Application Support Dir. That directory would be Plex Media Server. Under there is where you find the Metadata, Logs, Cache, Application Support, and Plug-in Support directories with all their contents.
Perhaps some logs (ZIP) from the “Logs” directory are due at this point?
Let me see what it’s trying to do?
May I see your override.conf as well please?
As for the SSD, if you use dd or make a tar image of the APPLICATION_SUPPORT_DIR and all below it, you can clone it anywhere . Just remember to take care of any UID/GID changes for user plex
I’m having my doubts about the SD card. I’m pretty sure I used an alternate when I did the OS/PMS refresh. However, I will proceed with the dd, regardless.
If I understand you correctly, PMS should not be trying to /bin/mkdir /mnt/music/plexmediaserver when it is starting up?
By Logs you are talking about the PMS Logs, yes? If so, the .zip I sent earlier today should have timestamps that correlate with the times stamped at 03:17 AM today when it was attempting to mkdir.
Let me know if there are alternate logs you need and where from.
If you have media in /mnt/music and add /mnt/music as a directory to monitor, PMS will blow up because it will end up monitoring itself. You’ll never get scanning to stop. I urge caution as you do this with /mnt. Perhaps you meant /mnt/plexmediaserver which is independent of your /mnt/music ?
/mnt/music/plexmediaserver/Library/Application Support - working directory
/mnt/music/music - music library (yes, music/music twice)
/mnt/video - video library
However, in the interest of what you’ve stated above, I’ve added another SSD to the system (maybe with the idea of booting and running the OS and Plex from it, rather than the SD) and migrated the entire plexmediaserver contents to it. I don’t believe Plex was scanning itself due to how I had the music in a further music subdirectory and pointed to that subdirectory as the library location, but I understand how that may have been perceived.
So, I now have 3 SSDs:
/mnt/pms/plexmediaserver/Library/Application Support - working directory
/mnt/music/ - music library (yes, single music/)
/mnt/video/ - video library
I updated the entry in override.conf to reflect the location change and PMS no longer tries to mkdir when it starts. I still do not understand why it was doing that in the first place - the directory already was there and it has been using it for years.
I am still, however, seeing Warning: The unit file, source configuration file or drop-ins of plexmediaserver.service changed on disk. Run 'systemctl daemon after a reboot.
