TL;DR - This is about moving metadata to a NAS fileshare and bad things happening. Moving the metadata back fixed it. I’ve read the FAQs and am hoping someone has the magic xattrs stuff to make it work. I detail things below.
Synology NAS ONLY Server Version#: DSM 7.1.1-42962 Update 2
Linux Player Version#: 1.30.1.6562-915986d62
This is all current as of 202301161004z
What this is NOT:
- I am NOT discussing using content stored on the Synology NAS. That works flawlessly.
- I am NOT discussing using any apps on the Synology NAS, including Plex Media Server, as the NAS is merely being used to share files/filesystem/mountpoints/shares.
What this IS:
- If I move the metadata as per the June 2018 “Linux Tips” article (systemd override, etc.) then there are issues. I detail below with a link to the article thread.
- If I move the metadata back to a local drive all these errors immediately stop.
What have I looked up: Every article her on NFS, CIFS, and the FAQs on Linux and Synology. Most of the latter deal with “the Plex app running ON the Synology NAS” or “Problems with media being served from the Synology.” The exception is the article previously cited, but here’s a direct link: Linux Tips - #5 by ChuckPa – look for the 5/21 article “Moving Plex’s metadata…”
Method and Errors:
-
Create a NAS served NFS filesystem, set permissions, bring up mountpoint on Linux server. Verify that access to /Plex/MOVIES and /Plex/TV is full read/write with proper user, group, and permissions. This has worked FLAWLESSLY for over 6 months. Zero issues, no need to restart pms, no end-user complaints, etc.
-
Follow the instructions as per the link above, create /Plex/PlexData, rsync /var/lib/plexmediaserver to it, set the override file, reload systemd and restart plexmediaserver. Yes, I did the first rsync with pms on, then stopped pms, rsync’d again and did the override and the start. Plex reported NO issues and worked great for 2 days.
-
After each 2-3 days there would be SQlite corruption locking errors in the systemd journal. Following Plex’s faq on db corruption each time for the PRAGMA integrity_check; showed “Ok”. Following other instructions (mainly move the db and have Plex regenerate it) worked great for 2-3 days, and then the errors would return.
-
There were times that attempting to connect would not find a working server on the listening port. Restarting plexmediaserver would correct that. In these cases no errors report prior to stop or post start.
-
There were times that the server responded fine, allowed selecting a username, but when selecting media (say a movie) it would fail to load. Error messages on the Roku or Chrome players were inspecific and NO errors were recorded on the server.
-
Changed from NFS mount to CIFS mount thinking perhaps some extended attributes supported on local file systems but not NFS/CIFS were being used by Plex. CIFS performed just as poorly as NFS. Poorly=errors, stoppage of server usability.
-
I hate disappointing friends and family so, did the rsync back to /var/lib/plexmediaserver, stopped plex, rsync again to catch loose ends, remove the override file, start plexmediaserver, and everything WORKS PERFECTLY.
As mentioned I’ve searched for this online (Google) and on these forums, read the FAQs, and have not been able to locate any reference material to suggest why the metadata works fine on the local drive, and has consistent “issues” on the NAS drive, whether through NFS or CIFS.
Just to weed out potential issues, I did set mode 0777 on the NAS shares and did verify uid/gid and did verify complete write access. The NAS was rebooted as was the Linux pms server. The network is solid – media streams great when the server isn’t “stuck”, and mrtg shows less than 1Mbps on a 100Mbps link through a MikroTik 100Mbps switch/router. Nothing fixed the issue other than moving the metadata back onto the local drive.
To add further detail for anyone reading this far:
Server is an HP 8300 4 i5s with 4 cores each, 8GB with 2+ free with the plex scanner doing its thing and 3+ free when it’s not. /swap is 99% free. io wait is about 10% during plex scanner and around 1-2% normally. Nas is Synology RS1221+ network no issues, 5 drives in RAID 10+1. lav is below 1 and iow below 2%.
Any help appreciated, and yes, if there’s a definitive answer that doesn’t require downtime to fix, that’s worth your time and my money. If downtime is required, there will be a RAID10 to RAID5 conversion in about 5 months which will have a few minutes of downtime (irrelevant to this thread so put in as a P.S. below my sig) so I can test things then.
Thanks!
Ehud
P.S. How you can convert a RAID10 to a RAID5 without ages of downtime for data copy:
-
Don’t let your usage size on the array exceed 50%. In other words, if you have 4x16TB drives which are two stripes of two mirrors, do not exceed 16TB. By “don’t exceed” I mean plan ahead and upgrade prior to that point.
-
Use any spare 16TB drive, and rsync all the media data to it.
-
Use a second spare 16TB drive and do the same.
— Downtime #1 starts here —
4. Stop the plexmedia server and do the rsync to the drive in Step #2 until all data is sync’d with no straggling changes. This should take less than 20 seconds depending on content being downloaded, scanned, etc.
- Move the spare from Step#2 to the Server and mount it up with the mountpoints being your media library directory(ies). Pre-lay the USB cable. This is temporary. Pre-setup a new fstab to mount the new drive instead of the NAS but at the same mountpoint. A script isn’t really necessary since one script command is just as good as an cp /etc/fstab /etc/fstab.old && cp /etc/fstab.new /etc/fstab. This should take less than 30 seconds. I’ve done it in 3 … it’s all a question of what redundancy, uptime, SLA, etc. you want to tolerate
— downtime #1 ends here — 60s typical
6. Restart pms
-
Redo the sync to drive in Step #3
-
Reconfigure your array from RAID10 to RAID5 or whatever
-
Sync from drive in Step #3/#6 to new RAID array
-
On pms Server mount up new RAID array on temporary mountpoint, and sync from plex media data to new RAID array.
— downtime #2 starts here —
11. Stop plexmediaserver
12. Redo sync from step #9 to catch any straggling/updated data
13. Return mountpoints to pre Step #5 and remount new array as pms media mountpoints
— downtime #2 ends here — 60s typical
14. Restart plexmediaserver
Of course if you don’t care about the downtime just do what everyone else does:
- Copy the media data to a spare drive
- Shutdown pms
- Redo the array
- Copy the media data to the new array
- Ensure mountpoints and permissions are still good
- Restart pms
This is a lot less work, but step 4 will take a relatively much longer time than the entire process detailed above. Still, 13 steps with 1 cable, 2 fstab changes, 2 spare drives, and 6 rsyncs vs 6 steps with 1 spare drive and 3 rsyncs at a cost of 30-120min (depending on size of data being copied and medium of transmission such as USB3.2 vs 3.0 vs network)… that’s up to you. I hate downtime with a passion, but then again, there are a lot less SPOFs in method 2.
