Ubuntu 19.04 running 1.16.6.1592-b9d49bdb7

Server Version#: 1.16
Player Version#: ?

Well after running a steller micro box computer (intel NUC), that’s been running for near 9 months without fail, PLEX finally crashed, and will NOT restart.

Where do I start, and how do I fix?

Thanks
-Richard

Manually collect the log files please in a tar.gz (the Logs) directory and I will look.

Thank you… however, I nuked the whole thing, and started with a new ubuntu 19.04 build from scratch. If it fails again, Im going to look into the hardware.

I do have a back up of my entire database though… Im going to look thought that for you and try and post it here.

OK, so it crashed again… this time, all was running correct, until I added in my Wifes photo collection (which is huge). So, the whole thing results in my nuking it and starting the plexserver from scratch again… I guess Im just NOT going to ever be able to put in the old photos on the plex server. (a bummer)

If anyone wants to look at it… here is my last core dump of the database.

http://helix.no-ip.biz:8080/share.cgi?ssid=0k5jfwC

be warned… 3Gig download. Hopefully someone can take a look at it, tell me whats going wrong.

Thanks
-Richard

Not to worry about the download. Getting it now. For future reference. LogNumFiles doesn’t need to be set for more than about 10 now. We’ve increased the logfile size to 50 MB by default (10 MB per file)

About to call it a night here. I’ll look in the morning.

Thanks for the .tar.gz. The reason it was so big was you’d sent me the entire Plex directory I only need the Logs directory. :slight_smile:

You will unfortunately need to redo this one more time. Because of this, I’m going to make an additional suggestion. Use Ubuntu 18.04 LTS (Long Term Service branch). 19.04’s support will end abruptly on April 2020. You’ll be stuck. The really unfortunate part is: If you have a problem with PMS because of the OS, we won’t be able to report it upstream and get it fixed. That puts me in the unfortunate position of telling you that I can’t support it. All Ubuntu even number distributions offer LTS, and all odd number distributions are for the single calendar year only.

Looking at your error specifically points to the kernel’s iNotify service.

Oct 03, 2019 04:25:40.678 [0x7fbcd5ffb700] DEBUG - [Notify] Now watching "/media/shares/Plex_Photos_Sharon/2009"
Oct 03, 2019 04:25:40.949 [0x7fbcd5ffb700] DEBUG - [Notify] Now watching "/media/shares/Plex_Photos_Sharon/2010"
Oct 03, 2019 04:25:41.095 [0x7fbcd5ffb700] DEBUG - [Notify] Now watching "/media/shares/Plex_Photos_Sharon/2011"
Oct 03, 2019 04:25:41.262 [0x7fbc977fe700] DEBUG - HTTP 200 response from GET https://plex.tv/api/v2/release_channels?X-Plex-Token=xxxxxxxxxxxxxxxxxxxx
Oct 03, 2019 04:25:41.280 [0x7fbcd5ffb700] DEBUG - [Notify] Now watching "/media/shares/Plex_Photos_Sharon/2012"
Oct 03, 2019 04:25:41.284 [0x7fbc977fe700] DEBUG - PublicAddressManager: Obtaining public address and mapping port.
Oct 03, 2019 04:25:41.309 [0x7fbcf7dae700] DEBUG - Jobs: '/usr/lib/plexmediaserver/CrashUploader' exit code for process 2383 is 0 (success)
Oct 03, 2019 04:25:41.485 [0x7fbcd5ffb700] DEBUG - [Notify] Now watching "/media/shares/Plex_Photos_Sharon/2014"
Oct 03, 2019 04:25:41.650 [0x7fbcd5ffb700] DEBUG - [Notify] Caught exception on directory.

Am I correct that you have photos well curated and/or have a lot (more than 10,000) photos?
Given this is calling out iNotify, I’m suspicious of the number of directories involved.

The Linux kernel, by default, allocates a notification table to hold 8192 directories. If you have more than that, it throws an exception. To figure out what’s happening here in your usage case, I’d like to suggest the following:

  1. Install Ubuntu 18.04.02 LTS and apply all updates
  2. Get all the network mounts in place and ready
  3. Count (using find /media/shared -type d -print | wc -l or similar) how many actual directories are in use for all your media Plex manages which includes movies, television, music and photos.
  4. If we find more than 8192 are in use, we address that by customizing in /etc/sysctl.conf with a fs.inotify.max_user_watches entry (rounded up to the next multiple of 32768 directories to give you some growing space)
  5. In event it’s simple a number of files issue, then we will have to manage how you add photos. (Plex has always been fussy at adding photos because so much is happening with each photo – thumbail & tagging of each generates a lot of database activity)

If you can get down through steps 1-3 and get back to me with the totals you find, we’ll proceed from there.

Thank you for your very very detailed overview of this… But I have a few questions.

Are we talking about directories of files?? The photo libraries are in the form of the years. Example, There’s a folder called, “Plex_Photos_Sharon”, and in this is a yearly directory, and in each of these is the photos taken for that year.

So, when you say 8192, there’s no way there are more then 8000 photos in each year directory?? Do you mean 8192 in total over these years??

I looked up this /etc/sysctl.conf file… no entry in this for fs.inotify.max_user_watches so Im figuring making one. Maybe something like fs.inotify.max_user_watches=32768?

Kind of a bummer that I can’t get photos on… I loved how plex uploaded photos automatically, it’s one of Plex’s best feature if you ask me.

So, I have to scrap 19.04, and go 18.04 again?? Ouch… I’m assuming that I can tar the Library, re-install 18.04, and restore the Library and it should work?

Thank you for your help!

Before going too far:

  1. Ubuntu releases follow a pattern. 15,17,19 (odd years) no longer receive support from Canonical (the head company) as soon as the next year (16,18,20, etc) is released the following April. This means we can’t get anything fixed by Canonical if there is an OS bug (which often happens). This is why we recommend staying with the even numbered years. You get bug fixes for at least 3 years. 5 years if the LTS (Long Term Service) release.

  2. Regarding your photos, I’m not concerned about how many photos there are. I am concerned about how many directories (folders) you created. If the total is more than 8192, we will make an entry in /etc/sysctl.conf so PMS can monitor more directories for you. It’s a simple task. I have a full How-To written.

  1. Copy… I’ll schedule to go to 18.04

  2. There are only 17 folders in the “Plex_Photos_Sharon” folder.

p.s. on a side note, there are 27,835 items in those 17 folders, totaling 68.96GB.

Thanks

Ok… Now for a few suggestions / questions about how your disk is partitioned?

  1. Do you have a separate /home partition ?

  2. Regarding Plex, Making a backup (tar.gz) of ./Library after you cd /var/lib/plexmediaserver as root will make short work, without losing your PMS installation. I have a nice How-To

Thanks for the response…

All files are located on a QNAP NAS via File Server.

In my fstab is

‘10.0.10.4:/Library/Extras /media/shares nfs4 auto,defaults 0 0’

Hope this helps.
Thanks
-Richrd

that tells me about the file server.
what about the host itself ?

So sorry… Im not sure what you mean… The PLEX server is on an intel NUC computer, running Ubuntu 19.04. The plex hooks into the File Server… There’s nothing more.

I don’t think I understand the question. so, sorry.

-Richard

How are the NUC SSD(s) partitioned?

Do you have one massive partition for root (/) and everything in it or do you have

/, /home. and a few others ?

what does df show?

Filesystem                  1K-blocks       Used  Available Use% Mounted on
udev                         16304056          0   16304056   0% /dev
tmpfs                         3268264       1388    3266876   1% /run
/dev/nvme0n1p2              238797468   20409340  206188144  10% /
tmpfs                        16341308          8   16341300   1% /dev/shm
tmpfs                            5120          4       5116   1% /run/lock
tmpfs                        16341308          0   16341308   0% /sys/fs/cgroup
/dev/nvme0n1p1                 523248       7716     515532   2% /boot/efi
/dev/loop0                      91392      91392          0 100% /snap/core/6673
/dev/loop1                      55936      55936          0 100% /snap/lxd/11985
/dev/loop3                      56064      56064          0 100% /snap/lxd/12100
/dev/loop2                      91264      91264          0 100% /snap/core/7713
10.0.10.4:/Library/Extras 15485782016 7531476992 7953764352  49% /media/shares
tmpfs                         3268260          0    3268260   0% /run/user/122
tmpfs                         3268260          0    3268260   0% /run/user/1000

MODERATOR EDIT: Legibility

With everything being on the / partition, you’ll unfortunately need a full, wiping, installation.

What I’d like to recommend as you do this is:

  1. As Ubuntu offers partitions, let it make a home partition for you.
  2. Of that 256 GB SSD, take about 128 GB away from the / partition (root) and use that to create /home

When done, it will look like this (I have a 1TB SSD and this is dual boot, so my sizes are larger)

My root (/) is 128 GB, and I have given it 300+ for /home. I do this because I have a development system. You don’t need / to be so big. ~64-90 GB will do if you create /home. Let the system take the other partitions it wants.

[chuck@lizum ~.99]$ df
Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/sda5      134152192  84033364  50118828  63% /
/dev/sda3         534180    230788    303392  44% /boot
/dev/sda7      335849244 189426672 146422572  57% /home

Doing this now has separated your data from the OS. Anything you place in /home can be preserved as you go from version to version or even a Re-installation as we are now.

My system is Fedora but will likely be changed to Ubuntu soon. My /home partition will survive intact.

How are we doing on the complexity of the task? You still with me?

ummmm… Im with you, but Im not sure that I can do it. :slight_smile:

Im going to have to think about this, and read it over multiple times.

-Richard

I will try to create / capture screenshots of performing a Ubuntu installation and place it in the Linux-Tips section.

Would that be of help ?

I have a draft of the How-To ready .

I have the screenshots of what the process looks like.
Ubuntu makes it very easy to partition the disk however you want.

On SSD systems, it usually takes longer to define the partitioning than it does to perform the actual full remaining installation.

If you would like to take a look at it (in progress), it’s here.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.