Plex crashing server - out of inodes on tmp

Server Version#: 1.31.2.6739-a87e876bd
Player Version#: N/A
Ubuntu 22
I recently had/have an issue that plex is creating tens of thousands of thumb-xxxxxxx.jpeg files in the tmp folder, the only way to recover is to boot into recovery and delete the folder that holds them (there are generally too many to use rm *.jpeg). They seem to be being created post the 4 am maintenance run - I created a new tmp disk with formatted with “-T small” so it would be optimized for small file usage but I am still getting swamped. I have just reset the tmp again and will turn off any automatic processes as it only fills the disk overnight.

Has anyone any ideas about this, have I done something stupid or perhaps is there a way to move the PMS tmp folder to some other path so only PMS brings itself down each night.

This is a known problem and it appears (:crossed_fingers: ) the correction is coming in the next release.

For now, You can move the TMP folder PMP uses by customizing your configuration.

Using:

Set an overide Environment variable for Environment="TEMP=/pathname"

Reload the service definition
Restart PMS

Thanks, just for a bit of clarification on this, the files it is creating are actually temp files and can be removed or are they some sort of cache files that PMS needs? I have already moved the /tmp to its own partition now and given it quite a lot of space and inodes so it does not bring the system down at the moment.

Do you have some kind of bug reference for this so i can read more on the issue?

They are temp files which can be removed.

What you’re seeing is an unfortunately “development OOPS” where the software failed to clean up after itself.

I looked at the changes made for 1.31.3 and see that Engineering has fixed that.

I prefer to keep /tmp as the RAMFS (tmpfs) as the kernel wants.
That’s why I suggested setting the environment variable in PMS startup to make PMS move elsewhere.

That way, PMS moves to someplace safer and you keep your fast /tmp partition as it’s intended.

How soon will 1.31.3 get pushed out, do you have any idea?

I don’t know where this falls in relationship to the code-freeze line.

It looks like it made it but I won’t know until after tomorrow’s team meeting.
I suspect they will get it in.

If it hasn’t made the cut, I will ask they do include it.

OK, thanks, I look forward to upgrading to something that does not fill up my tmp

This is occurring on my PMS as well, I changed override.conf to change TEMP but it appears not to be working. is there a way to specify TWO environment vars in override.conf?

Environment="TEMP=/media/nas/plexdata/tmp"
Environment="PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR=/media/nas/plexdata/Library/Application Support"

I suspect this syntax is wrong, perhaps:

Environment="TEMP=/media/nas/plexdata/tmp; PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR=/media/nas/plexdata/Library/Application Support"

@plex321a

Turn off Credit Detection for the interim.

They are working on fixing this cleanup.

I had the syntax wrong - this is the workaround that works for me:

Environment="PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR=/media/nas/plexdata/Library/Application Support" "TEMP=/media/nas/plexdata/tmp" "TMP=/media/nas/plexdata/tmp"

I probably don’t need the TMP= but i set it regardless

@plex321a

Caution - You don’t want Transcoder temp on a network device.

All Dolby audio requires file locking. File locking is not guaranteed on network storage.

It might not fail at first but it will fail.

Gotcha! The path is misleading - a legacy name – which today is actually SATA/ext4 drive. Without the TEMP, pms was scribbling to a root fs with limited space

@plex321a

You can always create a service override (/etc/systemd/system/plexmediaserver.service.d/override.conf)

and specify TMP to be wherever you want it.

I know most distro installers think root only needs to be 16-32 GB. (I run 128 GB).
(You can customize the size of root when you next do a fresh instal of the OS / create the partitions … ( /boot/efi, /, swap, /home)

something like this:

[chuck@lizum Downloads.2013]$ df
Filesystem     1M-blocks   Used Available Use% Mounted on
/dev/nvme0n1p2    124870  30533     94338  25% /
/dev/nvme0n1p4    789911 541984    247927  69% /home
/dev/nvme0n1p1       499     26       474   6% /boot/efi

As for moving TMP,

yes, i am using override.conf - i was just oblivious to the Environment= syntax for systemd with more than one ENV var (that is Environment="A=fooA" "B=fooB" "C=fooC") once I had that I was GTG.

BTW: nice set of Linux Tips!

1 Like

I am currently having this issue, I’ll be turning off all automatic processes, is there an ETA for this fix?

Greetings!
Any update on this? I have found this to be the culprit of causing instability on my unRAID system as when the issue occurs fully uses my systems memory and locks up.
Thanks!

1 Like

Just came to say I stumbled upon this thread because I’m on version 1.32.4.7195 running in a docker container and I too am having this issue. If this is an issue on older versions as well I guess I may just disable credit detection. Also wanted to add it looks like this is happening AFTER credit detection runs. I will watch the credit detection run and see the inodes for my tmp directory increase and eventually decrease when it shows it ends in the Plex dashboard. But soon after it ends CPU usage stays high and the credit detection folder continues to increase until inode exhaustion reaches 50% at which point the files are not removed.

1 Like

Any update on the status of this bug getting fixed?

There’s no word yet from Engineering about when a fix will be available for release.

The testing I’ve done shows it is cleaning up after it finishes.

I would like to ask –

Does this happen when ingesting a lot of media (fresh library load) ??

Last time was with an existing library when I finally noticed all the folders. This week it was with a new re-install on unraid/docker. One folder has upwards over 6,000,000 jpeg files. I don’t really know for sure how many there are but I’m currently trying to delete the first folder and the naming is up to “thumb-6853822” so far. Unraid struggles with small file sizes so it times out when I try to load the folder in gui. Currently trying to delete through mc. Last time I had to reformatted the drive to get rid of all of them.

I have turned off credit detection so it doesn’t happen again now after I found this thread.