Out of space issue with /dev/shm for Live TV transcoding

Live TV & DVR
, , ,
1

Server Version#: Latest (1.30.1.6562)
Player Version#: AppleTV (latest)
Tuner Make/Model: HDHomerun Prime
Guide/Lineup name: N/A
Using XMLTV?: N/A
Channel number/Name: N/A

I’ve got a relatively complex setup, but in short I have a Proxmox VM running Ubuntu and within that I have the Plex Docker image running PMS. I pass through an NVidia Tesla P4 for HW transcoding, and that works well. Media files are stored in another VM running TrueNAS Scale, but that isn’t relevant.

I’ve been having issues with the transcode temp directory. In the past, I passed through a NAS folder for this, but I think b/c of it being attached via the LAN, the performance was quite flaky. My VM disks are all on SSD and I really don’t want to destroy those by using them for transcode data.

SOOO, I decided to pass through /dev/shm (tempfs…i.e., ramdisk). I have 32GB RAM allocated to the VM, so 16GB is available to /dev/shm. I pass through this folder as “/transcode” to the Plex docker container, and this is what I see from a console inside:

Filesystem      Size  Used Avail Use% Mounted on
tmpfs            16G     0   16G   0% /transcode

Totally cool. All is good. I see that and /dev/shm on the host filling up over time as I stream TV. Things are pretty peppy. BUT after a couple of hours of streaming a single channel, I get a message on the client saying I’ve run out of space.

So, I’m assuming Plex is just blindly saving an unlimited number minutes into my scrub buffer…until it can’t and then it just goes boom. I have a hard time believing it is this “naive” about this. But I can’t figure out any other reason for it failing.

Any thoughts? Is there a hidden “max scrub buffer size” setting I can tweak? Any other tips for making this setup work as it is intended?

PS - I even tried creating a dedicated transcoding tmpfs/ramdisk thinking perhaps the shared /dev/shm disk was perhaps getting used by another process and causing an issue, but I saw the same results.

2

Plex will buffer the whole session until the program changes in the EPG. This happens so rewind etc works.

A Live TV stream uses an avg of 28Mbit/sec. Give or take. That’s 3.5MB/sec times 60sec equals 210MB times 60min divided by 1024 equals 12.3GB per hour. Assuming a show is lasting an hour.

Times that by how many Live TV streams you have going… that 16GB mount fills up easily.

Now factor in if the stream is 4K resolution. That little mount is nothing.

Make it bigger or use the hard disk instead of RAM.

3

Thanks, though this happens regardless of show changes. The most recent example happened after 2 hours of back-to-back 30-minute shows with nobody else watching. So either the EPG program change info is no longer correct or there’s a bug in Plex.

Also, if it’s transcoding, it’s compressing to h.264 (certainly has to for ATV as it doesn’t support MPEG2). Does Plex REALLY use an NVENC profile that outputs at 28Mbit/sec??? That’s crazy. Should be able to get good quality 1080/30 well under 10Mbit/sec. Even at 10, you’d be looking at 1.2MB/sec or 4.3 GB/hr giving me 4 hours of buffer. It sounds like you may be right given the observed results, but that’s…broken.

No matter what, crashing instead of just trimming your available back buffer when you reach your drive capacity is also…broken.

So I dug deeper into this. It appears that it may be even more “broken” than I thought. Unless I’m mistaken, Plex is storing the original MPEG-2 AND the transcoded h.264 in the Transcode folder.

drwxr-xr-x 2 paul shared 200 Jan 14 17:37 plex-transcode-7dd5ba9f-0f95-4370-8d94-318d1fe9093b/
drwxr-xr-x 2 paul shared 180 Jan 14 17:37 plex-transcode-D366BD2F-A78E-4038-BD9A-98FEDE3DA11C-1473f31a-702f-48e3-a053-71f963df3704/

(transcoded h.264) plex-transcode-7dd5ba9f-0f95-4370-8d94-318d1fe9093b/
-rw-r--r-- 1 paul shared  684132 Jan 14 17:37 media-00048.ts
-rw-r--r-- 1 paul shared  587124 Jan 14 17:37 media-00049.ts
-rw-r--r-- 1 paul shared  711580 Jan 14 17:37 media-00050.ts
-rw-r--r-- 1 paul shared  501772 Jan 14 17:37 media-00051.ts
-rw-r--r-- 1 paul shared  666836 Jan 14 17:38 media-00053.ts
-rw-r--r-- 1 paul shared  971584 Jan 14 17:38 media-00052.ts

(mpeg-2) plex-transcode-D366BD2F-A78E-4038-BD9A-98FEDE3DA11C-1473f31a-702f-48e3-a053-71f963df3704/
-rw-r--r-- 1 paul shared 2291156 Jan 14 17:37 media-00048.ts
-rw-r--r-- 1 paul shared 2440992 Jan 14 17:37 media-00049.ts
-rw-r--r-- 1 paul shared 2267844 Jan 14 17:37 media-00050.ts
-rw-r--r-- 1 paul shared 2025136 Jan 14 17:38 media-00051.ts
-rw-r--r-- 1 paul shared 2255436 Jan 14 17:38 media-00052.ts
-rw-r--r-- 1 paul shared 2454904 Jan 14 17:38 media-00053.ts

So it appears that both the MPEG-2 (@ around 20-25Mb/s) and the h.264 (@ around 5-6Mb/s) are being stored.

AND I just confirmed that they are NOT being cleared out at program change. They are just continuing to pile up. Until it breaks.

That just seems broken to me. Maybe there’s some reason for storing both raw and transcoded copies that escapes me. But letting it build up until it dies…I can’t think of any reason for that.

4

I’m going to bump this as there are definitely two bugs at play. There seems no other support channel for paid customers, I’m hoping maybe a Plex Employee can respond with an acknowledgement at the very least.

  • BUG 1: there seems to be no logic built in to the Live TV transcoder to limit the retention of transport stream (TS) files…they appear to just accumulate until a channel is changed or the stream fails.

  • BUG 2: when transcoding MPEG-2 to h.264, both the original AND the transcoded files are retained indefinitely (see BUG 1), thus increasing the space consumed by a factor of 5 over what seems necessary.

Although I can delay the inevitable by utilizing a much larger disk, this is still a bug that will surface with time. In essence, this is a memory leak.

5

No bugs, just doesn’t work the way you want it to. Plex has never officially recommended using ram for storage and all it does is amp up the number of support threads. A spinning hard drive is less than a $100 dollars and gives terabytes of temporary storage.

To achieve what you are prescribing people would have to give up Tuner Sharing, Time Shifting and Transcoding on the Fly for Closed Captioning.

1 Like
6

What? The main solution is simply to implement a rolling queue that deletes old TS files from the tail of the queue after x minutes or as the queue(s) reach a certain size or percentage of free space. This would in no way impact a single one of the features you listed. Those just aren’t related to the size of the scrub buffer. Certainly not when we’re still talking about keeping many tens of minutes at a minimum.

The lesser issue (storing only the transcoded TS files) I suppose might not be a bug, at least not a breaking one. And fixing it MIGHT impact someone who has a player that can only decode MPEG-2 sharing the tuner with someone who can only decode h.264. Though I highly doubt such a player even exists. Otherwise, the only downside is that all users would playback only the transcoded version. And if that was undesirable for someone who REALLY wanted the less efficient codec, I suppose there could be a toggle.

No matter what, storing temporary files until it blows up is a bug. Period.

7

I tire of stating this over and over: Don’t use a RAM disk as your transcode temp. Seriously, DON’T!

Nearly every case I’ve seen where someone is recommending this includes a serious misunderstanding of how operating systems work. The only legitimate use case is to prevent wear on an SSD but a much better solution is to use a spinning disk (as already stated in this thread).

Your “bugs” are not that in the least. They are intentional design features. The first is the capture buffer (which does have a limit) which is shared by all consumers of that channel. Additionally there are cases where the transcode has to restart and when it does so all the previous transcoded data is lost. Saving the capture buffer still allows seeking outside the range of what is currently transcoded.

8

Thanks for sharing. I can relate. I tire of Plex, as well.

As already stated in the original post, this was needed because performance using the spinning disks was abysmal. Stuttering, crashing, etc. No doubt this was due to the disks being networked, as all 12 of the spinning disks on my server are passed through to TrueNAS. And I DO know that this is something you don’t suggest doing.

And you know how I know that?? Because unlike the thing you apparently tire of stating over and over, this is actually in the documentation (https://support.plex.tv/articles/200250347-transcoder/). Right after the incomplete sentence and surrounded by all of the missing material about how this is also used for Live TV regardless of whether you are actually transcoding, and how it differs significantly from the transcoding of static material (I guess it handles this pretty much the same way…just not the media optimizer).

Here, let me give you hand:

Transcoder temporary directory

Setting this will override the default directory that Plex uses when transcoding temporary files for streaming. It is also used to buffer Live TV streams to allow for rewind and fast-forward of live broadcasts even if no actual transcoding is required. This setting does not affect where transcoded files are stored for background transcoding tasks (sync or Media Optimizer).

You might want to set this if your primary drive has limited space. Additionally, you must set this if you are using our official Docker image or one of the containerized plugins from your NAS vendor.

Please keep in mind the following:

  • This directory must maintain a significant amount of free space. We recommend the following as minimums for each simultaneous transcoding session:
    • Audio File Transcoding: _____ GB
    • Video File Transcoding: _____ GB (or at least 100MB more than your largest video file)
    • Live TV Streaming: ____ GB
  • Because the high number of write operations can significantly impact the lifespan of most SSD’s, we recommend that this directory reside on a traditional spinning disk.
  • This directory must not reside on a network share.

Even better, include the ability to access documentation like this from the admin UI (perhaps even using a little blue help icon with a tooltip or pop-up directly next to the input fields) instead of making your users search dense online documentation and then (far worse) dig through a massive number of “community” posts that are filled with inconsistent and/or incomplete and/or inaccurate information, only a small fraction of which actually provided by Plex employees.

A little bit of documentation can go a long way. This is paid software. Expectations are different that in free and open source alternatives.

Reluctantly back to Channels, I guess. It’s expensive (which is why I decided to give Plex another shot), but it is a far better overall experience for Live TV. It’s definitely not perfect, but at least the Apple TV client embeds an MPEG-2 transcoder, the guide doesn’t constantly go whack forcing you to exit (at least on ATV, AFTV, and Google TV), the TV remote control operates as expected (up/down, channel numbers, jump/prev, etc.), and didn’t come with these issues. Or attitudes.

And sorry but having a live tv stream crash your client after a couple hours with a cryptic error dialog rather than gracefully handling the exception (or even considering an alternative approach that avoids the issue entirely)…is a bug. If your product management is accepting that as “by design” then in the immortal words of Adam Savage, well, there’s your problem.

9

There is no bug. You just don’t like the amount of space the cache is taking up. It absolutely does not store files until it blows up. There is a limit. Your little ramdisk is soooooo tiny. A grain of sand compared to a boulder. Unfortunately for you 16 gig is not enough so I will repeat what I said earlier.

Make it bigger or use the hard disk instead of RAM.

There is zero advantage to using ramdisk because 1) you’re not trying to encode files at the fastest speed possible because it only needs to work as fast as the stream comes in. We are not encoding a 50 gig file where this may save a few minutes. 2) unless you have a lot of RAM you’re going to run into extreme limitations.

Also, what happens when somebody transcodes a 4K movie and you have that ramdisk still in place? You wanna see that thing fill up even faster then transcode a 4K movie. The idea is good but it’s not practical.

10

Thanks and good thought, but the normal transcoding ironically doesn’t use this folder. It actually could be called “Streaming Temporary Folder”, as it seems only to be used when “real-time” streaming (whether or not any transcoding actually takes place). The process for movies appears quite similar to live tv, where it saves transcoded data in “chunks.”

-rw-r--r-- 1 paul shared  975351 Jan 20 15:58 media-02690.ts
-rw-r--r-- 1 paul shared 1004044 Jan 20 15:58 media-02691.ts
-rw-r--r-- 1 paul shared 1106190 Jan 20 15:58 media-02692.ts
-rw-r--r-- 1 paul shared 1093932 Jan 20 15:58 media-02693.ts
-rw-r--r-- 1 paul shared 1108807 Jan 20 15:58 media-02694.ts
-rw-r--r-- 1 paul shared 1179304 Jan 20 15:58 media-02695.ts

That’s a few seconds of 4K Top Gun Maverick (HEVC) transcoded to “medium” 1080 h.264. Even DirectPlay stores ts files in there because they need to ship the video to the client in smaller chunks. Same for audio track transcoding, actually, with chunks of m4a or whatever.

So the only thing that would happen is that the scrub buffer (“capture buffer”) would decrease in size to match the space available and maybe you’d have to re-transcode if you had low space available and you scrubbed far enough back.

Actually, it’d be far better than with live TV because you have the original media and the buffer would quickly refill with new transcoded chunks at whatever position the user scrubs to. This actually appears to happen anyway, as any time you scrub back or forward significantly (presumably beyond the buffer), one or more new directories in /Transcode/Sessions/ are created, the old ones are destroyed, and the process starts anew.

Doesn’t matter. The way it currently works is obviously faultless and the only possible solution, so there’s little reason for me to beat this dead horse.

EDIT: actually I’ve not tested whether this directory is used temporarily for background transcoding (sync & Media Optimizer), so it could conceivably be an issue if that’s how that process works…but the documentation seems to imply otherwise.

EDIT 2: confirmed that it does not appear use this directory for optimization…the transcoding output is stored alongside the original media in an “.inProcess” temporary directory below the /Plex Versions/ directory. So ignore the first edit.

11

Sounds like a feature request and not a bug.

12

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.