Plex Transcoder RAM Drive Experience
I was experimenting with some RAM drive options for transcoding on my Plex server today and found some interesting take-aways through experimentation I wanted to share with the community.
- Plex does it's own transcoding garbage collection mid-stream.
- Plex tries to keep at least 100 MB of free drive space at all times.
- The size of the RAM drive only needs to be big enough to support:
- The number of simultaneous streams you want to be able to transcode
- The amount of cached playback you want to retain during the transcoding process
- for rewinding without restarting the transcoder process (which kills the current transcoder session and starts a new one)
- Performance improvement was modest-to-negligible in comparison to transcoding to a "slow" consumer grade HDD.
- If the stream is paused while the transcoder continues to process the media, and the RAM drive fills up, the transcoder with fail. The playback (transcoder session) will need to be restarted and all that caching will be for nothing!
How I tested
I made a 150MB NTFS RAM drive and played a x265 encoded 5Mbps video hosted on a locally JBOD pooled NTFS (separate from where Plex is installed) to a Roku client. This forced Plex to transcode the media to h264, so I can test how the transcoded stream was handled. My server is on the edge of providing smooth playback (Transcode speed ~= 1) for the particular media file so it was a good test for identifying where hiccups in the transcoding processing came. I also recreated the RAM drive a number of times with some other options (dynamic memory allocation, only allow physical memory [AWE, no pagefile]) and another size (1 GB) to see if there were any differences in the experience or file system handling. With all these options, the size of the resulting transcoded media would be larger than the RAM drive.
I first tried to stream the media live (as fast and as the transcoder would give it to me) and observed the filesystem and transcoded stream pieces (.ts files) as they appeared on the RAM drive. I found that Plex would keep as many pieces in the session as possible before removing the oldest viewed pieces to make room for for the new pieces. This garbage collection process seemed to kick in when there was ~100MB left of free space left on the drive. At this point, Plex reclaimed a fair amount of memory all at once (~300MB) of early stream pieces. It then filled this space up again until there was 100 MB free and did the same.
Next, I started the transcoding session and waited for the RAM drive to get ~95% full before starting playback. Even though space was getting close to full, Plex seemed smart enough to know that the first few pieces of the stream were not yet consumed, so it didn't delete them. Once started, the experience was close to the first test were Plex kept as many old pieces of the stream around as it could while continuing the transcode.
In both these cases, memory needed to be reclaimed by Plex to continue the stream. I noticed that this reclamation was also somewhat smart so I wanted to look at it a bit more.
Using a larger RAM drive (1 GB) I started playback and about 20 minutes in I paused the stream. This was long enough such that the earlier pieces of the media needed to be removed while live streaming to make room for the next pieces, but since my server wasn't very good at streaming the media, what I was watching right before pausing was the newest piece of the transcode. So I could have rewinded a bit and still had the pieces in the cache, but I didn't to see what happened with the garbage collection process. I noted that again when the free space got to 100MB, Plex did it's garbage collection and reclaimed a large chunk of memory as expected. I waited and noted that as the space filled back up to the 100MB of free space, Plex started reclaiming smaller pieces; ~15-30 MB at a time. Seemingly, Plex is trying to balance how much memory it reclaims with how much rewinding space it leaves for the user. It continued to take these smaller chunks until the transcode caught up to the current playback position and continued to fill the drive keeping the oldest piece to continue the playback.
I continued watching the free memory on the drive with the media paused. When the RAM drive was completely filled to 100% usage, the transcoder again seemed to recognize the earlier pieces were still needed so it didn't delete them, but instead, it killed the whole session providing the error. "Conversion failed. The transcoder crashed or failed to start up.". I'd like to think this is an edge case and that a better option for the transcoder would be to go into an idle mode waiting till there were consumed pieces before continuing. Maybe a dev sees this study and adds a ticket for a patch. Maybe not. Know this is a limitation to transcoding with a small-ish destination drive.
In all test cases, the transcode was CPU limited in my circa 2007 server. I have been transcoding to a spare WD Green drive (5400 RPM) to save SSD wear on the system drive. Also, all the metadata in my Plex library makes it too large for my system SSD, but I probably wouldn't house it there anyway. After over 3 years of Plexing this way, it hasn't been an issue, but I was curious if I gained any performance with the RAM drive. Apparently, I did ... all 0.2-0.3x of it. Pretty negligible ... possibly in the noise of a streaming experience, but I'll take it. If you're transcoding to an SSD (which is reasonable, I don't want to start that flame war here), you likely won't see any improvement.
Testing with both static and dynamic memory allocation for the drive yielded the same results as far as transcode speed performance. Though, there was not much else going in the server at the time ... so it's feasible that the memory was almost sequential in dynamic mode anyway. So unless you just want to have a known amount of memory for Plex right off the top ... there was not much reason to use static allocation.
The one thing that is notable that I haven't seen anywhere else, due to Plex trying to be smart, if you opt to use a RAM drive for the transcoder (or any small drive really), things like Plex-Free-Mem and PlexTidy are not required. There's no reason to auto-manually do what the software does for itself.
I can not speak to multiple streams because I didn't test that case. I imagine that the magic free space number is 100 MB to allow a few streams some leeway to do garbage collection individually and not completely fill the drive prematurely. This is pure speculation though on my part.
IMO, even though this is really easy to set up, if you're using a HDD, this is likely more trouble than it's worth. Speed gains are modest, and the caching constraints I noted can make the experience less than stable. If your using an SSD and are worried about excessive wear, or are physically limited (# SATA ports, power load, etc.), this is an option without introducing another piece of hardware to the system.
Though, I think the best solution is probably to just get a really small SSD that you don't care about and use that.
I hope this was helpful to someone!
OS: Windows Server 2012 R2
Plex Server: 126.96.36.19929
RAM Drive: ImDisk