I was “optimizing” some 4K movies I had on my server. I noticed that as time went on, the plex transcoder process continued to consume more and more memory until it had eaten through all 6GB of my RAM, and 500MB of my swap. I am running on linux, and was using the Optimized for TV preset. Is this a known bug or is this something new? What information should I provide to help out?
Server Version#: 1.13.5.5291
Player Version#: 3.65.1
Linux distribution and version please?
Log files please?
Linux Version: CentOS Linux release 7.5.1804 (Core)
Kernel Version: Linux 3.10.0-862.9.1.el7.x86_64 #1 SMP Mon Jul 16 16:29:36 UTC 2018 x86_64 GNU/Linux
Log Files (not a new enough account to upload): Logs
In general? QEMU, unless you’ve enabled GPU passthrough from the host is going toto be software based and needs more memory thereby needing a good 4GB of memory for Plex use, 8 GB is preferred and what I recommend for NAS systems for 4K transcoding.
If you’ve only given 6 to the VM, Please do increase.
My VM setup is pretty simple tbh. It’s running libvirt with the guest under KVM. I have 6 of 8 cores dedicated to this VM with currently nothing else running on the host. 6GB of 12GB is available to the VM. Currently, on the VM itself, I’m showing a write speed of 55MB/s, and 307MB/s read speed verified by the following commands:
[ian@ian-primary ~]$ dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 19.4428 s, 55.2 MB/s
[ian@ian-primary ~]$ dd if=/tmp/test1.img of=/dev/null bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 3.49688 s, 307 MB/s
Personally, I don’t think those are terrible speeds, and according to the server’s IO wait (which is around 10% of the total core allotment), it doesn’t seem to be the bottleneck here, but I may be mistaken.
It seems that the OOM reaper came and killed the plex transcoder near the end there, so I will have to re-encode. This time I’ll enable debug logging and get back to you on that.
EDIT: The odd thing doesn’t seem to be the amount of memory used, as I’ve run other transcodes that start at about 1GB - 2GB throughout the transcode. The issue here is that it seems to be a memory leak, as the process starts low, and as it continues it grabs more and more RAM until it OOMs.
EDIT 2: Forgot to mention, that the movie in particular that triggers this issue is “Ready Player One”, the other process is direct playing except for audio transcode, and is using about 11% of the total 600% available CPU.
With nothing transcoding, from the host OS, please show me free -m
then show me the same output from within the VM.
Host OS:
ian@1uracksrv:~$ free -m
total used free shared buff/cache available
Mem: 12014 10309 243 114 1462 948
Swap: 0 0 0
Guest OS:
[ian@ian-primary ~]$ free -m
total used free shared buff/cache available
Mem: 5805 138 2051 12 3614 5348
Swap: 1019 119 900
According to top and ps aux, the guest is currently reserved and “occupying” 50.1% of the Host’s available RAM. I forgot that I actually had another VM on this machine that is running, which has started to occupy RAM but isn’t actually doing anything from within the guest. I used it to test the ballooning memory system that libvirt uses for guests.
The transcoder is currently sitting at about 13.4% usage on the guest OS and rising. I’ll let you know when it either dies or completes.
EDIT: Totally forgot to stop the transcodes before performing free -m, updated with correct values.
Another point to consider, Different codecs have different memory requirements based entirely on what the target codec is.
I presume HEVC HDR (based on your CPU load) is the most difficult and demanding of all resources when done in software.
Is that available or installed? It’s showing as if “installed”.
ian@1uracksrv:~$ free -m
total used free shared buff/cache available
Mem: 12014 10309 243 114 1462 948
That’s 10 of 12 GB used just sitting there idle. Not a healthy position to be in.
To be certain of what’s happening, please obtain the XML for the file in question. Please let me see the particulars of all the datastreams in it. Also, what are the optimzation settings? Burning subtitles, etc.
Would you mind pointing me to where I can generate this XML file? I’ve never done so in the past…
And the high usage on the host is because the ballooning memory system in libvirt doesn’t start to free unused memory in guests unless it’s needed by another guest or by the host OS. When I somewhat tested this function with a simple C program designed to OOM the guest, then OOM the host. When the guest needed the memory it allocated more from the host, and when the host needed memory it freed the unused memory from the guest to allow it to be allocated on the host.
Sure.
Hover over the item, exposing the Ellipsis, click it -> Get Info
Get Info gives the popup. Lower left corner you will find “View XML”
I only need from <media> through </media> tags . Above, is the filename info (not needed). Below is the genre and metadata (also not needed)
It looks like you were right, it is 4K HEVC. Here’s a link to the XML with the non <media> tags removed. Link
If you look, it’s Main 10 and bt2020nc color space, indicating HDR. That is currently the toughest codec to decode in software on the planet (it was deliberate too). Add to that you have the image-based subtitle planes in there. (PGS)
All this is taking memory, because the transcoder has to allocate memory for all of them .
My best advice to you is:
Yeah, looking at it now, it looks like it’s picked up every single language of subtitle + audio. I definitely don’t need it in Swedish. I’ll go ahead and remux those out of the file and report back.
Out of curiosity, how does Plex determine which version of the file to transcode when selecting “Optimize”? I am attempting to use my muxed version, but I’m not sure if it’s picking the right version.
I too have seen this in the last few versions of the PMS. This did not happen in the past. In my case the 4k files are being remuxed (Direct Streamed). The only thing that is being transcoded is the audio.
Here is a graph of what is happening when watching a 4K movie. It goes from about 4GB to over 10GB of memory usage.
Edit: I too am running Centos 7 in a VM
That’s almost the exact behaviour that I’ve been experiencing. As a software developer I would understand if the RAM usage stayed constant at a high usage, but since it seems to be gradually increasing as the transcode occurs that would indicate to me a memory leak. Starting at the beginning and watching through the end the process incrementally gets to take up more and more RAM until it OOM, but if I start near the end the RAM usage stays low as if it was just starting.
I would agree. I did not see the behavior in the past with 4K this seems to be something that has happened in the last 2 months according to my graphs.
With that type of data in hand, the best thing for me to do is hand this case off to someone better.
@sa2000 , can you assist please?
All:
I see a developer’s note about this. I’ve sent a message to ask if the concerns here are the same as was just fixed.
When I get his reply, I will relay here.
