If you have not already, we suggest setting your Plex username to something else rather than email which is displayed on your posts in forum. You can change the username at https://app.plex.tv/desktop#!/account
Welcome to our forums! Please take a few moments to read through our Community Guidelines (also conveniently linked in the header at the top of each page). There, you'll find guidelines on conduct, tips on getting the help you may be searching for, and more!

[Ubuntu 17.04x64 Hyper-V VM] Complete system hang while transcoding

Robbie_CrashRobbie_Crash Posts: 9Members, Plex Pass Plex Pass
edited April 21 in Linux

Hi there, I'm running PMS on an Ubuntu 17.04 VM named Plex, running on Hyper-V 2016 and after it's been transcoding for a while, the system will become completely non-responsive (no pings, response in the app, unable to switch ttys on the virtual console, locked with two cores at 100%CPU) but Plex will continue to log activity. This happens almost every time that Plex has been transcoding for longer than a few minutes. Hardware is a dual Xeon E5504 with 48GB RAM with 6 NLSAS 10K 1TB drives in RAID 6 on a PERC 530.

I was having similar problems with this same setup after upgrading to 1.5, however that was upgraded along the way from 0.8 or 0.9 something on Ubuntu 15.04 or something all the way up to 17.04 so I decided to destroy the VM and build a new one to try to eliminate the issue.

Everything was freshly installed, no settings are customized, plex is running under plex:plex. All media is stored on NFS mounted volumes, temporary transcode directory is set to /tmp on the PMS VM so as not to try to put it on the NFS volume. Plex is configured with 7CPUs which crosses a NUMA line, but numa=off in Grub, as well as elevator=noop. However I had these issues without elevator and noop settings in grub. It's got 8GB of RAM assigned, 18788MB of RAM is the maximum amount of memory per NUMA node.

I've attached the kernel.log from the start of the freeze til the reset, as well as the PMS log from the same time.

Any suggestions?

Best Answers

  • Robbie_CrashRobbie_Crash Posts: 9Members, Plex Pass Plex Pass
    Accepted Answer

Answers

  • trudgetrudge Posts: 2,288Members, Plex Pass Plex Pass
  • Robbie_CrashRobbie_Crash Posts: 9Members, Plex Pass Plex Pass

    Neither of those settings really has an analogue on Hyper-V. Nothing is being offloaded to the NIC, and no NIC teaming is involved.

  • ChuckPAChuckPA Posts: 17,602Members, Plex Pass, Plex Ninja, Plex Team Member Plex Team Member

    There is no reason to deal with NUMA here. You should be running a simple SMP instance of Ubuntu (no need to play with grub's config) with 'virtual CPUs' the only parameter you define in your guest's config. Please confirm you mean 7 virtual cores (using current terminology) and not 7 physical CPUs (each which can have 4 or more cores in it). Also please confirm, in GB physical system RAM and system swap (Windows swap file and Linux swap partition) size in GB

    The symptoms you describe (complete hang of the OS) is indeed similar to that experienced with ESXi and the VMNET3 adapter. The root cause there was a fault with ESXi and Ubuntu not properly handling any offloaded packets. This also hints of an over-provisioned thin instance.

    Further, there is something else going on. Not being able to console switch (CTRL ALT Fn) , is outside PMS control. Gnome's desktop manager handles this in conjunction with systemd (pty login) and the pty driver of course.

    If you are using hardware accelerated graphics in your VM and Chrome (also with acceleration enabled), Since chrome on the desktop and chrome in the VM are not aware of each other, they will conflict / lock a GPU quite easily. When this happens, you have the lockup you describe. Disabling hardware accelerated graphics at the VM level will prevent this from happening. VirtualBox suffers from the same vulnerability.

    If the above does not resolve the situation, before we go too far here.

    I will need the full logs, Verbose debugging OFF. You may PM them too me. Include the browser link to this thread along with the ZIP file you obtain from Settings - Server - Help - Download Logs

    Please DISABLE Verbose logging until requested

    Please search before posting

    Primary support forums: Linux, Synology, and QNAP

    Please remember to report back. This benefits others.

    Useful links

     Installation and Basic Setup |  Media Preparation (How to name your media files)  |  Linux Permissions 

     Handling TV Specials | Handling Movie extras  |  Nas Compatibility List

     Reporting Plex Server issues | Plex Media Server FAQ | Linux Tips

     

    Other useful guides: Local Subtitles | The Plex "dance" | Synology FAQ | PMS Release Announcements

    No technical support via PM unless offered

    Please remember to mark the appropriate answer(s) which solved your issue.

     
  • Robbie_CrashRobbie_Crash Posts: 9Members, Plex Pass Plex Pass

    Thanks for the info. I've removed the scheduler and the NUMA reference from Grub.

    There were 7 virtual processors assigned to the VM, I dropped this down to 4 last night for testing and haven't seen the problem since (though transcoding has been light since then). The VM has 8GB of RAM, with 8GB for swap as well. The Windows host has 48GB of physical RAM and a system managed page file which is currently sitting at 5888MB. The VHDX is thin provisioned, but the storage on the host has about 1TB free, other VMs on the host are all generally idle.

    I'm running Ubuntu server, not Ubuntu desktop, so there's no Gnome, or hardware accelerated graphics at all. The host system doesn't have a dedicated GPU. It's a Dell T410, using whatever the onboard graphics are. However, the system runs headless, aside from one Windows VM all the other VMs on the box are also Ubuntu server, and are only accessed through the virtual console for installation, and in this case to try to see what's going on when the VM stops responding to network traffic.

    I'm going to force a big batch of transcoding now and will get back to you on the results. If it gets through that batch, I'll set the assigned vCPUs back to 7 and see what happens and transfer logs if needed.

    Thanks again.

  • Robbie_CrashRobbie_Crash Posts: 9Members, Plex Pass Plex Pass

    THis happened again some time between the last post and now. I've left the VM with 4 vCPUs and still 8GB RAM.

    PM'd as requested.

    THanks for your help.

  • Robbie_CrashRobbie_Crash Posts: 9Members, Plex Pass Plex Pass

    Thanks for the pointer. I'll poke around in my kernel logs more and see if I can figure anything out.

    All the advanced features on the NIC have been disabled, so I'm not sure where to go from here. Thanks for giving me a starting point. I'll post back with updates assuming I make progress.

  • Robbie_CrashRobbie_Crash Posts: 9Members, Plex Pass Plex Pass
    Accepted Answer
Sign In or Register to comment.