How come this happens to me :'(

General Discussions
1

Hey guys,

I’m running my PLEX server on a VM, running on ESXi 6.5, on a DELL R710 with 2x Intel Xeon L5520 CPUs.

It’s running on Ubuntu Server 16.04.

Here is me launching 1 transcoding stream.

I can’t launch more, it makes everything buffer… I’m lost here.

My VM has 8 vCPUs, 12 gigs of ram…

I’ve activated fast transcoding preferred and hardware acceleration in the server’s options without success. My other VMs are using 1 vCPUs and I have 16 in total, so at most only 12 vCPUs are used at the same time, this way I know I’m not surapprovisionning…

I read everywhere on the internet that PLEX on a VM, on ESXi, on a R710 works just fine, people can have multiple users transcoding at the same time without sweatin’…

Please help me :smile:

Thanks!

2

Some more informations :

My PLEX VM’s specs :

801×254 14.2 KB

My R710 pCPUs :

607×158 3.88 KB

Transcoding an .mp4 :

967×517 168 KB

Transcoding an .mkv :

967×517 179 KB

3

Before diving down the hardware & configuration rabbit hole,

What is the source media? HEVC? HEVC HDR ? 1080P or 2160P?

4

@ChuckPA said:
Before diving down the hardware & configuration rabbit hole,

What is the source media? HEVC? HEVC HDR ? 1080P or 2160P?

Hi, thanks for taking time to answer me!

All my medias are 1080P - H264.

5

please do the following:

  1. make sure the server is otherwise quiet
  2. Settings - Server - General – DEBUG logging only (NOT verbose)
  3. Start playback of one your movies showing this problem and let it play for 15 seconds then stop
  4. Collect the logs. (Settings - server - help - Download Logs) Attach the ZIP file it gives you
  5. Also, Hover over the movie -> Get Info -> View XML. Copy & paste the XML here with your logs
6

@ChuckPA said:
please do the following:

  1. make sure the server is otherwise quiet
  2. Settings - Server - General – DEBUG logging only (NOT verbose)
  3. Start playback of one your movies showing this problem and let it play for 15 seconds then stop
  4. Collect the logs. (Settings - server - help - Download Logs) Attach the ZIP file it gives you
  5. Also, Hover over the movie → Get Info → View XML. Copy & paste the XML here with your logs

There you go!

XML : `This XML file does not appear to have any style information associated with i - Pastebin.com

7

In looking at the loading with your logs:

  1. The transcoder is active.
  2. It appears to be transcoding (burning in) the subtitles.
  3. What are the player’s settings regarding subtitles? “Automatic” or “Burn Always” ??
8

It’s on automatic !

9

let’s do this…

  1. Stop PMS
  2. clear out the HTTP caches (the agents) sudo rm -rf "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Caches/*"
  3. Start PMS

This will clear out the agent caches… My intent here is to unwind any backlog until we find the trigger

10

Alright, done! Do you want me to send you another log file now?

11

See how the server behaves this way.
You showed what appeared to be one process working at full capacity.
I’m trying to isolate the root cause.

Some obvious questions:

  1. New music added? (this requires audio analysis which is CPU intensive)
  2. New moves/episodes ?
  3. New Photos?
12

1082×462 174 KB

1503×333 79.5 KB

Here is how it behaves now. The first “little mountain graph” is all streams without subtitles. I have had a few buffering. Then the big one when I enabled subtitles on each of them.

  1. No, I do not use PLEX for Music.
  2. Yeah, I have a separate Qbit server that is linked to Sonarr & Radarr, adding new content constantly (movies & TV shows) but I’ve turned it off for our experiments!
  3. No, I do not use PLEX for Photos.
13

Okay so, I’ve been running 6 streams for the past 10 minutes now :

  • 4 of them were transcoding, 2 on WAN, 2 on LAN
  • 2 of them were direct play
  • Once they all got going, I had no buffering at all in 10 minutes
  • I noticed the transcoder state was either “Transcoding speed 1 or more” or “Throttling” which is a good thing
  • The only problem that I noticed is that, once I had the first transcoding stream running and I started the second, the first one buffered for a few seconds. Then same for the second and the third, and then the third and the fourth.
14

Please remember, Video transcoding is done by the GPU (ASIC) on those machines with QSV support. Subtitle burning (whatever may cause it) is done only by the CPU.

If you’re seeing the transcoder with speed > 1.0, that tells us it’s generating video faster than 1.x playback speed .

I’m starting to wonder what impact the other VMs are having on your machine because the data you present does not justify the stuttering you’re seeing unless a) physical resources are running low b) network outbound to the players can’t send fast enough.

15

@ChuckPA said:
Please remember, Video transcoding is done by the GPU (ASIC) on those machines with QSV support. Subtitle burning (whatever may cause it) is done only by the CPU.

If you’re seeing the transcoder with speed > 1.0, that tells us it’s generating video faster than 1.x playback speed .

I’m starting to wonder what impact the other VMs are having on your machine because the data you present does not justify the stuttering you’re seeing unless a) physical resources are running low b) network outbound to the players can’t send fast enough.

Hi!

Again this morning, a friend of mine was watching a 1h episode and sent me a message, again, telling me “after 40 minutes, black screen, yellow loading circle, nothing happening. Gotta cut the stream and restart it.”

I just don’t get it. Yesterday I was able to stream 4x transcoding without interruption, this morning my friend is using PMS alone, he’s not even transcoding, direct play, and he gets cut off randomly. It appears, when I look at PMS stats through Tautulli, that his client switched to transcode mode randomly and that’s what caused the cut-off.

Now, regarding your questions about the other VMs, here you go :

Those are the VMs running on my ESXi server.

Qbit Server : 2 vCPUs, 2 gigs of RAM
Tautulli Server : 1 vCPU, 1 gig of RAM
PMS : 8 vCPUs, 12 gigs of RAM
VPN & DDNS server : 1 vCPU, 1 gig of RAM

My R710 has a total of 16 vCPUs available and 24 gigs of RAM available.

I’m getting more and more confused about these random cut-offs which are really unpleasant for my users… I just can’t understand why it happens and it bugs me so much!

(PS : I have 1 gig optic fiber to the home and my friend also does, rj45 cat 6 on both ends, 900Mbps download and 500 Mbps upload speedtests on both ends, so network ain’t the problem here :pensive: )

16

Here is my friend’s player setup (he is using Plex app for Windows, updated to the last version as myself) :

1012×494 428 KB

He told me how the buffer happened and I find that odd so let me tell you :

Suddenly, image freezes and yellow loading circle starts turning. He then got up from his fat ■■■■ and went for a coffee. When he returned like 5 minutes later, STILL buffering, at the exact same spot, nothing had changed. He closed the Plex app, relaunched it, resumed his stream and it went playing instantly.

This is freaking odd.

17

Alright, it did it again, and this time I noticed it on Tautulli, here is how it looked :

Ot8dNY6.png1063×311 103 KB

Looks like the stream doubled right when it started buffering, wtf is goin on

Seems like, no matter if PMS is transcoding or not, streams will just randomly stop after a few minutes…

18

Uh… Seems like my PMS is loosing a lot of packets…

Number of active TCP connections :

Okay, I think I’ve narrowed it down to a network-related matter after all.

At 12:02, my friend sent me “Oops, plex did it again”.

I looked at Netdata, searching for answers, and… here is what I found. I mean, it’s pretty logical, when you think about it : player starts to buffer but doesn’t resume alone? It buffers without limits? Seems like a lost packets issue…

Now to know where it comes from. I’d need some help over here! :smile:

19

4MFrBpR.png1585×196 16.2 KB

Noticed that right when it stops working, my server’s TCP memory got “resetted”, seems like. Should I maybe tweak my Ubuntu Server, allowing more RAM to this TCP memory?

20

TCP network tweaking?
Are you really confident you know what you’re doing? (that stuff is REALLY tricky)