Help: stream is constantly buffering, libmpv sends TCP RESET

[Yohoo]

Help, my playback is literally pausing & buffering EVERY SINGLE SECOND.

What I observed:

• When this happens, there’s a huge hike in the Bandwidth chart.
  

  

• When I check my server with the neghogs utility it reads around 6MBps from the connection.
  

  

• Playback in browser works fine (transcoded). However Direct Play playbacks from the macOS app / iOS app are having this issue. Since both clients have libmpv based players (I saw libmpv in the User-Agent header in the HTTP requests), there’s a chance this is a problem associated with libmpv.

• The problem happens only with a couple of video files in my library. I have a large library of thousands of episodes.

• When I tried to tcpdump the HTTP traffic, it turns out the clients are sending TCP RESET to server ALL THE TIME, and it looks like this caused the playback had to re-transmitting ALL THE TIME, and end up with the buffering & high bandwidth usage problem.

• I’ve been using plex since 2018. I first encountered this problem earlier this year so this might be a recent bug. Also found this Reddit thread that is relatively new.

• I usually have my plex server behind haproxy but the exact same problem happens regardless has proxy or not. (I tested both scenarios). When behind haproxy, I got CD--- state from the logs when this happens – which means, the client terminated the connection while data (HTTP body) was been transmitted. This corresponds to the TCP RESET that appeared during tcpdump.

My setup

• Plex Web Version 4.76.1
• Plex Server Version 1.26.1.5798 (latest LinuxServers docker image)
• Plex iOS Version 8.2
• Plex for Mac Version 1.44.0.2981-e2d72ac2

Media info

Codec H264
Bitrate 1120 kbps
Bit Depth 8
Chroma Location left
Chroma Subsampling 4:2:0
Coded Height 1088
Coded Width 1920
Color Primaries bt709
Color Range tv
Color Space bt709
Color Trc bt709
Frame Rate 23.976 fps
Height 1080
Level 4.0
Profile high
Ref Frames 3
Stream Identifier 1
Width 1920
Display Title 1080p (H.264)
Extended Display Title 1080p (H.264)
Codec AAC
Channels 2
Bitrate 128 kbps
Audio Channel Layout stereo
Profile lc
Sampling Rate 44100 Hz
Stream Identifier 2
Display Title Unknown (AAC Stereo)
Extended Display Title Unknown (AAC Stereo)

[OttoKerner]

Is this an mp4/m4v file? If so, you might wanna read: Help, Plex LAN traffic at 280Mbs - #5 by OttoKerner

[Yohoo]

Hi OttoKerner,

Converting to MKV did solve the problem.

For reference, I used ffmpeg for the conversion:

ffmpeg \
-i input.mp4 \
-vcodec copy \
-acodec copy \
output.mkv

output from ffmpeg:

ffmpeg version 4.2.4-1ubuntu0.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)
  configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'input.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.20.100
  Duration: 00:23:32.08, start: 0.000000, bitrate: 1253 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 1120 kb/s, 23.98 fps, 23.98 tbr, 90k tbn, 47.95 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 127 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
Output #0, matroska, to 'output.mkv':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.29.100
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 1120 kb/s, 23.98 fps, 23.98 tbr, 1k tbn, 90k tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(und): Audio: aac (LC) ([255][0][0][0] / 0x00FF), 44100 Hz, stereo, fltp, 127 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame=33854 fps=0.0 q=-1.0 Lsize=  215842kB time=00:23:32.05 bitrate=1252.2kbits/s speed=1.52e+03x
video:193112kB audio:22064kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.309594%

Side-by-side comparison using the “Play Version” feature and I can confirm the MKV file plays flawlessly, while the MP4 file clogs everything like before.

Tested with both iOS & macOS apps they have the exactly same behavior (so it’s very likely a libmpv thing)

The bandwidth chart looks calm when playing MKV file:

@OttoKerner Thanks for the tip. MKV conversion worked like a charm. But are we going to fix the root cause eventually? Since MP4 is still a widely used format. From the information I’ve collected it looks like a issue in libmpv – perhaps is a bad timeout setting on HTTP connections?

@anon18523487 Just FYI since your analysis in this thread explained my bandwidth burst issue perfectly.

[OttoKerner]

@Yohoo

But are we going to fix the root cause eventually? Since MP4 is still a widely used format. From the information I’ve collected it looks like a issue in libmpv – perhaps is a bad timeout setting on HTTP connections?

The root cause is your file. It is not the mp4 container format as such, but rather an incorrect makeup of this particular file. This is also explained in the linked article. Repair the affected files, as they are not suitable for streaming applications.
If you can remux the files to mp4 (but this time with proper interleaving) then there is no need to switch the container format.

[Yohoo]

proper interleaving

But in that case, the MKV file I got should suffer the same problem, right? Since I literally used -vcodec copy option and it should’ve carried any possible problems over into the MKV.

[Volts]

Not necessarily. Interleaving is about how the video and audio packets are stuffed into the file, not the contents of the packets.

Bad interleaving/muxing for streaming:

VVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVAAAAAAAA

Better interleaving/muxing for streaming:

VVVVAVVVVAVVVVAVVVVAVVVVAVVVVAVVVVAVVVVA

The client needs to have both Video and Audio packets available for playback.

In the first case, the client has to ask the server for packets at the beginning of the file (to get some Video packets) and then at the end of the file (to get some Audio packets) and then at the beginning of the file (to get some Video packets) and then at the end of the file (to get some Audio packets), and so on. This defeats the client-side buffer cache.

In the second case, the client can just ask for the next few packets, in a nice orderly manner, and it will have the upcoming Video or Audio packets it needs already cached.

[Volts]

@Yohoo the ffmpeg command you used may also be sufficient to rebuild a well-interleaved .mp4, just change the output filename.

If it isn’t, there are a few other easy things to try.

[Yohoo]

You’re right, ffmpeg straight to an mp4 file worked as good as mkv.

I noticed in ffmpeg log says muxing overhead: 0.532856% so it must be what you’ve mentioned for creating a well-interleaved file.

Thanks for the information. It’s nice to learn something new.

[Yohoo]

So in conclusion, while the clients could have done a better job buffering the abnormal video files, we can still do ffmpeg -vcodec copy -acodec copy -i problematic_video_file.mp4 fixed_video_file.mkv as a workaround.

Since converting to mp4 is a speed=809x and for mkv speed=1.52e+03x (numbers are base on my single-shot tests in my library stored on HDD), this operation is fast enough to deal with large libraries – provided the user is able to write some sort of scripts to feed all problematic video files through ffmpeg.

[Volts]

Alternatively, turn off Direct Play in the clients.
(Leave Direct Stream enabled if that option exists.)

If Quality is set to Maximum or Original the result should be an on-the-fly remuxing of the requested video & audio streams into a new container.

[gbooker02]

@Yohoo

I noticed in ffmpeg log says muxing overhead: 0.532856% so it must be what you’ve mentioned for creating a well-interleaved file.

No, this is the overhead of the format.

Take as an example where a file has all the video data at the beginning and all the audio data at the end. To play this, the client must read some video data, seek to where the audio data is, read some there, seek back to the video, read some, seek back to the audio, etc. There’s a bunch of seeking back and forth. A properly interleaved file has the audio and video data mixed together in the file so that the video at time 1:00 and the audio at time 1:00 are right next to each other. So the player doesn’t need to seek at all but rather can just linearly read the file.

It’s the seek back/forth that’s causing the bandwidth usage and problems. Seeking back/forth on a file stored on your PC isn’t really much of an issue (even with spinning HDs) but over a network it’s a killer and wastes a lot of bandwidth.

When you run the mkv conversion or the ffmpeg -vcodec copy, that command will do the many seeks required on the file and give you a resulting file that is properly interleaved. It’s copying the audio/video data as it is in the original file but placing it in proper positions in the file so the client playing the resulting file doesn’t need to seek all over the place to play it.