I brought up this subject in another thread but would like to make it official here. Please create a benchmark which can be used to ascertain the actual performance of hardware in PMS. I will quote myself from the other thread here to make my case:
@techguy888 said:
After doing some more research on Passmark results of various Xeons and considering an upgrade to an E5 2683 V3 chip (14 cores, 2.5GHz all-core turbo speed), I have to once again question the validity of using Passmark as an analog to Plex Media Server transcoding performance. My chip, a 5820k @ 4.4GHz achieves a score of 16836. Not bad for a mere 6-core chip. How about that 2683 v3? 17627, a mere 4.6% performance improvement PassMark - Intel Xeon E5-2683 v3 @ 2.00GHz - Price performance comparison
How about a balls to the wall 2699 v4 22-core 2.8GHz all-core Turbo chip? 23344. PassMark - Intel Xeon E5-2699 v4 @ 2.20GHz - Price performance comparison Better, but still only a 38.7% improvement for a massive 3.66x core count. If we ignore any minor IPC differences between these chips (Broadwell is only a couple % faster than Haswell for all but the most specialized tasks) and we perform a basic “total core performance” calculation wherein we mulitply the number of cores by the clock speeds, we arrive at figures of 26.4, 35, and an incredible 61.6 for the 5820k, 2683 v3, and 2699 v4, respectively.
Now I’m skeptical. We all know that video transcoding is a highly parallel task. How can Passmark, which seemingly scales poorly with respect to core count beyond a certain point, be a good analog for this task? Maybe it was back when this rule of thumb was created, but I contend that it is not today, given the results I have provided you above.
At this point I would like to reiterate my stated desire for a true Plex benchmark built by the devs. At the very least, I would like to know if there are limitations to the scalability of the transcoding engine with respect to core count. Do > 6-core chips make sense for Plex?
Is there a dev that could give some feedback to this? I don’t expect someone to just come out and say “ok, benchmark good idea, you’ll have it tomorrow” but it would be great to get a dialog going on this subject.
A developer probably will not give feedback unless this gets some traction from users wanting it… which doesn’t seem to be the case either way.
As for me… Meh… not needed
Then it would be great to hear from other people who want to figure out just what their system is capable of if that’s what it takes to get a response from the dev team. I can’t believe I’m the only person who has invested thousands of dollars into this endeavor, I also can’t believe I’m the only one that wants to know if I have the right hardware for the job.
You know if you have the right hardware if the hardware is doing what you ask it. I actually see the problem you are talking about but I don’t find it Plex’s job to provide a benchmark. As passmark is widely used for raw CPU-Power (which still is but maybe won’t be in a few months (Integration of QuickSync and HW-accelerated Transcoding) the most important factor when transcoding) Plex provided a rule of thumb.
But generally speaking Plex utilizes FFMPG for transcoding. So I’d suggest you check for an ffmpg benchmark and you probably will find your answer.
As to all the other possible bottlenecks when rating a Plex Hardware it just is to complex to be taken serious. E.G. what kind of NAS are you using, what’s the NAS Wake Time, what’s the transfer speed, are other devices accessing your network storage and so on and on.
Weighing all those different factors seems almost impossible.
So if you just want to see how your cpu scales for transcoding check those ffmpg benchmarks.
Just my 2 cents
While true that FFMPEG is at the heart of Plex’s transcode engine, we have no idea what chagnes have been made or how they impact performance. We’re back in the “rough analog” scenario again, just like with Passmark. Admittedly, it should be a better analog than Passmark but my purpose with this request is not to improve the accuracy of a guess but to get hard, quantifiable data.
@techguy888 said:
but to get hard, quantifiable data.
Even then the “hard, quantifiable data” is going to depend on a lot of things out of Plex’s control. Such as the Mobo, memory bus, bus speed, HDD speeds, etc.
The only way to get the type of information you want is to test it yourself on an individual basis. And even then, things such as codecs used, bitrates and container types are going to affect each media item’s ability on each system. But again, it’s going to rely on how well you did the job on preparing your media, as much as the choices you made in purchasing the hardware.
Passmarks give a rough idea of what to expect for a given CPU. And that’s only when transcoding. There are too many variables to take into account for anything else.
Case in point my own PMS device. It’s an Asustor AS-7004T NAS with an i3-4330 CPU in it, rated at just over 5K passmarks. This says I should be able to handle 2 1080p 8Mbbps transcodes at a time, based on Plex’s stated requirements. That’s only the TRANSCODE expectations… ALL of my media is converted to MP4, H264 video codec with an AAC stereo audio as the first track. In theory, this is the most Direct Playable file container and codecs. And in my own testing this holds very true. I can get well over 5 streams at one time if they are Direct Played. And with the HW Transcoding beta, my CPU usage goes down to a mere 20-25% on a single core instead of an average 300% across all 4 cores.
You want to know what to expect for your given hardware/files? Test it yourself by opening streaming sessions on multiple devices, and multiple transcode or Direct Play streams. Find out where the “make or break” point is on your system.
The time P the Plex Team would need to develop this type of testing, and verify it’s accuracy, would be better spent working on Feature Requests, bug fixes and making the Betas part of the main stream suite, IMO.
@MikeG6.5 said:
Even then the “hard, quantifiable data” is going to depend on a lot of things out of Plex’s control. Such as the Mobo, memory bus, bus speed, HDD speeds, etc.
Yeah… This isn’t a counter-point. I’m not asking for Plex to run benchmarks and provide us with the information, I’m asking for a benchmark (i.e. a utility) so that I can run it on my system and produce my own data.
@MikeG6.5 said:
The only way to get the type of information you want is to test it yourself on an individual basis.
Right… That’s exactly what I’m asking for. A standardized method which can be used in order to determine under what scenarios does my current hardware fall short. Perhaps users could even share results and a database could be formed for comparison purposes. Strictly maintained by the users, of course. Not asking for the Plex devs to handle this bit of busy work.
@MikeG6.5 said:
And even then, things such as codecs used, bitrates and container types are going to affect each media item’s ability on each system. But again, it’s going to rely on how well you did the job on preparing your media, as much as the choices you made in purchasing the hardware.
Yep. That’s why 3rd party benchmarking utilities like Passmark do such a poor job of representing performance in Plex. The workload in Passmark doesn’t vary between runs. Plex’s workload obviously does.
@MikeG6.5 said:
Passmarks give a rough idea of what to expect for a given CPU. And that’s only when transcoding. There are too many variables to take into account for anything else.
I’m all for people chiming in and sharing their opinions. That being said, you seem not to have read the initial post in which I clearly demonstrate that Passmark does not scale well with core count beyond 6 cores.
@MikeG6.5 said:
You want to know what to expect for your given hardware/files? Test it yourself by opening streaming sessions on multiple devices, and multiple transcode or Direct Play streams. Find out where the “make or break” point is on your system.
I’ve already done this. What I’m looking for here is data. I need to know if more cores can produce better performance in Plex in my workloads. There are cheap Xeons out there which on paper should offer a significant upgrade for me but only if Plex’s transcode engine scales well with core count when serving multiple clients.
@MikeG6.5 said:
The time P the Plex Team would need to develop this type of testing, and verify it’s accuracy, would be better spent working on Feature Requests, bug fixes and making the Betas part of the main stream suite, IMO.
How do you know? FFMPEG has benchmark functionality built-in already. For all we know this would be a trivial matter.
