Hi all, this is not meant to be a negative post, however like many of you I am regularly having to deal with ongoing issues with primary and basic features not working as intended, changing as each release is rolled out.
As a very seasoned IT professional (not perfect, but seasoned - 20+ years but I’m not a coder) my basic assumption is that testing of these core and basic features are being performed by someone somewhere before the ‘early releases’ are rolled out. Yes, I am aware that things are not easy with so many devices to support etc and the reasons or excuses behind that and am grateful for the wonders that Plex brings.
However, in recent weeks and months primary features such as scanning a library and streaming an item (in my case music) have failed, seemingly due to lack of testing. I can’t think what else it would be, albeit there may be decisions made around which scenarios would be tested.
So, questions and suggestions below:
Questions
- Is Plex officially using Plex Pass members to do UAT? (That stands for user acceptance testing).
- If yes, what is the process around providing stable product to the ‘public’ channel? I don’t see a traditional ‘no new features in a version, bug fix only’ policy anywhere, nor do I see a statement about how exactly this is kept stable, especially considering it is named public. The early access & preview release page also does not currently make this any clearer either.
- If no, why are we getting so many bugs?
A fundamental outcome of however code is made (Waterfall, Agile, DevOps, BiModal etc) has to be that it is fit for purpose. I don’t think this is working particularly well at present, close, but not quite there.
Suggestions
I’m expecting, this is not a particularly easy problem to solve and keep everyone happy, so wouldn’t you know it, I have a few suggestions to get the ball rolling:
- Plex ask the community to vote on which the ‘core’ features are that must not break
- Agree a method (maybe add a beta channel) to do real UAT on those core features before deploying to the Plex Pass channel
- Perform UAT on any other changed areas at the same time.
- Use the public channel for beta testing
- Give Plex Pass customers a stable channel.
Core features might be:
- Streaming of all media areas (Play files of different types in library and confirm working)
- Scanning of all media types (add new file, scan it, confirm metadata is received and is correct etc)
- Syncing of offline content e.g. music works
- Can log in
- Any others.
It doesn’t need to be labour intensive.
Plex will deny it but they do use Plex Pass subscribers as a beta group for a lot of things. The “privilege” of having early access to many features and products is actually paying to beta test.
However that is not a bad thing it is just what we must be aware of. Some things I do not mind beta testing and others I want good stability so I will let other folks test it a period of time before I start using it.
As an example if I was able to use the DVR functionality (I cannot because of no access to any OTA or cable channels) I would download and use knowing there is likely to be bugs and features missing. On the other hand I delay getting new server software until it has been out long enough for others to prove its stability and viability. I do not wany my server system to be impacted by bugs but I can always switch clients if the Plex Pass versions of software renders one of mine unusable.
I wish Plex used a different model for their beta testing but they have hit upon a rather unique one where we pay in order to be their testers. It is not bad because, mostly, we have choice.
As an example if I was able to use the DVR functionality (I cannot because of no access to any OTA or cable channels) I would download and use knowing there is likely to be bugs and features missing.
Well to be fair, that one is actually marked as a beta.
Regardless, I have always looked at plex pass releases as a paid beta. If you want stable, you always have the option to stay on the public release and wait for the new stuff to be stabilized.
Early access to software or preview releases are one of the several benefits of having a Plex Pass subscription. I can’t say if it is the only reason for you to subscribe, but it is not the only thing you get with it. We realize that some users do purchase a Plex Pass to get early access, while others do so to get premium features. It is not a requirement to use our beta software as a subscriber. That is entirely your choice. If you want to live on the edge a little, install it in a production environment. If that could potentially cause issues with the people you share with and you would rather not assume that risk, then stick with a stable release. Some users install betas on secondary machines to poke around with. The choice is yours.
Thank you for your questions and feedback regarding UAT, our release cycle, and how Plex Pass subscribers are involved. I’m going to see if I can get someone from QA to answer those.
@kinoCharlino said:
… We realize that some users do purchase a Plex Pass to get early access, while others do so to get premium features. …
You left out those of us that purchased a Plex Pass to support Plex. I, as I said, do use some of the beta products like Plex on Roku but I do not use others like the server. (Actually I guess I do use the server a bit but I wait until others have shown that there not any serious bugs) But I would still have bought a Lifetime Plex Pass when I did without those “perks.”
I also do not share my media and do not have multiple users, even when my granddaughters visit they just use Plex when they want.
For me my Plex Pass is kind of a badge of support and really has little other purpose but I do take advantage of some of the perks they are just not important to me.
@Elijah_Baley said:
You left out those of us that purchased a Plex Pass to support Plex.
You are absolutely correct! We heard that in the roundtables, that people purchase Plex Pass subscriptions to support Plex and I am really appreciative of that. Everyone in the company I have spoken to since joining Plex 5 months ago shares with me how much they appreciate those who get a Plex Pass subscription because it literally funds everything we do.
I also do not share my media and do not have multiple users, even when my granddaughters visit they just use Plex when they want.
One of the things that drew me originally to Plex back in 2012 was that I could choose how I wanted to share my media. I’m not fan of posting family photos and videos on social networking, and this gave me a way to have control over who gets to see my kids in the digital world.
For me my Plex Pass is kind of a badge of support and really has little other purpose but I do take advantage of some of the perks they are just not important to me.
Thank you very much… seriously. I don’t think enough people say things like this as openly and candidly as you, and I dig that. I’m not sure if you recall me from my username, but we met in a Google Hangout a few months back to gather your feedback. I think it’s important to listen to customers, even if we can’t do everything they want right away.
I agree with Elijah…I’d have bought it regardless.
That said, I do think it has grown to a point (PMS, anyways, maybe not any other packages), where “Plex Pass” is too broad, and you now need a 3rd branch of releases:
- Public/Stable
- Plex Pass (Premium)/Stable
- Plex Pass (Premium + Pre-Release)/Beta
@jkalland said:
That said, I do think it has grown to a point (PMS, anyways, maybe not any other packages), where “Plex Pass” is too broad, and you now need a 3rd branch of releases:
- Public/Stable
- Plex Pass (Premium)/Stable
- Plex Pass (Premium + Pre-Release)/Beta
I have a colleague from QA that is about to jump in and shed some light on our release process. I do agree with you that a 3rd bench is necessary, and we do actually have it in place already. These are the Plex Ninjas. They are involved with us internally in chat and engage in issue reporting and tracking as they review releases before they hit Plex Pass subscribers. It would however, be naive for us to assume that all bugs are crushed before going out to a wider beta release. We have been working this past quarter, and continuing to do so into the next quarter, on improving the way Ninjas can contribute and are themselves supported by Plex.
@jkalland - Isn’t that what we have already? Downloads are split between the official download section - stable/public vs plex pass, and then Plex Pass only forum postings for pre-release software. That part seemed pretty clear to me.
@jkalland said:
I agree with Elijah…I’d have bought it regardless.
That said, I do think it has grown to a point (PMS, anyways, maybe not any other packages), where “Plex Pass” is too broad, and you now need a 3rd branch of releases:
- Public/Stable
- Plex Pass (Premium)/Stable
- Plex Pass (Premium + Pre-Release)/Beta
That is a great idea!!
I wonder if Plex still has the vision to see the advantages of that approach? I am quite sure they will still get enough people that want to be on the bleeding edge to find plenty of testers for the beta level and it would free those of us that want/need stability in the server from having to pay attention to the time of a release.
I hope Plex implements that soon!
@jkalland said:
- Public/Stable
- Plex Pass (Premium)/Stable
- Plex Pass (Premium + Pre-Release)/Beta
This is essentially already implemented. The public/stable exists, and a Plex Pass subscriber can choose to use the current (stable) release with premium features or grab the pre-release version.
@drinehart said:
@jkalland - Isn’t that what we have already? Downloads are split between the official download section - stable/public vs plex pass, and then Plex Pass only forum postings for pre-release software. That part seemed pretty clear to me.
Nope it is not, quite. The server only links to the Plex Pass version or the public version and the Plex Pass version is often updated without any testing by a “special” download from here or the forums.
The server update channel should be able to be set to three update channels:
- Stable/public
- Stable/Plex Pass
- Beta/Plex Pass
and releases should go into #3 first and, after they are shown stable for a week or so, they should move to #2 and then after they pass acceptance by Plex Pass users they should move to #1.
This would assure, much more than now, that those of us that need stability in the server and have a Plex Pass are mostly shielded from serious bugs but we still get previews of new features in a timely manner.
It seems to have few drawbacks and many advantages.
@Elijah_Baley - I have only noticed the beta/pre-release server versions posted in the forums first. DVR, new transcoder, etc… In my experience, there has not been a stable/plexpass release (unless you include the one directly before releasing it as public - which typically occurs very quickly with little time between - not worth of a separate thing, IMHO). Have you seen a clearly labeled beta version that was not posted to the forum before being posted to the PP download section?
All of that being said, I do agree. I don’t like to download PP releases of PMS that have issues that are obvious the first time it loads. Or that have broken functionality with metadata or playback. But, I have gotten to where I take a wait and see and treat the PP released more like a beta and if no one complains after hours/days, I might install it.
You can download stable and Plex Pass versions from the Downloads page. I think the confusion here is in part because the pre-release version is called the Plex Pass version. As a Plex Pass subscriber, you can download the stable (or non-Plex Pass version) and get most of the premium features included, except for bleeding-edge features like DVR which are currently only in the Plex Pass version of the download.
@drinehart: Just because a Plex Pass server version is not labeled “beta” does not mean it is not truly one. Many Plex Pass releases get placed in the update without any public or Plex Pass testing and that is why there are many cases of bug as soon as a Plex Pass release appears.
I think that just because a Plex Pass release is not called a “beta” does not mean it is not one. It is sort of like the old children’s riddle:
Q: How many legs does a dog have if you call a tail a leg?
A: Four, calling a tail a leg does not make it one.
So, not calling a Ples Pass release a beta does not mean it really is not one. The “preview” releases that occasionally appear in these forums are really “post alpha” but “pre beta” the Plex pass releases are the real beta level that Plex uses.
Crap, I accidentally marked as answered on a random post while scrolling through the screen - that just shows 20 years of IT experience and you can still click something you shouldn’t! If one of the admin’s could fix that?
Regarding Elijah_Baley’s post I’d argue that the super stable branch should be a plex pass feature, at least it’s something that should be discussed.
Also, while I can find Plex documentation that states the Plex Pass Channel is 'Early Access / Preview Release), this does not say that this is an unstable branch, nor can I find anything referring to the public branch as officially branded stable. Indeed, when I first started looking at Plex Pass (and I believe still), the pre-release branch was sold as a way of getting new features before everyone else, not as a beta branch. I think Plex Pass Subscription takeup would be affected if it was branded as beta and unstable.
I think there is a bit of a branding / value issue here that needs to be solved.
The idea of three branches, where those of us technically inclined can ‘opt-in’ to early testing and provide logs back to a bug tracker seems perfect to me. Even the very closed shop Apple are doing this now with their public beta’s and Microsoft has started doing it also with their Windows Insider program.
For my money though, Plex need to document exactly what to expect from the existing channels, or commit to a change to some new method and document that.
One thing all of us ‘Plex Pass’ subscribers have in common is that we love Plex and want to continue using it, I see this thread as a discussion about how we could make that world better, while potentially improving the Plex branding in the process.
Hi @marshalleq,
Thank you for your interest in our release process. I appreciate your well thought-out feedback and have captured it for us to review. I would be happy to address your concerns regarding product quality, testing, and how Plex Pass subscribers can become involved.
Our QA team provides Plex Pass subscribers with the opportunity to test various features in the application to verify that our software is stable and functionality is operating as expected. We typically receive a significant amount of feedback from users in our community forums that quite often results in discovering and tracking issues that may not have been visible to us without the multitude of customer scenarios. However, before a beta release is made available to Plex Pass subscribers, we provide it to the Plex Ninjas for review. This in-between layer has been helpful in surfacing and investigating issues as we prepare the features and user experience for a wider Ples Pass release.
During each release, we focus on key areas that have been impacted and develop a test plan with maximum coverage - typically with an eye towards ensuring we haven’t regressed in adjacent areas. If time does not allow for a full regression test, then we do a sanity test of the core functions. The application is delivered to our Plex Ninja community for a period of time to help surface any issues and is a last defense towards improving our software quality. During this entire process, developers will be making bug fixes along the way and the cycle repeats. We continuously look at our test results to determine if an application is ready for release to Plex Pass subscribers.
On top of that, we also run automated testing daily for core functions to help identify bugs as early as possible, with daily builds of the application. The automated testing also utilizes a Plex pass account as well as restricted accounts for testing.
I hope that addresses the concerns. Please let us know if there are any additional questions or feedback. Thank you.
Hi @cpoon, thanks for your reply.
I appreciate there is a process. In particular I note:
- You mention that there is a test plan with maximum coverage and where time does not allow a sanity test of core functions. How is it then that basic core functions like media scanning and streaming are still released to Plex Pass subscribers broken? It seems to me that the core feature testing process is not well scripted.
- You mention that the QA team provides the Plex Pass subscribers with the opportunity to test features to verify stability. This is not how the Plex Pass subscription benefit is sold. Further, as far as I know, Plex does not currently offer a method to downgrade the server or switch back to the ‘Stable’ channel when things break. These things seem to be at odds with each other and in my view should be clarified in documentation. For example, I have not yet read anywhere officially that the public channel is a stable channel. Predicting one possible response here, I don’t think the uptake will be as strong if we are told we must run the Plex Pass channel in our own beta test environment, separate from production.
I’m not opposed to providing beta testing capability to Plex Pass subscribers, in fact I encourage it, however fundamentally this situation is unclear and I see causes a lot of confusion. In my view there should at least be a knowledge base article created explaining exactly what each channel is targeted at with links to support documentation for e.g. how to revert to stable.
In my view ‘Public’ is a poor naming convention for stable anyway and leads to further confusion. The state of the Plex Pass channel is largely undefined and asides from your response above, how we all fit into the testing process has been a mystery.
Is there a chance we could raise these things and get them clarified, officially in documentation? Or perhaps I just haven’t found it yet?
Resolving the above should I suspect create an environment with increased happy customers and a more stable product. Win / win!
Thanks.
I also note the confusing wording in the forums around ‘Plex Media Server Preview Releases’, which is a newer version than is available in the Plex Pass channel. Perhaps this is the same release as a Plex Ninja gets or, it’s something else…
