I ran a script to compare the ratings from the new agent vs the IMDb dataset. I had to do this by abusing the “match” function of Plex, where I basically iterated through each movie and matched against the old IMDb agent to pull the guid. It was incredibly slow (roughly 10 hours for 2000 movies), so it is not an actual solution. I have not found any other way to get the IMDb ID from the movie using the new agent.
The results were promising. The agent was just released, so obviously they may have just updated the ratings in their crowdsourced platform, but outside of a few cases where the matching was not accurate, the difference in scores was almost always correct (like over 80%). The few cases where it differed, it differed by an absolute value of 0.1. I did not see any cases above this difference that could not be attributed to a wrong match.
Prima facie, it does seem like the ratings are more accurate. Maybe that’s simply because of the release, and as time goes on, the ratings on their Plex Cloud may become stale (unclear). More importantly, I do not think there’s any mechanism to keep ratings in sync with a Plex server, so depending on when a movie was added, those ratings can drift over time and become stale. Usually, this would be a concern for mainly new movies where the number of votes is low when the movie is added. Finally, there are some movies that do not display IMDb ratings still and use TMDb/RT instead. My script found IMDb ratings, so they exist, but I guess they are not in their crowdsourced metadata.
