Let’s see if we can shortcut this a bit. Are the videos that will not play 10-bit H.264? Try playing them with subtitles disabled and AAC or AC3 audio.
Also, what version of PMS are you running (see Authorized Devices), and what is your system like - CPU & RAM.
Conjecture: Your clients cannot direct play 10-bit H.264 video, so the Plex server has to transcode it to 8-bit H.264. Your server is unable to perform the transcode for some reason, so you get the error message.
Test: Play one of the 8-bit H.264 videos that is OK. During playback, reduce the playback resolution (if 1080p then reduce to 720p or 480p, etc). Does the video play OK?
Reducing the resolution forces PMS to transcode the video. This tests if your system can transcode a known good video.
Look in Authorized Devices. This shows clients authorized to play media from your server.
Also, for SmartTV, stick/box devices (Roku, AppleTV, etc), you can view the version in the Plex app on the device. Usually somewhere in the app settings.
For TVs, look up the specifications for each make/model.
Example: I’ve an LG B7 OLED. The specification page tells me it supports HDR10 & Dolby Vision playback and AC3, EAC3, & dts audio.
For Rokus, etc, check the specs for each of those devices, just like for TVs.
Web browsers are easy. They don’t support anything. 
Seriously, browsers are very limited compared to other clients. None support HEVC video, and audio is limited as well.
If you are playing on Windows or Mac desktop/laptops, you use use the Plex for Windows/Mac or Plex Media Player client. They both handle video & audio formats that web browsers do not.
