There’s been some issues with Big Sur in combination with certain network filtering apps (e.g. Little Snitch). This was fixed with the update to Big Sur 11.1.
I haven’t seen any general issues / problems since.
So not sure what is causing this for you. Doesn’t seem to be a general issue w/ Plex or macOS.
You could use the information showing in the “now playing” tile for that playback on the Plex server dashboard. If any part of the media is being transcoded, this will displayed (where the exemplary screenshot says “Direct Play”).