Plex Media Server has no player. It streams media to Plex clients.
How are you playing the videos - Plex Web (Chrome/Firefox/etc), Plex for Windows, or Plex Media Player?
Plex Web is limited by the capabilities of the web browsers.
If you want multi-channel output, use Plex for Windows or Plex Media Player. Both available at plex.tv/download. Either can be configured to passthrough multi-channel audio such as Dolby Digital, dts, and TrueHD.
Plex for Windows & Mac is the current desktop app. It has a look & feel similar to the web interface. It is updated on a regular basis as new features are added and bugs fixed.
Configuring audio settings for Plex for Win/Mac is similar to configuring Plex Web settings. See Windows & Mac Desktop documentation for details.
Plex Media Player, PMP, has the “10 foot interface” (TV Mode) similar to the Smart TV, Android TV, etc apps. It is use by many people with HTPCs. PMP is essentially in maintenance mode, not receiving new features. Its long term future is unclear (unless I’ve missed an announcement).
When first launched, PMP appears in Desktop Mode, similar to Plex Web. To configure audio options, place it in TV Mode by selecting the TV icon (rectangle w/ “TV” inside) in the top right portion of the application window.
Edit:
Select your username/icon to TV Mode settings to configure audio passthrough, etc.
Also, while in TV Mode settings you will have to use the arrow keys, escape key, etc for some navigation. Not all choices are available via mouse clicks.
See Plex Media Player documentation for additional details.