Where is better to install Plex Media Server; in a QNAP NAS or in a Windows Sever?

I have the following scenario in a warehouse: a disk server QNAP TS-EC1280 with Intel Xeon E3-1200 v3 processor and 12 disks of 10Tb WD Gold practically full and thinking about expanding the capacity with a QNAP RAID expansion chassis REXP-1620U- RP.

The connection from my home address to the contents of the mentioned disk server through Plex is via fiber optic connection at both ends of 600Mb symmetric (effective 60Mb down and 30Mb up).

The question is: where is it better to install Plex Media Server, on the QNAP disk server itself or on a dedicated Dell PowerEdge 860 server connected via LAN to the QNAP?

You need to provide more information. How many users? How many remote users? Transcoding or Direct Play?

Any particular reason why you are considering a Dell PowerEdge 860? It’s a very low powered Xeon server so won’t be able to do much (any?) transcoding.