As for bare performance…
The DS218+ uses a Celeron J3355 which as a PassMark score of approx. 1345. The NUC7I5BNK has a Core i5-7265U processor which as a PassMark score of approx. 5094.
The 218+'s CPU is not powerful enough to transcode 1080p content on its own (purely by CPU). However its integrated GPU can be used for hw accelerated transcoding. That will handle 1080p transcoding just fine and can do so for 4K videos (but only using h264).
https://ark.intel.com/content/www/us/en/ark/compare.html?productIds=97539,95597
The NUC can handle a 1080p transcode using just the CPU or about 4 parallel 720p transcodes. The NUCs iGPU is more powerful as well. That being said I cannot find a specific reference as for how many streams it’ll manage – should however provide more capacity…
Best OS?
Go with what you’re most comfortable with. If you’re familiar with Linux and know your ways with it… go Linux. If you’re not… there’s no harm in using Windows – just don’t go with Windows 7 anymore.
NFS & Automatic library updates
Plex cannot detect changes to your folders over NFS (or network shares in general). Just manually trigger Scan Library Files from the library context menu.
If you’re using Linux, make sure to properly mount the NFS share.
There’s a helpful post in the tips & tricks section on how to do that.
