Even tho PMS is running when I try to add my own library files it asks me to install PMS

Was there a Plex server installation on this PC before?
If yes, you need You need to reclaim your server, but don't dare to touch the Windows Registry

Is this regular Windows, or Windows Server Edition?
If it’s a server editon, see Plex install fix for Windows Server (Claim issue)