This costs me several hours of tirelessly removing/installing/log off/log on etc, but I found the solution.
“if Plex is installed on a remote Linux server, you need to set up a SSH tunnel by executing the following command on your local computer. Replace 12.34.56.78 with the IP address of the remote Ubuntu server.”
ssh 12.34.56.78 -L 8888:localhost:32400
Then you can access Plex web interface via the following URL.
So make sure you connect from the server itself, either via Remote Desktop or via SSH tunnel.
thanks to Linuxbabe: https://www.linuxbabe.com/ubuntu/install-plex-media-server-ubuntu-18-04