Removing Devices not working

[FordGuy61]

@TechSavvy

Also, I have disabled remote access in the server settings, but somehow some devices can still connect remotely. If I disable remotely it should not allow any remote access.

Maybe connecting via Plex Relay service?
Disable Secure Connections in Settings → Server → Network and restart PMS.

Reference:

Plex.tv public ip-adresses

Plex Relay is only disabled by either: disabling remote access in general under Settings - Server - Remote Access disabling ‘secure connections’ under Settings - Server - Network (restart of plex server required)