I know how to remove authorized devices and this is not the problem. The server is not in this list.
A friend was sharing their server and then stopped sharing. Now I have their server in my left menu and it is constantly refreshing. I just want to remove it but am unable to see how I can.