Generally you would click on the friend in the Plex Home or Manage Library access settings (like if you wanted to change their restrictions) then click edit on their server and then the leave server button.
Though it sounds like it may be a ghost server on their account which I can remove manually from their account if you don’t see it there,
