@EvanVanVan said:
Any suggestions? Do I need to remove the device AND change my password?
It is at least worth a try.
Go here: https://plex.tv/users/edit
set your cursor into the ‘New Password’ field and make sure to check the checkbox underneath
This should take care of all non-local devices.
For local devices (those in your own home network and those connected to it by VPN-tunnels)
you may have to take additional measures.
(Which depend on whether you use Plex Home or not)