Users Only Connect "Indirectly"

Start here: The Basics of Remote Access Troubleshooting

It is perfectly possible that your swapping ISPs has put you behind a Carrier-Grade NAT. This is pretty much the default nowadays.