Make the server "invisible" to specific users

Hi guys, could be possible to turn off the server to specific users?

I share my server with some friends (naturally for free), and that is ok to me when I don’t need it or when I’m home and I use local network to see my contents.
But, when I’m out and I’d like to watch something via remote connection, I’d like to be the only one connected, since my bandwidth in upload is not so large, and of course I don’t want to watch a lower quality or not watch it at all because of them. Instead of calling them personally and tell them to stop watching (which is not nice at all), it could be useful some option to make the server as it would be off ONLY to them (maybe selecting them both specifically and all)

Early 2021 clean-up: duplicate (focusing on the same requirement)