What are you using as your server? I use an Intel NUC running Ubuntu, then use the Shield as the client mostly. It can take up to 5-6 seconds for me to see playback of Live TV start, but then it’s fine. 
I’ve not tried using the Shield directly as both the server and client at once though
