Short Answer: The Plex app on the Sony TV has limited capabilities compared to Plex on the Nvidia Shield. That is why the movie transcoded when you played it using the TV app. It has nothing to do with HDMI or Ethernet cables.
When using the TV app, do not choose TrueHD audio. It is not supported by your TV and Plex will transcode it to a supported format. Choose the Dolby Digital, dts, or AAC audio track if available.
The TV and Plex app apparently do not support VC1 video, so Plex must transcode it to a supported format. The CPU in your NAS may struggle with this. The solution is: a) avoid VC1 video; b) use Hardware Transcoding, which requires a Plex Pass; c) use a client, such as the Shield, which supports VC1 video; or d) use Handbrake or similar tools to convert the media into supported formats before adding to the Plex library.
Note that if you enable subtitles while audio is transcoding, the video will also transcode. I do not know what happens when audio is direct playing. Enabling subtitles may not force a transcode in that case (I’ve an LG, so no way to verify how a Sony handles subtitles).
You can monitor what is transcoding during playback in the Plex Dashboard. Plex Web → Activity → Dashboard. If needed, click on the gold, rectangular-ish icon on the right to expand the information (to see detail for audio & video).
Long Answer:
It depends on what audio, video, and subtitle formats are supported by the Plex client.
Not all Plex clients support the same formats, even if they run on the same OS (android, etc). If a movie direct plays on the Shield, it may or may not direct play using the Plex TV app, even though both the TV and the Shield run the Android OS.
For example, if the Plex app on your TV does not support VC1 video, then your Plex Server will transcode the video. The Plex app on the Shield supports VC1 video, so if you watch a VC1 movie via the Shield, the video will direct play.
The same applies to audio formats. Supported formats direct play. Unsupported formats transcode.
Subtitles are a separate can of worms. If the audio stream is transcoding and you enable subtitles, expect that the video will also transcode. If the audio stream is direct playing or direct streaming, enabling subtitles may or may not force a video transcode. It depends on the capabilities of the client.
I know this all sounds like you’re just tossing darts at a moving board hoping to hit the direct play bullseye. It really isn’t that bad. You just have to know what audio & video formats are supported by your devices.
I rarely use the Plex app on my LG TV due to its limitations - no TrueHD audio support and very limited subtitle support. That makes it hard to direct play Blu-ray rips, especially if I want to enable subtitles.
The Shield + Denon solves those problems. The Shield supports the video formats on DVD & Blu-ray discs (MPEG2, VC-1, H.264, H.265). The Denon handles the audio formats found on DVD & Blu-ray discs (PCM, (E)AC-3, dts, dts-HD MA, TrueHD, TrueHD + Atmos).
Hope this helps and doesn’t confuse things. Tried to provide detail w/o overloading. 