Thanks for the updated information.
You may be running into a problem with the AndroidTV app.
There are many reports on the forum with people having trouble with the FireStick 4K (some linked below). It transcodes when it should direct play.
Per this thread, it seems v7.30 fixes the problem. However, it is still beta and not yet available via Amazon (currently v7.28.0).
It seems you have two options: 1) Wait for the fixed version to become available via Amazon; or 2) sideload the beta version and see if it works. Note that the Amazon store version always lags the Google Play version, sometimes by several weeks. This is due to Amazon release processes. Do not sideload the Google version on your FireTV. It won’t hurt anything, but it is built differently and will probably be unstable.
The beta version is available via the Android TV Beta Announcement thread. The thread says Plex Pass required, but I was able to download the Amazon FireOS Devices version (the one you would want) using a non Plex Pass login.
There is a possible third option: Fall back to a working version of the app, probably a 7.27 release. Scan the below threads and see if anybody mentions an earlier, working version. You could then download it from the announcement thread and sideload it to your FireTV (if the download links have not expired).
Sideloading apps on the FireStick is straightforward. Google “sideload firestick” or similar to find instructions. If I remember correctly, you set the FireStick to allow unauthorized apps (somewhere in FireStick settings). Then use the Downloader app to download and install the beta version.
Regarding Optimized Versions and 4K HDR video: Don’t.
You can do it. However, you may not like the results.
Plex transcodes all video to H.264 SDR (Standard Dynamic Range). At this time, Plex does not tonemap the 10-bit HDR picture to 8-bit SDR. As a result, the colors usually appear washed out and the video quality is quite poor.
Optimizing HEVC SDR or H.264 SDR is not a problem as the source and destination are both 8-bit video.
[INFO] Plex, 4K Transcoding and You; AKA The Rules of 4K is a long thread, but has some good information regarding 4K video and Plex.