On my iPhone or my Android (since it could play the video), I’d go to Playback Settings > Quality > Convert Automatically, then it would correctly transcode the audio (was set to Original).
On my Android TV devices, if I was quick enough I could select HD > Manual > pick any option there. Under Auto it wasn’t correctly trying to transcode, which I believe is why the optimized version wasn’t working.
I was able to fix the issue overall by manually reencoding the audio and then muxing it back in with the original video.