From the log file:
transcode bitrate=2147483647 encoder=h264_qsv width=1920 height=800
This can indicate Plex has not performed a deep analysis of the file, to determine the bandwidth at various points.
Look at the XML info for the file.
You should see an entry with deepAnalysisVersion="6". Also a requiredBandwidths entry for the overall file and each video/audio/subtitle track. Ex: requiredBandwidths="39082,38821,38529,38195,37863,37534,36106,36100"
In Settings → Scheduled Tasks, check that both Upgrade media analysis during maintenance and Perform extensive media analysis during maintenance are selected. This will tell Plex to perform deep analysis during the maintenance window. You may need to extend the window for maintenance tasks as well.
The deep analysis helps Plex determine the bandwidth needed for transcoding.
It might help with the transcode quality issues you are experiencing.
Also, look at some of the movies/shows that have been in your libraries for a long time. Plex might have analyzed those. You could test with one of those files until Plex analyzes Watchmen.
Example XML Info with deepAnalysis and requiredBandwidths
<Media id="1837" duration="5262091" bitrate="40648" width="1920" height="1080" aspectRatio="1.78" audioChannels="6" audioCodec="dca-ma" videoCodec="h264" videoResolution="1080" container="mkv" videoFrameRate="24p" audioProfile="ma" videoProfile="high">
<Part accessible="1" exists="1" id="1837" key="/library/parts/1837/1651125657/file.mkv" duration="5262091" file="/media/movies-hd/Airplane! (1980)/Airplane! (1980).mkv" size="26736476943" audioProfile="ma" container="mkv" deepAnalysisVersion="6" indexes="sd" requiredBandwidths="42673,42286,41525,40899,40838,40838,40838,40838" videoProfile="high">
<Stream id="5040" streamType="1" default="1" codec="h264" index="0" bitrate="39112" bitDepth="8" chromaLocation="left" chromaSubsampling="4:2:0" codedHeight="1088" codedWidth="1920" frameRate="23.976" hasScalingMatrix="0" height="1080" level="41" profile="high" refFrames="4" requiredBandwidths="37979,37675,37112,36895,36890,36890,36890,36890" scanType="progressive" width="1920" displayTitle="1080p (H.264)" extendedDisplayTitle="1080p (H.264)"> </Stream>
<Stream id="5041" streamType="2" selected="1" default="1" codec="dca" index="1" channels="6" bitrate="1536" language="English" languageTag="en" languageCode="eng" audioChannelLayout="5.1(side)" bitDepth="24" profile="ma" requiredBandwidths="4803,4640,4194,3921,3921,3921,3921,3921" samplingRate="48000" title="dts-HD MA 5.1" displayTitle="English (DTS-HD MA 5.1)" extendedDisplayTitle="dts-HD MA 5.1 (English)"> </Stream>
<Stream id="79633" streamType="3" default="1" codec="pgs" index="2" bitrate="46" language="English" languageTag="en" languageCode="eng" requiredBandwidths="58,58,58,58,58,58,58,58" title="English SDH" displayTitle="English (PGS)" extendedDisplayTitle="English SDH (PGS)"> </Stream>
<Stream id="5043" streamType="3" codec="pgs" index="3" bitrate="29" language="English" languageTag="en" languageCode="eng" requiredBandwidths="294,294,294,294,294,294,294,294" title="Trivia" displayTitle="English (PGS)" extendedDisplayTitle="Trivia (English PGS)"> </Stream>