I came across this a few days ago aswell…
So I had a movie that I had encoded at my usual 6.5 Mbps 10Bit x265, but when I played it in Plex, both the Dashboard and Tautulli were reporting that I was using 18.3 Mbps.
Now I am aware that the total bitrate does obviously include the Audio track, and yes, the movie had been on my system for some time, and therefore Plex had already done a Deep Analysis of it.
After much reading, I found out that when using ABR, there are times when a much higher bitrate may be used for scenes with a lot of motion.
So I re-ripped the original, and used Bitrate Viewer to take a look, and indeed there were a few rather large spikes.
So looking further I found this article… https://eldar.cz/kangaroo/binarni-sxizofrenie/howto-handbrake-limit-peak-bitrate.html and although it refers to x264, most it is still applies to x265.
So I have now added the following line into the Advanced Options box…
–pme:vbv-maxrate=8500:vbv-bufsize=17000
Which ultimately sets a Max Bitrate value to ensure it doesn’t go above.
After re-encoding the movie, it now runs at 13.9 Mbps… Much better 
