If this is a mp4 file, see: Help: stream is constantly buffering, libmpv sends TCP RESET - #2 by OttoKerner
P.S. an initial spike is fully expected, because at playback start the client is filling its network buffer as fast as it can.
Only if the bandwidth remains abnormally high over almost the whole file, refer to the link above.
