Having the exact same issue. After doing some online research I discovered I’m not alone and this seems to be a VERY common problem (going back years) with NO solution.
I’ve tried different clients (iPad, Phone, PC, TV) and different streaming options and I still can’t do more than a couple of FFs until I get the endless spinning wheel.
My UnRaid server doesn’t report any errors and I have a pretty powerful media server (64GB RAM, Intel 12700K CPU, M.2 2TB, 5 fast HDD 40TB) and run Cat 8 cable 1G from server to TV.