I have cleaned up at least what you had listed plus some other folders that I could think of. Still running into the same issue. I will keep cleaning the folder structure to improve things but dont think this has solved the issue yet. Server has also been rebooted and library scan ran.
EDIT
Cleaned up more of the file structure and after another reboot, rescan and database optimization, things are playing correctly again. Thanks for the help