My media takes a while to play.

When I start to stream media, it takes like 15-20secs to even start the media. This is a pms on a 10gbs seedbox server. Any help is appreciated!

Normal if Plex server has to transcode the media file for your particualr client or due to bandwidth restrictions or the need to burn in subtitles
https://support.plex.tv/articles/201575036-why-is-my-video-stream-buffering/

No, I only use direct streams, but when the media does start playing, it doesn’t buffer.

@ethanboucon said:
No, I only use direct streams, but when the media does start playing, it doesn’t buffer.

You can’t control that directly.
If Plex has determined that it must transcode a file, it (tries to) transcodes it.

Reasons, why Plex would activate the transcoder are manifold.
For a detailed explanation, please show us the Plex XML info of a video and tell us which Plex client was used and whether a subtitles was enabled or a bandwidth restriction was in effect.