If its FreeNAS the jail needs to use VNET for transcoding to work.
Check out PMS not sending transcoded files to clients
