Oddly enough AC3 5.1 seems to be working now for the time being.
It seems a hard reset (completely shutting down the Xbox) fixes a lot of issues that come up occasionally. The Xbox seems to be very picky which .mkv files it will play without issue though.
Have you had any issues with no sound from AC3 5.1 audio? Sometimes it will fail to direct play and force transcode mid stream as well. Weird!