I tried this myself and this is what Plex logs are returning to me:
May 25, 2013 15:16:18 [0xacef1b40] DEBUG - Job running: /usr/lib/plexmediaserver/Resources/Plex Transcoder transcode http://192.168.9.6:8080 /tmp/plex-transcode-50e30fa7-0361-4cfe-bccd-92012eed63ad-6ccf0cb1-7c9d-4526-9b60-8d3264b3acc0/media 4 10 0 any nil 2 yes UTF-8 default 0.9375 a none aac 3355443 yes no http://127.0.0.1:32400/video/:/transcode/session/50e30fa7-0361-4cfe-bccd-92012eed63ad/progress
May 25, 2013 15:16:18 [0xa83ffb40] DEBUG - [Transcoder] Starting transcoder v306 (t=10s, q=4)
May 25, 2013 15:16:18 [0xad6f2b40] DEBUG - [Transcoder] Registered components, about to parse arguments.
May 25, 2013 15:16:18 [0xad6f2b40] DEBUG - [Transcoder] Video caps: ca=1, cv=0, ac=libvo_aacenc, mp3=-1, aac=0,1, ac3=-1, dts=-1, pcm=-1, vp=4d, vr=0, vl=0, mss=3355443, anam=1
May 25, 2013 15:16:18 [0xad6f2b40] DEBUG - [Transcoder] About to apply initial settings.
May 25, 2013 15:16:18 [0xa83ffb40] DEBUG - [Transcoder] Opening the input file.
May 25, 2013 15:16:23 [0xa83ffb40] DEBUG - [Transcoder] Thread count: 2
May 25, 2013 15:16:23 [0xa83ffb40] DEBUG - [Transcoder] Input #0, asf, from 'http://192.168.9.6:8080':
May 25, 2013 15:16:23 [0xa83ffb40] DEBUG - [Transcoder] Stream #0.0, 124, 1/1000: Video: h264, yuv420p, 1280x694, SAR 1:1 DAR 640:347
May 25, 2013 15:16:23 [0xa83ffb40] DEBUG - [Transcoder] Applying output settings.
May 25, 2013 15:16:23 [0xacef1b40] DEBUG - [FFMPEG] Duration: -1
May 25, 2013 15:16:23 [0xacef1b40] DEBUG - Read line, and done: 0
May 25, 2013 15:16:23 [0xad6f2b40] DEBUG - [Transcoder] Frames per second is either 24.390244 or 23.976024.
May 25, 2013 15:16:24 [0xad6f2b40] DEBUG - Request: GET /video/:/transcode/segmented/stop [192.168.9.4:53870] (2 live)
May 25, 2013 15:16:24 [0xabeefb40] DEBUG - * Cookie: SESSION-GUID=50e30fa7-0361-4cfe-bccd-92012eed63ad; expires=Thu, 13-Aug-2020 23:59:59 GMT; Path=/;
May 25, 2013 15:16:24 [0xabeefb40] DEBUG - Stopping transcode session 50e30fa7-0361-4cfe-bccd-92012eed63ad
May 25, 2013 15:16:24 [0xabeefb40] DEBUG - Killing job.
May 25, 2013 15:16:24 [0xabeefb40] DEBUG - Signalling job ID 15933 with 9
I've tried mux as "asf", "mp4", and "mpegts" but no go no matter how I do it. Any thoughts?