Transcoder stops at 60 seconds of buffer even with larger buffer defined

No, it’s not possible. See [Very Popular] myPlex: buffer content Youtube-style for more discussions on this topic.