Will i be able to set up my server on Roku ultra box? Will Hvec/.265 files direct play while using Roku?
Rokus and almost all other players that are not standalone computers are just players. They cannot run the Plex server at all. The only exception I know of is the Shield TV where you can, mostly, run a server. But even on the Shield I do not recommend it because it has too many little inconsistencies and outright bugs, particularly if you have a large library.
The server software absolutely cannot run on any Roku box and I doubt that even the Ultra, the most powerful Roku, could even remotely run the server if it ever were to be somehow ported to it.
It will play hevc/h.265 but it cannot host a server like the shield can.