Using Webkit & HTML for UI?

Dev Community Dev/API Corner
1
Random thoughts. Possibilty a tangent?
I am using Plex as my HTPC UI and have been thinking of ways of going forward, that would allow us to maximise development efforts from multiple parties, rather than trying to get one team trying to solve all the issues. I would like to share with you some of the ideas, and see what the feeling is (Note: I am not part of the Plex dev team, so this is just me in my corner):

What I have now is an idea to use HTML/Javascript for the UI, so that anyone with the necessary skills can easily create a skin or customise it. This would also allow us to leverage the development effort being into WebKit (the rendering engine used by Safari & Google Chrome). Also, this would allow Dashboard, Yahoo Widgets (formerly Konfabulator), type widgets to be created for anything a developer wishes to do.

For the media player, either this could be done as it is now, or simply handeded off to a regular plugin, which is embedded in a web page.

The front-end would be a 'thin-client' with a server component handling the media catalogue and discoverable by something like Bonjour, so you could have one server and then multiple clients throughout the home. The protocol would be XML based, with XSL rendering according to the view being used.

A rough presentation of what could be possible is here (best viewed with Safari, due to use of CSS3):
http://ajmas.dyndns.org/misc/web-htpc/menu.html
For the link above make sure to use maximise the window. The left arrow will toggle visibility of the left pane.

Video (use Safari & Quicktime, then move the mouse in an out of the player):
http://ajmas.dyndns.org/misc/web-htpc/video.html
The link above demonstrates the use of using of HTML for the control layers.
2

Looks like I am not the only one entertaining this sort of solution for their HTPC solution: http://developer.apple.com/safaridemos/movie-trailers.php (Note you will need a WebKit based browser, such as Safari or Google Chrome to view this).

3

Just to throw my two cents in on this, I think its a great idea.



Safari is adding an extensions function similar to FireFox, which allows devs to create web applications using HTML5 and javascript… the same elements used to create the example that you posted above.



If the htpc app was built as a Safari Plugin, it would be easy to create it to run cross platform, and not be limited to macs.



Also, sites like Hulu will see the app as being the Safari browser, and maybe wont block it?



I wonder if something like this would be able to connect with Plex Media Server?

4


I'd love to see html5/CCS3/JS (and therefor Ajax) as a Skinning Option. I don't like the current xml-based skin technic, its too complicated. I'm a professional programmer, I looked into skinning but I will not touch it!

The Safari Extensions are Sandboxed and I'd say that will prevent them from communicating with the Plex Server, just a guess.
I'd prefer it no to run in Safari 'cause you are limited in extending it. It should be relatively easy to replace/extend the current UI with the latest open source webkit engine within Plex itself. This would make it easier to play media files with codecs not supported by html5 and interacting with various remotes. JS can do all the dynamic code manipulation, while html5 and css3 does the rest.
It would be interesting at least to try. ;)

I'd like to know what the plans/ideas are for Plex 1.0 in the UI part of Plex.

My major problems with the current xml-based skinning is debugging, the constant trial and error, and the long, long search for the matching in all those skin-files. I think it's a real mess. :D Its time to take another step away from xbmc ;)

An established standard like html/css/js with thousands of already existing coders would be a major step forward. (Almost) Everybody could easily customize their favorite skin.

It would also make App/Plugin development easier. A special Plex Browser, Email Reader etc no problem.
5


I agree, the current skinning engine is not easy to work with and very limited.
6


Nice to hear from a skilled skinner. I have the utmost respect for you and your work!

If we would know the plans/ideas of the developers for UI especially if they do NOT intent to change the UI engine, then others could do this for them / the community. html does not have to replace the current one but it could be an alternative for skinners.
7


I know they do plan on changing it, but don't know to what. It's on the list, but that new library system is more important to them right now.
The skinning engine is bad but it's working, the new library system is much better and a better use of the developers time.
8


I absolutely agree, the library has to be to top priority and I do not want them to change that. :D
9

I think we all agree that the new library is the most important thing for the devs to be working on at the moment but that shouldn’t preclude any discussion(s) around changes to or the implmentation of a new skinning engine.



I for one would welcome a complete rewrite of the skinning engine by the Plex devs, especially if they show the same kind of innovation they are applying to the library. When I first tried skinning Plex the fact that it used the same engine as XBMC was a bonus but seeing as though Plex and XBMC continue to diverge then a rewrite makes sense.



While I personally have no problem with XML the skinning engine would benefit from being simplified somewhat, at the moment it’s a mess and getting anything working often requires hours of trial and error.



I’d also like to see it utilising hardware acceleration, the API’s are there to be used :slight_smile:



Other ideas anyone?

10

i imagine that with the skinning engine we’re more likely to see slower incremental changes/improvements, rather than an abrupt total replacement like the library is getting. and personally i really don’t think a webkit/html based UI is a good idea for something like plex. just speculating here (and disclaimer: I am not a skinner), but long-term, i wouldn’t be surprised to see the skinning engine evolving more in a core animation sort of direction.

11


my thoughts exactly. you'd be able to create some stunning interfaces using the core api's. i know people seem very against the current xml system but i have to say, once you take some time to get to grips with it, it's not THAT bad. for someone building something from the ground up and with an organised structure, it makes a lot of sense. i managed to get an 'okish' handle on it within a few months. the hardest part of using it currently is picking up someone elses work, which isn't too bad when the original creator has commented their work. i posted it in the feature request section a while back but i don't think it got much of a response, i think a great first step would be to add the ability to access various features of core image/animation through the current xml system. this way existing skins could be used still rather than everything we currently have needing to be dropped.

that said, as it stands, the amount of information currently accessible to the skin is somewhat limited and due to that, certain things cannot be done. conditions would be the one thing that springs to mind that i'd like to see improved in that sense. i would be nice to be able to say 'if THIS... then THIS' no questions asked rather than being limited to using certain bits of information in certain situations.

so anyway, it's just my opinion but i think that would be a good direction to head in first BUT i do genuinely believe that there's no point in having a pretty interface when the foundation underneath it remains unfinished. XBMC looks better than Plex in some respects but when it comes down to it, XBMC still doesn't handle all my bluray rips without stuttering when changing chapter whereas Plex does. so while i would love to see all this stuff, i don't expect it any time soon and i'm pretty excited to see the underlying changes in the library and video player in the meantime.

marc
12


Well said :)

So I think the common theme here is that we all agree that Plex rocks and that the changes the devs are working on are the most important things. Conditions sound useful, doesn't XBMC have something like that implemented already? Core animation support I think is a must in any skin engine rewrite. From what I can tell from Elans blog Alexandria is addressing some things that affect skins (ie media tags) so that's going well. What else would make our lives easier/better when skinning?


I'd like more options for text effects, at the minute the engine doesn't even compare to what's possible with CSS. I'd also like to see 3D tackled sensibly in the skin engine, at the moment any kind of 3D effects are resource intensive and so time consuming to write that tbh their not worth the effort (to me).


I'd also like to be able to move/resize my skin elements dynamically based on what is currently being displayed.

Any other thoughts?

Alistair
13

It probably didn't get much of a response because it simply isn't possible :( We can't just add in bits of Core Animation here & there. All the GUI code (and thus the skinning engine) is based on SDL (a cross-platform graphics library) which can't be mixed with the Quartz-based Core Image/Core Animation graphics technologies. To use any Mac-specific APIs, we'd need to rewrite the entire engine to use them instead of SDL. It's not an impossible task (the engine has already been ported from DirectX to SDL by the XBMC team) but it's certainly not a trivial one, and not something we're particularly interested in committing to, at least in the near future. Sorry to be the bearer of bad news!
14


No, that's cool. I wasn't sitting round waiting for a reply or anything. :) I did have a read about Core Image before posting in the feature requests section but I'm no programmer so I guess I didn't fully understand that you can't mix the two. My bad. Anyway, thanks for the reply and at least that gives us a rough idea of where you guys don't want to head with it. I look forward to seeing what you come up with.

Marc
15


Yeah, that's exactly the problem. It is impossible to combine SDL with Core. That's why an already existing "engine" is the best way IF once plan to replace the current one.
And although Core Image and Core Animation sounds "fun" it is (in my opinion) overhead. I first thought too that the Core Library provided by Apple would be cool as UI Engine. But with html5 and CSS3 and JavaScript you can do everything you need. JavaScript brings you a really powerful script language to dynamically change the DOM provided by html. I don't see what the Core Library has, that is impossible with the current "browser technology", even 3D is no problem. JavaScript can actually tab into the Objective-C Code/Classes and therefor access all data you need. Or just use Python/Perl/Ruby.

A plus side is you can extend the current App Technology for Plex and extend it like Safari Extensions or even further. These Apps don't need to be Media Server depended but since its only standard web tech ... . Your own Games written in JS, no Problem (see: [Chrome Experience](http://www.chromeexperiments.com/))

I have nothing against XML, but a redesign of the Skinning Engine, no matter how, is needed for Plex 1.0. And the past computer history shows us that "brewing your own stuff" is not always the best way. ;)

Like El Massman said: "I look forward to seeing what you come up with." :D
16

Don't you **dare** badmouth Core Animation in front of me! We've become very close friends during the development of the 0.9 media server (check out the grid view in the Media Manager video on the blog). ;)

Seriously though, it's actually very lightweight, but extremely powerful and flexible - it was only considered when other "regular" drawing methods proved too expensive performance-wise, and I've been blown away by how good it is. If it could help in a GUI app with a limited amount of extra graphical touches, the possible applications for a 10-foot UI are astounding when you think about it.

The skinning engine is "good enough" for the time being. I don't like that it isn't "perfect", and I'm sure the other devs don't either, but being a small team we have to pick our battles. Also, if we suddenly brought out something that broke existing skins (which a HTML/CSS/JS based replacement would definitely do) there would be uproar on the forums - just check the third-party skins forum to see how passionate people are about their various UIs.

Also, *not* "brewing your own stuff" isn't always the best thing to do either - it may have taken a long time, and admittedly I'm biased, but I haven't seen anything else out there quite like Alexandria and I certainly don't regret the decision we made to start from scratch with the library. And in this case, "brewing our own" would imply writing an engine from scratch in OpenGL - CA is a fairly mature, widely used framework, where another team with far more developers than us (and a much more fearsome boss!) has already worked out the kinks & bugs before releasing it into the wild. There are things CA has been able to do since its release that are only just becoming available in HTML5/CSS in the latest WebKit builds, with much better performance.
17

Was not my intention ;)


Then I will definitely look into the Core Library in near future ;)


I did not mean the engine per se but the way to interact. Like the current xml to "interact" with the skinning engine. Its from xbmc and not an open standard (as far as I know). Yes my previous formulation was bad (not my native language).
Alexandria is not "own stuff" since its based on an standard database engine. The Database Scheme has to be (of course) unique for your purpose and shouldn't be copied :D But I hope you are not writing a complete Database Management System (witch would be the "engine"). ;)
I would certainly not call a c++ class "own stuff" since its using a widely used programming language.
So don't get me wrong: I didn't mean to rewrite an complete rendering engine or storage engine but the language to communicate with such that should be an open, already known language, no matter if it's rendered via openGL, Core or something else. Thats why I said html & co would be perfect. WebKit as rendering engine and the open standard html&co as "language". If the Core Library has such an "language" or an (not open) intuitive one that's perfect to use for skinning Plex, then use it and I'm happy.

No matter what happens in future with the skinning engine/skinning language, I'm happy with Plex and I eagerly await the first release of 0.9. I think the Developers (and the skinners) are all doing a great job.

PS:

Never said suddenly, I said (earlier) as an option ;) Later I meant "If and only if" without suggesting that anyone plans it ;)



18

You are correct.


Haha, no, of course not :) But Alexandria is a lot more than just a database - it contains many separate components, including several that were developed from scratch.


I see where you're coming from, but in my opinion, the full HTML spec is probably overkill for what we'd need - there's a lot in it that wouldn't be suitable, or required, for developing a 10-foot UI for Plex. And we'd still need to include some non-standard "additions", since Plex has features that need accessing/controlling that standard web browsers don't. XML is considered to be an open standard, in the sense that "open" means something that can be easily read/parsed by other tools. CSS & JavaScript, however, present some interesting possibilities, and would be much more suitable for some things that have been forced into the current XML skin format.


Thanks! Glad you like it :D


"As an option" is something we probably wouldn't even consider :) Having multiple ways of doing the same thing is part of the reason Plex is as bloated as it is today! We firmly believe in choosing the *right* way of doing something, then making that the *only* way. It makes the software simpler, more efficient, and easier to use, develop & understand :)
19

Steve, is that you? ;)

Seriously though, it makes total sense to do this with a Mac-exclusive app. The leap in becoming an Apple customer is making the decision that trading customization for solid functionality is worth it. Some people not only disagree with that decision, but do not even understand it. And an Apple hater is born.

And yes I find the irony in this statement in a thread about customizing the UI <_<
20


This is part of the reason I suggested something like Webkit for the rendering engine. The under pinnings of Webkit are probably taking advantage of plenty of MacOS X features already and as it evolves it will potentially take advantage of more. Consider things like Canvas2D and Canvas3D that bring extra stuff to the party. For the front-end scripting we could extend the Javascript API to support Plex specific stuff. The way I see it is that the Plex team already has plenty of work on their plate and anything that allows some of the development to be 'delegated' helps.

Also given the server is going to be using SOAP based messages, then the front-end transformation could be done in XSL? Another approach would be for the server to support REST and JSON?

I meant to take a look at Adobe Air to see whether it could form the basis of an experimental front-end platform for doing this, but I haven't had the time.

Edit: I believe Webkit is already using Core Animation for some stuff, but I can't seem to find a good link for this.

Edit: This may also be interesting to look at (requires a Webkit based browser): http://webkit.org/blog/324/css-animation-2/