Summer of Code ™ 2006

After a lot of hacking, I finally decided to release the next mini-version of Beacon. 0.2 has a lot of changes from the previous version; the most significant of them being a complete architectural change. Beacon 0.1 completely relied on JavaScript for most of the XML parsing; but I switched to PHP in this version. Hey, the server is bound to have more resources than the client, so the application is now a whole lot faster than before

I also finally got the DOM Tree to work. So now we have nice tree-like display of the GuideXML document. I’ll be adding DragAndDrop support to the tree itself sometime soon, but for now you have an easy of selecting and deleting Nodes (Chapters and Sections). Check out this screenshot:

Beacon 0.2 also using the Geshi Syntax highlighting engine to display the XML source instead of the earlier JavaScript engine. The XML is coloured on the server using PHP and subsequent changes are synchronizsed via AJAX calls:

Gentoo Infra still hasn’t removed the GuideXML SoC site, so you can check out the latest Beacon version there, while it lasts

Cheers!

I’ll be giving a talk on `Building web applications using Javascript’ on September 16th at Freedel. It’s my first talk, so I’m both nervous and excited! Anyway, if you happen to be anywhere near the NCR on 16th and 17th, do drop in. I’ll be focussing on the MochiKit and Dojo toolkits, both of which were extensively used to make Beacon, the GuideXML editor. I’ll also be offering some insight into how Beacon was built at the presentation.

Keep smiling!

Wow, time flies. The magnificent Summer of Code 2006 is finally nearing conclusion, with just a few more minutes to go.

Looking back at the entire experience, I cannot help but smile. Beginning from the proposal preparation, right upto the final moments of convincing your mentor that your project really deserves the last $2000 cheque; you keep feeling happy that you are part of something big.

I was especially lucky to be in the company of the totally awesome people at Gentoo, they made the whole thing a lot more enjoyable. One of the great things I liked about this years SoC program was how it brought so many different people from all corners of the world together; we got to interact with one helluva cross-section of folks

Thank you Google, and thank you Gentoo, for making this years summer one that I will remember for a long time to come! Looking forward to another great summer next year (maybe as a mentor? ;))

On another note, I’ll be trying to get Beacon to be officially accepted by the GDP. Which means lots more work to do, many improvements to be made. If Gentoo decides to retain the repository, I’ll continue developing on it there. Additionally, I wanted to be as generic as possible while building Beacon, which means that it has the capability of being an editor for almost any XML dialect. I’ll also be looking into those possibilities sometime in the future. Meanwhile, I’ll be working towards deploying both Beacon and repodoc-web on http://guidexml.soc.gentoo.org/, hopefully before next week.

As far as my other accepted applications are concerned, I’ve already begun work on the PHP-GNOME proposal. The other two would have to wait a while I guess

So, that’s it. Hope all of you have had a great summer! Cheers!

Update: Newer Screenshots
Phew! At last we have a working prototype of the GuideXML editor, and it’s called Beacon. Before we get bogged down with the technical details, lets have a look at the good stuff first

The welcome screen for Beacon is very simple, two options: Create new GuideXML document, or edit an existing one:

Once you’ve entered the essential details, you’re taken to the actual editor. It’s mostly based on Dojo’s InPlace Editor widget, but the more complicated tags like <p> and <pre> are handled by a textarea instead (basically, tags that can have more tags in them). Although they are just stupid textareas for now, watch out for shiny new RTEs like TinyMCE, they are bound to replace the textareas anytime in the near future!

With Beacon, you can even edit existing GuideXML documents with ease. Er, almost, don’t try editing the Gentoo Installation Handbook yet! (to be specific, <book> based GuideXML documents are not yet supported) But basically, everything remains the same as how it was when you create new document except that the content is already there:

Okay, the cool toolbox on the left allows you to drag and drop GuideXML elements on the document to add them. Dojo’s DnD kind of sucks, but MochiKit’s packed version doesn’t have DnD support yet…

…Now the problem is that you can’t remove elements yet. Lame, I know, but I guarantee that the space below the toolbox is reserved for the DOM tree of the document. That neat little widget will allow you to re-arrange or delete your GuideXML elements at will.

Neat. Now onto the gory details. The whole application is meant to be a generic XML editor, yeah, that’s right, Beacon is not entirely specific to GuideXML alone! Thanks to Dojo and MochiKit (both of whom still don’t like to talk to each other, I made them sit in the same room with great difficulty; Dojo wouldn’t let MochiKit in unless it was `packed’!) the entire application is driven by powerful XML configuration files that control all aspects of the application.

Beacon maintains dual representation of the document at all times; that means we have two DOM trees, one representing the GuideXML document itself, and the other representing the HTML (and editable) version of the document which you see in the `Design’ tab of the editor. Thanks to XSLT, interconversion is quick (I won’t say painless though, XSLT is TOUGH!).

Well, I better get back to making the editor look more beautiful while thinking about that DOM Tree widget… Meanwhile, you guys, enjoy!

So, it’s been a while since I posted anything, but that only means I’ve been in my coding hole all day all night

Lots of progress on the editor front, I finally got started on the UI and the AJAX backend. In priniciple, this is how it works: We have the welcome page which first asynchronously makes a request to the server for an XML file that contains everything needed to make the forms on the welcome page, essentially the fields that the `Create new File’  form requires. This may sound stupid, we already know what we need to create a new GuideXML document, but, I decided to take a more generic approach for two reasons:

• Easy Extensibility. What if there are changes to the GuideXML format itself?! This way, we also open up avenues for making a generic XML editor, not specific to GuideXML.
• Easy Translation. As mentioned in my proposal, I’m going to use the PO-file format to handle translations. Requesting for information at the very beginning allows me to get the strings to be displayed immediately.

True to the `AJAX’ concept, the PHP script returns an XML file (which is actually part of a larger `configuration’ file for the editor itself, more on that soon) which is then parsed by the friendly JavaScript and the form displayed to the user.

I’m going to be focussing on the creation of new documents at first, and so I’ve disabled the `Edit existing Document’ feature for now. So once the form for creating a new file has been filled, the JavaScript once again kicks into action and validates the form; if all is well, we submit the form to another PHP script that handles the initialization of the editor itself.

Coming on the editor, what we have here is a `Toolbox’ on the left with 3 menus and a tabbed pane on the right which acts as the editor itself. The first menu called `Primary Elements’ consists of buttons for modifying the Title, Author(s), Abstract, Version and Date of the GuideXML document. The second menu is called `Content Elements’ and has two buttons named `Chapter’ and `Section’. My plan is to be able to allow the user to simply drag-and-drop the Chapter/Section button to the editor, and, voila a Chapter/Section is inserted! Likewise with the 3rd menu called `Body Elements’, consisting of <p>, <pre>, <note>, <warn>, <impo>, <table>, <ul>, <ol>, <img> and <dl>.

In the editor area, every editable portion (title of the chapter/section, contents of any body element etc.) highlights when you move your cursor over it. Double-clicking it will open up a textbox (in cases of one-line contents or when formatting is not required/allowed, the chapter title for example) or a textarea (in cases where formatting is required, like the contents of any body element). Now the textarea is where TinyMCE kicks in, all textarea’s on the page will be replaced by a heavily-modified and stripped down version of TinyMCE that I am working on. Essentially, this WYSIWYG widget will allow you to format the text as <path>, <c>, <b>, <e>, <sub>, <sup>, <mail> and <uri> in the case of all body elements other than <pre>, and <i>, <comment>, <keyword>, <ident>, <const>, <stmt> and <var> in the case of pre-tags. Now this is kind of tough, I need to develop a plugin for each of these tags, the only functionality I can reuse from TinyMCE is the spellcheck plugin and the <sup>, <sub> tags.

Moving on to the backend, I was thinking of an XSLT file that translates the GuideXML source to a div-filled HTML format suitable for display in the editor and vice-versa. Actually we need two of these to convert from one format to another, the one to be used will be determined by the active tab; if the `View Source’ tab is active we convert the source to HTML, HTML to source otherwise.

Some knick-knacks to sort out would be stuff like how frequently the file should be saved, and how exactly that is to be done. All-in-all, pretty interesting stuff, please do have a look at the code in the SVN repository if you’re interested

So until next time, Cheers!

Stuart inspired me to do an objective evaluation of all the toolkits out there and then make a choice. So, here it is, my personal evaluation and reasons for choosing what I chose!

Okay, so that’s the brief summary. I haven’t included prototype in the list because its too fundamental and a lot of the toolkits are actually based on it. I also have probably missed many more toolkits, but these were the ones that caught my eye.

The main factors that we kept in mind while making the choice were:

• License. It is important for the toolkit to have an open license that falls within the limits of the Gentoo Social Contract.
• Infrastructure. The toolkits’ deployment requirements must be something that the Gentoo-Infra team can provide.
• Documentation. It shouldn’t take a long time to learn the toolkit itself. One of the important requirements of the project is rapid prototyping since the UI forms a core component of the editor.
• Stability & Support. The chosen toolkit should obviously not be one that is dead. It is important for the toolkit to be stable, and under active development; since the application may be used by hundreds of people.
• Features. Of course, the stuff that the toolkit offers should match with what we require. Ideally, it shouldn’t offer anything more nor less.

Based on the above, all Java based tollkits are out. The GWT has a particularly restrictive license. I wasn’t too comfortable YUI because it wouldn’t validate. I ruled out Dojo because it wouldn’t work in Opera. I am not too particular about the application working on IE, but I would expect it to work smoothly in Firefox and Opera at the very least. I started out with OpenRICO but then realised that I’m spending too much time finding out what functions it offers.

MochiKit offers most of the effects that script.aculo.us has, besides being one of the most awesome toolkits out there. One look at its documentation, and I knew this was it. It’s very programmer friendly, and work exactly like how you expect it too. MochiKit forms a very nice abstraction layer over JS to hide all the gory browser quirks. Its visual library however, is a bit broken, basic functions such as `Move’ or `Scale’ are broken, although they’re working on it for 1.4.

I couldn’t not use MochiKit just because of its Visual Effect backdraw, the other stuff that it offers are just too amazing too miss. So here’s where the X library comes in. I believe MochiKit and the X Library to perfectly complement each other; use MochiKit for the Model and Event Handling, while leaving X to do all the jazzy stuff. It’s working out quite well uptil now, although ideally I would prefer to use MochiKit alone (If only the `Move’ function would work!). But then, X is quite amazing too, and its fun to use both the toolkits.

So that’s that, I’m going to go ahead with MochiKit and the X Library. More on how the UI is going to turn out later.

(If you’re looking for the right AJAX toolkit to use for your application, I’d suggest you start your search with MochiKit. There are also good listings at OSAF and AJAX Patterns)

At long last, I have managed to create an ebuild for the repodoc web service and its sample client. This should help all interested Gentoo devs in installing it and giving us feedback. If your overlay is located at /usr/local/portage, you must first build the Manifest: `ebuild /usr/local/portage/www-apps/repodoc-web-0.0.1_alpha.ebuild digest’. After that, `emerge repodoc-web’ should do the trick.

But that’s not all. As the post-install instructions say, you must do two more things before you can start using the service.

1. Change the first 2 define lines in validate/index.php:
   define(’HTDOCS’, ‘/path/to/htdocs’);
   define(’REPODOC’, ‘/path/to/repodoc/executable’);Please note that the service will work well only with the latest GIT checkout of repodoc from Ferdy’s repository. The repodoc-0.0.1_beta ebuild will still work, although the output may not be as predicted as there are module changes between the ebuild and GIT versions.
2. You need to create the /.repodoc directory and assign its ownership to the web server (usually ‘apache’ or ‘nobody’). This step could not be included in the ebuild because of access violations. This is strictly a repodoc requirement and not related to the web-service. The repodoc team and myself will be working on a solution for this soon.

Once you’ve done these two steps, you may access the client at http://localhost/repodoc-web. Select a GuideXML file and click on Validate to get a tabulated result of the validation.

Now, I’d like to point out that the system is far from perfect. There are two basic issues that I need to tackle:

1. Eliminating the `modify define lines’ step since webapp.eclass knows the location of htdocs. Also repodoc is usually at /usr/bin/repodoc, but this is not entirely true if you’re not using the ebuild and just the GIT version.
2. Error Handling for non GuideXML files. Trying to validate a non GuideXML file will result in a non-helpful error message. Proper handling is necessary here.

Well, of course these aren’t the only flaws, so I’d love to hear about what you have to say about everything!

Building the ebuild was quite tricky, considering this is the first one I’ve made. The webapp and depend.php eclasses made things a little easier. The webapp eclass was nicely documented, and looking at other ebuilds like phpmyadmin or tikiwiki also helped quite a bit.

On other news, I was frantically trying to finalize on a particular AJAX toolkit. After further discussion with Stuart, who very kindly let me make the choice, it looks like we’ll go with Dojo. Although I’m not entirely sure yet. I’ve seen better eye-candy with Echo2, another Java based toolkit. Also, GWT is out because of licensing problems. Apparently, the license states that:

“you may not distribute Google Web Toolkit Development Tools or any services or software associated with or derived from them, or modify, copy, license, or create derivative works from Google Web Toolkit Development Tools, unless you obtain Google’s written permission in advance.”

Quite a day, I’m still undecided on the toolkit, but atleast I can sleep with the comfort of having written my first ebuild. Go Gentoo!

So, I’ve been pondering over how exactly the editor is going to be implemented. I’ve decided not to use any of the existing web-based WYSIWYG editors, but rather use one of the AJAX toolkits out there to build the whole application. I’ve narrowed it down to 3 choices, and here they are, in order of my preference:

1) Google Web Toolkit - Yep, it’s a Google product, so it should be quite fun to use! They have the good old swing-type widget concept here, so the main editor would consist of a custom widget that I will extend from the basic textarea. Yeah, if you notice, I will have to code in Java, which is not a problem for me at all. Before you arrive at any wrong conclusions, I would like to make it clear that GWT uses Java classes only as a base to generate code. The end-result is pure JavaScript + XHTML that can work with any server-side language (which in our case, is PHP). The advantages I see here are: support for a wide variety of browsers, a good solution for the triple-b (Browser Back Button) problem, and a robust framework to lean on.

2) Yahoo UI Library -  Ok, so I mentioned Google, it would be unfair not to mention the competition. The Yahoo UI library is damn stable and widely used (it’s not in “beta” like most of Google’s products!). Rich widget library, although building a custom editor widget here would be much tougher. Great effects, cool CSS and design patterns to go along. Pretty much everything to get the editor up and running.

3) Dojo - I don’t know why, but this reminds me of Django. Well Dojo uses the Moxie editor, which is to be modified to handle GuideXML instead of plain HTML. One thing that caught my eye here was the persistent storage functionality, that would allows users to temporarily save the document they are editing, on the server. I’m not sure we want that feature in the editor, but even otherwise, this toolkit is quite interesting.

Other than these 3, there is also Rico and Mochikit. This is the worst part about FLOSS, there’s just too much choice! Although my scales tip heavily towards the GWT ( this being the Google Summer of Code and all ); I plead all of you to please send me your opinion of what toolkit would be the best fit for the project.

Okay, so I got together a decent client for all of you to test out the repodoc web service. It's pretty basic, just about 35 lines of code, uses the cURL and XSL extensions of PHP5, and displays a tabulated page that shows how tha validation went. I haven't concentrated too much on the eye-candy yet, I'm waiting until I decide on a nice theme for the editor itself.

Check out the client, the upload form, and a sample result file.

Meanwhile, I'm steadily moving on the ebuild, it should be done now that the client is ready. I'm also going through "AJAX Patterns and Best Practices" to see which pattern applies to the editor the best. Needless to say, if any of you have ideas, comments, suggestions or cristicisms, be sure to send them my way

A brief update on what's been happening. So, according to my schedule, the web service is supposed to be all ready to go, which it is. So now onto the next phase, Interface Design. Interesting stuff, now this is where I get to design all the eye-candy and decide on how usable we want the editor to be

No surprises here, the good old XHTML + CSS + JavaScript combination can do wonders. I don't think I'll need to employ anything else to get a nice UI up and running. I'll have to keep in mind to keep the JS open enough for AJAX integration later on though.

But, before I get into that, Stuart wanted an ebuild for the repodoc web service. Which is an awesome idea, fits into the `Gentoo Way' ™, besides making the installation for the web service a lot more generic and painless. My ebuilding skills are amateurish at best, but hey, I think I can pull it off. To begin with, the webapps.eclass should help immensely. The man page for that is quite informative, so I don't thnk I'll have any trouble here. OTOH, there is nothing much right now to package, just 3 files.The package would be a lot more substantial if I wrote a client for the web service and included that too. So I guess I'll do that before writing the ebuild.

And my totally awesome mentor has posted a blog entry on Mentoring SoC Participants! I'm very glad to be working with a mentor who puts so much thought into his work, thanks Stuart!

That's about it, stay tuned for updates on the ebuild and the UI. Cheers!