Category Archives: programming

GSoC 2013: Text, images, delays

image by ericwa

image by ericwa; please observe 'textedit' info window and its preferences on the right side

Given the delay since the last update, this should be a lengthy post. Since I'm writing this at a very late hour, it won't be.

Shortly after the last update, I added painting of images, as long as they're 32-bit. I finally began considering the 'gstateless' variant of the backend, where the OpalGState class would be removed, as well as the dependence on GSContext family of classes. The idea was that all state switching could be managed by Opal. Unfortunately, Fred explained how AppKit treats GState and the idea does not map well to Core Graphics's one-graphics-state-stack-per-context model.

Then I went to Dublin for a job interview. When I got back, I was (shall we put it mildly) highly distracted by the experience of another travel to a foreign country. After that, a two week summer camp where I almost no progress was done. A place with no good internet connectivity (except several cafe bars) would be tolerable were there a place for me to unpack my laptop as permanently as possible. Unfortunately, any such place was a computer classroom, and my accommodation was anything but appropriate for the activity. Other distractions of a summer camp also came in play, such as 'Hey, come here, let's work on the movie.'

Would coming back home change things? Well, after a massive headache due to low amount of sleep during the last day of camp and during the return trip, after a cold, and after a toothache, I finally got to work.

Thanks to a lot of help with Eric on the Opal side (and even a bit on the backend side, because he's cool like that), we now have font rendering! Hooray!

Unfortunately, you may notice that the above images feature rectangles again, instead of icons. This time they're yellow rectangles instead of red ones. Why? -compositeGState:... method is being called after -DPSimage:... was called, so we end up with a 'not-implemented' marker for -compositeGState:....

What's next? Well, wrapping up as many things as possible by the end of the week, when GSOC ends.

Yep, I could and should have done far better this year. Oh well.

GSoC 2013: App content now visible!

I completely agree with you, dear reader — I don't like it either that it took so long to get somewhere useful. 🙂

But, here's two screenshots that are right now making me happy. (Aside from lunch, that is.)

GSoC 2013: first useful screenshot of SystemPreferences

GSoC 2013: first useful screenshot of Calculator

Red boxes are images. On the top left of SystemPreferences you can see the menu. The white blob in SystemPreferences is where WindowMaker drew its HUD to display window position while moving the window around; doublebuffering is disabled at the moment.

Calculator uses NSWindows95InterfaceStyle, so the menus are embedded inside the window. (A necessity under XQuartz.)

So, how did I get here?

Doublebuffering

I had no idea where to start from fixing the issues. So I turned to trying to write the backing image into a file. That's easy to do with Core Graphics; but, there happened to be a small bug with Opal. Eric stepped in once again, and fixed saving to files. I'm really thankful to him for doing this so quickly.

Now that I could output files, I saw that backing images for menus were correctly drawn. What went wrong was sending things to the screen. Menus generally worked. Hooray! That was a big confirmation that something small was going wrong. Images painted on screen were incorrectly positioned, and when running under XQuartz on OS X, even sized incorrectly. (XQuartz forces a minimum window width even for menu windows.)

At some point, I figured out, what the hell – let's check what CGContextDrawImage() specification says. Lo and behold: the argument specifies only the destination rectangle, and does not relate to the parent rectangle! The solution is to use CGImageCreateFromImageWithRect(). I ran into a small Opal bug here, which I reported to Eric; the new image still includes the original image (just like the documentation says it should), but under Opal, this means CGImageGetWidth()/Height() report original image's values. Also, writing such a sub-image still writes out the entire image to disk.

Unfortunately, I didn't get this to work yet. I'm still getting weird offsets, but now I'm on the right track. Currently committed version has doublebuffering disabled (which means moving anything over the window clears that part of the surface and does not cause correct redraw).

Saving and restoring graphics state

So… -DPSgsave and -DPSgrestore. Although I was aware that this is the thing to look at in case things mess up with matrices, and I spent hours and hours looking around for what went wrong (very small amount of code, but a lot of exploration), it didn't occur to me that I, perhaps, did not implement the methods.

Eric pointed it out to me that I didn't.

I checked, and indeed: I was going around in circles just because I did not push state to the stack where appropriate.

So I implemented these methods under names -DPSsavegstate and -DPSrestoregstate. When it didn't help, I looked around for ideas about what else may have went wrong.

Then today, having run out of ideas, I triplechecked what I did, and the method names were incorrect. With Eric, I only discussed that state needs to be saved; I have absolutely no idea where I pulled the method names out from or why I even though they were correct.

Then, as I went to check what the method names should be, I found they were named -DPSgsave and -DPSgrestore. And they are in, for example, CairoContext — not in CairoGState. Which is perfectly reasonable: GState classes are exactly what these methods manipulate, wrappers for state of backends whose underlying libraries do not contain a state stack, or need to maintain additional state.

So having overridden these methods, called super and then passed them on to OpalGState, I finally got something sensible on screen. Hooray!

Next: Cleanup and moving DPS methods to OpalContext

Eric has suggested moving (nearly) all methods from OpalGState to OpalContext. This definitely makes sense; if there's some state that happens not to be maintained by Opal, it probably makes sense to add the required functionality to Opal and not to the backend.

I'll also be cleaning up the implementation before moving on to adding fonts et al. There's leaks all over the place because I (intentionally) did not pay enough attention while researching what the problems are.

Mid-term self-evaluation for this year's GSoC is between July 29th and August 2nd. I'll be submitting relatively low scores for myself; I'm personally unsatisfied with the time it took me to reach this stage. I'm hoping that the worst is behind me. On the other hand, I still have to deal with fonts. sigh

GSoC 2013: Doublebuffered painting!

Thanks to a fix that Eric applied to Opal's handling of painting CGBitmapContext->CGImage on top of another CGContext, the backend now actually paints something sensible on a doublebuffered window. Hooray!

Well, actually, it only paints something sensible on a menu window. And as soon as the backend has to paint an update, for some reason the updated stuff is painted (mostly) upside down. I've tested with the Gtk theme, and the in-window menus also seem to be updated upside down.

I also seem to be unable to get menu items to be wider despite various hacks when it comes to fonts. It seems that OpalFontInfo's -widthOfString: is not used at all, and a series of calls to -advancementForGlyph: is instead performed, with no effect on the width of the menu whatsoever.

I'll explore further what's wrong with both of these issues tomorrow; preparing for a job interview today will take the rest of this day.

Note: I did not commit my yesterday's work described above, as it mostly consists of hacks that have no useful effect onscreen. Eric's changes to Opal, combined with a change of #if 0 to #if 1 in gnustep-back's OpalSurface's init method do produce some of the visible changes described above.

GSoC 2013: Drawing some actual content

Looking at the existing [...GState DPS...] methods and comparing them to the Core Graphics functions, it looked like implementing the backend will be as much of a cakewalk as I originally presumed.

Turns out it is not meant to be.

(Today's text is a report as much as it is a way for me to contemplate next steps. It's a noisy stream of thoughts. Sorry about that.)

All is drawn, but nothing is seen (a.k.a. doublebuffering)

The Cairo backend contains code intended to render into a backing store and then draw the content on screen when requested to do so by X. (Presumably, also to blit the content when it updates.) Theoretically, it should be trivial to add this functionality: add a new CGBitmapContext, redirect all drawing into this new context, and then blit this content on screen when requested.

When creating the surface that targets the screen, one gets a gswindow_device_t* that describes the X11 window. To tell the rest of the gnustep-back that this surface can handle its doublebuffering (although it really can't), we set this gswindow_device_t's gdriver to self, and we add flags GDriverHandlesExpose and GDriverHandlesBacking to its gdriverProtocol bitmask. This is done only if the surface is not NSBackingStoreNonRetained.

(From the above description one would think by default, backing stores would be already handled elsewhere in gnustep-back. Unfortunately, although it seems like gnustep-back might be trying to handle the case where the surface does not handle X11's Expose event nor does it handle having the backing store, it does not do so, at least not properly.)

So, I've tried to do it. All drawing is redirected to a CGBitmapContext, and then in -handleExpose I'm trying to blit the content on the X11-backed Opal surface. It seems that Opal currently has some issues with the following code (i.e. I get everything drawn in pale yellow):

- (void) handleExposeRect: (NSRect)rect
{
  CGRect cgRect = CGRectMake(rect.origin.x, rect.origin.y, 
                      rect.size.width, rect.size.height);

  CGImageRef backingImage = CGBitmapContextCreateImage(_backingCGContext);        
  CGContextDrawImage(_x11CGContext, cgRect, backingImage);  
  CGImageRelease(backingImage);
}

Debug output from Opal suggest it's going through all the expected dance moves: creating that image, converting it into destination context's colorspace, et al. I'll poke Eric to see if he has a better idea of what's going wrong.

An interesting behavior is that if I simply don't redirect drawing to the bitmap context, then I get some content on screen. Despite not having an actual backing store. Maybe GDriverHandlesBacking has side effects? I'll have to dig in more.

(Oh, and gotShmCompletion methods seemed like they might be related. No change if I implement them.)

Everything is flipped

When I first got some content on the screen, it turned out that GNUstep's zero-zero seems to be based on top-left corner of the drawable — opposite from what Opal expects.

So as a first attempt I went in and implemented matrix functions. While things don't get drawn in the appropriate place yet, it seems like I'm on the right track. The thing that worries me is that I'm not sure what's the orientation of DPS matrix as opposed to the orientation of the CG matrix. Also, I'm now overriding DPS methods and not calling super, yet there's a bunch of code in GSGState's implementation that makes me think this is not such a good idea. DPS functions seem to be maintaining matrix state; hopefully they don't ever use it directly. If they don't, then it might be safe to simply override the getter for the matrix.

To be able to reset the Opal matrix to the identity (or, near-identity, considering it's also flipping the coordinate system), I've also added a method to Opal that allows this precise operation. This is an extension over what Apple provides: Apple provides absolutely no way to set the CTM (presumably, this stands for current transform matrix) directly, which makes only a little bit of sense. I don't see a good reason this functionality is not provided to developers. (I can think of one that's just a tiny little bit good.)

Menus have a single item

Now all I get in a menu window is a single menu item. Drawn in place of the menu window's title.

Meaning, drawn on a completely incorrect place. I'm not really sure if this is a menu item drawn in an incorrect place, or if this is caused by the fact that -compositeGState:..., which seems to permit blitting one GState's backing surface's content on top of another GState's content, is not implemented.

If this is so, this may prove very troublesome to implement without adding further hacks into Opal. Core Graphics seems to provide no real way to copy a portion of a context on top of another context, apart from using CGBitmapContext.

Fonts. Oooh, boy.

Cairo backend does a lot of stuff when it comes fonts. A lot, lot of stuff. I wonder why most of that is not exposed to the rest of gnustep-back, so I'll probably be moving that out of the Cairo backend and into an additional module within gnustep-back called fonts, or something like that. Or perhaps even into the existing gsc module.

Just taking a quick look, art backend seems to be supporting just fonts with the .nfont extension (explaining Helvetica.nfont), but I may be wrong. Taking a quick look at xlib backend, and it seems to be doing something similar to the cairo backend.

It's going to be a pain to implement fonts, especially to do it properly. I'll have to think about how to deal with refactoring this — cairo backend is popular and accidentally breaking it won't be looked at kindly by the community. Still, it'll be worth it.

Unless it turns out I should have dealt with Opal. In which case Cairo backend's implementation will stay Cairo backend's implementation, and Opal backend will leverage Opal for its text needs.

Yes — just as the title of this section implies, I'm not sure what I'm talking about.

Other upcoming steps

Among other things, I'll look into getting image painting to work. GNUstep does a lot of things with images, so seeing the images might help me figure out what's going on more easily.

GSoC 2013: Red rectangle!

It turns out I'm slower with GSoC progress than anticipated. GNUstep meeting in Cambridge, which I'll discuss in upcoming posts (and do so as soon possible, so I don't forget all the wonderful sights and experiences, and all the wonderful people I met), was informative, but it turns out not quite supportive of my GSoC productivity. I spent a week being nervous before my first international trip, and I spent the last week (after returning) being impressed with how wonderful Germany and England are.

So, what progress did I make?

Preparations

First, before the trip, I prepared the backend. I figured out the simple things, such as where do I have to modify configure scripts in order to add support for the new backend, as well as which classes are expected by gnustep-gui. I identified the minimum set of methods that need to exist in the relevant classes (mostly OpalGState) and I added stubs.

Opal exposing X11 drawables as contexts

In Cambridge, thanks to help from Opal's author Eric Wasylishen, we started exposing the creation of CGContext specific to X11 drawables. The function already existed in Opal; we just refactored the code a bit. (Rest of the hacking time in Cambridge was spent on other tasks.)

Drawing onscreen

Now that I finally cleared my head of amazement at how beautiful Cambridge is and how great hosts we had, I decided that it's really long overdue that we get some content drawn on-screen. It took me a while to figure out where to draw in order to actually get the content drawn on-screen. It turns out the safest place to insert my dummy draw method is -[OpalGState DPSfill] — probably because it's intended as the place to draw stuff, as opposed to various initialization methods such as -[OpalGState DPSinitgraphics].

The dummy draw method draws a red rectangle 1024×768, so I'm very happy to report that the test application's NSWindow is, as of today, fully red.

Thoughts on next steps

Overall, I hope that things will roll much, much faster from here onward. There are bound to be many bugs along the way — and many assumptions by existing apps and themes. It'll slow implementation down, but I hope that along the way I'll figure out a way to clean up the current messy situation with gnustep-back's numerous classes such as GState, Context, et al.

The scariest part is that the Cairo backend looks like it handles many complex behaviors and different use situations. Let's see how many bugs the Opal backend will have in the end and how much of Cairo backend's behavior I'll have to replicate to fix them. 🙂

Google killing XMPP federation with their Google Hangouts?

According to the Ars Technica article on Hangouts, we can expect Google to drop support for XMPP federation.

We should apparently be happy that Google is not dropping XMPP client-to-server connections.

The instant messaging space is apparently turning into a duopoly of Microsoft's Skype and Google's Hangouts with everyone else shoved to the sidelines. I'm not counting Facebook Chat as a serious alternative, and iMessage is not intended as an instant messaging service.

I hopefully don't have to point out how much this frustrates and annoys me. Google is turning out to be worse than Microsoft ever was: they're actively backpedaling on their past promises. They're backstabbing the "open". Scratch that — they're throwing a stake through open's heart, ripping it in pieces, then gorging on its still beating remains. Instead of reading RSS through Reader, we're supposed to read custom posts via the closed and locked down Google+. They are killing iGoogle. They're basically killing the open web and open Internet, while at the same time paying lip service to open.

Android, which is just-enough-Linux-but-not-really. Rich authorship markup, which requires two-way linking to Google+ instead of using the semantic web techniques.

I'll keep on looking for ways to back out of Google ecosystem as much as possible. What's next — Gmail that can't send emails out?