I am in Toronto right now, spending a week with Mobile folks. It has been about 6 weeks since I started working on a mobile extension, and the list of dirty tricks we employing to get things done is growing. Some of these hacks are there because there is no alternative, and some are there because I just didn’t figure out the right way to do it.

Here is a short list:

Gestures

I reinvented them, poorly. Our extension relies heavily on the ability to support a rich set of multi touch gestures. We have a hack, similar to how it was done in Touching is Good (I know!), where we swap out the callbacks in MouseModule and GestureModule for our own. This doesn’t seem right, for a few reasons. The main one being that this doesn’t seem like a real public API, and it will be broken sooner or later. The second thing is the fact that we are relying on shaky JS to re-interpret the platform-interpreted events. So maybe there is a better way to do things. The nsIDOMSimpleGestureEvent only serves us to a limited degree, I  think there is room for a more low-level multi touch event API that is similar to the Android one. Or maybe a comprehensive high level API with more event types.

Speech

This should probably be implemented natively and have multi-platform support. Right now, in Android, we are doing Javascript <-> js-types <-> jni <-> Java. What could go wrong?! But seriously, it is kind of ugly right now, but I would like to make our JS solution prettier and useful to other extension writers who want to tap in to Android’s API and services.

Security

The Mozilla profile directory in Android is not world-readable, presumably for good reasons. Our extension’s media files need to be world-readable so that Android’s TextToSpeech service could pick them up and use them as earcons. This requires an install and uninstall hook that uses Android’s Context.getDir to get/create a subdirectory in Firefox’s top-level app directory where we could copy over those media files for system consumption. Maybe this is the best way to do it… But I can’t imagine anyone being happy about an extension writing to that directory. Sorry!

–enable-accessibility

Our extension depends on accessibility being enabled at build time. Since I am getting some face time with folks this week, it seems like a good thing to bring up.

I am sure there are other horrible horrible hacks in there. But the list above is really what I needed to confess to. Thanks for empathetic attention.

After making a non-binding resolution to report my Caribou progress on a weekly basis, I flaked. Of course. But luckily Nohemi has picked up the slack and have kept you all up to date about the libcaribou powered GNOME Shell keyboard in her more binding GSoC reports. So no more architecture diagrams are needed, you all get the idea. But if you didn’t, let me make it clear: The goal of Caribou is to make it easy to implement new on screen keyboards where you would only need to provide the view, and libcaribou will be your model and controller.

It is better to admit now than later: I will not have the bandwidth to continue to work on Caribou as my time is slowly running out. So…

What we need:

• A maintainer.
• A GTK module, Nohemi is working on this, but we will need similar solutions for other toolkits and fallbacks (XIM?).
• Unit tests for the library – libcaribou is gaining features and getting complex. We need some tests here.
• More keyboard layouts/languages.
• More function keys for the switch scanning keyboard.
• Finer interaction modes:
• Modifiers, use latching and traditional key gestures with multitouch.
• Respect AccessX settings for sticky/slow/bounce keys.
• A more hardware-like interaction where keys will auto-repeat when held down.
• etc.
• … and automatically choose the right mode for the keyboard without comfusing the user.
• Revisit the “X adapter” and maybe use something more high level.

Anyway, plenty of exciting work. Are you at the summit, please find me if any of this interests you.

I’ll be talking about inclusive design.

Since the last time I wrote to you, dear bloggy, I have been working a lot on Caribou, made a trip to welcome Jenny back from Haiti, and crossed the continent by rail.

Let’s talk about Caribou, so much has changed!

Antler

Antler is the Keyboard UI that is bundled with Caribou. It does not have the pretension of being ready for users any time soon, it is more a sample implementation of a libcaribou keyboard,and a place for me to try stuff out and see if it would work in our platform. You could follow Nohemi‘s progress to catch up on how libcaribou is being used to power the new GNOME Shell keyboard UI. With all that said, Antler is still kinda cool. Here is a crappy video of Antler’s touch keyboard in action:

Scanning Support

I redesigned Caribou’s switch support from the ground up with the goal of simple configuration. There is still plenty of more work to do, but after looking at commercial alternatives I feel like we could do a pretty good job. Here is me typing text solely with the right shift key:

Input Method Support (bye AT-SPI!)

This is an experimental tangent, that might or might not be worth the time I spend on it. Recently Caribou master received GTK2/GTK3 input modules that perform DBus calls to the Caribou keyboard, and have it show up when and where it is needed. This has proven to be pretty tricky. I will hopefully follow up with a post about this, and some interesting hacks innovations surrounding these methods. Future work includes writing similar modules for QT3 and QT4. And having the keyboard emit key activation signals that the modules could use for inserting text instead of using XTest which feels so hackish and wrong.

If I could be sentimental for just one paragraph and say that all the hard work folks have put into GObject introspection is finally paying off. Big time. Transitioning between all these different languages at will is really amazing. I think we are entering a perfect storm where we have an extremely competitive developer story. There is really nothing quite like this. Yay.

Since I spent this week shuffling deck chairs, I don’t have any exciting new visuals. Hopefully soon.

I didn’t get a chance to talk about my plans for Caribou in this little private SOC I am having here. Well here are two oil rigs is a diagram to illustrate it:

Caribou Daemon

The daemon, through AT-SPI and any means necessary figures out when a typing task is needed and activates the keyboard through DBus. When it does this, it provides as much information as possible regarding the text entry task: The location and size of the text area, the location of the cursor and the type of text expected (plain, email, url, number, full name). The daemon is written in Python, at least for now as Python is equipped with the best AT-SPI client library. The daemon has a relatively small body of code, so porting it to C is a worthy mission for some time in the future.

libcaribou

This is a library for keyboard implementors. It is written in C/GObject. It handles all common Caribou keyboard tasks such as:

• De-serializing Caribou keyboard definition files into KOMs (Keyboard Object Model, I just made that up. You are welcome), with respect to the user’s XKB configuration and keyboard type preferences.
• Implementing defined keyboard behavior.
• Exposing the keyboard over DBus with interfaces known to the Caribou Daemon.
• Implement scanning.
• Do all the ugly X tasks, XKB, XTest, you name it. This could be replaces/supplemented with a higher level input method in the future.

Keyboard UI

This is the view for libcaribou’s model. It can be written in any language and toolkit that supports GObject introspection. The packaged UI is called Antler, and currently it is less than 200 lines of Python/GTK+. The idea here is that GNOME Shell (or anyone) could easily implement their own keyboard that can take advantage of Caribou’s facilities. If we pull this off well, keyboards will be advertising capabilities and co-exist, so the appropriate keyboard UI is invoked for the given task.

Thanks for reading! In a future post I will describe Antler and plans I have for it.

I have been spending the last while working exclusively on Caribou, trying to get it out of the miserable shape it is in and make sure that it not only remains relevant, but really shines. I have been doing this for two weeks. But the first week was mostly spent on chasing down this bug in GDK. So I really only got down to business this last week.

Here is a photo:

What I have done:

• Created an introspectable C library, initially to do the keyboard emulation and XKB stuff, but I have grand plans for it. What already landed in master is just simple key synth methods to replace Caribou’s reliance on python-virtkey which was never approved as GNOME external dependency.
• Created an experimental git branch with a revamp of how we do layout in Caribou. Users will no longer choose between layouts, this is inferred from the current keyboard group the X server is set to. If the user will have any choice it will be between geometries, either a more natural geometry similar to what you have on tablets, or a fuller keyboard emulation.
• Made “sub keys”, all the latest screen keyboards seem to do this, so now Latin accents and Semitic vowels could be easily entered.

Ok blog, I have to go. I’ll need to write a brief roadmap of where I think we should take this, and how Caribou could play a role in GNOME Shell, etc.

Here is a screencast of what Caribou looks like today.

1. Install a pre-release distro like Ubuntu 11.04 or Fedora Rawhide. While the Caribou package is not necessarily up to date on these platforms, the prerequisites generally are, so you just need to clone Caribou from git.
2. Use JHBuild. This is what all serious GNOME testers and developers use. Unfortunately Caribou is not in the current moduleset, and neither are some dependencies. Fortunately, I wrote a small moduleset file that takes the shortest path to having a working version of Caribou for your testing needs. It also has Orca, and other modules will be added in the future. This should probably go into upstream jhbuild eventually.

Once you have installed JHBuild, use the configuration below in your .jhbuildrc, or create .jhbuildrc.a11y, and use jhbuild -f if you have a working jhbuild environment you don’t want to mess with.

build_policy = 'updated'
moduleset = 'http://people.gnome.org/~eitani/a11y/gnome-a11y-3.0.modules'
modules = ['meta-gnome-a11y']
checkoutroot = os.path.expanduser('~/gnome-a11y/source')
prefix = os.path.expanduser('~/gnome-a11y/install')
skip = ['gudev']

Next, run jhbuild bootstrap to install build tools, and install some other development libraries that are worth skipping in jhbuild (for now it’s just gudev).

In Fedora:
yum install libgudev1-devel

In Debian/Ubuntu:
apt-get install libgudev-1.0-dev

Run jhbuild build, and watch things compile successfully.

Now, the hairy part about GNOME accessibility is the fact that there is an AT-SPI registry that needs to start with the session. Optimally the registry runs on a dedicated accessibility D-Bus bus. First you need to make sure you installed and enabled at-spi2 through your distro. On Ubuntu you might want to install it via PPA, on Fedora 14 it already is in the repositories, search for at-spi2. Next you need to enable desktop accessibility and D-Bus support:

gconftool-2 --set /desktop/gnome/interface/at-spi-corba --type bool false
gconftool-2 --set /desktop/gnome/interface/at-spi-dbus --type bool true
gconftool-2 --set /desktop/gnome/interface/accessibility --type bool true

There are a lot of tricks to get the newly built registry to run in your session, but the easiest way is also the most hackish, simply edit /usr/share/dbus-1/services/org.a11y.atspi.Registry.service and substitute /usr with the prefix to your newly installed registry, for example /home/eitan/gnome-a11y/install. Log out and back in again, and you should be using the latest AT-SPI2 registry.

If everything worked, running jhbuild run caribou should bring you the latest version of Caribou.

Good luck!

Collabora

I have been drafting Telepathy specs, improving Gabble, drafting more specs, and generally learning a lot. Some (potential) user-visible features include bomb-proof XMPP invisibility, interactive authentication, power saving, communication policy (contact blocking), improved codec-conflict user messages in VoIP and video sessions. And more that I can’t remember right now. Of course this is on top of the client work that is not public yet. It’s been a great experience, working with energized smart people like Simon, Will and Sjoerd is fantastic. I would love to head to FOSDEM this year and see everyone, but I will be turning 30, so probably not.

GNOME Accessibility Team

Early last year Willie Walker’s Orca team in Oracle was finally liquidated and we were not sure where things were headed. I was working on putting a hackfest together, giving a talk, organizing a booth, and generally having a presence in CSUN. This was the first a11y meetup, and I spent time preparing a presentation, dealing with sponsorship and travel, getting resources for the hackfest and booth. Besides saying goodbye to Will at CSUN, it was also a disengagement for me, as my job and other projects were keeping me elsewhere.

Since that hackfest there has been another one in Seville, Alejandro and Joanie have been leading the GNOME a11y team like nobody’s business. There is even a weekly meeting that I never attend on a regular basis. Things are looking good, it is really humbling. Joanie started doing this work pretty much at the same time I got involved in a11y, and she is still plugging away. Somebody please hire her so she could do this full time.

I also gave a talk at GUADEC about Universal Design. I learned a lot preparing it, and I hope to continue polishing it.

Getting Closer

At some point this year, I found the time to develop an iPhone application, a GIS Django backend and a web-based map authoring tool! The idea of the app is basically aural augmented reality. For example you could walk around in a city neighborhood and listen to past events and stories of different places. The main difference from an audio tour is that you are being led by sound as it gets louder when you approach as opposed to visible landmarks. Of course the technology is the trivial part, the real magic is in the well-produced audio. Jenny Asarnow has been doing that, and created fantastic soundscapes with stories and music. Full discloser, she is also my sweetheart. Getting Closer was showcased in two events: Megapolis in Baltimore and Third Coast in Chicago. We hope to collaborate with more audio producers and find new and exciting uses for this. As for my end, it was fun to step out of my niche Desktop programming and do some mainstream stuff, like iOS, Django and HTML/JS, just to know I could.

Causing A Nuisance

I joined other young Jewish activists in New Orleans this fall where we successfully agitated the Jewish American community into conversation by interrupting the Jewish General Assembly, the largest annual event surrounding Jewish philanthropy. Disrupting the Israeli prime minister’s keynote was really just an added bonus. This is one of the most important things I have been part of this year, and I am completely flattened by the amount of positive responses we have gotten for this. I wrote an essay about the experience that I hope it will be published soon. I am looking forward to continuing to be part of this conversation.

Caribou

After things settled down, and I caught back up on work, I started giving Caribou some desperately needed attention. I asked that it be pulled from the GNOME 2.32 module set in the last cycle because I just never had time to really get it ready for release. No excuses! I spent the holiday slowness porting Caribou to GTK3 and GObject introspection. Let me tell you, it was not trivial! I ended up rewriting whole chunks of Caribou, which was not all that bad. Since taking over maintainership I have done little in doing major changes, and accepted some large patches that I never should have, so it was cleanup time. This whole ordeal did not introduce any user-visible changes besides a new prefs window and stability. Um, I guess that is something. Anyway, I will be merging it to master today. That bad news (and this is what sucks about GNOME 3.0), is that other contributors and testers will need to do a jhbuild dance before being able to run any of this. I have a moduleset I will share with the world that does the minimum required stuff for a development a11y stack, with the new AT-SPI2, Orca, Accerciser and Caribou. There are some amazing artists in the GNOME community, anyone fancy on designing an icon for Caribou?

That’s Not It!

I am sure I forgot stuff.

I also prepared my talk. You should come, it’s on Wednesday at 11:15 AM in the Seville room. It’s going to be extremely nontechnical, but I hope you could follow. I don’t think a single acronym will be mentioned, although I don’t hold me to that. We will be discussing the software development process, design, things we take for granted every day, and door knobs.