The Open Sourcerer

Announcing ExceptionalEmails.com

If you are a sysadmin or developer or similar you probably get a bunch of emails from systems telling you they are doing just fine. You probably have mail rules to shove these off into some folder you never look at so you can get on with life. If one should happen to not turn up, that would be kind of interesting, but there is no email rule you can make to alert you about an email that didn’t happen. Over the last couple of weeks I have been building a system to fix that http://exceptionalemails.com. You basically shove all the emails you get at a set of special email addresses, one for each type of regular email, and set up rules saying what you expect to happen. You then get on with your life, and if an email fails to happen, or perhaps contains the wrong words (fail/error/out of disk space/etc.) then and only then we will send you an email – you only need to see the exceptions.

This is the form to set up the rules for an alert, so in this example I would set my fileserver backup schedule to email alanbell1+fileserver@exceptionalemails.com when it is done (or leave it emailing me, and set up a rule to put the mail in a folder of my email and forward the mail to alanbell1+fileserver@exceptionalemails.com)

This was my first project using MongoDB as a back end and I have been really impressed by it, I have a background in NoSQL and it all made sense to me in terms of performance expectations and optimisations. I load tested it with a million emails and it was still really fast. It is running on Ubuntu server, with a user interface is written in PHP. The back end jobs that receive emails and check for alerts going overdue are written in Python.

I would be really interested in any feedback on the site, I have some plans for improving the analysis of past emails with sparklines so you can see when failures happened, and maybe fluctuations of arrival times of emails. Any other suggestions would be welcome. There is an outside chance that I might write a JuJu charm for it – and probably do a bit of a refactoring of the code to make deployment easier. One of the reasons for choosing MongoDB at the back end and a separate process to receive the emails was to allow it to scale horizontally across a bunch of servers. Based on my load testing I couldn’t hammer it hard enough to slow things down noticeably so I am not sure my grand clustering plans are going to be required.

The code is on Github, under AGPL3 and I am tracking issues there.

Ubuntu Smart Scopes

A new feature of Ubuntu was discussed today (which is like an announcement but without overhyping it), it is called Smart Scopes and is documented here https://wiki.ubuntu.com/SmartScopes1304Spec go read that first and then I have a video for you to watch.

http://www.youtube.com/watch?feature=player_embedded&v=CBeQur7VBDM

Now go back and read the spec that I told you to read earlier, but all the way to the end this time.

In the video from left to right is Alan Bell (me), David Callé, Jono Bacon, Michael Hall, Roberto Alsina and Stuart Langridge, all discussing this new framework for searching. It is coming soon, to the Ubuntu Raring desktop and then to phone and TV and tablet etc. The objective is to make searching really really effective and helpful to the user, but as with the previous efforts in this direction there will be some concerns around how it is implemented.

In short, Canonical will be running a server much like the existing productsearch.ubuntu.com server which will accept queries and return a bunch of results as json. The current implementation searches Amazon and the Ubuntu One music store and a few other places. The new one will do the same, plus more server-side searches, plus a new feature altogether which is a list of good scope names for the client to search. Your client will now send a list of all locally installed scopes to the server (actually a list of scopes you have added and a list of scopes you have removed or turned off from the standard set) along with your query. The server then returns results it found and wants to put in your dash, plus a subset of the local scopes you sent it, in order, that the server thinks would be good places to hunt for your search term. This means that your client might have 100 or more locally installed search scopes, but the server will advise it which are likely to give good results. Now for the scary bit, once you have looked at the results and perhaps clicked on something then your client pings the server again to tell it which scope produced the most relevant result. This means that the server can learn from this feedback about which scopes produce high quality results for that keyword, and perhaps rank that one a bit higher in future recommendations lists.

• Lenses are now called master scopes
• You control each individual scope that you want to search in or not search in, not the master scopes so you will have 100 or so things to turn on or off.
• You can still have locally installed scopes that search authenticated data sources
• You could in principal run your own search server if you write one to implement the API and patch the home master scope to look at your own server
• The server isn’t open source
• You can’t opt out of the feedback process (without turning off the smart scope altogether – which you can do)
• If you install a local scope then your client will tell the server the name of that scope
• Every query to the server is going to include a list of locally installed scope names (100 or so perhaps?)
• You can focus a search at a particular scope by using a keyword, for example “omlet: chicken house” to only search the Omlet scope and not the chicken stuff master scope.
• The rather poorly thought out remote-content-search checkbox to disable local scopes from doing online searches remains in place – however you don’t need it as you have per-scope controls.
• There may be some code quality checks introduced to stop scopes that don’t pay attention to the remote-content-search setting from getting into the Ubuntu distribution. – but you don’t need it.
• This probably won’t put more adverts on your desktop while you are trying to do work.
• This is probably a more private way of searching for stuff than googling for it.
• This won’t be opt-in, all the good stuff in Ubuntu is turned on by default.
• Your IP address gets logged on the web server logs, but not in the database of the smart scopes application running on the server. The developers working on the smart scopes don’t have access to the web server logs.
• It would be relatively trivial (I could do it in a day or so if I felt like it) to write a gnome-shell client for this smart scopes server to display the remote results, however doing something with the scope recommendations list would be a bit of a struggle.
• The home master scope (dash) search box will contain the help text “search your computer and online sources” to make it clear that it isn’t just a local search.

Now to the big question. How much are people going to freak out about this? Well if they read the spec all the way to the end they will see all the stuff that is being collected, how it is aggregated, how much or how little privacy this is costing them and why it is being done for the greater good of having decent search results. The feedback data collection process is likely to be slightly freakout causing. I can see why the developers want this turned on and I can see why it is antisocial to turn it off, like leeching on bittorrent while downloading an Ubuntu iso or whatever. I think they would be wise to have a checkbox in the privacy settings dialogue so that antisocial people can turn this off. I imagine the developers will stick with the current policy that if you want to use smart scopes you have to participate in the feedback process to make it better.

I think we need to do some education around the lack of an applications launcher though. Currently people think that Super + name of application is a replacement for the Gnome 2 applications menu. It isn’t. Super+a + name of application is how to start applications. This is going to focus the search on just applications and will work a lot faster than doing an omniglobaleverywhere search which is what the superkey does by itself.

For me this is a good development overall. The privacy debacle will be solved to my satisfaction when you can locally and personally blacklist scopes. This will mean that I can write a scope without it being co-dependent on all the other online scopes and I don’t have to worry about whether intranet access constitutes internet access. All scopes can simply stop if remote-content-search is set, but nobody needs to set it, the flag will basically just break all searching and be a bit pointless.

Shopping lens for Gnome Shell

The shopping lens in Unity has been one of the most talked about features, not always in a good way. Personally I quite like it, I have purchased several items through it and it seems to be a faster experience at simply finding stuff than using Amazon itself. Producing a lens that makes money clearly caused a lot of people to look up and examine the situation in more depth and it was then discovered that it should have been using https for the transport and issues around privacy and sending search terms off the local machine were raised. All valid issues which have been addressed to a greater or lesser extent, but it got me thinking about privacy  and prompted me to have a play with Gnome Shell and in due course prompted me to look into Gnome Shell extensions.

You can do cool things with Gnome Shell extensions, they are written in Javascript and can manipulate the user interface in quite flexible ways. One particular type of extension is the search provider – which corresponds pretty well to the Unity lens. There are search providers for wikipedia, virtualbox, ssh and a few other searchable things and today my shopping search provider for Amazon was approved and published.

In simple terms to get the thing running visit the extension page using chromium or a supported version of Firefox whilst running Gnome Shell. This should give you a slider on the left that you can use to activate the extension, this will download and install it in ~/.local/share/gnome-shell/extensions/shopping@libertus.co.uk (that looks like an email address, but it isn’t). Once installed you can hit the settings icon on that very same web page to configure the extension – or just hit super and type “a toaster” to search for “toaster” on Amazon (“a” is the default search prefix to tell it you want to do an Amazon search)

Configuring it looks like this:

If you set the keyword to blank then it will function like the Unity shopping lens does – it searches as you type for everything you type. I kind of like the prefix, I know when I want to search on Amazon and when I don’t actively want to do that I am not interested in the performance hit of a redundant search, but you can make up your own mind about that.

The shop domain is the bit after www.amazon and it can be any one of:

• co.uk
• de
• com
• ca
• fr
• co.jp
• it
• cn
• es

Different stores return different products and prices and descriptions.

The affiliate code can be set to anything, gnomestore-20 will give money to Gnome as you shop, electronicfro-20 will give money to the EFF, and leaving it as theopesou-21 will give money to the LBF (Libertus Beer Fund).

Why this is not very evil

• The user is given control over the affiliate code
• It is opt-in for each user of the machine as it is installed in the user home directory.
• There is a user controllable keyword prefix for searching
• It uses https for transport client to server and server to Amazon
• The client side is fully open source
• I am being transparent about how it works
• It does not use geoIP to decide what Amazon store to use, it does not pass your IP address to Amazon or anywhere else.
• I am not interested in logging searches or IP addresses, I plan to monitor logs for a few days to check it is working OK for people then turn off logging altogether for the server.
• Enabling this does not require you to enable other online search providers, they are not co-dependent.

Why it is a little bit evil

• The server side is not in the control of the user – or published source (it is about 5 lines of code, but includes secret API keys) it is now published in the repository with the client side code, the API keys are in a separate file.
• It uses my affiliate ID as a default
• It is sending search queries off the local machine – and has the capacity to log or do evil with all queries typed into the search box (check the code to make sure it doesn’t. The code starts executing on every single keystroke, but it gives up if the regular expression with the keyword doesn’t match what you typed).
• It gets pictures from Amazon and they could data mine the logs for those and indirectly track your searches that way (highly unlikely – they track you in much more useful ways as soon as you click through to anything you are interested in. Image logs have little informational value.)
• It advocates for a materialistic society based on the existential nihilism of unfettered consumerism

How it could be less evil

• If someone wants to fork it and run a server on behalf of some foundation that is considered less evil than me then feel free to do so, it is GPLv2 code. What I don’t want you to do is fork the extension and change the default affiliate ID but keep it pointing at my server by default as a back end.
• You could run your own local search provider and register for your own API keys if you want to trust nobody but yourself and Amazon. I don’t think you can earn money purchasing items yourself on your own affiliate code so don’t think there is a magical discount facility waiting for you to set it up.

Future enhancements

I would like to make the store list  a dropdown selection – maybe auto selected based on locale or something. It would be good to provide a list of common affiliate IDs for charities and organisations that people might want to support, but I want to retain the ability for a user to type in one of their own choosing. Maybe it could take a list of affiliate IDs that it uses at random. If you have suggestions then do leave a comment, or fork it on github https://github.com/AlanBell/shopping-search-provider

Happy shopping (it is the reason for the season) and have a happy new year!

She sells sea shells

Unity isn’t the only desktop environment that Ubuntu has. There are many and as they said, Unity is a shell for Gnome but it is not Gnome-Shell. I have been using Unity for a few years now and figured I would have a bit of a play with Gnome Shell for a bit. It is very easy to install, on Ubuntu clicking here: gnome-shell will with a bit of luck set it up for you. At the lightdm login screen you can then select gnome shell from the list of desktops and you are done.

Shell has two modes, normal and overview, and you get to the overview mode by hitting the super key (that is the one with the anticompetitive Microsoft advert on it in all probability). Overview mode allows you to see a launcher on the left (a lot like the unity launcher) and a workspace switcher on the right which flips workspaces for your primary monitor only, secondary monitor stays static, which actually works rather well for me, I prefer the 1 dimensional list of workspaces to the 2×2 grid that unity has by default. Also in overview mode at the bottom of the screen is a notification area, where you can see notifications that have happened, I like this too. Notifications seem to have a good balance between being ephemeral and reviewable, plus when you click on notifications in shell you actually get to the thing that made them – so you get a notification about an email and decide to go read it, click the notification and you activate Thunderbird rather than having to go and find it yourself. The menu bar  has cool looking speech bubble menus on the indicators and best of all it has an accessibility indicator in the menu bar so it is easy to turn on the various interesting assitive tools we have.  There isn’t a global menu which is great, after two years of using it I never really understood the benefit or got used to having the menu in a different place to the thing it belongs to. Overall it seems quite polished, quite productive to use and really rather pleasant.

So this is all rather fun, but is there cool stuff I am missing out on? I can see the HUD has gone, and the dash with it’s lenses and omni-global super search. Or are they? Gnome shell has extensions https://extensions.gnome.org/ and they can do rather a lot. I haven’t seen a HUD extension yet, but I think that would be entirely possible. As for Lenses, well yes, they are absolutely there, have a look at the Wikipedia search provider. You type stuff in overview mode and as you type it searches both your local applications and Wikipedia, returning article summaries that you can click on to open them fully in the browser. When running gnome-shell you can install the extension right from that web page with a little slider control, it then automatically downloads (into ~/.local/share/gnome-shell/extensions) and starts working, no sudo access required as it is just your session, and you opted into it personally as these are not installed system-wide (you probably can install them system-wide by putting them somewhere else). They are written in Javascript and have quite a lot of control over the user interface and can do all kinds of things, in fact looking at the capabilities of extensions it strikes me that pretty much all of the Unity features could be re-implemented as gnome-shell extensions. Right now I am mostly interested in extending the search provider area and the overlap of that with lenses. I want to see if the gnome-shell approach has the same problems as Unity with regards to privacy or different ones. I have been doing a bit of tinkering and in the next post I should have something to show.

Privacy is hard. Lets go shopping!

It is possible that you have read some stuff about the Ubuntu shopping lens recently. Unity and lenses are pretty cool, it kind of gives one place to search for all kinds of stuff, and you plug in additional search engines to throw your query at. So you might search for a word and find an application, some local files, some wikipedia articles, a file in a document management system and some products on Amazon that relate to your search terms. A kind of super dooper omni-global search for things as well as web pages. Some people are concerned that searching for things means you have to send your search terms about all over the place, you can’t truly anonymously just search for stuff, you have to tell these various search engines what you are looking for – and if you are using an omni-global super search that means that search terms might go to inappropriate places.

There they are!

Imagine you are searching for your socks, this is like looking in a drawer that normally contains socks and at the same time phoning up all your friends and asking them if they have seen your socks anywhere. This is not ideal and when people started noticing this (after about a year of it having this behaviour – people only noticed when there was obvious money involved) a switch was added to “implement privacy”. It was a rush job, nobody is arguing with that, and it satisfied most people at first glance. There is now a switch in the privacy controls that can be flipped to turn off online searches. Sounds good right? Well diving a little deeper this just sets a flag, and the individual search scopes (the things that put results in the lenses) have to be modified to check for that flag and not go searching. To stretch our sock analogy this is like contacting each of your friends individually and saying “when I set this flag, and phone you, please just hang up on me”. If you acquire a new friend you have to remember to tell them this important condition of friendship.

Now if you want to ask each of your friends “have you got my Justin Beiber CD?” you have a problem, because they will just hang up on you. So you have to unflip the flag to perform that search – your friends are entirely useless with the flag set because they will just hang up on you – or they will ignore the flag and might end up spying on your morning sock related searches if you accidentally phone them. You do still have the option of doing a focussed lens specific search rather than using the home lens omni-global everywhere search. Using lens specific searches you can “search in my sock drawer” or “ask that friend if they have seen my socks” without other friends finding out about your sock related issues.

A lot of people got hung up over whether the shopping lens should be opt in or opt out. This is a bogus thing to get hung up about, the problem isn’t the default setting, the problem is that it isn’t granular enough. Lets say you install the shopping lens and opt-in to online searches. Awesome, you can now have lots of fun purchasing toasters and shoes and better CDs than that Justin Beiber one that you lent to someone or other. Now lets say you install the rather fine Google Docs lens you would expect that to be opt-in too right? Well, bit late, sorry you opted in to online searches already and there is no more fine grained control (as it happens the Google Docs lens is one of those friends that doesn’t know about your flag and it totally ignores it and picks up the phone).

So, there are issues, the developers know about them and they are a clever bunch, I am sure that they will come up with a more considered and comprehensive approach to this by the next version of Ubuntu. In the mean time I am not particularly comfortable writing lenses because they either have to depend on all other online search lenses being turned on, or they have to ignore the flag. Both options kind of suck, so I have been exploring other interesting things that are similar to lenses, one of which is the topic of my next post.

Actions from Ubuntu UDS-R

This week was the Ubuntu Developer Summit in Copenhagen, this is kind of an internal series of workshops, but as Ubuntu is an open project it isn’t a closed doors event. Anyone can attend in person, some people are sponsored by Canonical to attend. Others (like me this time) can follow along remotely via live video and audio streams and IRC talk back to the workshop rooms. The full schedule was here, and the proceedings of each session are recorded on an Etherpad page. I wrote some scripts to download all the text of these pages and scoop out the action items and you can find that over here. These will all get processed and end up on status.ubuntu.com as pretty burndown charts for ongoing tracking, but this is a quick and dirty list of all of them.