The Open Sourcerer

Ubuntu Coaster and other animals

My son James (12) has presented me with two gifts he’s made at school recently. Both are terrific and he designed and made them himself.

As a proud dad I felt it only right to show them off…

Note the clever location of the hole for use as a keyring…

Then yesterday he presented me with this very nice USB Device modelled into a Seahorse:

There’s a clip to hold the two parts together and I’m really impressed how the two halves line up; seeing as they are cut from two different sheets of perspex!

Ubuntu Smart Scopes

A new feature of Ubuntu was discussed today (which is like an announcement but without overhyping it), it is called Smart Scopes and is documented here https://wiki.ubuntu.com/SmartScopes1304Spec go read that first and then I have a video for you to watch.

http://www.youtube.com/watch?feature=player_embedded&v=CBeQur7VBDM

Now go back and read the spec that I told you to read earlier, but all the way to the end this time.

In the video from left to right is Alan Bell (me), David Callé, Jono Bacon, Michael Hall, Roberto Alsina and Stuart Langridge, all discussing this new framework for searching. It is coming soon, to the Ubuntu Raring desktop and then to phone and TV and tablet etc. The objective is to make searching really really effective and helpful to the user, but as with the previous efforts in this direction there will be some concerns around how it is implemented.

In short, Canonical will be running a server much like the existing productsearch.ubuntu.com server which will accept queries and return a bunch of results as json. The current implementation searches Amazon and the Ubuntu One music store and a few other places. The new one will do the same, plus more server-side searches, plus a new feature altogether which is a list of good scope names for the client to search. Your client will now send a list of all locally installed scopes to the server (actually a list of scopes you have added and a list of scopes you have removed or turned off from the standard set) along with your query. The server then returns results it found and wants to put in your dash, plus a subset of the local scopes you sent it, in order, that the server thinks would be good places to hunt for your search term. This means that your client might have 100 or more locally installed search scopes, but the server will advise it which are likely to give good results. Now for the scary bit, once you have looked at the results and perhaps clicked on something then your client pings the server again to tell it which scope produced the most relevant result. This means that the server can learn from this feedback about which scopes produce high quality results for that keyword, and perhaps rank that one a bit higher in future recommendations lists.

• Lenses are now called master scopes
• You control each individual scope that you want to search in or not search in, not the master scopes so you will have 100 or so things to turn on or off.
• You can still have locally installed scopes that search authenticated data sources
• You could in principal run your own search server if you write one to implement the API and patch the home master scope to look at your own server
• The server isn’t open source
• You can’t opt out of the feedback process (without turning off the smart scope altogether – which you can do)
• If you install a local scope then your client will tell the server the name of that scope
• Every query to the server is going to include a list of locally installed scope names (100 or so perhaps?)
• You can focus a search at a particular scope by using a keyword, for example “omlet: chicken house” to only search the Omlet scope and not the chicken stuff master scope.
• The rather poorly thought out remote-content-search checkbox to disable local scopes from doing online searches remains in place – however you don’t need it as you have per-scope controls.
• There may be some code quality checks introduced to stop scopes that don’t pay attention to the remote-content-search setting from getting into the Ubuntu distribution. – but you don’t need it.
• This probably won’t put more adverts on your desktop while you are trying to do work.
• This is probably a more private way of searching for stuff than googling for it.
• This won’t be opt-in, all the good stuff in Ubuntu is turned on by default.
• Your IP address gets logged on the web server logs, but not in the database of the smart scopes application running on the server. The developers working on the smart scopes don’t have access to the web server logs.
• It would be relatively trivial (I could do it in a day or so if I felt like it) to write a gnome-shell client for this smart scopes server to display the remote results, however doing something with the scope recommendations list would be a bit of a struggle.
• The home master scope (dash) search box will contain the help text “search your computer and online sources” to make it clear that it isn’t just a local search.

Now to the big question. How much are people going to freak out about this? Well if they read the spec all the way to the end they will see all the stuff that is being collected, how it is aggregated, how much or how little privacy this is costing them and why it is being done for the greater good of having decent search results. The feedback data collection process is likely to be slightly freakout causing. I can see why the developers want this turned on and I can see why it is antisocial to turn it off, like leeching on bittorrent while downloading an Ubuntu iso or whatever. I think they would be wise to have a checkbox in the privacy settings dialogue so that antisocial people can turn this off. I imagine the developers will stick with the current policy that if you want to use smart scopes you have to participate in the feedback process to make it better.

I think we need to do some education around the lack of an applications launcher though. Currently people think that Super + name of application is a replacement for the Gnome 2 applications menu. It isn’t. Super+a + name of application is how to start applications. This is going to focus the search on just applications and will work a lot faster than doing an omniglobaleverywhere search which is what the superkey does by itself.

For me this is a good development overall. The privacy debacle will be solved to my satisfaction when you can locally and personally blacklist scopes. This will mean that I can write a scope without it being co-dependent on all the other online scopes and I don’t have to worry about whether intranet access constitutes internet access. All scopes can simply stop if remote-content-search is set, but nobody needs to set it, the flag will basically just break all searching and be a bit pointless.

GeoTools: Geolocation services for vtiger CRM

As many of you know already, our company Libertus Solutions does quite a lot of work with the open source CRM called vtiger. It’s a very competent and accomplished product made even more so by its well thought out extension capabilities.

In this post I’m really pleased to announce our first open source vtlib module for vtiger called GeoTools.

It was derived from another project on the vtiger forge called Maps, which we have taken and extended in true open source style. Standing on the shoulders of giants, and all that…

GeoTools introduces Geolocation features to vtiger in a standard vtlib module package. It adds the ability to perform distance-based searches on your data.

GeoTools uses the Google Maps API to gather positional data, that’s latitude and longitude coordinates, for the entity records that have been configured in the GeoTools Settings area. Once we have acquired this positional data we can then perform location-based calculations to display the results on an embedded Google Map, and as a list view of entity records.

Anyway enough of the words already! Here’s a video:

As soon as the forge site is up I’ll update this and provide links to the code.

Update: Here we are: This will be a moving target for some time yet – it’s still rather “beta” grade code…

Finding VirtualBox IP addresses

I have been running some server instances in VirtualBox recently and as I move between networks it is a pain to have to log in and get the IP address from ipconfig before being able to access the test web sites I have running in them. I also prefer to SSH to them rather than use the VirtualBox instance (it gives better character screen size, although I could reconfigure things; I also tab my terminals).

Anyway, in order to make things easier I put together two scripts, one that handles getting the IP address of the Virtualbox instance, and the other that handles connecting via SSH just by telling it the instance name. In order to do this you need to install the VirtualBox Guest Additions. This package is generally used in connection with video drivers for GUI based guests, but it also has some extensions that present extra information about the guest to the host machine.

Installing on Ubuntu is quite easy. My setup is using Ubuntu 12.10 on the desktop (host) and Ubuntu 12.04 LTS on the server (guest). To start with there is a package with the Guest Additions ISO in, so start by installing it with:

sudo aptitude install virtualbox-guest-additions-iso

Next you need to mount the ISO in the guest OS. To do this choose the Install Guest Additions option from the Devices menu. Since this is a CLI server OS it won’t automatically mount the ISO, so you will have to do this manually with:

sudo mount /dev/cdrom /media/cdrom

Once you have done this you need to install dkms and then run the install script with:

sudo aptitude install dkms
sudo /media/cdrom/VBoxLinuxAdditions.run

It will complain about not having found X to install the graphics drivers, but this isn’t a problem.

Once you have done this you can use the command:

VBoxManage guestproperty enumerate

where <vname> is the name of your guest. This will list out all of the available information that can now be accessed.

Using this command in a little bit of bash I created the two scripts. Firstly to get the IP address of a named guest:

#! /bin/bash
VIP=`VBoxManage guestproperty get $1 "/VirtualBox/GuestInfo/Net/0/V4/IP" | awk '{print $2}'`
echo $VIP

Secondly, to SSH to a named guest:

#! /bin/bash
VIP=`VBoxManage guestproperty get $1 "/VirtualBox/GuestInfo/Net/0/V4/IP" | awk '{print $2}'`
ssh-keygen -f "/home/paul/.ssh/known_hosts" -R $VIP
ssh $VIP

This second one is a little more involved because it first deletes the entry from the known_host file (remember to change the location). I’ve done this to stop an error coming up if the IP address has already been used, which isn’t uncommon with DHCP leases (you often get the same one, but not always!). You will have to confirm the authenticity of the host each time you connect, but since this is scripted and the IP has been automatically obtained locally to the machine this shouldn’t present a security risk.

Each of these scripts takes the guest name as a parameter, eg:

vbip Alfresco

or

vssh Alfresco

Lastly, to make these new scripts easy to use I created a .bash_aliases file in my home directory with the following:

## custom aliases
alias vssh='~/scripts/vssh'
alias vbip='~/scripts/vbip'

You will need to adjust for whever you have put these scripts, I tend to have a scripts directory in my home directory for this purpose.

OT: Chillies for 2013

I’ve sown my seeds for this year and have decided to document the process on G+ for a change.

So if you want to find out about what I and other chilli heads will be growing this year head over to the Chillies Community on Google Plus…

Shopping lens for Gnome Shell

The shopping lens in Unity has been one of the most talked about features, not always in a good way. Personally I quite like it, I have purchased several items through it and it seems to be a faster experience at simply finding stuff than using Amazon itself. Producing a lens that makes money clearly caused a lot of people to look up and examine the situation in more depth and it was then discovered that it should have been using https for the transport and issues around privacy and sending search terms off the local machine were raised. All valid issues which have been addressed to a greater or lesser extent, but it got me thinking about privacy  and prompted me to have a play with Gnome Shell and in due course prompted me to look into Gnome Shell extensions.

You can do cool things with Gnome Shell extensions, they are written in Javascript and can manipulate the user interface in quite flexible ways. One particular type of extension is the search provider – which corresponds pretty well to the Unity lens. There are search providers for wikipedia, virtualbox, ssh and a few other searchable things and today my shopping search provider for Amazon was approved and published.

In simple terms to get the thing running visit the extension page using chromium or a supported version of Firefox whilst running Gnome Shell. This should give you a slider on the left that you can use to activate the extension, this will download and install it in ~/.local/share/gnome-shell/extensions/shopping@libertus.co.uk (that looks like an email address, but it isn’t). Once installed you can hit the settings icon on that very same web page to configure the extension – or just hit super and type “a toaster” to search for “toaster” on Amazon (“a” is the default search prefix to tell it you want to do an Amazon search)

Configuring it looks like this:

If you set the keyword to blank then it will function like the Unity shopping lens does – it searches as you type for everything you type. I kind of like the prefix, I know when I want to search on Amazon and when I don’t actively want to do that I am not interested in the performance hit of a redundant search, but you can make up your own mind about that.

The shop domain is the bit after www.amazon and it can be any one of:

• co.uk
• de
• com
• ca
• fr
• co.jp
• it
• cn
• es

Different stores return different products and prices and descriptions.

The affiliate code can be set to anything, gnomestore-20 will give money to Gnome as you shop, electronicfro-20 will give money to the EFF, and leaving it as theopesou-21 will give money to the LBF (Libertus Beer Fund).

Why this is not very evil

• The user is given control over the affiliate code
• It is opt-in for each user of the machine as it is installed in the user home directory.
• There is a user controllable keyword prefix for searching
• It uses https for transport client to server and server to Amazon
• The client side is fully open source
• I am being transparent about how it works
• It does not use geoIP to decide what Amazon store to use, it does not pass your IP address to Amazon or anywhere else.
• I am not interested in logging searches or IP addresses, I plan to monitor logs for a few days to check it is working OK for people then turn off logging altogether for the server.
• Enabling this does not require you to enable other online search providers, they are not co-dependent.

Why it is a little bit evil

• The server side is not in the control of the user – or published source (it is about 5 lines of code, but includes secret API keys) it is now published in the repository with the client side code, the API keys are in a separate file.
• It uses my affiliate ID as a default
• It is sending search queries off the local machine – and has the capacity to log or do evil with all queries typed into the search box (check the code to make sure it doesn’t. The code starts executing on every single keystroke, but it gives up if the regular expression with the keyword doesn’t match what you typed).
• It gets pictures from Amazon and they could data mine the logs for those and indirectly track your searches that way (highly unlikely – they track you in much more useful ways as soon as you click through to anything you are interested in. Image logs have little informational value.)
• It advocates for a materialistic society based on the existential nihilism of unfettered consumerism

How it could be less evil

• If someone wants to fork it and run a server on behalf of some foundation that is considered less evil than me then feel free to do so, it is GPLv2 code. What I don’t want you to do is fork the extension and change the default affiliate ID but keep it pointing at my server by default as a back end.
• You could run your own local search provider and register for your own API keys if you want to trust nobody but yourself and Amazon. I don’t think you can earn money purchasing items yourself on your own affiliate code so don’t think there is a magical discount facility waiting for you to set it up.

Future enhancements

I would like to make the store list  a dropdown selection – maybe auto selected based on locale or something. It would be good to provide a list of common affiliate IDs for charities and organisations that people might want to support, but I want to retain the ability for a user to type in one of their own choosing. Maybe it could take a list of affiliate IDs that it uses at random. If you have suggestions then do leave a comment, or fork it on github https://github.com/AlanBell/shopping-search-provider

Happy shopping (it is the reason for the season) and have a happy new year!