The Open Sourcerer

Shopping lens for Gnome Shell

The shopping lens in Unity has been one of the most talked about features, not always in a good way. Personally I quite like it, I have purchased several items through it and it seems to be a faster experience at simply finding stuff than using Amazon itself. Producing a lens that makes money clearly caused a lot of people to look up and examine the situation in more depth and it was then discovered that it should have been using https for the transport and issues around privacy and sending search terms off the local machine were raised. All valid issues which have been addressed to a greater or lesser extent, but it got me thinking about privacy  and prompted me to have a play with Gnome Shell and in due course prompted me to look into Gnome Shell extensions.

You can do cool things with Gnome Shell extensions, they are written in Javascript and can manipulate the user interface in quite flexible ways. One particular type of extension is the search provider – which corresponds pretty well to the Unity lens. There are search providers for wikipedia, virtualbox, ssh and a few other searchable things and today my shopping search provider for Amazon was approved and published.

In simple terms to get the thing running visit the extension page using chromium or a supported version of Firefox whilst running Gnome Shell. This should give you a slider on the left that you can use to activate the extension, this will download and install it in ~/.local/share/gnome-shell/extensions/shopping@libertus.co.uk (that looks like an email address, but it isn’t). Once installed you can hit the settings icon on that very same web page to configure the extension – or just hit super and type “a toaster” to search for “toaster” on Amazon (“a” is the default search prefix to tell it you want to do an Amazon search)

Configuring it looks like this:

If you set the keyword to blank then it will function like the Unity shopping lens does – it searches as you type for everything you type. I kind of like the prefix, I know when I want to search on Amazon and when I don’t actively want to do that I am not interested in the performance hit of a redundant search, but you can make up your own mind about that.

The shop domain is the bit after www.amazon and it can be any one of:

• co.uk
• de
• com
• ca
• fr
• co.jp
• it
• cn
• es

Different stores return different products and prices and descriptions.

The affiliate code can be set to anything, gnomestore-20 will give money to Gnome as you shop, electronicfro-20 will give money to the EFF, and leaving it as theopesou-21 will give money to the LBF (Libertus Beer Fund).

Why this is not very evil

• The user is given control over the affiliate code
• It is opt-in for each user of the machine as it is installed in the user home directory.
• There is a user controllable keyword prefix for searching
• It uses https for transport client to server and server to Amazon
• The client side is fully open source
• I am being transparent about how it works
• It does not use geoIP to decide what Amazon store to use, it does not pass your IP address to Amazon or anywhere else.
• I am not interested in logging searches or IP addresses, I plan to monitor logs for a few days to check it is working OK for people then turn off logging altogether for the server.
• Enabling this does not require you to enable other online search providers, they are not co-dependent.

Why it is a little bit evil

• The server side is not in the control of the user – or published source (it is about 5 lines of code, but includes secret API keys) it is now published in the repository with the client side code, the API keys are in a separate file.
• It uses my affiliate ID as a default
• It is sending search queries off the local machine – and has the capacity to log or do evil with all queries typed into the search box (check the code to make sure it doesn’t. The code starts executing on every single keystroke, but it gives up if the regular expression with the keyword doesn’t match what you typed).
• It gets pictures from Amazon and they could data mine the logs for those and indirectly track your searches that way (highly unlikely – they track you in much more useful ways as soon as you click through to anything you are interested in. Image logs have little informational value.)
• It advocates for a materialistic society based on the existential nihilism of unfettered consumerism

How it could be less evil

• If someone wants to fork it and run a server on behalf of some foundation that is considered less evil than me then feel free to do so, it is GPLv2 code. What I don’t want you to do is fork the extension and change the default affiliate ID but keep it pointing at my server by default as a back end.
• You could run your own local search provider and register for your own API keys if you want to trust nobody but yourself and Amazon. I don’t think you can earn money purchasing items yourself on your own affiliate code so don’t think there is a magical discount facility waiting for you to set it up.

Future enhancements

I would like to make the store list  a dropdown selection – maybe auto selected based on locale or something. It would be good to provide a list of common affiliate IDs for charities and organisations that people might want to support, but I want to retain the ability for a user to type in one of their own choosing. Maybe it could take a list of affiliate IDs that it uses at random. If you have suggestions then do leave a comment, or fork it on github https://github.com/AlanBell/shopping-search-provider

Happy shopping (it is the reason for the season) and have a happy new year!

How to install OpenERP 7.0 on Ubuntu 12.04 LTS

Introduction

Welcome to the latest of our very popular OpenERP installation “How Tos”.

The new release of OpenERP 7.0 is a major upgrade and a new Long Term Support release; the 7.0 Release Notes extend to over 90 pages! The most noticeable change is a complete re-write of the User Interface that features a much more modern look and feel.

OpenERP 7.0 is not only better looking and easier to use, it also brings many improvements to the existing feature-set and adds a number of brand new features which extend the scope of the business needs covered by OpenERP. Integration of social network capabilities, integration with Google Docs and LinkedIn, new Contract Management, new Event Management, new Point of Sale, new Address Book, new Fleet Management,… are only some of the many enhancements in OpenERP 7.0.

The How To

Following that introduction, I bet you can’t wait to get your hands dirty…

Just one thing before we start: You can simply download a “.deb” package of OpenERP and install that on Ubuntu. Unfortunately that approach doesn’t provide us (Libertus Solutions) with enough fine-grained control over where things get installed, and it restricts our flexibility to modify & customise, hence I prefer to do it a slightly more manual way (this install process below should only take about 10-15 minutes once the host machine has been built).

So without further ado here we go:

Step 1. Build your server

I install just the bare minimum from the install routine (you may want to install the openssh-server during the install procedure or install subsequently depending on your needs).

After the server has restarted for the first time I install the openssh-server package (so we can connect to it remotely) and denyhosts to add a degree of brute-force attack protection. There are other protection applications available: I’m not saying this one is the best, but it’s one that works and is easy to configure and manage. If you don’t already, it’s also worth looking at setting up key-based ssh access, rather than relying on passwords. This can also help to limit the potential of brute-force attacks. [NB: This isn’t a How To on securing your server…]

sudo apt-get install openssh-server denyhosts

Now make sure your server has all the latest versions & patches by doing an update:

sudo apt-get update
sudo apt-get dist-upgrade

Although not always essential it’s probably a good idea to reboot your server now and make sure it all comes back up and you can login via ssh.

Now we’re ready to start the OpenERP install.

Step 2. Create the OpenERP user that will own and run the application

sudo adduser --system --home=/opt/openerp --group openerp

This is a “system” user. It is there to own and run the application, it isn’t supposed to be a person type user with a login etc. In Ubuntu, a system user gets a UID below 1000, has no shell (it’s actually /bin/false) and has logins disabled. Note that I’ve specified a “home” of /opt/openerp, this is where the OpenERP server code will reside and is created automatically by the command above. The location of the server code is your choice of course, but be aware that some of the instructions and configuration files below may need to be altered if you decide to install to a different location.

[Note: If you want to run multiple versions of OpenERP on the same server, the way I do it is to create multiple users with the correct version number as part of the name, e.g. openerp70, openerp61 etc. If you also use this when creating the Postgres users too, you can have full separation of systems on the same server. I also use similarly named home directories, e.g. /opt/openerp70, /opt/openerp61 and config and start-up/shutdown files. You will also need to configure different ports for each instance or else only the first will start.]

A question I have been asked a few times is how to run the OpenERP server as the openerp system user from the command line if it has no shell. This can be done quite easily:

sudo su - openerp -s /bin/bash

This will su your current terminal login to the openerp user (the “-” between su and openerp is correct) and use the shell /bin/bash. When this command is run you will be in openerp’s home directory: /opt/openerp.

When you have done what you need you can leave the openerp user’s shell by typing exit.

Step 3. Install and configure the database server, PostgreSQL

sudo apt-get install postgresql

Then configure the OpenERP user on postgres:

First change to the postgres user so we have the necessary privileges to configure the database.

sudo su - postgres

Now create a new database user. This is so OpenERP has access rights to connect to PostgreSQL and to create and drop databases. Remember what your choice of password is here; you will need it later on:

createuser --createdb --username postgres --no-createrole --no-superuser --pwprompt openerp
Enter password for new role: ********
Enter it again: ********

Finally exit from the postgres user account:

exit

Step 4. Install the necessary Python libraries for the server

sudo apt-get install python-dateutil python-docutils python-feedparser python-gdata \
python-jinja2 python-ldap python-libxslt1 python-lxml python-mako python-mock python-openid \
python-psycopg2 python-psutil python-pybabel python-pychart python-pydot python-pyparsing \
python-reportlab python-simplejson python-tz python-unittest2 python-vatnumber python-vobject \
python-webdav python-werkzeug python-xlwt python-yaml python-zsi

With that done, all the dependencies for installing OpenERP 7.0 are now satisfied (note that there are some new packages required since 6.1).

Step 5. Install the OpenERP server

I tend to use wget for this sort of thing and I download the files to my home directory.

Make sure you get the latest version of the application: at the time of writing this it’s 7.0. I got the download links from their download pages (note there are also deb, rpm and exe builds in this area too). There isn’t a static 7.0 release tarball as such anymore, but there is a nightly build of the 7.0 source tree which should be just as good and will contain patches as and when things get fixed. The link below is to the source tarball for the 7.0 branch.

Note: As an alternative method of getting the code onto your server, Jerome added a very useful comment showing how to get it straight from launchpad. Thanks!

wget http://nightly.openerp.com/7.0/nightly/src/openerp-7.0-latest.tar.gz

Now install the code where we need it: cd to the /opt/openerp/ directory and extract the tarball there.

cd /opt/openerp
sudo tar xvf ~/openerp-7.0-latest.tar.gz

Next we need to change the ownership of all the the files to the OpenERP user and group we created earlier.

sudo chown -R openerp: *

And finally, the way I have done this is to copy the server directory to something with a simpler name so that the configuration files and boot scripts don’t need constant editing (I called it, rather unimaginatively, server). I started out using a symlink solution, but I found that when it comes to upgrading, it seems to make more sense to me to just keep a copy of the files in place and then overwrite them with the new code. This way you keep any custom or user-installed modules and reports etc. all in the right place.

sudo cp -a openerp-7.0 server

As an example, should OpenERP 7.0.1 come out soon, I can extract the tarballs into /opt/openerp/ as above. I can do any testing I need, then repeat the copy command so that the modified files will overwrite as needed and any custom modules, report templates and such will be retained. Once satisfied the upgrade is stable, the older 7.0 directories can be removed if wanted.

That’s the OpenERP server software installed. The last steps to a working system is to set up the configuration file and associated boot script so OpenERP starts and stops automatically when the server itself stops and starts.

Step 6. Configuring the OpenERP application

The default configuration file for the server (in /opt/openerp/server/install/) is actually very minimal and will, with only one small change work fine so we’ll simply copy that file to where we need it and change it’s ownership and permissions:

sudo cp /opt/openerp/server/install/openerp-server.conf /etc/
sudo chown openerp: /etc/openerp-server.conf
sudo chmod 640 /etc/openerp-server.conf

The above commands make the file owned and writeable only by the openerp user and group and only readable by openerp and root.

To allow the OpenERP server to run initially, you should only need to change one line in this file. Toward to the top of the file change the line db_password = False to the same password you used back in step 3. Use your favourite text editor here. I tend to use nano, e.g.

sudo nano /etc/openerp-server.conf

One other line we might as well add to the configuration file now, is to tell OpenERP where to write its log file. To complement my suggested location below add the following line to the openerp-server.conf file:

logfile = /var/log/openerp/openerp-server.log

Once the configuration file is edited and saved, you can start the server just to check if it actually runs.

sudo su - openerp -s /bin/bash
/opt/openerp/server/openerp-server

If you end up with a few lines eventually saying OpenERP is running and waiting for connections then you are all set.

On my system I noticed the following warning:

2012-12-19 11:53:51,613 6586 WARNING ? openerp.addons.google_docs.google_docs: Please install latest gdata-python-client from http://code.google.com/p/gdata-python-client/downloads/list

The Ubuntu 12.04 packaged version of the python gdata client library is not quite recent enough, so to install a more up-to-date version I did the following (exit from the openerp user’s shell if you are still in it first):

sudo apt-get install python-pip
sudo pip install gdata --upgrade

Going back and repeating the commands to start the server resulted in no further warnings

sudo su - openerp -s /bin/bash
/opt/openerp/server/openerp-server

If there are errors, you’ll need to go back and find out where the problem is.

Otherwise simply enter CTL+C to stop the server and then exit to leave the openerp user account and go back to your own shell.

Step 7. Installing the boot script

For the final step we need to install a script which will be used to start-up and shut down the server automatically and also run the application as the correct user. There is a script you can use in /opt/openerp/server/install/openerp-server.init but this will need a few small modifications to work with the system installed the way I have described above. Here’s a link to the one I’ve already modified for 7.0.

Similar to the configuration file, you need to either copy it or paste the contents of this script to a file in /etc/init.d/ and call it openerp-server. Once it is in the right place you will need to make it executable and owned by root:

sudo chmod 755 /etc/init.d/openerp-server
sudo chown root: /etc/init.d/openerp-server

In the configuration file there’s an entry for the server’s log file. We need to create that directory first so that the server has somewhere to log to and also we must make it writeable by the openerp user:

sudo mkdir /var/log/openerp
sudo chown openerp:root /var/log/openerp

Step 8. Testing the server

To start the OpenERP server type:

sudo /etc/init.d/openerp-server start

You should now be able to view the logfile and see that the server has started.

less /var/log/openerp/openerp-server.log

If there are any problems starting the server you need to go back and check. There’s really no point ploughing on if the server doesn’t start…

If the log file looks OK, now point your web browser at the domain or IP address of your OpenERP server (or localhost if you are on the same machine) and use port 8069. The url will look something like this:

http://IP_or_domain.com:8069

What you should see is a screen like this one (it is the Database Management Screen because you have no OpenERP databases yet):

What I do recommend you do at this point is to change the super admin password to something nice and strong (Click the “Password” menu). By default this password is just “admin” and knowing that, a user can create, backup, restore and drop databases! This password is stored in plain text in the /etc/openerp-server.conf file; hence why we restricted access to just openerp and root. When you change and save the new password the /etc/openerp-server.conf file will be re-written and will have a lot more options in it.

Now it’s time to make sure the server stops properly too:

sudo /etc/init.d/openerp-server stop

Check the logfile again to make sure it has stopped and/or look at your server’s process list.

Step 9. Automating OpenERP startup and shutdown

If everything above seems to be working OK, the final step is make the script start and stop automatically with the Ubuntu Server. To do this type:

sudo update-rc.d openerp-server defaults

You can now try rebooting you server if you like. OpenERP should be running by the time you log back in.

If you type ps aux | grep openerp you should see a line similar to this:

openerp 1491 0.1 10.6 207132 53596 ? Sl 22:23 0:02 python /opt/openerp/server/openerp-server -c /etc/openerp-server.conf

Which shows that the server is running. And of course you can check the logfile or visit the server from your web browser too.

That’s it! Next I would suggest you create a new database filling in the fields as desired. Once the database is initialised, you will be directed straight to the new main configuration screen which gives you a fell for the new User Interface in OpenERP 7 and shows you how easy it is to set up a basic system.

She sells sea shells

Unity isn’t the only desktop environment that Ubuntu has. There are many and as they said, Unity is a shell for Gnome but it is not Gnome-Shell. I have been using Unity for a few years now and figured I would have a bit of a play with Gnome Shell for a bit. It is very easy to install, on Ubuntu clicking here: gnome-shell will with a bit of luck set it up for you. At the lightdm login screen you can then select gnome shell from the list of desktops and you are done.

Shell has two modes, normal and overview, and you get to the overview mode by hitting the super key (that is the one with the anticompetitive Microsoft advert on it in all probability). Overview mode allows you to see a launcher on the left (a lot like the unity launcher) and a workspace switcher on the right which flips workspaces for your primary monitor only, secondary monitor stays static, which actually works rather well for me, I prefer the 1 dimensional list of workspaces to the 2×2 grid that unity has by default. Also in overview mode at the bottom of the screen is a notification area, where you can see notifications that have happened, I like this too. Notifications seem to have a good balance between being ephemeral and reviewable, plus when you click on notifications in shell you actually get to the thing that made them – so you get a notification about an email and decide to go read it, click the notification and you activate Thunderbird rather than having to go and find it yourself. The menu bar  has cool looking speech bubble menus on the indicators and best of all it has an accessibility indicator in the menu bar so it is easy to turn on the various interesting assitive tools we have.  There isn’t a global menu which is great, after two years of using it I never really understood the benefit or got used to having the menu in a different place to the thing it belongs to. Overall it seems quite polished, quite productive to use and really rather pleasant.

So this is all rather fun, but is there cool stuff I am missing out on? I can see the HUD has gone, and the dash with it’s lenses and omni-global super search. Or are they? Gnome shell has extensions https://extensions.gnome.org/ and they can do rather a lot. I haven’t seen a HUD extension yet, but I think that would be entirely possible. As for Lenses, well yes, they are absolutely there, have a look at the Wikipedia search provider. You type stuff in overview mode and as you type it searches both your local applications and Wikipedia, returning article summaries that you can click on to open them fully in the browser. When running gnome-shell you can install the extension right from that web page with a little slider control, it then automatically downloads (into ~/.local/share/gnome-shell/extensions) and starts working, no sudo access required as it is just your session, and you opted into it personally as these are not installed system-wide (you probably can install them system-wide by putting them somewhere else). They are written in Javascript and have quite a lot of control over the user interface and can do all kinds of things, in fact looking at the capabilities of extensions it strikes me that pretty much all of the Unity features could be re-implemented as gnome-shell extensions. Right now I am mostly interested in extending the search provider area and the overlap of that with lenses. I want to see if the gnome-shell approach has the same problems as Unity with regards to privacy or different ones. I have been doing a bit of tinkering and in the next post I should have something to show.

Privacy is hard. Lets go shopping!

It is possible that you have read some stuff about the Ubuntu shopping lens recently. Unity and lenses are pretty cool, it kind of gives one place to search for all kinds of stuff, and you plug in additional search engines to throw your query at. So you might search for a word and find an application, some local files, some wikipedia articles, a file in a document management system and some products on Amazon that relate to your search terms. A kind of super dooper omni-global search for things as well as web pages. Some people are concerned that searching for things means you have to send your search terms about all over the place, you can’t truly anonymously just search for stuff, you have to tell these various search engines what you are looking for – and if you are using an omni-global super search that means that search terms might go to inappropriate places.

Imagine you are searching for your socks, this is like looking in a drawer that normally contains socks and at the same time phoning up all your friends and asking them if they have seen your socks anywhere. This is not ideal and when people started noticing this (after about a year of it having this behaviour – people only noticed when there was obvious money involved) a switch was added to “implement privacy”. It was a rush job, nobody is arguing with that, and it satisfied most people at first glance. There is now a switch in the privacy controls that can be flipped to turn off online searches. Sounds good right? Well diving a little deeper this just sets a flag, and the individual search scopes (the things that put results in the lenses) have to be modified to check for that flag and not go searching. To stretch our sock analogy this is like contacting each of your friends individually and saying “when I set this flag, and phone you, please just hang up on me”. If you acquire a new friend you have to remember to tell them this important condition of friendship.

Now if you want to ask each of your friends “have you got my Justin Beiber CD?” you have a problem, because they will just hang up on you. So you have to unflip the flag to perform that search – your friends are entirely useless with the flag set because they will just hang up on you – or they will ignore the flag and might end up spying on your morning sock related searches if you accidentally phone them. You do still have the option of doing a focussed lens specific search rather than using the home lens omni-global everywhere search. Using lens specific searches you can “search in my sock drawer” or “ask that friend if they have seen my socks” without other friends finding out about your sock related issues.

A lot of people got hung up over whether the shopping lens should be opt in or opt out. This is a bogus thing to get hung up about, the problem isn’t the default setting, the problem is that it isn’t granular enough. Lets say you install the shopping lens and opt-in to online searches. Awesome, you can now have lots of fun purchasing toasters and shoes and better CDs than that Justin Beiber one that you lent to someone or other. Now lets say you install the rather fine Google Docs lens you would expect that to be opt-in too right? Well, bit late, sorry you opted in to online searches already and there is no more fine grained control (as it happens the Google Docs lens is one of those friends that doesn’t know about your flag and it totally ignores it and picks up the phone).

So, there are issues, the developers know about them and they are a clever bunch, I am sure that they will come up with a more considered and comprehensive approach to this by the next version of Ubuntu. In the mean time I am not particularly comfortable writing lenses because they either have to depend on all other online search lenses being turned on, or they have to ignore the flag. Both options kind of suck, so I have been exploring other interesting things that are similar to lenses, one of which is the topic of my next post.

Camera Control

This is based on a £35 wireless pan and tilt camera, but with a bit of python running on a separate web server to control it. The camera comes with a rather messy user interface, that isn’t tablet/phone friendly but is fairly easy to pick apart and see how it operates by poking a few cgi files on the camera to give it instructions. The camera itself is running Linux (with a bit of a GPL violation as there is no source published as far as I can tell). The mjpeg video stream is direct from the camera and it has a clickable imagemap overlay which chops up the image into 40px squares. Clicking a square requests a simple web service running on a computer close to the camera which starts the camera moving and stops it again after a fraction of a second multiplied by the number of pixels it needs to move. The camera itself has no absolute or relative positioning (not quite true, you can set 8 preset absolute positions), it is just done by careful timing of the start and stop signals.

The camera can move in diagonal directions, it would be nice to get it to smoothly move to a new location smoothly first by panning and tilting, then the rest of the way on one axis. Might be nice to have something round the outside of the image to allow you to move double the distance, at the moment you can move a half frame in one click would be nice to be able to go a whole frame away.

The camera stream has a maximum of 4 concurrent viewers, a few more if I drop the resolution to 320×200. For lots of viewers I would use ffmpeg to reencode from mjpeg to something else on the fly and possibly pass it to a streaming server of some kind, the overlay control should work just as it is over a multicast video stream. In reality, if you are expecting more than a couple of concurrent viewers they are going to argue over control too much.

Source code is on launchpad and I will tidy it up somewhat as time goes by and make it more of a collaborative project (starting with some installation instructions, it uses apache and mod_python at the moment). Please share ideas for features in the comments here.

If you can’t see the video, try coming back later, it probably hit the limit of viewers.

The Quantal Quetzal takes flight

 A week today on the 18th October is the release date of Ubuntu 12.10 the Quantal Quetzal. This release was pitched to be all about quality and from my testing I think the quality has improved quite a lot. If you were thinking of trying Ubuntu with the Unity interface then 12.10 is the release to go for, a lot of niggles have been unniggled and sharp edges smoothed over.

As is traditional with Ubuntu releases there will be a party in London where users, enthusiasts and the Canonical release team get together to consume some adult beverages and generally have a fun evening. This release is no exception to the tradition and you are most welcome to join us at the George Inn from about 7ish (or whenever you can). There is an optional sign up sheet with more details, anyone on the list will probably have a name badge waiting for them (this is based on feedback from previous events).

If you want to get your hands on an official Ubuntu 12.10 DVD we are now taking pre-orders for the UK local team allocation just send in an SAE they will probably arrive around the end of the month or early November.