The Open Sourcerer

Ubuntu Smart Scopes

A new feature of Ubuntu was discussed today (which is like an announcement but without overhyping it), it is called Smart Scopes and is documented here https://wiki.ubuntu.com/SmartScopes1304Spec go read that first and then I have a video for you to watch.

http://www.youtube.com/watch?feature=player_embedded&v=CBeQur7VBDM

Now go back and read the spec that I told you to read earlier, but all the way to the end this time.

In the video from left to right is Alan Bell (me), David Callé, Jono Bacon, Michael Hall, Roberto Alsina and Stuart Langridge, all discussing this new framework for searching. It is coming soon, to the Ubuntu Raring desktop and then to phone and TV and tablet etc. The objective is to make searching really really effective and helpful to the user, but as with the previous efforts in this direction there will be some concerns around how it is implemented.

In short, Canonical will be running a server much like the existing productsearch.ubuntu.com server which will accept queries and return a bunch of results as json. The current implementation searches Amazon and the Ubuntu One music store and a few other places. The new one will do the same, plus more server-side searches, plus a new feature altogether which is a list of good scope names for the client to search. Your client will now send a list of all locally installed scopes to the server (actually a list of scopes you have added and a list of scopes you have removed or turned off from the standard set) along with your query. The server then returns results it found and wants to put in your dash, plus a subset of the local scopes you sent it, in order, that the server thinks would be good places to hunt for your search term. This means that your client might have 100 or more locally installed search scopes, but the server will advise it which are likely to give good results. Now for the scary bit, once you have looked at the results and perhaps clicked on something then your client pings the server again to tell it which scope produced the most relevant result. This means that the server can learn from this feedback about which scopes produce high quality results for that keyword, and perhaps rank that one a bit higher in future recommendations lists.

• Lenses are now called master scopes
• You control each individual scope that you want to search in or not search in, not the master scopes so you will have 100 or so things to turn on or off.
• You can still have locally installed scopes that search authenticated data sources
• You could in principal run your own search server if you write one to implement the API and patch the home master scope to look at your own server
• The server isn’t open source
• You can’t opt out of the feedback process (without turning off the smart scope altogether – which you can do)
• If you install a local scope then your client will tell the server the name of that scope
• Every query to the server is going to include a list of locally installed scope names (100 or so perhaps?)
• You can focus a search at a particular scope by using a keyword, for example “omlet: chicken house” to only search the Omlet scope and not the chicken stuff master scope.
• The rather poorly thought out remote-content-search checkbox to disable local scopes from doing online searches remains in place – however you don’t need it as you have per-scope controls.
• There may be some code quality checks introduced to stop scopes that don’t pay attention to the remote-content-search setting from getting into the Ubuntu distribution. – but you don’t need it.
• This probably won’t put more adverts on your desktop while you are trying to do work.
• This is probably a more private way of searching for stuff than googling for it.
• This won’t be opt-in, all the good stuff in Ubuntu is turned on by default.
• Your IP address gets logged on the web server logs, but not in the database of the smart scopes application running on the server. The developers working on the smart scopes don’t have access to the web server logs.
• It would be relatively trivial (I could do it in a day or so if I felt like it) to write a gnome-shell client for this smart scopes server to display the remote results, however doing something with the scope recommendations list would be a bit of a struggle.
• The home master scope (dash) search box will contain the help text “search your computer and online sources” to make it clear that it isn’t just a local search.

Now to the big question. How much are people going to freak out about this? Well if they read the spec all the way to the end they will see all the stuff that is being collected, how it is aggregated, how much or how little privacy this is costing them and why it is being done for the greater good of having decent search results. The feedback data collection process is likely to be slightly freakout causing. I can see why the developers want this turned on and I can see why it is antisocial to turn it off, like leeching on bittorrent while downloading an Ubuntu iso or whatever. I think they would be wise to have a checkbox in the privacy settings dialogue so that antisocial people can turn this off. I imagine the developers will stick with the current policy that if you want to use smart scopes you have to participate in the feedback process to make it better.

I think we need to do some education around the lack of an applications launcher though. Currently people think that Super + name of application is a replacement for the Gnome 2 applications menu. It isn’t. Super+a + name of application is how to start applications. This is going to focus the search on just applications and will work a lot faster than doing an omniglobaleverywhere search which is what the superkey does by itself.

For me this is a good development overall. The privacy debacle will be solved to my satisfaction when you can locally and personally blacklist scopes. This will mean that I can write a scope without it being co-dependent on all the other online scopes and I don’t have to worry about whether intranet access constitutes internet access. All scopes can simply stop if remote-content-search is set, but nobody needs to set it, the flag will basically just break all searching and be a bit pointless.

GeoTools: Geolocation services for vtiger CRM

As many of you know already, our company Libertus Solutions does quite a lot of work with the open source CRM called vtiger. It’s a very competent and accomplished product made even more so by its well thought out extension capabilities.

In this post I’m really pleased to announce our first open source vtlib module for vtiger called GeoTools.

It was derived from another project on the vtiger forge called Maps, which we have taken and extended in true open source style. Standing on the shoulders of giants, and all that…

GeoTools introduces Geolocation features to vtiger in a standard vtlib module package. It adds the ability to perform distance-based searches on your data.

GeoTools uses the Google Maps API to gather positional data, that’s latitude and longitude coordinates, for the entity records that have been configured in the GeoTools Settings area. Once we have acquired this positional data we can then perform location-based calculations to display the results on an embedded Google Map, and as a list view of entity records.

Anyway enough of the words already! Here’s a video:

As soon as the forge site is up I’ll update this and provide links to the code.

Update: Here we are: This will be a moving target for some time yet – it’s still rather “beta” grade code…

Finding VirtualBox IP addresses

I have been running some server instances in VirtualBox recently and as I move between networks it is a pain to have to log in and get the IP address from ipconfig before being able to access the test web sites I have running in them. I also prefer to SSH to them rather than use the VirtualBox instance (it gives better character screen size, although I could reconfigure things; I also tab my terminals).

Anyway, in order to make things easier I put together two scripts, one that handles getting the IP address of the Virtualbox instance, and the other that handles connecting via SSH just by telling it the instance name. In order to do this you need to install the VirtualBox Guest Additions. This package is generally used in connection with video drivers for GUI based guests, but it also has some extensions that present extra information about the guest to the host machine.

Installing on Ubuntu is quite easy. My setup is using Ubuntu 12.10 on the desktop (host) and Ubuntu 12.04 LTS on the server (guest). To start with there is a package with the Guest Additions ISO in, so start by installing it with:

sudo aptitude install virtualbox-guest-additions-iso

Next you need to mount the ISO in the guest OS. To do this choose the Install Guest Additions option from the Devices menu. Since this is a CLI server OS it won’t automatically mount the ISO, so you will have to do this manually with:

sudo mount /dev/cdrom /media/cdrom

Once you have done this you need to install dkms and then run the install script with:

sudo aptitude install dkms
sudo /media/cdrom/VBoxLinuxAdditions.run

It will complain about not having found X to install the graphics drivers, but this isn’t a problem.

Once you have done this you can use the command:

VBoxManage guestproperty enumerate <vname>

where <vname> is the name of your guest. This will list out all of the available information that can now be accessed.

Using this command in a little bit of bash I created the two scripts. Firstly to get the IP address of a named guest:

#! /bin/bash
VIP=`VBoxManage guestproperty get $1 "/VirtualBox/GuestInfo/Net/0/V4/IP" | awk '{print $2}'`
echo $VIP

Secondly, to SSH to a named guest:

#! /bin/bash
VIP=`VBoxManage guestproperty get $1 "/VirtualBox/GuestInfo/Net/0/V4/IP" | awk '{print $2}'`
ssh-keygen -f "/home/paul/.ssh/known_hosts" -R $VIP
ssh $VIP

This second one is a little more involved because it first deletes the entry from the known_host file (remember to change the location). I’ve done this to stop an error coming up if the IP address has already been used, which isn’t uncommon with DHCP leases (you often get the same one, but not always!). You will have to confirm the authenticity of the host each time you connect, but since this is scripted and the IP has been automatically obtained locally to the machine this shouldn’t present a security risk.

Each of these scripts takes the guest name as a parameter, eg:

vbip Alfresco

or

vssh Alfresco

Lastly, to make these new scripts easy to use I created a .bash_aliases file in my home directory with the following:

## custom aliases
alias vssh='~/scripts/vssh'
alias vbip='~/scripts/vbip'

You will need to adjust for whever you have put these scripts, I tend to have a scripts directory in my home directory for this purpose.

OT: Chillies for 2013

I’ve sown my seeds for this year and have decided to document the process on G+ for a change.

So if you want to find out about what I and other chilli heads will be growing this year head over to the Chillies Community on Google Plus…

Shopping lens for Gnome Shell

The shopping lens in Unity has been one of the most talked about features, not always in a good way. Personally I quite like it, I have purchased several items through it and it seems to be a faster experience at simply finding stuff than using Amazon itself. Producing a lens that makes money clearly caused a lot of people to look up and examine the situation in more depth and it was then discovered that it should have been using https for the transport and issues around privacy and sending search terms off the local machine were raised. All valid issues which have been addressed to a greater or lesser extent, but it got me thinking about privacy  and prompted me to have a play with Gnome Shell and in due course prompted me to look into Gnome Shell extensions.

You can do cool things with Gnome Shell extensions, they are written in Javascript and can manipulate the user interface in quite flexible ways. One particular type of extension is the search provider – which corresponds pretty well to the Unity lens. There are search providers for wikipedia, virtualbox, ssh and a few other searchable things and today my shopping search provider for Amazon was approved and published.

In simple terms to get the thing running visit the extension page using chromium or a supported version of Firefox whilst running Gnome Shell. This should give you a slider on the left that you can use to activate the extension, this will download and install it in ~/.local/share/gnome-shell/extensions/shopping@libertus.co.uk (that looks like an email address, but it isn’t). Once installed you can hit the settings icon on that very same web page to configure the extension – or just hit super and type “a toaster” to search for “toaster” on Amazon (“a” is the default search prefix to tell it you want to do an Amazon search)

Configuring it looks like this:

If you set the keyword to blank then it will function like the Unity shopping lens does – it searches as you type for everything you type. I kind of like the prefix, I know when I want to search on Amazon and when I don’t actively want to do that I am not interested in the performance hit of a redundant search, but you can make up your own mind about that.

The shop domain is the bit after www.amazon and it can be any one of:

• co.uk
• de
• com
• ca
• fr
• co.jp
• it
• cn
• es

Different stores return different products and prices and descriptions.

The affiliate code can be set to anything, gnomestore-20 will give money to Gnome as you shop, electronicfro-20 will give money to the EFF, and leaving it as theopesou-21 will give money to the LBF (Libertus Beer Fund).

Why this is not very evil

• The user is given control over the affiliate code
• It is opt-in for each user of the machine as it is installed in the user home directory.
• There is a user controllable keyword prefix for searching
• It uses https for transport client to server and server to Amazon
• The client side is fully open source
• I am being transparent about how it works
• It does not use geoIP to decide what Amazon store to use, it does not pass your IP address to Amazon or anywhere else.
• I am not interested in logging searches or IP addresses, I plan to monitor logs for a few days to check it is working OK for people then turn off logging altogether for the server.
• Enabling this does not require you to enable other online search providers, they are not co-dependent.

Why it is a little bit evil

• The server side is not in the control of the user – or published source (it is about 5 lines of code, but includes secret API keys) it is now published in the repository with the client side code, the API keys are in a separate file.
• It uses my affiliate ID as a default
• It is sending search queries off the local machine – and has the capacity to log or do evil with all queries typed into the search box (check the code to make sure it doesn’t. The code starts executing on every single keystroke, but it gives up if the regular expression with the keyword doesn’t match what you typed).
• It gets pictures from Amazon and they could data mine the logs for those and indirectly track your searches that way (highly unlikely – they track you in much more useful ways as soon as you click through to anything you are interested in. Image logs have little informational value.)
• It advocates for a materialistic society based on the existential nihilism of unfettered consumerism

How it could be less evil

• If someone wants to fork it and run a server on behalf of some foundation that is considered less evil than me then feel free to do so, it is GPLv2 code. What I don’t want you to do is fork the extension and change the default affiliate ID but keep it pointing at my server by default as a back end.
• You could run your own local search provider and register for your own API keys if you want to trust nobody but yourself and Amazon. I don’t think you can earn money purchasing items yourself on your own affiliate code so don’t think there is a magical discount facility waiting for you to set it up.

Future enhancements

I would like to make the store list  a dropdown selection – maybe auto selected based on locale or something. It would be good to provide a list of common affiliate IDs for charities and organisations that people might want to support, but I want to retain the ability for a user to type in one of their own choosing. Maybe it could take a list of affiliate IDs that it uses at random. If you have suggestions then do leave a comment, or fork it on github https://github.com/AlanBell/shopping-search-provider

Happy shopping (it is the reason for the season) and have a happy new year!

How to install OpenERP 7.0 on Ubuntu 12.04 LTS

Introduction

Welcome to the latest of our very popular OpenERP installation “How Tos”.

The new release of OpenERP 7.0 is a major upgrade and a new Long Term Support release; the 7.0 Release Notes extend to over 90 pages! The most noticeable change is a complete re-write of the User Interface that features a much more modern look and feel.

OpenERP 7.0 is not only better looking and easier to use, it also brings many improvements to the existing feature-set and adds a number of brand new features which extend the scope of the business needs covered by OpenERP. Integration of social network capabilities, integration with Google Docs and LinkedIn, new Contract Management, new Event Management, new Point of Sale, new Address Book, new Fleet Management,… are only some of the many enhancements in OpenERP 7.0.

The How To

Following that introduction, I bet you can’t wait to get your hands dirty…

Just one thing before we start: You can simply download a “.deb” package of OpenERP and install that on Ubuntu. Unfortunately that approach doesn’t provide us (Libertus Solutions) with enough fine-grained control over where things get installed, and it restricts our flexibility to modify & customise, hence I prefer to do it a slightly more manual way (this install process below should only take about 10-15 minutes once the host machine has been built).

So without further ado here we go:

Step 1. Build your server

I install just the bare minimum from the install routine (you may want to install the openssh-server during the install procedure or install subsequently depending on your needs).

After the server has restarted for the first time I install the openssh-server package (so we can connect to it remotely) and denyhosts to add a degree of brute-force attack protection. There are other protection applications available: I’m not saying this one is the best, but it’s one that works and is easy to configure and manage. If you don’t already, it’s also worth looking at setting up key-based ssh access, rather than relying on passwords. This can also help to limit the potential of brute-force attacks. [NB: This isn't a How To on securing your server...]

sudo apt-get install openssh-server denyhosts

Now make sure your server has all the latest versions & patches by doing an update:

sudo apt-get update
sudo apt-get dist-upgrade

Although not always essential it’s probably a good idea to reboot your server now and make sure it all comes back up and you can login via ssh.

Now we’re ready to start the OpenERP install.

Step 2. Create the OpenERP user that will own and run the application

sudo adduser --system --home=/opt/openerp --group openerp

This is a “system” user. It is there to own and run the application, it isn’t supposed to be a person type user with a login etc. In Ubuntu, a system user gets a UID below 1000, has no shell (it’s actually /bin/false) and has logins disabled. Note that I’ve specified a “home” of /opt/openerp, this is where the OpenERP server code will reside and is created automatically by the command above. The location of the server code is your choice of course, but be aware that some of the instructions and configuration files below may need to be altered if you decide to install to a different location.

[Note: If you want to run multiple versions of OpenERP on the same server, the way I do it is to create multiple users with the correct version number as part of the name, e.g. openerp70, openerp61 etc. If you also use this when creating the Postgres users too, you can have full separation of systems on the same server. I also use similarly named home directories, e.g. /opt/openerp70, /opt/openerp61 and config and start-up/shutdown files. You will also need to configure different ports for each instance or else only the first will start.]

A question I have been asked a few times is how to run the OpenERP server as the openerp system user from the command line if it has no shell. This can be done quite easily:

sudo su - openerp -s /bin/bash

This will su your current terminal login to the openerp user (the “-” between su and openerp is correct) and use the shell /bin/bash. When this command is run you will be in openerp’s home directory: /opt/openerp.

When you have done what you need you can leave the openerp user’s shell by typing exit.

Step 3. Install and configure the database server, PostgreSQL

sudo apt-get install postgresql

Then configure the OpenERP user on postgres:

First change to the postgres user so we have the necessary privileges to configure the database.

sudo su - postgres

Now create a new database user. This is so OpenERP has access rights to connect to PostgreSQL and to create and drop databases. Remember what your choice of password is here; you will need it later on:

createuser --createdb --username postgres --no-createrole --no-superuser --pwprompt openerp
Enter password for new role: ********
Enter it again: ********

Finally exit from the postgres user account:

exit

Step 4. Install the necessary Python libraries for the server

sudo apt-get install python-dateutil python-docutils python-feedparser python-gdata \
python-jinja2 python-ldap python-libxslt1 python-lxml python-mako python-mock python-openid \
python-psycopg2 python-psutil python-pybabel python-pychart python-pydot python-pyparsing \
python-reportlab python-simplejson python-tz python-unittest2 python-vatnumber python-vobject \
python-webdav python-werkzeug python-xlwt python-yaml python-zsi

With that done, all the dependencies for installing OpenERP 7.0 are now satisfied (note that there are some new packages required since 6.1).

Step 5. Install the OpenERP server

I tend to use wget for this sort of thing and I download the files to my home directory.

Make sure you get the latest version of the application: at the time of writing this it’s 7.0. I got the download links from their download pages (note there are also deb, rpm and exe builds in this area too). There isn’t a static 7.0 release tarball as such anymore, but there is a nightly build of the 7.0 source tree which should be just as good and will contain patches as and when things get fixed. The link below is to the source tarball for the 7.0 branch.

Note: As an alternative method of getting the code onto your server, Jerome added a very useful comment showing how to get it straight from launchpad. Thanks!

wget http://nightly.openerp.com/7.0/nightly/src/openerp-7.0-latest.tar.gz

Now install the code where we need it: cd to the /opt/openerp/ directory and extract the tarball there.

cd /opt/openerp
sudo tar xvf ~/openerp-7.0-latest.tar.gz

Next we need to change the ownership of all the the files to the OpenERP user and group we created earlier.

sudo chown -R openerp: *

And finally, the way I have done this is to copy the server directory to something with a simpler name so that the configuration files and boot scripts don’t need constant editing (I called it, rather unimaginatively, server). I started out using a symlink solution, but I found that when it comes to upgrading, it seems to make more sense to me to just keep a copy of the files in place and then overwrite them with the new code. This way you keep any custom or user-installed modules and reports etc. all in the right place.

sudo cp -a openerp-7.0 server

As an example, should OpenERP 7.0.1 come out soon, I can extract the tarballs into /opt/openerp/ as above. I can do any testing I need, then repeat the copy command so that the modified files will overwrite as needed and any custom modules, report templates and such will be retained. Once satisfied the upgrade is stable, the older 7.0 directories can be removed if wanted.

That’s the OpenERP server software installed. The last steps to a working system is to set up the configuration file and associated boot script so OpenERP starts and stops automatically when the server itself stops and starts.

Step 6. Configuring the OpenERP application

The default configuration file for the server (in /opt/openerp/server/install/) is actually very minimal and will, with only one small change work fine so we’ll simply copy that file to where we need it and change it’s ownership and permissions:

sudo cp /opt/openerp/server/install/openerp-server.conf /etc/
sudo chown openerp: /etc/openerp-server.conf
sudo chmod 640 /etc/openerp-server.conf

The above commands make the file owned and writeable only by the openerp user and group and only readable by openerp and root.

To allow the OpenERP server to run initially, you should only need to change one line in this file. Toward to the top of the file change the line db_password = False to the same password you used back in step 3. Use your favourite text editor here. I tend to use nano, e.g.

sudo nano /etc/openerp-server.conf

One other line we might as well add to the configuration file now, is to tell OpenERP where to write its log file. To complement my suggested location below add the following line to the openerp-server.conf file:

logfile = /var/log/openerp/openerp-server.log

Once the configuration file is edited and saved, you can start the server just to check if it actually runs.

sudo su - openerp -s /bin/bash
/opt/openerp/server/openerp-server

If you end up with a few lines eventually saying OpenERP is running and waiting for connections then you are all set.

On my system I noticed the following warning:

2012-12-19 11:53:51,613 6586 WARNING ? openerp.addons.google_docs.google_docs: Please install latest gdata-python-client from http://code.google.com/p/gdata-python-client/downloads/list

The Ubuntu 12.04 packaged version of the python gdata client library is not quite recent enough, so to install a more up-to-date version I did the following (exit from the openerp user’s shell if you are still in it first):

sudo apt-get install python-pip
sudo pip install gdata --upgrade

Going back and repeating the commands to start the server resulted in no further warnings

sudo su - openerp -s /bin/bash
/opt/openerp/server/openerp-server

If there are errors, you’ll need to go back and find out where the problem is.

Otherwise simply enter CTL+C to stop the server and then exit to leave the openerp user account and go back to your own shell.

Step 7. Installing the boot script

For the final step we need to install a script which will be used to start-up and shut down the server automatically and also run the application as the correct user. There is a script you can use in /opt/openerp/server/install/openerp-server.init but this will need a few small modifications to work with the system installed the way I have described above. Here’s a link to the one I’ve already modified for 7.0.

Similar to the configuration file, you need to either copy it or paste the contents of this script to a file in /etc/init.d/ and call it openerp-server. Once it is in the right place you will need to make it executable and owned by root:

sudo chmod 755 /etc/init.d/openerp-server
sudo chown root: /etc/init.d/openerp-server

In the configuration file there’s an entry for the server’s log file. We need to create that directory first so that the server has somewhere to log to and also we must make it writeable by the openerp user:

sudo mkdir /var/log/openerp
sudo chown openerp:root /var/log/openerp

Step 8. Testing the server

To start the OpenERP server type:

sudo /etc/init.d/openerp-server start

You should now be able to view the logfile and see that the server has started.

less /var/log/openerp/openerp-server.log

If there are any problems starting the server you need to go back and check. There’s really no point ploughing on if the server doesn’t start…

If the log file looks OK, now point your web browser at the domain or IP address of your OpenERP server (or localhost if you are on the same machine) and use port 8069. The url will look something like this:

http://IP_or_domain.com:8069

What you should see is a screen like this one (it is the Database Management Screen because you have no OpenERP databases yet):

What I do recommend you do at this point is to change the super admin password to something nice and strong (Click the “Password” menu). By default this password is just “admin” and knowing that, a user can create, backup, restore and drop databases! This password is stored in plain text in the /etc/openerp-server.conf file; hence why we restricted access to just openerp and root. When you change and save the new password the /etc/openerp-server.conf file will be re-written and will have a lot more options in it.

Now it’s time to make sure the server stops properly too:

sudo /etc/init.d/openerp-server stop

Check the logfile again to make sure it has stopped and/or look at your server’s process list.

Step 9. Automating OpenERP startup and shutdown

If everything above seems to be working OK, the final step is make the script start and stop automatically with the Ubuntu Server. To do this type:

sudo update-rc.d openerp-server defaults

You can now try rebooting you server if you like. OpenERP should be running by the time you log back in.

If you type ps aux | grep openerp you should see a line similar to this:

openerp 1491 0.1 10.6 207132 53596 ? Sl 22:23 0:02 python /opt/openerp/server/openerp-server -c /etc/openerp-server.conf

Which shows that the server is running. And of course you can check the logfile or visit the server from your web browser too.

That’s it! Next I would suggest you create a new database filling in the fields as desired. Once the database is initialised, you will be directed straight to the new main configuration screen which gives you a fell for the new User Interface in OpenERP 7 and shows you how easy it is to set up a basic system.