Open Source with the Home Office and the British Computing Society

Recently there has been a lot more interest from the government in Open Source software than we have ever seen before, both at Cabinet Office level, departmental level and in Local Authorities. Last night was the first of two sessions hosted by the British Computing Society’s Open Source Specialist Group to help the Home Office IT team to gain a better understanding of why they are not taking advantage of as much Open Source software as they feel they should be doing, and to examine some of the issues and obstacles that have led to them being locked in to solutions that don’t give them the freedom and cost benefits that they are seeking.

The format of the evening was a panel debate with Mark Elkins of the BSC chairing and Tariq Rashid of the Home Office proposing the topics for discussion. On the panel were representatives from a number of large system integrators (SIs) who work on large scale government projects. The panel was:

  • Darren Austin, UK Chief Engineer, Atos Origin.
  • Adam Jollans, Program Director – Open Source and Linux Strategy, IBM Systems & Technology Group.
  • Mike Robertson, Head of Public Sector Business, Savvis.
  • Gurpritpal Singh, CTO, UK Technology Consulting, Hewlett Packard.

The format of the evening was that Tariq would pose a question and the panel members gave their responses before it was opened to the floor for questions and comments from the audience. This format worked quite well – although some members of the audience were clearly unused to requesting, and then waiting to be called to speak, and rather disrespectfully interrupted the proceedings on a number of occasions to spout their opinion during the panel responses – please, if you go to an event with a set format, don’t disrupt it, that just makes the community seem unprofessional.

I won’t break down the responses question by question (there will be audio published at some point I believe and I didn’t take good notes) but some of the key points raised were:

The System Integrators are perfectly happy to work with Open Source. The customer just has to ask for it. All the SIs on the panel said this. They already provide Open Source solutions to other countries, they already use Open Source software where they are providing just a service (cuts their costs and gives them more control). They just pitch proprietary stuff at procurement contracts because that is what wins them here.

When the customer asks for a service to be performed to open standards (yes there was a discussion of the definition of an open standard, the problems of FRAND and the need for Free standards) then the integrator will generally use Open Source software because it reduces their costs (a little) but much more importantly allows them the freedom to commercialise the overall solution in the way that they want to, without complicated negotiations with a third party supplier. The implication of this seemed to me to be that the government still gets screwed over, but only by the SI, and possibly not so badly.

Purchasers of smaller solutions rather than multi-million pound services projects buy from a catalogue, the  G-Cat or something like that. This is a list of approved, vetted, commercial off the shelf (COTS) solutions that are safe to use (“safe” in this context meaning you won’t get fired if the thing you bought was on the catalogue). This catalogue is hard to get on to. Suppliers of proprietary software have to jump through hoops to prove that they are good enough as a company to supply the licenses and there may be some technical appraisal, I don’t really know the details. The point is that the process is hard, it takes time, and probably money. Suppliers go through that process and write it all off as cost of sales, because they know that if they get on the list then the gravy train is on it’s way into town. Open Source projects, with great code, a solid and active community, but no real concept of “financial stability” (and equally no concept of “financial instability”) often have no budget to jump through hoops and fill out documents as a presales exercise because they get, and want, no financial reward at the end of the process when someone in local government downloads and uses the software for free. If the government wants Free Software in the catalogue, they are going to have to pick up the tab in the short term for the presales activity and engage with some knowledgeable consultants (yes, we will do that kind of thing) on a project to go through the evaluation process and fill out all the forms to enable, in the longer term, better value selections to be made from the catalogue.

There was quite a discussion about the ownership of risk, this is important to government purchasers, but more as a concept, than as a reality. Large projects have big penalty clauses, which means that the government likes to work with suppliers who have the financial wherewithal to live up to these clauses. I don’t think I am revealing that much about my company finances to say that we would struggle to demonstrate that we could pay up on a penalty clause running into tens of millions of pounds. Does the government exercise these penalties on a regular basis? No. As one of the pannelists mentioned they would swiftly end up owning all the SIs if they did, and whilst the UK government nationalising IBM is a fun thing to contemplate, it really isn’t going to happen. I made the point at this stage that the government seems to get a lot of comfort from knowing “who to sue”, if things break. What they need to do is learn how to gain comfort from knowing “how to fix it”, and knowing that they can engage with any other supplier to fix broken things. Having open code and the legal right to modify it to your requirements and to have other people modify it to your requirements actually reduces risk. Having financial penalties does not in fact reduce risk at all, it just mitigates your liability when things go wrong.

Next week there will be another debate covering slightly different topics, I believe the format and panel will stay the same which I think works very well (subject to a well behaved audience of course). The topics are listed below, feel free to discuss them in the comments and I will try and pass on some of the most insightful at the event.

Evening Debate 2 – Tuesday 1st March

1. Security. OSS is insecure compared to commercial software?

  • By what criteria can we select software to minimise security risks?
  • Does OSS need a different approach to patching?
  • Can we simply use empirical evidence when comparing OSS with closed software? Statistics for internet browsers are common – published vulnerabilities, known exploits, time to fix
  • Key question for HMG is – all things being equal, open code means vulnerabilities can be discovered and exploited before there is time to fix

2. Buy-not-Build. Can OSS actually benefit HMG because HMG doesn’t want custom or re-engineered software?

  • HMG generally asks IT suppliers to build systems from COTS components and minimise customisation and re-engineering – it doesn’t want to maintain special code because of cost and risk. So does a significant benefit of OSS not apply to HMG?

3. Legal advice for OSS

  • OSS has some unique legal aspects compared with commercial software – where to get advice? Myths around legal obstacles and obligations are going unchallenged.
  • Patents and liability issues are often raised – resolved by major OSS suppliers who will shield customers?

4. Long Term Strategy

  • OSS won’t happen overnight.
  • Should we work backwards from insisting on open information formats for HMG interactions with the public and other sectors? This way the use of open standards compliant software filters back into HMG organisations.

5. Other Ideas

Edubuntu at BETT 2011

Just before my trip to Belgium I was asked to say a few words about the Edubuntu project at the BETT 2011 educational technology show as part of a larger presentation on Open Source software for education. It was quite encouraging to see the packed room of teachers and educational leaders all wanting to learn more about doing more for less with Free Software.

OpenERP vs Lotus Domino

I spent last week out in Belgium, the home of fine chocolates, waffles and Open Source Enterprise Resource Planning applications. I was lucky enough to sample all three as I was on a training course in the OpenERP head office. OpenERP 6 has just been released and it is an amazing thing to have a full ERP system that is Free Software and has Ubuntu as the preferred platform (we were all given an Ubuntu VMware/Virtualbox virtual machine for the training course). The training I did covered the technical aspects of the OpenObject framework rather than the accountancy and business management angle of the functional training, in fact throughout the course we did nothing related to moving money and stock about.
Back in the past I used to be a Lotus Notes and Domino developer, building business applications for medium to large enterprises all over the world. Notes has a bit of a reputation for being unpopular with users for a variety of reasons, but if you forget about the email client aspects of it the underlying platform is the granddaddy of all the NoSQL database engines and a forms based development environment which, despite IBM’s best efforts to break it, is still rather powerful. The OpenObject platform is similarly powerful, it has a few advantages and a few drawbacks, what I want to do here is provide a bit of a comparison and terminology cross reference between the two platforms and see what concepts are common to both. I will be referring throughout to OpenObject rather than OpenERP because I am talking about the framework on which OpenERP is written, just like Notes Mail is an application on the Notes framework.

Database / Server

The OpenERP server can host multiple databases, each one contains totally isolated set of applications and a different set of user profiles, I think of this like multiple Name and Address books and as each openERP database effectively has a separate NAB I would compare them to separate servers or perhaps domains in the Notes world.

Modules / Databases

Yeah, I know databases sort of got renamed to Applications in the Notes client but everyone still calls them databases. In OpenObject this maps most closely to a Module. A database is a collection of related stuff that comes together to peform some useful function. A bunch of modules might work together to form an integrated suite, just like a suite of databases in Notes or the OpenERP suite of modules.

Model and Form View / Form

In OpenObject you list the fields an object has in a python class. This is the back end list of fields that corresponds to the fields on a Notes form. There is a separate file in XML format where the layout of the form is defined. There is a slightly confusing terminology clash here, in OpenObject this is called a form view. If you don’t define a form view it will create an automatic one for you just laying out all the fields defined on your model with their labels in a 4 column grid. So in summary forms in OpenObject are done in two parts, back end field definitions in python and front end layout in XML.

_defaults / Field default values

default values can be pre-filled in new documents by creating a _defaults dictionary in your model class. Basically this is a list of the back end fields that you want to give a starting value to and the value you want.

_constraints and _sql_constraints / Validation formulas

The _constraints dictionary is a set of rules that are enforced just in the user interface, these are implemented as python functions that get passed the relevant field values and return true or false and if neccessary give a message to the user on what is required. _sql_constraints are written down to the database layer so will be enforced even if the document is edited programatically without using the forms user interface.

Records / Documents

OpenObject uses PostgreSQL engine as the underlying database engine, this is a big grown up RDBMS, more comparable to Oracle than the fast and lightweight MySQL. The back end of a document in OpenObject is a record in a table. These are abstracted a bit by the ORM (Object Relational Mapper) so you don’t need to worry too much about the underlying tables, text fields can have up to 1GB of text in them and you can do multivalue fields (called selection fields) and have relational fields that you can think of like an array of doclinks but actually they add columns to tables or create new tables to allow joins and things to happen in the SQL layer as appropriate. You don’t need to worry about this bit too much, it just works. Changes work too. If you add a new field to your model you can just start using it, when the server starts and updates your module it will adjust the schema without losing data to accomodate your new field. If the new field is mandatory then you must provide a default value so it can backfill the existing documents, but if not you can just add the field and start using it like in Notes.

Functional fields / Computed and Computed for display fields

Functional fields can be set to store the value to the database or not.

Read only fields / Computed when composed fields

If a field has a default value and is read only then it will act like a computed when composed field in Notes.

Tree Views / Views

So in OpenObject the UI of a document is called the “form view”, when looking at a bunch of documents this is known in the user interface as “list mode” but in the back end it is actually a “tree view” which you might correctly surmise means it can correspond to a categorised view in Notes. There have been some optimisations in version 6 which allow progressive loading of collapsed tree views which allows the client to drill down into a huge view without loading all the rows, a bit like Notes does. Unlike Notes I don’t think it does progressive loading of flat views, so you probably have to be a bit careful of lists with many thousands of rows as it will send all of the data to the client on opening the view.

Calendar Views / Calendar Views

These really correspond rather well from the OpenObject concept to the same thing in Notes, it is just a treeview but you specify the column that is your start time and which one is your duration. Time zone support is a tricky subject in both, but there is no real perfect solution for all situations when it comes to timezones.

Gantt views / n/a

I never really figured out why Notes didn’t grow a nice gantt chart view format, it would have been so easy to do, probably simpler than the calendar view. OpenObject has one and it works fine, similar to the calendar it is just a tree view and you tell it which column is the start, duration and which contains links that draw dependency lines to other documents. Simples.

Diagram Views / n/a

These allow you to map out related objects, I think it was implemented to allow a visual workflow editor to be built but I could see how you would use this to have a dynamically drawn org chart in an HR application for example.

Graph Views / n/a

Graphing data is great and this works in both the GTK desktop client and in the web client with a rather fancy looking Flash graphing object. Again, these are simple to make, just views with a few extra rules to tell it to draw a bar chart or pie chart and whether to group by a particular value and what operator to use when grouping, this allows you to have bar charts that sum values from related documents. This isn’t supposed to be a sophisticated graphing toolkit, just a nice easy way to visualise some of the data in the ERP system.

Domain / Selection formula

Domains in OpenObject are expressed in a rather odd python syntax as a list of tuples and conditions half in reverse polish notation and half in the normal order. They can be used to restrict the number of options in a dropdown list (e.g. a dropdown list of states that shows the right options based on what you choose in the country field) and can also be used to restrict what’s displayed in tree views just like a view selection formula.

Python / Lotusscript

OK so I am going to be a language snob for a bit here. Lotusscript is nice and easy, it is a dialect of BASIC and does allow a reasonable amount of modern object oriented coding, but it just isn’t an actively developed language and nobody wants to be a BASIC programmer. IBM understand this and have been trying for years to get Java to do half the stuff you can do with Lotusscript but without credible front-end integration in Java it just isn’t going to happen. Python is a proper modern language that has an active development community around it. It is as easy to use as Lotusscript, it lacks all the syntactical punctuation of Java and C++ and there is none of the indignity of having to do your own pointer arithmetic like you get with C. There are Python libraries to do pretty much anything you could possibly imagine, there is no great app store of .lsx files waiting to be called in to extend your Lotusscript capabilities.

Access Controls List / Access Control Lists

All the modules installed on your server add options to the central consolidated Access Controls List rather than each application having a separate ACL. Access can be given to groups of users at the record type or object level and you can give read/write/create/delete access. This is kind of like setting the access rules on forms in Notes (the key tab on the form properties that nobody uses because readername fields on a form make bad things happen). For more fine control there are Record Rules, this is where you would implement Author and reader field type security to allow an Employee to see HR records where the EmployeeID is equal to the current user ID for example.

n/a / Replication

This doesn’t really exist in the OpenObject world. It is a client-server model and if you want to do anything you need access to the server. You can have multiple servers in a high availability cluster but there is no multi-server replication. There is support for using the server over XMLRPC which allows a dumb client (i.e. with no locally installed code other than python) to access the full OpenObject API including methods defined in your custom modules. This could be used to write a simple offline client that can then synchronise objects with the OpenERP server. You could even write an interface to synchronise data to a Notes client. Hmm, I might actually do that at some point.

n/a / What You See Is What You Get

Notes is a WYSIWYG development environment. If you want a field label green or change a table border width you just make it so because you are simply editing a block of rich text. This makes form development in Notes falling-off-a-log easy. It also gives you the flexibility to make a right old mess on screen. OpenObject is more declarative, which leads to less addition of random bling and more consistency across the entire database, which is what you want really. A corporate theme (which I haven’t figured out how to do yet) would apply to all modules in the database regardless of where they came from.

n/a / Public Key Authentication

This is just baked in to the Notes API at a really low level and simply isn’t there in OpenERP. Yes, administering ID files can be a pain in Notes and users tend not to want to invest any time in understanding the value of them over a username and password, but having real encryption and signatures as a fundamental feature is awesome, even if they are woefully missunderstood by the development community and undervalued by the users.
One interesting and somewhat related feature on the roadmap is that OpenERP is going to grow OpenID authentication, thus it will allow you to log on to OpenERP using credentials from Google, Facebook, Launchpad accounts etc.

Internationalisation / Domino Global Workbench

OpenObject uses a common internationalisation framework used in Open Source projects based on the GNU gettext format, all modules are written in US English then all the strings are exported to a .pot template files and .po files are created with translated strings for each target language. The Launchpad website has a great collaborative interface for allowing anyone to contribute and validate translations of your modules. I have done multi-lingual notes databases and yes, there was an attempt to integrate translations into Notes in Domino Global Workbench, last time I looked at that it was a bit rubbish.

Workflow / Lotus Workflow,formally known as Domino.Workflow

The graphical workflow engine is an optional extra on the Notes side (and a pig to integrate – yes I have done it) but built in as a standard feature across all modules in OpenERP. It has a nice graphical editor and you can do quite powerful operations and state transitions on multiple objects and call sub-workflows and so on. It does take a bit of getting used to, but it is the same across all modules.

Reporting and Printing / n/a

Actually I always had stuff printing out OKish from Notes, but OpenERP has a proper reporting engine built in based on the RML Report Markup Language It allows you to create fancy looking reports combining data from multiple objects plus images, barcodes and other complicated things.

Text & HTML / Rich Text

Well there is no native concept of rich text in OpenObject, there are text fields which can be big and can contain HTML and in the web client you can shoehorn in a web HTML editor. Basically it doesn’t compare to Notes if you want formatted text as a native datatype. That said, Notes Rich Text is a major pain when it gets down to doing stuff with it at CD record level or hacking about with rich text stored as MIME, or even using the NotesRichTextNavigator object to work with it. Rich text in OpenObject is limited, but that isn’t as much of an issue as you might expect and when you do integrate formatted objects things will at least be standards compliant and you will be manipulating HTML or perhaps ODF or possibly even MediaWiki markup – there is a wiki module but I can’t get it to render the pages – tips on this in the comments please if you know how.

AGPL / Proprietary

The OpenObject framework is licensed under the Affero GPL. This means that to comply with the license all your modules should be AGPL or a compatible license. If your business model relies on restricting the freedom of your customers you can look away now. Personally I always encouraged customers of Notes development work to review and look at my code and share and reuse it, plus I published interesting things developed or discovered for discussion with other developers in the community. I know this isn’t universal and some people do get all protective about intellectual property rights and like to hide their source code. There is actually no conflict at all between meeting the requirements of the GPL and being paid by a customer to build something they want. As long as you can get past this mental hurdle it really makes little difference to a bespoke software development business.

In Conclusion

Notes is OK, but proprietary and closed and quirky and carrying over 20 years of baggage with it. OpenERP has more of a rigorous and correct feel to it with standards compliance, a modern framework, nice APIs and a real grown up relational back end with transactions and atomicity and so on. The NoSQL engine at the core of Notes is a great idea, and new implementations of the architecture such as CouchDB are dead handy, especially for distributed applications.

All these detail comparisons are rather academic, the bottom line is that having poked about at the application development layer of OpenObject and looking back at all the Notes applications I have written over the last 15 years or so I am left thinking that most of them I could rebuild now in OpenObject, but slightly better.

One final note, I am going to delete uninformed and outdated Notes bashing in the comments, if you were going to say something like “I once used Notes 5 in a place I worked and everyone hated using it for email” then please save the wear on your keyboard and don’t bother, but if you have some clueful and current bashing you want to get off your chest (of either Notes or OpenERP) then pitch right in.

SSH Sessions Timing Out?

Now this might be common knowledge but it took me while searching and scratching about to find the right solution.

As many others do I’m sure, I use ssh terminal sessions a lot. I’ve often got half a dozen open at once either in separate terminals or in Terminator for example. Sometimes if I don’t update them for a reasonable length of time they simply lock up so that I have no response at all. I’ve not actually timed this issue but I expect it is happening somewhere between 10 and 30 minutes of inactivity.

After getting annoyed for the umpteenth time today I finally did some serious searching… I concluded that this lock-up is probably due to my broadband router. It wasn’t happening across the local LAN or when I was connecting from other locations so I already suspected my broadband service. I suspect that it’s a NAT cache timeout or something but that’s not really important.

Several howtos I found discussing various timeout problems suggested editing the ssh config file on the server. This is fine if you know which server(s) you will be connecting too all the time and of course if there don’t happen to be hundreds of them that you need to re-configure. But if you don’t know or do use many machines then that’s not an ideal solution at all.

I think this is though 🙂

If you haven’t already got one, make a file in your home ssh directory (on Ubuntu it is normally found in ~/.ssh):

touch ~/.ssh/config

Open it with your favourite editor and enter a line something like this:

ServerAliveInterval 120

Then save and close it. From my initial tests this seems to do the trick nicely.

According to the manual it:

Sets a timeout interval in seconds after which if no data has been received from the server, ssh(1) will send a message through the encrypted channel to request a response from the server. The default is 0, indicating that these messages will not be sent to the server, or 300 if the BatchMode option is set. This option applies to protocol version 2 only. ProtocolKeepAlives and SetupTimeOut are Debian-specific compatibility aliases for this option.

This is one of those things I should have done ages ago. I hope this help others who find their ssh sessions mysteriously hanging.

OverView Zoomy presentations with OpenGL

I was looking the other day at different ways to present concepts and demos of software than the traditional Impress style slide presentation. There are some online Flash based presentation tools which offer a quite different concept, Prezi in particular is nicely done with a great user interface for editing the presentation, but my overriding thought when using it was that if I was building such a tool, I wouldn’t do it that way, so I made a prezi presentation to outline how I thought it should be done:

A Good Pub Guide: The White Hart, Sherington

The White Hart, Sherington

The White Hart, Sherington

If you travel around the country on business it can be a very tiresome and soul-destroying experience. Especially when you end up having to stay in one of the bland, clinical, plastic and totally soulless hotels that the TV adverts would suggest are something altogether different. I’ve never liked these places and so I do try to find interesting, more characterful lodgings to stay that are not expensive. Often the best to look out for are Pubs with accommodation.

Earlier this week, we had a bit of a road trip on our hands… A day discussing OpenERP in Cambridge and the next day vtiger CRM consulting in Milton Keynes which is not too far away, so we needed somewhere to stay to avoid driving a 200+ mile round trip.

We found a real gem of place! A very attractive country pub with extremely pleasant rooms, very friendly service, fantastic food (the Mixed Grill was really, really excellent), good beer and entertaining and congenial locals too.

If you happen to be working around the Bedford, Milton Keynes, Newport Pagnell area and want somewhere to stay, I whole-heartedly recommend you check out The White Hart in Sherington.

« Previous PageNext Page »