Bash/Scripting Snippets

Over the last week or so I have been planning, preparing and then doing, a migration of the OS on my home server. It used to run a custom built Linux OS based on Linux From Scratch. Whilst this has been very reliable and fast, I have been always niggled by maintenance, upgrades and support issues that are part-and-parcel of a home-brew solution with no package management. In Linux From Scratch’s defence, it isn’t really meant to be a distro. It is a fantastic educational tool. Many people do use LFS for their desktops and servers however.

Anyway, my server is now running Ubuntu Server 8.10 and during the whole migration process I found it necessary to use a few little scripts to keep my sanity and help me do things a bit quicker.

Both as a record for me and for anyone else, here are few snippets which I found useful.

First comes rsync for backup/restore.

rsync is an open source utility that provides fast incremental file transfer.

I guess many of you will have used this before; I had not. It is now an invaluable tool in my arsenal. A simple line like this:

sudo rsync -azvv --progress -e ssh $SDIR $HOST:.$DDIR

can be used in a script and iterated in a loop if needs be. $SDIR is the source directory, $HOST is the host machine and $DDIR for the destination directory. I had some very large files and didn’t want to do the whole backup in one go so this was a useful way to step through specific directories. If you want to exclude any sub-directories or files simply add switches like --exclude 'backups' --exclude 'shared_drive' after the $SDIR. The exclude path is relative to the $SDIR.

Using rsync like this over SSH meant I needed no rsync daemon running on either machine. For a cron job, you would probably want to drop the --progress switch and the output verbosity and use SSH Agent.

I had a MySQL engine with lots of databases. Although a really brilliant tool, I’ve always been a little weary of phpmyadmin for large database backups/restores as I have found it susceptible to script timeouts etc. So another job for a little bash script.


#!/bin/bash
#Get a simple list of all the database names from MySQL
DBS="$(mysql -u root -ppassword -Bse 'show databases')"
DBDIR="/home/alord/backups/mysql"
mkdir -p $DBDIR
for DB in $DBS
do
echo "Doing a dump of $DB ..."
mysqldump -u root -ppassword $DB > $DBDIR/$DB.sql
gzip -9 $DBDIR/$DB.sql
done
exit 0

The idea for this came from here. Thanks marchost. The key line in this small script is this one: DBS="$(mysql -u root -ppassword -Bse 'show databases')". It calls the MySQL CLI, requests and retrieves into the $DBS variable, in plain text with no additional details or formatting other than line feeds, the names of all the databases that this MySQL instance is managing. Before I found this, I had had to manually create a list of the databases to backup in my scripts. This little line makes it fully automatic 🙂 The rest of the script should be fairly self explanatory I hope.

Warning: There is no error checking or anything here. It was only needed for a one time backup and it worked. But if you plan to use it more frequently you’ll need to add some tests and checks. I would also suggest creating a “backup” user on the MySQL database that has global privileges for just SELECT and LOCK TABLES which is enough. That way you are not exposing your databases’ root password to all and sundry.

And then, once my new server was up and running, I needed to drag loads of data back and do some judicious chmodding and chowning. Now, I’d always been niggled that I didn’t know how to do a recursive chmod on just files or directories. Well; I found out thanks to google and someone called AdaHacker.

find $DIRNAME -type f -exec chmod 640 {} \;

This is a recursive chmod command that works on files only. I didn’t use $DIRNAME as I was running this from the command line. Just replace it with ./ to work from the current directory. If you want to do a similar thing for just directories then replace the -type f with -type d.

Hopefully these will be helpful to some. They were to me.

Excellent Linux Humour

A bit of light relief… I came across a link to this very funny page on the Linux From Scratch mailing list.

I also really enjoyed the sig of the poster: Jeremy Henty…

“I compiled Linux From Scratch, and all I got was this lousy command line.”

Nice one.

Thanks Jeremy.

Pet Project

I’ve been meaning to write about this for a while now. But what with the flu, Microshaft’s ongoing corruption of the ISO and some other stuff cropping up, I just kept finding reasons to put it off. Finally however, after jotting some notes and so forth for a few days I’ve managed to get my act together.

A Pet [Open Source] Project I want to give some airtime to, and get some assistance for, is Linux From Scratch.

Linux From Scratch

History

Going back into the dark ages, I had been “playing” with one Linux distribution or other from the cover of PC magazines for sometime (I think it was a very early Mandrake or Suse product that first grabbed me), and found the whole system quite fascinating. The fact it was all free, and you could “LOOK” into it and see how it all worked together was a real eye opener. I was used to PCs and other computer platforms so it wasn’t all new… I grew up with VT100 terminals, DEC VAX VMS and then DOS so command line interpreters and such were nothing new in themselves but to get a complete OS, that did stuff, was free, and actually encouraged you to examine it, I remember it making me sit up and take notice even then.

One of the very first Open Source communities I came into active contact with was the Linux From Scratch (LFS for short) community. I cannot remember how I stumbled across the project or quite how long ago either, but it was quite a few years certainly. They have a feature which encourages newcomers to register their first LFS build when it is up and running. Checking on their website today, they have 19570 users registered so far. My LFS ID is 216 and the version of “the book” I recorded as having followed was 2.4.x when I registered. Although I certainly built (tried to build) a few before getting brave enough to register 😉 Anyway, I’m guessing this would have been around 1999/2000 some time.

LFS is still a project I follow closely and have a very warm opinion of. It has taught me a great deal over the years.

What is it then?

The project – if you can’t guess from the name – is all about building a functional Linux based operating system from scratch. That is, from nothing. You start with a spare partition on your hard disk and, by following the book, you learn what makes up a GNU/Linux operating system, how that operating system works and why bits of it behave the way they do. It is an educational project and it is a brilliant educational project. You gain knowledge of not just Linux itself but, Bash, compiling, device management and much, much more too. And what you also learn is what makes it all tick together… It is quite hard to explain but it’s a bit like the whole being worth more than just a simple sum of the individual parts.

LFS was started by a chap called Gerard Beekmans. The LFS project’s homepage explains the project thus:

What is Linux From Scratch?

Linux From Scratch (LFS) is a project that provides you with step-by-step instructions for building your own customized Linux system entirely from source.

Why would I want an LFS system?

Many wonder why they should go through the hassle of building a Linux system from scratch when they could just download an existing Linux distribution. However, there are several benefits of building LFS. Consider the following:

LFS teaches people how a Linux system works internally
Building LFS teaches you about all that makes Linux tick, how things work together and depend on each other. And most importantly, how to customize it to your own tastes and needs.

Building LFS produces a very compact Linux system
When you install a regular distribution, you often end up installing a lot of programs that you would probably never use. They’re just sitting there taking up (precious) disk space. It’s not hard to get an LFS system installed under 100 MB. Does that still sound like a lot? A few of us have been working on creating a very small embedded LFS system. We installed a system that was just enough to run the Apache web server; total disk space usage was approximately 8 MB. With further stripping, that can be brought down to 5 MB or less. Try that with a regular distribution.

LFS is extremely flexible
Building LFS could be compared to a finished house. LFS will give you the skeleton of a house, but it’s up to you to install plumbing, electrical outlets, kitchen, bath, wallpaper, etc. You have the ability to turn it into whatever type of system you need it to be, customized completely for you.

LFS offers you added security
You will compile the entire system from source, thus allowing you to audit everything, if you wish to do so, and apply all the security patches you want or need to apply. You don’t have to wait for someone else to provide a new binary package that (hopefully) fixes a security hole. Often, you never truly know whether a security hole is fixed or not unless you do it yourself.

Why LFS is a great platform

[When I discuss LFS I also imply the use of BLFS (Beyond Linux From Scratch) which is a fantastic resource for how to build and install the stuff that goes to make up a “useful” and “complete” Operating System.]

As some of the readers here will know, the little server I’ve built for home use is running LFS. It also runs, Apache, Tomcat, MySQL, PHP, Postgresql, is a Mail server, a Samba (Windows Networking) server, is our telephone exchange (running Asterisk) and a few other things too.

One of the main reasons for choosing LFS as the platform for this server is this: as it is built entirely from scratch there is no bloat, or unnecessary applications, the system is about as lean as you can get. The hardware I chose (very deliberately) is not the most powerful in the world; a mere 7Watts power consumption. But the applications running on the server currently seem very happy and there are plenty of system resources spare. This would be very hard to achieve using a mainstream distribution as they have to cater for as generic a host platform as possible and include a huge amount of features and supporting applications that are largely superfluous for a custom-built and tailored system.

Why LFS is not a great platform

LFS is not perfect however. The hurdle that causes most LFS users eventually to fall down and revert to a mainstream distribution is that of long-term maintenance of the LFS system. There is, by default, no concept of a package management system. When you install an application, you download the source code, and build the executable binaries and libraries and install them on your system. If there is a “dependency” issue like a missing library or something, this must be installed first before you can continue. In most respects this isn’t such a bad thing, but if you want to try some new app out it can involve building a great deal of software that you may realise, afterwards you don’t really want. Removing the unwanted can be a PITA.

My Desktop OS is Ubuntu. It works, and is very easy to upgrade and manage.

What happens next then?

In a few recent weeks, there has been a great flood of discussion and debate on the LFS mailing lists. The original thread for this debate, started by a long term LFS editor called Jeremy Huntwork, has sown the seed for a process to review what LFS is all about and how it could be taken forward whilst still maintaining the core principle of being an Educational Project first and foremost.

One area where I feel the project’s new direction and strategy could really benefit is from some “new blood” with few pre-conceived ideas or historical baggage.

If you use Linux, don’t really know what is going under the hood but want to, then please visit the LFS website, download or read on-line the current book and start working your way through it. Join the mailing lists (either directly or go through gmane and your favourite newsreader), and please contribute your views and experiences.

We really want to give LFS a new lease of life and that, IMHO, needs some fresh ideas and thinking too.