Bash/Scripting Snippets
Over the last week or so I have been planning, preparing and then doing, a migration of the OS on my home server. It used to run a custom built Linux OS based on Linux From Scratch. Whilst this has been very reliable and fast, I have been always niggled by maintenance, upgrades and support issues that are part-and-parcel of a home-brew solution with no package management. In Linux From Scratch’s defence, it isn’t really meant to be a distro. It is a fantastic educational tool. Many people do use LFS for their desktops and servers however.
Anyway, my server is now running Ubuntu Server 8.10 and during the whole migration process I found it necessary to use a few little scripts to keep my sanity and help me do things a bit quicker.
Both as a record for me and for anyone else, here are few snippets which I found useful.
First comes rsync for backup/restore.
rsync is an open source utility that provides fast incremental file transfer.
I guess many of you will have used this before; I had not. It is now an invaluable tool in my arsenal. A simple line like this:
sudo rsync -azvv --progress -e ssh $SDIR $HOST:.$DDIR
can be used in a script and iterated in a loop if needs be. $SDIR
is the source directory, $HOST
is the host machine and $DDIR
for the destination directory. I had some very large files and didn’t want to do the whole backup in one go so this was a useful way to step through specific directories. If you want to exclude any sub-directories or files simply add switches like --exclude 'backups' --exclude 'shared_drive'
after the $SDIR
. The exclude path is relative to the $SDIR
.
Using rsync like this over SSH meant I needed no rsync daemon running on either machine. For a cron job, you would probably want to drop the --progress
switch and the output verbosity and use SSH Agent.
I had a MySQL engine with lots of databases. Although a really brilliant tool, I’ve always been a little weary of phpmyadmin for large database backups/restores as I have found it susceptible to script timeouts etc. So another job for a little bash script.
#!/bin/bash
#Get a simple list of all the database names from MySQL
DBS="$(mysql -u root -ppassword -Bse 'show databases')"
DBDIR="/home/alord/backups/mysql"
mkdir -p $DBDIR
for DB in $DBS
do
echo "Doing a dump of $DB ..."
mysqldump -u root -ppassword $DB > $DBDIR/$DB.sql
gzip -9 $DBDIR/$DB.sql
done
exit 0
The idea for this came from here. Thanks marchost. The key line in this small script is this one: DBS="$(mysql -u root -ppassword -Bse 'show databases')"
. It calls the MySQL CLI, requests and retrieves into the $DBS
variable, in plain text with no additional details or formatting other than line feeds, the names of all the databases that this MySQL instance is managing. Before I found this, I had had to manually create a list of the databases to backup in my scripts. This little line makes it fully automatic 🙂 The rest of the script should be fairly self explanatory I hope.
Warning: There is no error checking or anything here. It was only needed for a one time backup and it worked. But if you plan to use it more frequently you’ll need to add some tests and checks. I would also suggest creating a “backup” user on the MySQL database that has global privileges for just SELECT
and LOCK TABLES
which is enough. That way you are not exposing your databases’ root password to all and sundry.
And then, once my new server was up and running, I needed to drag loads of data back and do some judicious chmodding and chowning. Now, I’d always been niggled that I didn’t know how to do a recursive chmod
on just files or directories. Well; I found out thanks to google and someone called AdaHacker.
find $DIRNAME -type f -exec chmod 640 {} \;
This is a recursive chmod command that works on files only. I didn’t use $DIRNAME as I was running this from the command line. Just replace it with ./
to work from the current directory. If you want to do a similar thing for just directories then replace the -type f
with -type d
.
Hopefully these will be helpful to some. They were to me.
Tags: Bash, cron, Linux From Scratch, MySQL, rsync, ssh, Ubuntu
Have you tried rsnapshot for backups? It uses rsync to transfer files over ssh, keeps multiple backups – which are space efficient as it uses hardlinks.
It’s great!
[…] Bash/Scripting Snippets […]