Database versioning: another thread

Just a quick note about another discussion that is going on regarding Database Versioning on the Subversion Users mailing list.

The thread can be browsed here:

http://subversion.tigris.org/servlets/BrowseList?list=users&by=thread&from=334363

An Open Source Project?

I've been thinking for some time about getting involved in an Open Source project. The question is whether to join an existing project or to start a new one.

The real trouble is of course time. I'm going to be emmigrating to the USA soon, so my time is going become very limited as I do the move and get accustomed to my new home (and re-aquaint myself with my family). That being the case, I really can't justify the time to spend on additional development outside of work.

However, with OSS it's always about scratching an itch, and recently I've found a rather big one. The problem is database versioning, and it's something that I've considered to be a real blight on the software development process. I've looked around and haven't really found anything to meet my (admittedly fluid) set of criteria:

• Store database schema state in the same place as the code that refers to it
• Generate an SQL script that will migrate a database from any arbitrary database state to another.
• Portable - Must either work with all database engines or (more likely) allow support for additional database engines to be added easily.
• Must be able to distinguish between delete/add operations and renames

This isn't an easy problem. I've been involed in discussions on this subject on the PostgreSQL mailing list:

http://archives.postgresql.org/pgsql-general/2005-06/threads.php#00469

I intend to think some more about the problem and if I think I can do it I will set up the project somewhere.

Why can't windows just work?

So there I was all smug about my nice shiny versioned WebDAV folder, when Windows goes and buggers it all up.

You see, it doesn't work.

You can quite happily create a 'WebFolder' (as Windows likes to call it) and you can browse through it. But most of the applications out there don't recognise it. Everything would be fine if you could only map a drive letter to the folder, but no. Micro$oft have to go only nearly finish the job and only make it nearly useful.

So, I've had to find something else. And so I have. It's a Novell product called NetDrive, which you should be able to find an any number of places online (4.1 is the version I've found). It works, and it's free (it's look like it's been replaced by a non-free program called WebDrive, but NetDrive is working fine for me).

The only problem is it's another application to have sitting in your system tray, but it's not too bad really. It supports https and maps the WebDAV folder to a drive letter, so you can happily use the folder with whatever application you fancy: they don't know the difference.

Bloody Micro$oft...

PC Naming

It has just occurred to me that in my last post I referred to the names of my PCs, and that many people might not know what the hell they mean.

Some time ago I wondered about a cool naming scheme for my personal PCs. I've heard of naming schemes based on beers, drugs, planets, Star Trek ship classes and Star Wars characters. I wanted to do something a little different, and so decided to base mine on musical modes.

Modes are basically scales, each with its own distinctive sounds. For more information, see here: Wikipedia (Musical Modes)

To briefly summarise:

• Ionian
• Lydian
• Mixolydian
• Dorian
• Phrygian
• Aeolian
• Locrian

So I can have five more PCs before I run out of names. I doubt that I'm going to have that problem any time soon!

Amusingly, the Wikipedia page includes in its description of the Locrian scale the following: "It is highly unstable". Funny that I happened to name a Windows PC after that mode...

Python on Windows

I've just written my first ever Python script for Windows.

And yes it is big, and it is clever.

My Linux box (mixolydian) is up all the time and every night it backs up a number of things (such as my WebDAV Subversion repository for example). Very nice, but what if that machine's hard drive was to die? Bye bye data.

The easiest solution is to back the files up to another machine: my Windows box (locrian). I much prefer that things like this are done automatically, so ideally I'd mount a Windows share hosted on locrian (from mixolydian) and copy the files that way.

The trouble is that locrian is not left on all the time: it's only on when I use it. To get around that I'd have to write code to make mixolydian poll for the availability of locrian and only copy when it's there. Definitely overkill and a waste of mixolydian's time (which currently is mainly concerned with maintaining a load overage of 0.00).

The solution is to make locrian pull the data rather than having mixolydian push it. To do that, I've written a short Python script in Windows which runs at startup. This lists the files in the two backup directories on each machine and copies across all files that are on mixolydian that aren't on locrian. It runs very quickly, and so hardly makes any difference to the startup time of WIndows.

You probably could do that with a Windows batch file but frankly the Windows command interpreter is such a bitch to work with I simply couldn't be arsed. Plus I just wanted an excuse to write a Python script for Windows (I installed Python on locrian some time ago and afterwards wondered why I had done it: I didn't have anything to run on it.)

So anyway, now I can sleep a little more soundly, knowing that my data is backed up on two hard drives.

Next stop, automatic backing up to CD/DVD...

WebDAV + Subversion

Lately I've been playing with the new autocommit features of Subversion 1.2.0, which allows Subversion to be used as a fully-functional WebDAV provider.

It sounds technical, and is an interesting toy to play with, but it's also really useful for me as a general user. What I now have is a folder on my home server, available online which I can use as any other directory and put files to. It's only accessible via SSL so only I can access it, and all accesses are encrypted to keep the contents of the folder secure.

That in itself is very useful for personal files, including my calendar which I can mount from something like Mozilla Sunbird, allowing me to access the same calendar file from work and home. But the really clever bit is that all changes made to the folder and its contents are fully versioned and tracked by Subversion transparently an automatically.

Take the case of my calendar. Imagine that I open the calendar file from work and add an event. Sunbird saves the new calendar file to the folder, and Subversion receives the new file and silently does a commit to the repository. Using Subversion's standard tools I can examine the repository and access any previous version of any file that has ever been stored in the folder.

To the general user the folder is just a folder, but effectively it has an automatic backup progress that backs up the file every time it changes.

Of course, this means the folder takes up more space than would be taken up by the files currently in it, but not as much as you think. Whenever a file is modified, Subversion only stores the differences between the old file and the new one. So in the case of my calendar change above, Subversion would only need to store the data representing the new event, and not the whole file again.

In terms of speed, using the WebDAV folder is slower than a local directory, but not much so. The slowest operation is saving a file, which involves transferring the file to the folder and waiting for Subversion to commit the change. It's not that slow, but slower than a normal save, making it less suitable for really large files. My use of SSL will also slow the process down somewhat.

In terms of what I use to get it working, the folder is served from Apache 2 with Subversion 1.2.0 underneath it.

On the client side there is davfs2 which allows a WebDAV resource to be mounted as a directory in the filesystem (though I can't figure how to get this to work with fstab).

For Windows there is the built-in WebFolders system, though this annoyingly doesn't yet allow you to map a network drive to the folder.

You can also gain read-only access to the current version of all files in the folder by just browsing to the folder's URL in any browser. This is a standard feature of Subversion which means you can quickly and easily get any file you like from anywhere without messing with local mounting.

Aside from the relatively minor issues mentioned above, the ability to securely share a folder from home accessible anywhere from the web and have it automatically version the changes is absolutely fantastic.

Fun at the checkout

Is it just me?

Please be honest: I can take it.

Every time I go shopping at the supermarket, I choose my checkout as tactically as I possibly can. It's a skillful balance between the checkout with the shortest queue, the least number of items on the belt, and the judgement of how long the people in the queue will take to pack and pay for their goods.

It's the last factor that is the hardest to evaluate, and can lead to you choosing a longer queue with the smug knowledge that you'll be through faster than the person who went for the shortest queue. You couldn't be happier.

Except that the plan is always foiled. Whichever queue I go for, there's always somebody in front of me who decided they've forgotten an item, or have trouble paying. Your smugness turns to frustration as the person who went for the shorter queue sails out, and you can almost see the smug expression on his face even through the back of his head.

There was even one time I was in the queue and wife of the bloke in front of me barged past me with a second full trolley of stuff, and proceeded to hand things from it to the checkout attendant.

The side rule is that if you are lucky enough to get a checkout that is completely empty with no queue you will have to deal with the attendant either having to get change out of bags or change the till receipt paper.

So tell me: is it just me, or does it happen to everybody?

Pocket Space

Well, Summer is fast approaching here in the UK, and that brings with it the usual problem that I have that this year is worse than usual: less pockets.

As it gets warmer, I stop taking a coat/jacket around with me, and that leaves me with fewer places to put things. This year I have an iPod to carry around as well as my Palm and phone; not to mention the usual items such as wallet, coins and keys.

I'm starting to think it's really time for one device that will do the job of a PDA, phone and MP3 player. The trick is to make the device as good at each of those tasks as a dedicated item is.

For example, an MP3 player has few controls on its body. They need to be large and easy to find and press without looking (for example, skipping track while the device is in your pocket). The PDA needs a large screen that can display anything. It's difficult to see how they can satisfy the design requirements for all three devices.

But, that's why I'm not a designer.

We've had PDA and phone combinations for a while. PalmOne's LiveDrive appears to be a stab at being a PDA and Phone combination. Who is going to take the next step and integrate all three?

For now I suppose I'll just put my hands in my pockets and saunter on. That's if I can find room for them of course.