Doing your regular backups is one thing, but doing it right is sometimes a quite different story. Especially when you happen to stumble upon some extreme situations - like e.g. a reasonably small (in terms of actual disk size) Subversion repository that has a rather high number of committed revisions.

Some time ago our backup software started to returned strange looking information about one of our SVN repositories - it reported that the process was taking a very long time to complete. A full weekly backup that started on Monday night, hasn't finished till Wednesday morning. That wasn't normal...

Although original repo was quite small (500MiB), after being dumped, compressed and encrypted, it weighted over 11GiB! Given the final size, the reason why it took so long for a full backup to complete (remember - dump, compress and finally encrypt) was quite obvious. But why was the dump so enormous? Was there anything wrong in our setup? Other repositories were backup up as normal, and all of them could be restored (we did a full diff, just to make sure).

A bit of a background

After some investigation it turned out that the repo was quite small in size (about 500MiB) but it had a reasonably high number of committed revisions (way over 25k).

It wasn't anything strange as we keep our whole configuration in Subversion, which is then distributed by Puppet among all of the servers. As a safe guard, all servers report and commit back all the changes made to their configuration.

They do it on regular intervals, run asynchronously from cron, every 4 to 6 minutes. If anything gets changed, machine sends back the current state of its configuration. Those changes are typically small, but given the number of servers, revisions build up quite fast.

What can be done about it?

The "problem" was in the way we did our backups. Initially, all our Subversion repositories were dumped using svnadmin dump and then piped through a set of other tools like bzip2, gpg etc. Finally, they were distributed among different destinations.

svnadmin dump uses a binary safe and portable text format to store the backup dumps. Unfortunately, if you have loads of revisions, and those commits are quite small in size, the overhead becomes very high - high enough to create such abnormal situations.

Even if the commit includes only a single line inside some file, it's still described by a number of attributes, that get dumped during the backup process. Multiply that by a factor of 25k and you get the point.

Solution

The solution turned to be rather easy.

Instead of doing svnadmin dump we make a hotbackup copy of the repository, tar and then feed through the original pipeline (bzip2, gpg and so on).

There is one catch, though. This requires extra space of roughly twice the size of the largest repository being backed up. But given the overall results, it was worth it in our case, since the size of the repository backup dropped dramatically - from 11GiB to something around 100MiB.

Current state

As a side note, currently our repository holds slightly above 700MiB of data and about 32k revisions. The backup still weights roughly 100MiB and takes only minutes to complete.

So what?

Always see through your toolbox before you start complaining or making panic movements :)

But there was a problem. I've even made a post on the Dovecot mailinglist:

http://www.dovecot.org/list/dovecot/2006-March/012064.html

In general - dovecot cannot mimic postfix' way of handling catch-all addresses - there was nothing even similar to the "table search order" found in my favorite MTA. So I made a rather ugly patch... and it worked. But it was such a dirty hack that merging it with the dovecot's trunk wasn't really an option. So whenever a new dovecot release was
published I had to manually prepare my "own" version. So I ended up having two separate setups depending on the client's demands - either a vanilla dovecot without my patch (that was updated on a regular basis) or a hacked version that... well... should be updated :)

Couple of days ago I was contacted by Maciej Paczesny asking if there was any progress with the problem I reported back in 2006.

After exchanging some emails and thoughts I think I've finally found a neat way of getting "things done right".

http://www.postfix.org/canonical.5.html

All you have to do in fact is to rewrite the recipient address using postfix' recipient_canonical_maps table. That's all. Thanks to this little trick, dovecot-deliver receives an address that is final and unique so it doesn't have any troubles locating a proper message store directory.

Technically speaking, you must define recipient_canonical_maps in such a way that it would return final and unique address for any of your users (here we can benefit from postfix' table search order, so catch-all addresses do work!). Then postfix will rewrite the envelope To: header, pass it to dovecot and voila - problem solved.

Thanks Maciej! :)

And that's it - the switch is upgraded!