On disk tag storage format

[David Bremner]

Austin outlined on IRC a way of representing tags on disk as hardlinks
to messages. In order to make the discussion more concrete, I wrote a
prototype in python to dump the notmuch database to this format. On my
250k messages, this creates 40k new hardlinks, and uses about 5M of
diskspace. The dump process takes about 20s on
my core i7 machine.  With symbolic links, the same database takes about
150M of disk space; this isn't great but it isn't unbearable either.

The assumption in both cases is that maildirsync is on, so most tags are
stored in the the original maildirs.

In principle such a representation (or some variation) could be be used
to interect with some external source of tagging information like gmail.
It could also be used (with rsync --hard-links?) to synchronize notmuch
databases between machines.

I'm still unsure about the runtime performance impact of updating the
file system and the Xapian index with every tag operation, but I thought
I would see if the representation itself was usable for most people
without bringing the filesystem to its knees. So I'd be interested to
hear other people experiences running this script. It _should_ be safe
since it opens the database in readonly form, but the smart money is on
backups before running other peoples experimental code.  Especially
since I don't claim to actually know python.

One technicality is that this hex-encodes ':' (compared to the other code
floating around); this is so hex_encode(message_id)+maildir_flags is a
valid maildir name. The uniqueness of the names comes from the (much
discussed) keying of messages on message-ids.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: dump-tags.py
Type: text/x-python
Size: 1952 bytes
Desc: not available
URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20121129/75bf508b/attachment.py>