notmuch/

Notmuch logo

Optimize notmuch's performance

  1. Optimize notmuch's performance
    1. Use of ecryptfs slows notmuch down.
    2. Xapian database format: chert vs. flint vs. glass
    3. Performance test suite

Use of ecryptfs slows notmuch down.

I noticed that my notmuch new was really slow (several minutes) for less then 20 new emails (around 20K in my database).

The performance tests in the notmuch source didn't show any real issue, even on a non-encrypted ext4 fs.

After investigation and running different tests, a switch to a LUKS-btrfs based filesystem to replace the ext4-ecryptfs resolved the issue. It now takes some seconds to index hundreds of messages.

Xapian database format: chert vs. flint vs. glass

Depending on the Xapian version that was used during the initial notmuch database creation, you might use another database format than the most recent one, which is often faster.

Check whether you are on chert, flint or glass:

$ ls $(notmuch config get database.path)/.notmuch/xapian/iam*
iamglass

If it does not say iamglass, but iamchert or iamflint, than you should migrate (read on).

There might be a file named flintlock, it is no indicator for your database format, but also used in case of chert.

Migrate from an older format to the most recent one (glass)

Make sure you are using xapian 1.4.x (depends on your distribution)

XXX: add examples for different distributions

Backup your notmuch tags:

$ cd
$ notmuch dump --output=notmuch-dump-oldformat

Move away your old database:

$ mv $(notmuch config get database.path)/.notmuch/ notmuch-db.bak

Find and import your messages, a new and fresh database will be created:

$ notmuch new

Restore your tags:

$ notmuch restore --input=notmuch-dump-oldformat

Check whether you are on glass now:

$ ls $(notmuch config get database.path)/.notmuch/xapian/iam*
iamglass

If it says iamflint or iamchert, most likely you are still using an old xapian version.

Clean-up: if you are sure everything is fine, you can delete the old things. The dump of your tags you might want to keep anyway as a backup - on the other hand you should have some mechanism in place to make regular backups of your notmuch database:

$ rm notmuch-dump-oldformat
$ rm -R notmuch-db.bak

Compact your database

In order to greatly reduce the size of your database, you should compact it. This can be easily achieved with notmuch compact:

$ notmuch compact

Note that this process can take a long time for large database and a write lock will be held for the duration to ensure data integrity. The process will create a new compacted database in a temporary directory and finally replace the original database if compaction is successful. For added safety you can provide --backup=<directory> option to preserve a copy of the original database. For more details on this process see notmuch-compact man page.

More information about this in the email archives:

Performance test suite

Notmuch ships with a performance test suite which uses a downloadable corpus of email.