[notmuch] Notmuch performance problems on OSX

[Carl Worth]

Hi Oliver, welcome to notmuch!

On Thu, 14 Jan 2010 15:30:48 +0000, Oliver Charles <oliver.g.charles at googlemail.com> wrote:
> I've installed the latest notmuch from Git at this time of writing,
> along with Xapian from SVN head. However, just tagging a single thread
> with only one message seems to take too long:
> 
> $ time notmuch tag +dissertation thread:7dc536441e6deade4256a46d46451221
> 
> real	0m0.812s
> user	0m0.022s
> sys	0m0.037s

Things work quite a bit faster than that on my machine:

$ time notmuch tag +foo id:5641883d1001140730l22832715ld6bdc95c9938d314 at mail.gmail.com

real	0m0.024s
user	0m0.012s
sys	0m0.004s

But that could just be system differences.

> And tagging all my messages is really horrible:
> 
> $ time notmuch tag +foobar tag:inbox
> 
> real	0m5.076s
> user	0m3.688s
> sys	0m0.105s

For this operation, I can't really compare. How many messages are you
tagging? Here's that operation for me with 525 messages in my inbox:

$ time notmuch tag +foobar tag:inbox

real	0m1.551s
user	0m1.504s
sys	0m0.016s

> That xapian-svn was built from svn HEAD right now, so I'm assuming it
> contains the #250 fix (http://trac.xapian.org/changeset/13808)

Which I think means that things could have been even *much* slower
before. ;-)

The Xapian defect #250 was just one, initial (and obvious) performance
problem. [Though, as I mentioned in a previous thread, if you're using a
Xapian flint database, (look for .notmuch/xapian/iamflint), then you
won't get the benefit of the Xapian fix until you rebuild your notmuch
database from scratch with a current notmuch.]

Once you've verified that you've got the #250 fix functional, there
could still be lots of performance bugs. And it would be time to start
profiling.

Perhaps the "notmuch daemon" idea (which we've proposed earlier for
other reasons) could help reduce overhead from reading the database and
writing it back out again. So that might be one avenue to explore for
fixing things.

I have no idea what OS X does, but Linux keeps my notmuch database in
its buffer cache so I can do these operations without even touching
"disk" (which is actually an SSD anyway, which also helps). I just
tried, and was able to get the single-message tag operation to be 3
times slower by dropping the cache:

$ sudo sh -c "echo 3 > /proc/sys/vm/drop_caches "
$ time notmuch tag +foo id:5641883d1001140730l22832715ld6bdc95c9938d314 at mail.gmail.com

real	0m0.062s
user	0m0.000s
sys	0m0.020s

But again, whatever the performance problem might be, the first step
would be to examine some profiles. (And I'm clueless, myself as to what
profiling tools might be available for OS X.)

-Carl
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20100114/f398ab58/attachment.pgp>