[notmuch] Notmuch performance (literally, in my case)

[Ben Gamari]

On Mon, 15 Mar 2010 09:29:35 +0000 (UTC), Olly Betts <olly at survex.com> wrote:
> On 2010-03-15, Hans Dieter Pearcey wrote:
> > On Sun, 14 Mar 2010 22:59:28 -0700 (PDT), Ben Gamari wrote:
> >> Notmuch is using xapian 1.08-1.99karmic from the Xapian backports PPA, which
> >> I believe includes the recent database update optimizations.
> >
> > As far as I know, it doesn't.  1.0.18 is the stable version in which it was
> > fixed.
> 
> 1.0.18 is also the version that's in the PPA - 1.08 has to be a typo as the
> PPA tracks currently releases closely, and 1.0.8 is 18 months old.

Yep, my bad. That was a typo.

> 
> I've seen a similar issue reported with apt-xapian-index in Ubuntu (it uses
> Xapian to maintain a database of packages).  But I've never seen anything
> like this myself, despite running Ubuntu on my laptop and spending a lot of
> my time building Xapian databases.
>
> Can you try this patch (you'll need to rebuild Xapian from source, and
> depending where you install it, perhaps set LD_LIBRARY_PATH to ensure the new
> build gets used):
> 
> http://oligarchy.co.uk/xapian/patches/xapian-1.0.18-flint-group-fsyncs.patch
> 
> What this does it to at least pair up the calls to fdatasync().  It's
> possible to move them all together, but requires more effort, so it'd be
> nice to know if this is actually going to help.
> 

This does seem to help. Of course, latency is a difficult thing to measure, but
notmuch does _feel_ faster. That being said, iostat still only shows
700kByte/second read and 300kByte/second write, so things haven't changed in
the throughput side of things.