Xapian commits unexpectedly slow

[Matthew Schauer]

Greetings,

I've been trying to migrate about 25K e-mails to Notmuch, and I'm seeing
some frustrating performance characteristics that don't seem to match
with the experience others report.  I'm dumping messages from
Thunderbird in batches and then running `notmuch new` to add each batch
to the database.  The indexing performance remains okay, at more than
200 per second, but after Notmuch has reported it's finished indexing,
it hangs for as much as several minutes before exiting.  A stack trace
confirms that this is Xapian committing the database, with most of the
time seemingly spent in `fdatasync`.  The time spent grows with the size
of the database, not the number of e-mails being imported, which means
this will remain a problem during day-to-day usage.

Has nobody else had a problem like this?  Is my setup just weird?  I'm
using Notmuch 0.29.3 from the Arch community repository, with Xapian
1.4.14 also from Arch repositories.  I am using a spinning-platter hard
disk, but I find it hard to believe that an SSD is required to get
Xapian to perform well at this scale.

Please let me know if you have any performance pointers or can help me
investigate this further.  Many thanks!


Matthew Schauer