Xapian commits unexpectedly slow

[David Bremner]

Matthew Schauer <matthew.schauer at e10x.net> writes:

> Greetings,
>
> I've been trying to migrate about 25K e-mails to Notmuch, and I'm seeing
> some frustrating performance characteristics that don't seem to match
> with the experience others report.

25,000 messages should really not be a strain, spinning rust or no.

> I'm dumping messages from
> Thunderbird in batches and then running `notmuch new` to add each batch
> to the database.  The indexing performance remains okay, at more than
> 200 per second, but after Notmuch has reported it's finished indexing,
> it hangs for as much as several minutes before exiting.  A stack trace
> confirms that this is Xapian committing the database, with most of the
> time seemingly spent in `fdatasync`.  The time spent grows with the size
> of the database, not the number of e-mails being imported, which means
> this will remain a problem during day-to-day usage.

It would be interesting if you could report the results of running the
notmuch performance test suite (under performance-test/ in the source).

The other thing I'm curious about is the actual size of the
database. This varies a lot, but in the past pathological performance
has sometimes been linked to indexing things that should not be,
bloating the database.

d