Xapian exception leading to database corruption

[David Bremner]

David Edmondson <dme at dme.org> writes:

> Using current git notmuch on Debian testing a rebuild from scratch of my
> database fails:
>
>> agrajag-testing ~/s/notmuch % ./notmuch new
>> Found 605510 total files (that's not much mail).    
>> add_file: A Xapian exception occurred 28m 37s remaining).
>> A Xapian exception occurred adding message: Unexpected end of posting list for 'G0000000000014364'.
>> Processed 137296 total files in 8h 4m 45s (4 files/sec.).
>> Added 135950 new messages to the database.
>> Note: A fatal error was encountered: A Xapian exception occurred
>> agrajag-testing ~/s/notmuch %                          
>

I can't duplicate this (probably not surprising) on debian testing. I
also have about 600k files, and I'm running debian testing. My total
index time is about 1h on a not-very-recent machine with an SSD. I'm
guessing you have a hard disk (or something is deeply wrong to take
that long). Even for a hard disk, 8h to index 137k files seems slow.
Did you happen to check for disk errors?

> I see that bremner reported something like this in #xapian, but not any
> resolution.

I guess your best bet is to write to

  xapian-discuss at lists.xapian.org

>
> Any suggestions? Is it possible to force a new chert database to be
> created rather than glass? (Mostly I'd like to get back to work!)

According to Olly Betts,

   With 1.4 you can pass Xapian::DB_BACKEND_CHERT in the flags when
   constructing the WritableDatabase object.

So you'd have to hack the notmuch source, I'm guessing around line 55 of
database.cc

#define DB_ACTION (Xapian::DB_CREATE_OR_OPEN | Xapian::DB_RETRY_LOCK | Xapian::DB_BACKEND_CHERT)

Do move the existing database out of the way, or apparently xapian can
get confused.