Error while compacting: Bad position key

[David Bremner]

Mike Hommey <mh at glandium.org> writes:

> Hi,
>
> When running `notmuch compact` today, it stopped with the following
> output:
>
> Compacting database...
> compacting table postlist
>      Reduced by 25% 648656K (2498904K -> 1850248K)
> compacting table docdata
>      Reduced by 15% 24K (152K -> 128K)
> compacting table termlist
>      Reduced by 1% 27008K (2211800K -> 2184792K)
> compacting table position
> Error while compacting: Bad position key

I had not seen anything like this before, but when I run xapian-check
from xapian 1.4.6-2, I see


termlist:
B-tree checked okay
doclen not within bounds
doclen not within bounds
doclen not within bounds
doclen not within bounds
doclen not within bounds
doclen not within bounds
doclen not within bounds
doclen not within bounds

[repeated 52498 times]

OTOH, notmuch compact completes for me, so that might be unrelated.
I'm copying the Xapian list for input.

> Compaction failed: A Xapian exception occurred
>
> Running xapian-check says:
>
> docdata:
> blocksize=8K items=2677 firstunused=19 revision=15425 levels=1 root=17
> B-tree checked okay
> docdata table structure checked OK
>
> termlist:
> blocksize=8K items=2986940 firstunused=276475 revision=15425 levels=2 root=271786
> B-tree checked okay
> doclen not within bounds
> (...)
> doclen not within bounds
> termlist table errors found: 107982
>
> postlist:
> blocksize=8K items=16090818 firstunused=312363 revision=15425 levels=3 root=249894
> B-tree checked okay
> postlist table structure checked OK
>
> position:
> blocksize=8K items=236476398 firstunused=653990 revision=15425 levels=3 root=598684
> xapian-check: DatabaseError: Block 459158 item 179: not in sorted order
>

What version of Xapian is this?  I guess an obvious question is if
there is some potential external cause of filesystem corruption
(e.g. smart errrors, power loss at an inconvenient time).

> Is there something I can do, or do I essentially need to completely
> rebuild the database (and if so, what's the best way to do it?)
>

xapian-check has an option F to attempt to fix things. It might not
help, but it shouldn't hurt. Otherwise something like

% notmuch dump > backup.txt
% mv Maildir/.notmuch $backup
% notmuch new
% notmuch restore < backup.txt

will rebuild the database.