Error while compacting: Bad position key

[Olly Betts]

On Thu, Jul 12, 2018 at 08:22:44AM -0300, David Bremner wrote:
> Mike Hommey <mh at glandium.org> writes:
> > When running `notmuch compact` today, it stopped with the following
> > output:
> >
> > Compacting database...
> > compacting table postlist
> >      Reduced by 25% 648656K (2498904K -> 1850248K)
> > compacting table docdata
> >      Reduced by 15% 24K (152K -> 128K)
> > compacting table termlist
> >      Reduced by 1% 27008K (2211800K -> 2184792K)
> > compacting table position
> > Error while compacting: Bad position key
> 
> I had not seen anything like this before, but when I run xapian-check
> from xapian 1.4.6-2, I see
> 
> 
> termlist:
> B-tree checked okay
> doclen not within bounds
> [repeated 52498 times]

That's a bug introduced in 1.4.6.  I've pushed a fix to master and will
backport for 1.4.7 (which I think will probably be this coming week as
there are several annoying issues found in 1.4.6).

> OTOH, notmuch compact completes for me, so that might be unrelated.

Yeah, that's unrelated to Mike's problem.

> > Running xapian-check says:
[...]
> > termlist:
> > blocksize=8K items=2986940 firstunused=276475 revision=15425 levels=2 root=271786
> > B-tree checked okay
> > doclen not within bounds
> > (...)
> > doclen not within bounds
> > termlist table errors found: 107982
[...]
> > position:
> > blocksize=8K items=236476398 firstunused=653990 revision=15425 levels=3 root=598684
> > xapian-check: DatabaseError: Block 459158 item 179: not in sorted order
> 
> What version of Xapian is this?  I guess an obvious question is if
> there is some potential external cause of filesystem corruption
> (e.g. smart errrors, power loss at an inconvenient time).

The "doclen not within bounds" message above means this must be 1.4.6
(though the database was probably built by an older version originally).

I don't think I've ever seen a report of "not in sorted order" before,
so if it's a Xapian bug it's either very rarely triggered or this is
an unusual manifestation.

I could make a patch which reports more details if you're happy to
build xapian-core from source.

> xapian-check has an option F to attempt to fix things.

It won't fix this case though.  Currently what it can do is regenerate
the small files in the database directory (which are sometimes truncated
to zero size after an OS crash).

Cheers,
    Olly