Database corruption after clean rebuild

[Gregor Zattler]

Hi notmuch developers,

I also had this database corruption, I waited for the fix to land
in notmuch 0.26.2, build it, moved the xapian directory away, did
a notmuch new and restored the tags from a dump.  But the problem
remains:

~$ xapian-check ~/Mail/.notmuch/xapian
docdata:
blocksize=8K items=10841 firstunused=75 revision=82 levels=1 root=2
B-tree checked okay
docdata table structure checked OK

termlist:
blocksize=8K items=1893162 firstunused=368983 revision=82 levels=3 root=177608
xapian-check: DatabaseError: 1 unused block(s) missing from the free list, first is 0


this is very similar to the old database which I had moved away:
~$ xapian-check ~/Mail/.notmuch/xapian-2018-04-29-00-22/
docdata:
blocksize=8K items=10863 firstunused=78 revision=59623 levels=1 root=2
B-tree checked okay
docdata table structure checked OK

termlist:
blocksize=8K items=1894648 firstunused=360821 revision=59623 levels=3 root=360580
xapian-check: DatabaseError: 1 unused block(s) missing from the free list, first is 0


Now I did notmuch compact and the database check says there are
no errors.


This seems to me as if the fix had not helped or there is another problem.


$ notmuch --version
notmuch 0.26.2+26~g9e158fb
~$ xapian-compact --version
xapian-compact - xapian-core 1.4.3

Thanks for developing notmuch, Gregor


* Javier Garcia <javiertury at gmail.com> [2018-04-07; 17:09]:
> Unfortunately I can't share my emails without the approval of other
> parties. The minimum subsets that trigger the error are in the range of
> 1000-5000 mails, so asking each and everyone of them is out of my reach.
> I tried to replicate the problem using just spam folders without success.
>
> The following is a solid workaround I've stumbled upon. Afew no longer
> complains and database corruption is gone.
>
> $ notmuch compact
> $ xapian-check ~/.mail/.notmuch/xapian
> Â Â  <check messages>
> Â Â  No errors found
>
> I built xapian-core 1.50 but I can't compile notmuch 0.26.1 against it.
> I will wait and test again in a few weeks.
>
> If you are interested in my setup, the error happens with this minimal
> configuration.
>
> #~/.config/afew/config
> [Filter.1]
> query = 'folder:"//(INBOX|Inbox|inbox)$/" AND (NOT tag:inbox)'
> tags = +inbox;-new
> message = Messages in INBOX folder are tagged as inbox
>
> [Filter.2]
> query = '(NOT folder:"//(INBOX|Inbox|inbox)$/") AND (tag:inbox)'
> tags = -inbox
> message = Messages not in INBOX folder cannot be inbox
>
> #~/.notmuch-config
> [database]
> path=/home-path/.mail
> [new]
> tags=new
>
> On 07/04/18 12:51, David Bremner wrote:
>> Javier Garcia <javiertury at gmail.com> writes:
>>
>>> I've applied the path to notmuch 0.26.1 without success.
>>>
>>> $ rm -rf ~/.mail/.notmuch
>>> $ LD_LIBRARY_PATH=/hidden-path/notmuch-0.26.1/lib/:$LD_LIBRARY_PATH
>>> ./notmuch new
>>> Â Â  Found 20065 total files (that's not much mail).
>>> Â Â  Processed 20065 total files in 58s (341 files/sec.).
>>> Â Â  Added 19605 new messages to the database.
>>>
>>> $ xapian-check .mail/.notmuch/xapian/
>>> Â Â  docdata:
>>> Â Â  blocksize=8K items=63 firstunused=1 revision=2 levels=0 root=0
>>> Â Â  B-tree checked okay
>>> Â Â  docdata table structure checked OK
>>> Â Â  termlist:
>>> Â Â  blocksize=8K items=43520 firstunused=8293 revision=2 levels=2 root=748
>>> Â Â  xapian-check: DatabaseError: 1 unused block(s) missing from the free
>>> list, first is 0
>> OK, so probably not related to reference loops (although that patch is
>> not very well tested).  It's not clear how notmuch is triggering it, but
>> this looks like the same bug in Xapian that olly fixed recently [1].
>>
>> A possible next step is to try building xapian master, and linking
>> notmuch against that.
>>
>> Maybe Patrick or Justus (in copy) has some idea why you're only seeing
>> problems in afew.
>>
>> Another debugging direction is to try to duplicate your problem with
>> some subset of mail that you're willing to share (bisection is the usual
>> strategy).
>>
>> [1] https://notmuchmail.org/pipermail/notmuch/2018/026369.html
>
> _______________________________________________
> notmuch mailing list
> notmuch at notmuchmail.org
> https://notmuchmail.org/mailman/listinfo/notmuch
Ciao; Gregor
-- 
 -... --- .-. . -.. ..--.. ...-.-