Bug: fatal error with notmuch new, second run starts indexing all over again

Fri Jul 14 03:14:36 PDT 2017

Hi David,
* David Bremner <david at tethera.net> [13. Jul. 2017]:
>> * David Bremner <david at tethera.net> [13. Jul. 2017]:
>>> Gregor Zattler <telegraph at gmx.net> writes:
>>>> I then downloaded parts of the archive, merged and sliced it and
>>>> now I have a sample of 25001 total files (that's not much mail),
>>>> on which notmuch new produces a xapian exeption:
[...]
>>> For me this corpus does not produce an exception (although it does have
>>> one thread with 5067 messages).

>>> I'm running Debian testing, so there could be slightly different
>>> versions of xapian and gmime. What versions of libxapian-dev and
>>> libgmime-2.6-dev have you built notmuch against?
[...]
> I tried on a debian stretch with the same library versions, and I still
> can't duplicate the crash.

This is strange.  I rerun the self compiled notmuch new on this
corpus and it did not throw an exeption, nore did the debian
stretch binary.  I have no idea why this is different with the
first run.

On my way reducing the corpus I had an intermediate corpus of
~43000 files on which notmuch new produced a xapian exeption.  I
kept this corpus but did not include the test results in my email
because the later, smaller corpus seemed more interesting.

I redid the test on this mid-sized corpus and I got an
exeption (again):

0 (master *) grfz at len:~$ gdb --args /home/grfz/src/notmuch/notmuch new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/grfz/src/notmuch/notmuch...done.
(gdb) b _notmuch_database_log
Breakpoint 1 at 0x1f6e0: file lib/database.cc, line 426.
(gdb) run
Starting program: /home/grfz/src/notmuch/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Found 43255 total files (that's not much mail).

Breakpoint 1, _notmuch_database_log (notmuch=0x5555557d2820, format=0x55555558d258 "A Xapian exception occurred adding message: %s.\n") at lib/database.cc:426
426     {
(gdb) bt
#0  _notmuch_database_log (notmuch=0x5555557d2820, format=0x55555558d258 "A Xapian exception occurred adding message: %s.\n") at lib/database.cc:426
#1  0x0000555555577f58 in notmuch_database_add_message (notmuch=notmuch at entry=0x5555557d2820, filename=filename at entry=0x555556fa4870 "/tmp/reduced-sample.xapian-exeption/new/1499897795.R10126604875560291557.len",
                                                                                              message_ret=message_ret at entry=0x7fffffffd2e8) at lib/database.cc:2597
#2  0x000055555556802f in add_file (state=0x7fffffffd560, filename=0x555556fa4870 "/tmp/reduced-sample.xapian-exeption/new/1499897795.R10126604875560291557.len", notmuch=0x5555557d2820) at notmuch-new.c:264
#3  add_files (notmuch=notmuch at entry=0x5555557d2820, path=path at entry=0x5555557d26e0 "/tmp/reduced-sample.xapian-exeption/new", state=state at entry=0x7fffffffd560) at notmuch-new.c:599
#4  0x0000555555567b44 in add_files (notmuch=0x5555557d2820, path=path at entry=0x5555557cf730 "/tmp/reduced-sample.xapian-exeption", state=state at entry=0x7fffffffd560) at notmuch-new.c:483
#5  0x00005555555689ed in notmuch_new_command (config=0x5555557ce1d0, argc=<optimized out>, argv=<optimized out>) at notmuch-new.c:1099
#6  0x0000555555561a27 in main (argc=<optimized out>, argv=0x7fffffffda88) at notmuch.c:456
(gdb) continue
Continuing.
Error: A Xapian exception occurred. Halting processing.
Processed 27611 total files in 21m 2s (21 files/sec.).
Added 27481 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
[Inferior 1 (process 6752) exited with code 01]
(gdb) 

I redid the test with this medium sized corpus with the notmuch
binary from debian stretch, the xapian exeption hits after the
same number of scanned/added files:

0 (master *) grfz at len:~$ gdb --args /usr/bin/notmuch new
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/notmuch...(no debugging symbols found)...done.
(gdb) b _notmuch_database_log
Function "_notmuch_database_log" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (_notmuch_database_log) pending.
(gdb) run
Starting program: /usr/bin/notmuch new
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Found 43255 total files (that's not much mail).
Error: A Xapian exception occurred. Halting processing.
Processed 27611 total files in 10m 21s (44 files/sec.).
Added 27481 new messages to the database.
Note: A fatal error was encountered: A Xapian exception occurred
[Inferior 1 (process 10408) exited with code 01]
(gdb) bt
No stack.
(gdb) continue
The program is not being run.
(gdb) quite
Undefined command: "quite".  Try "help".
(gdb) quit

In case your' not exhausted, the mid-sized corpus is here:
https://giku.de/reduced-sample-midsized.xapian-exeption.tar.xz

Regards, Gregor