notmuch python bindings corrupt db index (was: gmail importer script)

[Jason A. Donenfeld]

On Tue, Dec 11, 2012 at 7:26 PM, Patrick Totzke <patricktotzke at gmail.com> wrote:
> OK, i assumed i'd not have to download all messages because i already have them
> on disk -- synced via offlineimap. but then i remembered that OI stores uses
> its own naming scheme for the files in a maildir. so thats probably why your script
> did not realize theyre already there. (maybe it should? it could look up the MIDs from notmuch)

Gmail bandwidth restrictions make this unfeasible. Requesting mail
header IDs via IMAP will rate limit the listing to 2 mails per second.
The best gmail will allow us to do at quasi rapid speed (~180kB/s) is
ask for the IMAP sequence number and the unique gmail 64bit number.
So, the maildir scheme winds up being "${gmail-unique-number}.gmail",
and the script doesn't redownload based on the preexistence of a file
with the corresponding name.

> i am kind of reluctant to reproduce this again (on my work-desktop).

It'd really be a huge help if you could backup and try...

> All i did was start your script, writing to the index in another terminal in between (using alot)
> possibly my cronjob fired a "notmuch new; afew.." in between also.
> when I killed your script the index war broken.
>
> Can i turn on some global logging for notmuch to debug this?
> Maybe libnotmuch had trouble with so many db-writes at a time?

That is very wrong and bizarre. I open the database in read/write
mode, which is supposed to lock other programs from accessing it in
this way... Very strange.