[notmuch] notmuch new: Memory problem

[Carl Worth]

On Fri, 20 Nov 2009 09:56:50 +0100, Dominik Epple <dominik.epple at googlemail.com> wrote:
> I am strongly interested in giving notmuch a try.

Welcome to notmuch, Dominik! I'm sorry your initial attempt to use it
hasn't been quite as smooth as we might like.

>                                                   But I fail setting
> it up. The problem is that during "notmuch new", memory consumption
> and system load increases to values that make my system unusable. I
> then killed "notmuch new" at a memory consumption of 2.7G and at a
> system load of 7.

Yikes. That really sounds like something ran out of control consuming
memory. I certainly haven't seen anything like that before.

> After hitting Ctrl-C, it says "Stopping" but does not stop. I then
> killed "notmuch new" after some minutes with signal KILL.

After "Stopping" gets printed, the notmuch code won't be doing any more
work. It is expected that it will take some time after that message is
printed before notmuch will actually exit. The extra time is to wait for
Xapian to flush out to disk data that notmuch has already provided to
it.

I'm curious how big your .notmuch directory ended up after this
operation. (And how that compares in size to the total size of your
collection of mail.)

> Is there a problem with the number of my mails? I currently have over
> 40.000 Mails... they live currently in mbox files, I created a Maildir
> with mb2md-3.20.pl.

That's definitely not too much mail. I think you should expect "notmuch
new" currently to index on the order of 10 - 100 messages/sec.

Your "notmuch new" process should have been reporting a count once per
second as it progressed, (at least until things went wrong). How far did
you see that go?

I'm wondering if there's a particular file (or files) that are
triggering the bad behavior. Maybe we need a debug option for "notmuch
new" to print the filenames of messages as they are being processed.

-Carl