Slowness (search opens every email file?)

[Jason Woofenden]

Hi all,

I'm having a great time patching up the vim frontend, but I've got
an issue that is in the backend, and seems far above my head at
this point:

notmuch search tag:foo is slow!

(when my e-mail files are not already in the disk cache)

I saw on my activity monitor applet that it was using mostly i/o,
and started to wonder if it was opening every e-mail. I little work
with strace and searching revealed that this command was opening
many many e-mail files from my maildir(s). I spent a little while
digging around in the notmuch source, and didn't see where it was
opening the email files.

I don't think the search command should be opening the files. So my
questions:

1) Why is it opening the e-mail files? What information is being
read?

2) Do you agree that it should instead get this information from
the database?

3) How hard would it be make this fast? What would it take?

4) Who wants to do it?


I'd like it to be able to spit out 1000 threads in under a second.
Preferably under 100ms.


Thank you,    -- Jason


P.S. I mean really slow... 

notmuch search tag:foo  took  0.5 seconds for 32 threads

notmuch search foo took 6.4 seconds for 130 threads

Everything's getting into my cache, so I can't easily get lots of
numbers. For a while I had a simple search (tag:foo and tag:bar)
which returned about 600 threads, and it would frequently take
seconds.