Slowness (search opens every email file?)

[Jason Woofenden]

On 2011-07-11 10:58PM, Patrick Totzke wrote:
> Hi Jason,
> On Mon, Jul 11, 2011 at 03:07:21PM -0400, Jason Woofenden wrote:
> > notmuch search tag:foo is slow!
> > 
> yes, i've just used the vim ui for the first time and i agreee, its sluggish,
> searching for * takes a while.

It's not the vim ui that's the bottleneck. The underlying notmuch
search command is slow. I gave examples in my last e-mail:

>> notmuch search tag:foo  took  0.5 seconds for 32 threads
>> 
>> notmuch search foo took 6.4 seconds for 130 threads


> > (when my e-mail files are not already in the disk cache)
> > 
> > I saw on my activity monitor applet that it was using mostly i/o,
> > and started to wonder if it was opening every e-mail. I little work
> > with strace and searching revealed that this command was opening
> > many many e-mail files from my maildir(s). I spent a little while
> > digging around in the notmuch source, and didn't see where it was
> > opening the email files.
> 
> I cannot reproduce this. I'm no expert, but at least the output of 
>  strace vim -c ":NotMuch" 2>log

:NotMuch just shows the mailboxes. that's fast. It's showing the
contents that is slow. And it's not vim, it's because the notmuch
command is slow. Try this in a terminal:

strace notmuch search tag:flagged 2>&1 | grep 'open(.*/cur/'

Of course change the tag if you don't have flagged messages.


> A quick browse through notmuch.vim tells me that 
> 1) it doesn't use notmuchs json output, i think it should, as iirc
> this api is considered 'more stable' and is easier to parse than the 
> default output. More importantly,

I like this idea. I did some work earlier on improving the message
parsing in the vim ui. Might be better to use the json. I'll look
into json parsing in vim.


> 2) the output of notmuch is copied into a list. This will of course
> be slow if your query matches a lot of messages. Could this be done
> by asyncronously writing to the buffer somehow?

I'm pretty sure vim doesn't do asynchronous anything. That came up
in a vim vs emacs article I read.


Take care,   - Jason