folder and path completely broken in HEAD?

[dm-list-email-notmuch at scs.stanford.edu]

Jani Nikula <jani at nikula.org> writes:

> It's not going to help you, but I'll mention a few of the issues the old
> folder: search had, which we also had complaints about, and which would
> have been quite hard to fix while preserving the behaviour you want. In
> short, we considered the old folder: search broken.
>
> Given layout:
>
> 	Foo/{cur,new}
> 	foo/{cur,new}
> 	fooing/{cur,new}
> 	bar/foo/{cur,new}
> 	cur
> 	new
>
> It was impossible to refer to the top level folder.
>
> It was impossible to refer to foo without also referring to Foo, fooing,
> and bar/foo.
>
> In your layout, if you also had 2013/.bar.foo, folder:foo would match
> that as well. To not match that, you would have to include each
> folder:.foo.xxx in the search.

First, thanks for the response.  The responsiveness and friendliness of
the notmuch mailing list goes a long way towards compensating for any
missing features / customizability one might want.

I was already aware of the issues you raise, and had worked around them
by just renaming all my mail folders.  I agree that searching for a
particular folder is crucial functionality, and found it weird that I
had to abandon my main top-level mailbox (which I just renamed
.INBOX.Main).

However, currently it seems strange that there are *two* different
search terms (folder and path), and that neither one lets you search for
a portion of your folder name.  Admittedly the old folder code was one
of the parts of the notmuch source that didn't make sense to me (and now
I'm starting to understand why--e.g., the fact that it used stemming,
for instance, was just weird and maybe accidental).

I may be able to hack around this problem in the emacs lisp part or with
a wrapper script.  I'm already having to defadvice around
notmuch-call-notmuch-sexp to implement features I'm missing from
wanderlust and gnus (e.g., the inability to specify regexps matching all
my email addresses).

But to help me understand the current design, can you answer a couple of
questions?


First, are there people out there who do not use a collection of maildir
directories, with all mail in cur and new?  If not, why does notmuch try
to find mail in non-mail-directories, and why do you need search terms
differentiating new and cur?  Conversely, I find it particularly weird
that there's no convenient way to say "stop trying to index stuff that
isn't in a maildir (cur or new)."  You can do the inverse and blacklist
files, but then I end up with stuff like this in my .notmuch-config:

        ignore=dovecot-keywords;dovecot-uidlist;dovecot-uidvalidity;dovecot.index;dovecot.index.cache;dovecot.index.log;dovecot.index.log.2;dovecot.index.search;dovecot.index.search.uids;maildirfolder;

(And still I think there are other junk files that get left around by my
imap server or other software, so notmuch new still repeatedly tries to
index junk whenever I run it.)


Second, does anyone out there have a collection with more than a few
thousand maildirs?  I understand some people index millions of messages,
but even with tiny maildirs of a few hundred messages each, that's a
mere 10,000 maildirs.  So for that, who needs any kind of Xapian
indexing fanciness or dedicated XFOLDER prefix?  Scanning the entire set
of XDIRECTORY terms to apply an arbitrary predicate should take
negligible time.  So all you really need is a boolean search term
containing the directory docid (or with the existing schema the ability
to do a prefix search on XFDIRENTRYn:*).

Thanks,
David