notmuch.el: controlling what does and doesn't get expanded in searches

[Matt Armstrong]

Jani Nikula <jani at nikula.org> writes:

> On Thu, 04 Aug 2016, Matt Armstrong <marmstrong at google.com> wrote:
>> This question pertains to notmuch built from recent git HEAD, using
>> notmuch.el in show mode (i.e. not tree mode).
>>
>> I sometimes read a thread with a bunch of messages and notmuch.el
>> collapses a bunch of them (even if unread and the search matches tags
>> in every message).  I can't figure out the heuristic notmuch is
>> applying here.
>
> The idea is that all the messages matching the query are expanded, the
> others collapsed. Each expanded message must fully match the query. The
> unread or inbox tags are not special in this regard.
>
> I am not saying this is ideal, but this is how it's supposed to
> work. (Indeed I'd personally like to define e.g. saved search specific
> tags or queries to use for deciding which messages to expand.)

Thank you.

Yes, I find the query semantics with respect to tags and threads a bit
confusing at times.  This is not a problem specific to notmuch, as I
find the same kinds of issues in GMail.  Usually the problem occurs at
the semantic border between per-message tags and thread-based
operations.

I can now explain what I am seeing.  In my notmuch-post-hook I tag
messages according to various categories (mailing lists, etc.).  Every
time I do this I also tag them with 'filtered'.  My
`notmuch-saved-searches` has about 15 sub-queries of the form "tag:inbox
AND tag:<something>".

I also have a saved search to catch the "unfiltered" stuff:

  tag:inbox AND tag:unfiltered

So this occurred:

1) One message was sent to a foo-announce mailing list.  This was not
   caught by my filters, so it was simply tagged 'inbox'.

2) All replies were sent to the main "foo" mailing list, which *was*
   caught by my filters and tagged 'inbox' and 'foo' and 'filtered'.

This is bad for two reasons:

a) If I observed this thread by searching for 'tag:inbox AND tag:foo',
   the initial foo-announce message would not be expanded.  But, as the
   first message in the thread, it is the most important!

b) If I observed this thread by searching for 'tag:inbox AND not
   tag:filtered' (as I do to find all the "uncategorized" stuff in my
   inbox), then the foo-announce mail is expanded but none of the
   others.

This problem isn't specific to my 'filtered' tag approach.  I think it
generalizes to any approach that attempts to split incoming mail.

I've been seeing this problem quite frequently because I'm in an
environment where messages are cc:'d to different mailing lists all the
time.  It is common for threads to be cc'd to new mailing lists
mid-thread, or for people to pull lists off the cc: list mid-thread
(sometimes into private per-person distribution).

In this environment, different messages in those threads area expanded
depending on which query I use to find them.  This is undesirable
because, generally, I want to read every message I have not yet read in
these threads.


>> In particular, pressing SPC does not seem to navigate to the collapsed
>> messages (again, even if they are unread).
>
> SPC and n and p are supposed to navigate expanded messages only. N and P
> navigate all messages (but do not expand by default). Again, the tags
> the messages have do not matter. You can manually expand/collapse
> messages, and that'll affect the navigation.

Note that SPC also archives, and when it does, it archives the entire
thread, not just the expanded messages (i.e. those that match the
current search).  So, this gave rise to the above situation, where I
pressed SPC twice and archived a 40 message thread, with most messages
still unread.


>> Worst case: only the first messages is initially expanded and all
>> subsequent are collapsed.  I press SPC and the cursor goes to the end
>> of search results.  SPC again all the entire thread is archived.
>>
>> This behavior has caused me to accidentally skip messages.  First
>> step for me is understanding what is going on so I can fix it.
>
> Yes, let's first check that notmuch behaves as it is expected, and
> then figure out how to improve it.

I think the semantics make coherent sense for ad-hoc searches.  Things
break down for me when considering an inbox processing workflow heavily
based on archiving.

If I return to a thread after reading the first N messages I need not
see those messages expanded.  I have already read them, so I'd prefer
they be collapsed.  Not much usually does this for me because I archive
aggressively, but I don't always archive.  In these cases I think I'd
prefer expansion instead be based on whether I've read the message
(tag:unread).

Also, I do want threads consisting entirely of read messages to appear
in my inbox searches, so I do not want to simply add "AND tag:unread" to
all of my `notmuch-saved-searches`.

Additionally, if messages that don't match the query are pulled into
threads that don't match the query, and are "unread", I'd like to see
them.  Such messages are quite likely to provide important context.

I can think of two ideas:

 i) notmuch could have an "also expand tags" feature, where thread based
    results would auto expand matching tags.  I would set this to
    "unread".

 ii) notmuch could have an "expand query" feature, where thread based
     results would use an entirely different query to decide, within a
     thread, which messages to expand.  I would set this query to
     "tag:unread".

This would be most natural with the current archive semantics in
notmuch.el, which apply tags to entire threads on the assumption that
SPC advances over them in meaningful ways.