[notmuch] [PATCH 0/4] Make tags applied by 'notmuch new' configurable.

[Carl Worth]

On Tue, 24 Nov 2009 23:10:26 +0100, Jan Janak <jan at ryngle.com> wrote:
> I would like to propose that we make the list of tags applied by 'notmuch new'
> configurable. Right now notmuch applies two tags to all new messages added to
> the database, 'inbox' and 'unread'. The two tags are added by the C code in
> notmuch-new.c and they cannot be changed without editing the source file and
> recompiling notmuch.

Hi Jan,

Sorry to leave this thread sitting without review for so long. My travel
schedule, combined with a big US holiday last week combined to create a
fair amount of backlog.

> This change was motivated by my desire to remove both tags from newly added
> messages. My rules for adding these two tags are more complex and I do it in
> a script run after 'notmuch new'. Instead of 'inbox' and 'unread', I configure
> 'notmuch new' to add a new tag called 'new' (and only that one). This tag
> marks newly added messages that haven't been properly tagged yet by my 
> auto-tagging scripts. The last script I run after 'notmuch new' removes that
> tag. My auto-tagging scripts process only messages with the tag 'new'.

As a side-note, I would recommend against making your auto-tagging
scripts process only new messages. You can get a much more reliable
setup by having your auto-tagging scripts apply to the global
database. And this is not inefficient at all if you simply ensure that
you only match messages which need the tag added.

For example, in your current setup you presumably have something like:

	notmuch tag +foo <foo-specific-search-terms> and tag:new

If you change that to:

	notmuch tag +foo <foo-specific-search-terms> and not tag:foo

then you have the advantage of being able to change the search terms you
are using for the "foo" tag and know that this tag will be applied
globally, (and not only to messages incorporated since the rule
change). Also, you should find that this performs just fine.

You could also have a second specification that would support removing
the "foo" tag if the terms ever changed to become more restrictive:

	notmuch tag -foo tag:foo and not (<foo-specific-search-terms)

I've talked about doing "virtual tags" in the configuration file which
would implement basically the logic of these two "notmuch tag" commands
and be configured by simply providing the pair of:

	"foo" and <foo-specific-search-terms>

So, that's a slightly-off-topic recommendation for doing global tagging
rather than tagging based on new messages only. (Sup's support for
automatic tagging is only for new messages, and I learned there how
painful it can be to operate like that.)

OK, that was a rather long side-note.

Back to the patch, I do recognize that having "inbox" and "unread"
hard-coded within "notmuch new" is rather strict. So I'm definitely
interested in allowing something more flexible here. And I think the
configuration syntax you came up with makes a lot of sense for that.

> On a side note; It may seem logical to add/omit the tag 'unread' directly in 
> 'notmuch new' based on the Maildir flags extracted from the filename of the
> message. I suggest that we don't do that in 'notmuch new'. A better way would
> be writing a small script or command that can be run *after* 'notmuch new'.
> We could then distribute it with notmuch (maybe with other small tagging
> scripts for common situations). 

I do appreciate the sentiment here, (provide good support for
scriptability and let the user use that for extended functionality). But
the question is, how far do we take that idea? For example, we could
simply hard-code a single "new" tag and say that anyone that wants
"inbox" and "unread" on new messages should do:

	notmuch tag +inbox +unread -new tag:new

Would that make notmuch a more elegant system, or would it just be
harder to use than just allowing the inbox and unread tags in the
configuration file?

> I think Maildir flags should be processed after 'notmuch new' is because if
> there are multiple copies of a message with different flags, we may need to
> see all flags from all filenames to set corresponding tags properly and we may
> also need to take the directory part into consideration (i.e. the new mail is
> in 'new', not 'cur').

I think we're going to have to solve all of these problems anyway. We've
already got outstanding patches for dealing with message renames. And
the message rename case is identical to the case of duplicate messages
with different tags, (differing only in whether both filenames are still
present in the mail store).

> There is one catch for users who already have a configuration file and start
> using the patches. They will need to add the new section and the tags option
> manually if they want to preserve current behavior of applying 'inbox' and
> 'unread' automatically by 'notmuch new'.

Why not support backwards compatibility by simply treating an empty
configuration value as "inbox;unread"? We could even make the
notmuch-config code notice a case like this and write out the new
configuration file with the new option explicitly in it, (with
documentation).

I'd definitely like to see a smooth upgrade path where existing users
get as much benefit from a complete and well-documented configuration
file as new users. (I've been frustrated by too many programs where the
only way to get a well-documented configuration is to start from
scratch.)

> The last patch in the set adds a new command line option to 'notmuch new'.
> The name of the option is --tag and it can be used to override any tags
> configured in the configuration file. For example:
> 
>   notmuch new --tag=outbox --tag=read
> 
> adds the tags 'outbox' and 'read' and ignores any tags from the configuration
> file.

I don't like the idea of adding command-line-based tags to "notmuch
new", at least the way "notmuch new" currently works, (searching through
a mail store for any new files).

In the actual patch, you identified a use-case for this that is
important, (importing a new collection of messages and wanting them all
tagged together). The problem of doing this with the existing "notmuch
new" is that it's too easy to accidentally tag more than is
intended. The only way to make it work would be to:

	1. Turn off anything delivering mail to the mailstore

	2. Ensure that the delivery is actually complete somehow, (and
	   not still pending in local MTA queues, etc.)

	3. Run "notmuch new" to ensure all recently-delivered mail is
	   incorporated.

	4. Copy the new collection of mail into the mail store.

	5. Run "notmuch new --tag" with the collection-specific tag.

	6. Turn on mail delivery again.

That's too hard, and far too error-prone. (I don't know of any good way
to do step (2) on my system, for example.)

If we had something like "notmuch import" that took files (perhaps even
an mbox) and copied them into the mail store, then *that* command could
legitimately accept a --tag option.

Meanwhile, the way I think I'd like to see this operation supported is
by a search specification that operates on the filenames. People have
already asked for support for doing tagging based on directory names,
(which will apparently allow people to keep their collection of
Gmail-created tags). So I'd like to see something like this instead for
the use case:

	mv some-archive ~/mail-store/some-archive
        notmuch tag +some-archive filename:some-archive*

This idea seems very much in line with what you advocated earlier,
(handling the maildir "unread" flag after the fact, rather than in
"notmuch new").

And yes, I realize that I'm arguing on different sides of the issue
here. Trying to decide how much "notmuch new" should or shouldn't do is
a tricky thing, and I'm hoping that by arguing from both sides I'll
better be able to see the right answer. :-)

-Carl
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20091202/248e331f/attachment.pgp>