Notmuch scripts

[ccx at te2000.cz]

On Fri, Jun 24, 2011 at 11:29:21AM -0700, Carl Worth wrote:
> So much of the new functionality here consists of things I'd love to
> have implemented in the core command-line interface of notmuch

That's what I thought. I think it's actually beneficial to go the
prototyping route here and start with implementations that can be
written and altered quickly.

> The features I see that I'd really like to see in the core notmuch
> command-line tool are:
> 
> 	* Configurable "saved searches", (a syntax for expanding aliases
>           for often-repeated search specifications).
> 
> 	  That's an idea we've had for a while. What's new with the
> 	  zmuch implementation is the proposal of ":alias" for the
> 	  syntax. I think I might like that quite a bit. It looks a bit
> 	  easier to read (and type) than the previously-proposed
> 	  "{alias}".

Glad you like it. :-)

> 	* Delivery of search results to a maildir of symlinks
> 
> 	  The zmuch implementation has this functionality intertwined
> 	  with something that also invokes mutt. Obviously, people using
> 	  other MUAs might like to access this feature independently.

That's one of reasons I'm rewriting it. What I'm planning to have is:

  * notmuch-maildir-create -- make the symlinks, also save metadata in
                              the maildir

  * notmuch-maildir-merge -- reflect the changes on the original files

  * notmuch-maildir-refresh -- merge and recreate with same query

  * notmuch-maildir-mua -- create temp directory, invoke MUA (mutt) on
                           it, merge back

  * notmuch-maildir-search -- merge if old maildir is there, prompt for
                              query, create new maildir

The last one should be sufficient replacement for what notmuch-mutt
currently does, provided one has set up to run merge after mutt instance
exits. I'm not very satisfied with naming on the last two, so
suggestions are welcome :-)

> 	* Operations on files matching search terms (move, remove,
>           xargs)
> 
> 	  This isn't an operation I'd previously considered including in
> 	  notmuch, but it does seem generally quite useful.

These are just handy shorthands, the actual command could still fit on
one line.

> 	  Should we consider doing something like git does and allow
> 	  something like "notmuch xargs" simply find and invoke a shell
> 	  script named notmuch-xargs?

I thought of that also, but I wouldn't do that yet. Not until the
scripts are bit more polished and tested.

> 	  Doing that could let us get a bunch of this functionality in
> 	  place in the "core" sooner than if we waited for it to be
> 	  re-implemented in C.
> 
> 	  Though if we did this, I think I'd be highly inclined to port
> 	  the scripts from zsh to bash or even POSIX sh. How hard would
> 	  that be?

Not much of a problem for simpler scripts, but for the big ones like
saved searches or maildir merging it would mean a lot of added
boilerplate. That would reduce the readability and defeat the purpose of
prototyping.

I could rewrite the more complex ones in python, but on the other hand,
the zsh package is just 6MiB here including debugging symbols. I don't
think there will be many who will find it problematic to install.

> 	* Better date syntax for search specifications
> 
> 	  That's something that's obviously been missing from notmuch
> 	  core from the beginning. And there have been several proposals
> 	  with patches to do this in various ways.

I think it would be neat to steal this from the 'at' command.
By the way, this is another use-case for notmuch proxy.

> 	* Implicit concatenation of search terms with OR
> 
> 	  This seems like something easy to do with a command-line
> 	  arguemnt. Perhaps "notmuch search --or ..." ?

What would I personally like would be to use [] to mean the same as (),
just with implicit ORs inside. For example: "[ foo bar ( spam eggs ) ]"
would expand to "foo or bar or ( spam and eggs )". This way we can have
long sequences of both ORs and ANDs in one query.

 
> If we got all that into the core, then what would be left here would be:
> 
> 	notmuch-mailvars.sh
> 	notmuch-mutt.sh
> 
> 		These would provide integration of notmuch with mutt.

Actually I wish to make it generic enough that it can be used with
anything that can read a maildir.

> 	notmuch-spam.sh
> 	notmuch-unspam.sh
> 
> 		These would provide integration of notmuch with
> 		bsfilter, (and perhaps should be named to make that more
> 		clear---or generalized to justify the current name).

I'd love to make it generic, that's why I asked what people use.

> 	notmuch-pager.sh
> 
> 		I haven't looked at this to see what the colorization
> 		actually looks like, (I'm not always a huge fan of lots
> 		of color in my terminals). It seems that this would be
> 		more cleanly implemented as notmuch-colorize.sh or so
> 		and leave the pagination separate.

Actually I would like to have some common configuration somewhere, so
people can attach colors to certain tags and any app would be able to
utilize that.

By the way the flags for less I use make it behave like it was not there
at all if the contents fits on one screen or the output is not a
terminal.

> If we had that, I'd feel really comfortable having each of those in
> contrib. I think contrib should be restricted to things which provide
> integration of notmuch with some external tool, (and should make that
> obvious by having a name like notmuch-<tool> or notmuch-<tool>.sh or
> whatever).

I think contrib can serve also for experimental extensions and
interfaces, unless you want to move them somewhere else.

> All in all, there's definitely some very interesting functionality here,
> and I definitely appreciate you sharing it. Let's figure out the best
> way to get it all integrated into notmuch.

Thanks. I'm gonna alter most of the scripts so they better match unix
conventions and probably even make manpages for them. So don't take the
current state of the repo as final, I just wanted to give you heads up.

> Maybe in the meantime we throw everything into contrib even if some of
> it is seen as just proposals for better interfaces in the core tool? I
> don't know.

That's what I plan to do.

> >   * Every application that does not act as a proxy should use
> >     environment variable NOTMUCH to find the actual notmuch executable.
> > 
> >   * Every application that acts as a proxy should ignore the NOTMUCH
> > 	variable
> 
> That sounds reasonable enough to me. Perhaps these rules could go into a
> new contrib/README that would set out some guidelines for writing
> contrib tools, (such as notmuch-<tool> which I mentioned above).

Great. Should I write that too? :-) 

> > Configuration and temporary files:
> > I like XDG specification.
> 
> I'm missing some context to know what you're suggesting here.
> 
> > I think it's bit unnecessary to have to have
> > config files that belong only to few scripts littered all around my
> > homedir.
> 
> We should be able to put configuration for contrib tools into the main
> notmuch configuration file. If your tools don't want to read that file
> directly, they should be able to get by with the interfaces provided by
> "notmuch config set" and "notmuch config get". Obviously, each tool
> should create its own section in the configuration file.

I'm not so sure INI format is a good choice for every tool. It's not
conveniently parseable by a shell script and sometimes you need more
than just get/set functions. For example for saved searches I don't know
the key names beforehand and I need to retrieve them in defined order.

> Is that an insane plan?
I think nice thing to have common configuration. I'm unsure this one
size fits all approach is the best way, but I'm kind of okay with it.

> >   * Spam filter. Do you guys use any? What does it's interface look like?
> >     I currently use bsfilter which I've found does it's job pretty
> >     well.
> 
> I've currently got amavis and spamassassin adding extra headers, (and
> below a certain threshold I've got maildrop delivering detected spam to
> a separate maildir).

bsfilter can add headers too, though I don't use that feature now. I
didn't know how would offlineimap or any other mail-sync solution behave
if I went around changing actual content of the messages.

> Currently, notmuch never sees the detected spam. Ever since we got
> folder: support I've been meaning to let notmuch see it so that I can
> use notmuch to dig into my spam when I suspect something got
> mis-detected.
> 
> I don't currently have any system for getting user-provided feedback
> into my spam filtering. Do you get that with bsfilter?

Yes, bsfilter is short for bayesian filter, so it kind of needs sets of
ham and spam messages to learn from. That's why I have the scripts to
handle that.

> >   * Colors. I use bright fg on dark bg, but I understand somebody won't
> >     be happy with this choice.
> 
> I'm pretty-much black-on-white only. I really want a similar experience
> with my computer that I get from books. (Though I'm still waiting for
> much better contrast from my computer displays—e-ink definitely helps a
> lot for the very static use cases).

You'd definitely wouldn't like my cyan for unseen messages then. :-)
That's why I'd like to get that in globally configurable, see above on
notmuch-pager

> >   * New message processing. Currently I check for spam and I mute
> >     selected threads. I can see this can be made quite configurable.
> > 	Maybe create procmail equivalent for notmuch? :-)
> 
> I think lots of us have various hand-written scripts that call out to
> "notmuch tag" a bunch. It's definitely a common idiom to have "notmuch
> new" add a new tag, have the new-mail-processing script operate on
> tag:new, and then have that script remove tag:new from the things it
> processed.

I actually don't do much tagging, as saved searches seem more flexible
to me. The current implementation does three things (apart from having
nice progress meter):
  * applies 'sent' tag to emails from addresses in your config file.
    This can be accomplished by saved searches too, bot it was more
	convenient to me to implement it here.
  * runs spam check on the mail
  * makes 'mute' tag span whole threads

The last thing is not that easy to do in pure shell script.

I'm not sure if some domain-specific language would be that big
improvement over status quo, that it would be worth implementing.
It would be first necessary to asses which features are useful and what
people need. Apart from what it currently does I do think that it would
be useful if it could move messages around .

> An alternative approach has been proposed to make "notmuch new" able to
> act on specified messages, (and accept an explicit list of tags to
> add). That would make it much easier to actually use existing tools like
> procmail directly with notmuch. Some people are currently using the
> notmuch-deliver.sh script in use cases like this. (And that script is
> another existing candidate for contrib.)

I am actually considering using something like that together with
dovecot on my server, I want to get some experience with using notmuch
first.

Thanks for your elaborate response. :-)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20110625/6454c6da/attachment-0001.pgp>