[notmuch] Some thoughts about notmuch sync with other agents

[Paul R]

Hi,

before going into details, I'd like to tell you how much I like the
notmuch workflow, thank you for producing this nice piece of software !

Like most potential users, I can not really fully jump into notmuch
because of the current synchronisation issues with others MUA. One may
or may not like IMAP for good reasons, the fact is that it is here and
has allowed users to read mails from various places and terminals,
keeping important information synced. So I think that notmuch will have
to live with that, and provide what is required to integrate smoothly
into this environment. Redefining a new mail retrieval protocol and
a mail storing format are both really exciting projects, but they are
projects on their own that could only distract notmuch from its most
beautiful goal : giving *today* users the power to process mail in an
efficient way.

As you see, I advocate a NotMuch <-> IMAP synchronisation ASAP :)

At the moment, notmuch input are mail-as-file + user-defined tags.
OfflineIMAP allows to do the IMAP <-> mail-as-file transition, in both
directions, mail-as-file being namely MailDir. So we can simply aim at
a NotMuch <-> MailDir synchronisation, offlineimap will take care of
IMAP itself.

Of course, my proposal does not require to implement any MailDir
specific logic inside NotMuch, but rather describes how should notmuch
evolve to allow easy third-party-tool jobs.


Preliminary thoughts :
----------------------

First of all, processing mail with MUA involves some simple logic that
is shared by most MUA. This is about receiving *new* mails, *reading*
mail, *replying* to mail and so on... IMHO, this really belongs to the
mail processing logic and not to the user logic. Hence my first
request :

  1: Don't use user tags space to store MUA flags.

     That means no more "seen", "unread", "replied" as tags. These are
     MUA processing *flags*, that must belong to an established set.
     Tags, on the other hand, are user-land information. So no more
     [seen, replied, grandma, important] tag sets :)

Once this is done, notmuch will know, in a robust a predictable way,
what happened to a mail. Simply put, NotMuch will store and expose MUA
flags (Passed, Replied, Seen, Trashed, Draft, and Flagged [1]). For each
<flag>, notmuch should associate a <flag>_synced flag. When changing
<flag> from notmuch, it should set the <flag>_synced bit to 0. These are
MUA mail flags.

Additionally, in a third « space », notmuch should store its « new »
bit, as well as a « missing » bit probably. Again, this is neither MUA
logic or user logic, so this should not interfer with user
classification facility provided by tags, nor with MUA flags. It,
really, is something else, let's name it "status". Once this is done,
the 'notmuch new' command should find new mails and set the 'new' bit
for them, and find the missing mails and set the 'missing' bit for them.
This will allow for robust external processing.

Finally, notmuch should provide a switch to output a list of filenames
to stdout and to process a list of filenames from stdin.


NotMuch <-> MailDir synchronisation :
-------------------------------------

Provided the behaviour described above, notmuch <-> MailDir
synchronisation could be done fully externally, by a 'notmuch-maildir'
adapter.

Here is some pseudo code, that could be wrapped into a single
'notmuch-sync' command. The | are unix stream pipes, and everything
should be on a single line.

# 1/ Sync from NotMuch to MailDir

    notmuch list flags:(seen and not seen_synced) 
      | notmuch-maildir --mark-mail seen
      | notmuch move --stdin
      | notmuch set flags:+seen_synced --stdin

The output of the first command would be a list of filenames for mails
'seen' since last sync. The second one, an external notmuch--maildir
helper, would propagate this logic to the MailDir store (easy, this is
simply a rename), and output the list of (old-name ! new-name). Then
notmuch would use this information, via a generic 'move' switch, to know
that mail has been moved, and would output the list of the new places.
Finaly, notmuch would set the seen_synced flag.

Same would apply for the Replied, Trashed, Flagged and Passed flags.

# 2/ Then lets do the MailDir <-> IMAP sync

     offlineimap

... done ! that was easy :)

# 3/ notmuch new

     notmuch new

At this point, notmuch should set the 'new' or the 'missing' status bit
to the mails. Let's forget how to deal with the missing bit, that should
be easy to do.

# 4/ Sync from MailDir to NotMuch

  notmuch list status:new 
   | notmuch-maildir --filter seen
   | notmuch set flags:+seen+seen_synced --stdin

First command outputs newly discovered mail. Second one reads stdin and
output only files that are already seen (again, this is as easy as
a filter based on a regular expression). Third one reads stdin and
applies the seen and seen_synced flags.

Same applies for the Replied, Trashed, Flagged and Passed flags.


Conclusion:
-----------

As we have seen, this would allow most IMAP <-> MailDir <-> NotMuch
synchronisation, without having to implement any kind of
MailDir-specific logic inside notmuch. In fact, this notmuch-maildir
helper would be a simple script, and we could imagine doing similar
script for other stores, without having to touch the core of notmuch.


That was a long mail indeed, thank you for reading ! I'm waiting for
your comments.



Footnotes: 
[1]  http://cr.yp.to/proto/maildir.html

-- 
  Paul