unifying and automating draft ("postpone/resume") behavior [was: Re: notmuch "lost" email during power failure]

[Antoine Beaupré]

On 2019-11-12 18:03:58, Daniel Kahn Gillmor wrote:
> Hi Antoine--
>
> [epic story and funny rnat trimmed down to the salient bug reports and
>  feature requests because i'm a boring person]

Hi!

[I don't remember what RNAT (reverse network address translation) have to
do with anything, but it's true it was a rather long RANT so maybe I
forgot that I went through some weird addressing along the way. ;)]

> On Tue 2019-11-12 17:19:05 -0500, Antoine Beaupré wrote:
>> I would argue that notmuch should at least allow me to recover from a
>> power failure like this, as a MUA. It should "know" that message-mode
>> stores those messages there, and, why not, also store its tempfiles
>> there.
>
> This seems like a very specific ask.  I think what we want more
> generally is a sensible unification of the notmuch-emacs drafting
> behaviors such that message-mode autosaves are discoverable and
> recoverable in the same way that deliberately-saved drafts are.

Yeah. That is pretty much what I meant to say, but said in a more
useful, constructive and clever way. :p

> If that means that notmuch learns about message-mode drafts, that's one
> solution.  But it could also mean that notmuch-emacs overrides
> message-mode's autosave drafting approach, and fixes things there too.

I think the main problem right now is that "control-x control-s" does
something special in notmuch, in that it adds a message-id to the saved
message even if the message isn't saved yet.

But then when *message-mode* auto-saves those files, the message-id is
not present (just checked). For example this very message has 
References (hidden), From, To, Cc, Subject, In-Reply-To and Fcc headers
on disk, along with the fancy `--text follows this line--` magic blob on
disk, in ~/Mail/drafts/#...

While if I save this message to disk, it looks like a proper email
message. It has a (properly encoded) From header, To, Cc, Subject,
In-Reply-To, Fcc (so far pretty close to message-mode), but then also a
Message-ID, Date, X-Notmuch-Emacs-Draft, MIME-Version, Content-type and
Content-transfer-encoding. And of course, the content *is*
quoted-printable-encoded, which might be a problem for message-mode.

In my opinion, that's the fundamental conflict between the two modes:
message-mode expects buffers (and auto-saved files) to have this weird
exotic format to operate, but it's pretty much exactly the same on-disk
format, while notmuch expects a standard email message, which *differs*
from the actual buffer content. It will be fundamentally difficult to
reconcile those two.

> or, we somehow fix message-mode?

That could be a huge challenge. We'd need to teach it to parse real
RFC852 messages properly, without the magic separator, and to include
proper message IDs and other headers.

Alternatively, maybe message-mode could do the same translation magic
that notmuch does when it saves and loads those files, on write. By
moving the notmuch logic upwards in the stack, everyone could benefit.

But it's kind of a big change.

>> And indeed, if I hit [control-x control-s] on this message, it *does*
>> get saved as a "draft" in that it gets written in:
>>
>> ~/Maildir/drafts/cur/1573596264.M156307P32312.curie:2,DF
>>
>> That's a great improvement already. I don't remember exactly when or how
>> this happened, but that's great and I thank whoever did that for us
>> here.
>
> That was Mark Walters (in Cc), see the series from 2016 starting at:
>
>      id:1479046130-2716-1-git-send-email-markwalters1009 at gmail.com
>
> Thank you, Mark!

Yeeeaay! thank you Mark!

>> To make a long story short, I think the following should happen:
>>
>>  1. message-mode should automatically cleanup after itself a little
>>     better (not notmuch's job? yay double-negative, that means it's much
>>     job!)
>>
>>  2. encrypted emails should *never* be written in cleartext on the
>>     filesystem (not notmuch job, which also means it's much job!)
>>
>>  3. notmuch's draft subsystem should know about Emacs' autosave files
>>     and somehow show them in the UI
>
> Much of the above sounds like message-mode cleanup work to me.  It might
> be worth asking the message-mode folks to weigh on in this strategy.

For what it's worth, I reviewed the original patchset that was merged
from Mark, and notmuch *does* the right thing with encrypted messages
(issue 2 above): it asks users (configurable) what to do. This, again,
could be moved up into message-mode as well...

> Alternately, it might help to characterize the differences between
> message-mode autosaving and notmuch explicit drafting.
>
>  - message-mode autosaves happen without your knowledge.
>    notmuch draft saving only happens when you explicitly C-x C-s
>  
>  - message-mode autosaves are automatically cleaned up from the
>    filesystem after you send.  notmuch saved drafts persist in your
>    mailstore until you remove them (though they are tagged with
>    "delete")

Those two are equivalent for me in notmuch, as "deleted" emails get
periodically cleaned (yes, I'm an heretic).

>  - message-mode autosaves appear to save in cleartext even when the
>    message will be encrypted(‽).  if you try to C-x C-s on a notmuch
>    draft, notmuch-emacs gives you a big ol' warning and won't let you
>    save it in plaintext unless you agree (see
>    notmuch-draft-save-plaintext).

That is correct.

>    (Maybe there's a third behavior which
>    would be better than all of these, which is to explicitly save the
>    draft of encrypted messages encrypted-to-self-only, but that's not
>    implemented, afaict)
>
> Does this all sound right?  There's a lot to unpack here.

It does! thanks for clarifying my novel. ;)

Fundamentally, I think message-mode should treat emails the same way
epg.el treats encrypted files: the data in the buffer isn't the same as
the on-disk data, and there should be translation between the two. At
the very least, the on-disk data could be a properly formed email
message. If it's marked as needing encryption, then it should be
encrypted.

Arguably, I haven't looked in details at how epg.el works: maybe it
suffers from the same flaw. But from what I can tell, it simply bypasses
the autosave functionality. If you have edit an ecrypted file called
`foo.gpg` (note the non-standard extension), the `#foo.gpg` file is just
a symlink, a lockfile instead of the normal autosave.

This is probably because it would be prohibitively expensive to
constantly call gpg every time a buffer changes to do the autosave. I
think that is actually a fairly reasonable tradeoff.

A.

-- 
Tu connaîtras la vérité de ton chemin à ce qui te rend heureux.
                        - Aristote