notmuch "lost" email during power failure

[Antoine Beaupré]

Hi!

Today, in Canadaland(show.com), it's winter! There's snow everywhere,
it's beautiful but cold, which means we power up those heating system
and blow up breakers. Which means computers go down because those little
stupid things need power to keep going, especially when they don't have
UPS systems built in. Those backwards machines were previously known as
"desktops" but are now known as "what, you still have one of those? yes
I do leave me alone."

Anyways. Today my desktop lost power while I was writing a too long
email to dkg about internet standards. Then power went out halfway
through the email. When I rebooted, notmuch couldn't find the email and
I assumed it was dead and gone, wrongly, as it turns out.

I draw many conclusions from this:

 1. i shouldn't write long emails to dkg
 2. i shouldn't write long emails to replies of dkg to my long emails
 3. notmuch should never lose email drafts of long emails i write to dkg
 4. i should learn not to worry and love the bomb

Obviously, I come to you with conclusion #3.

Being the resourceful and friendly human being that he is, dkg pointed
out the `~/.emacs.d/auto-save-list/` directory. It contains a list of
files that have a list of file names in them (!?) pointing to various
auto-save files. I couldn't figure out where that thing comes from, but
it did lead me to a saved copy of the message.

Naturally, I only found dkg's message *after* I spent another hour
rewriting the damn message after the power failure, but it was accurate:
one of those files had a filename that pointed at:

~/Mail/drafts/#*message*-20191112-165309#

which contained the auto-save of the message! Hurray!

As it turns out, that location (~/Mail/drafts) is the default
`message-auto-save-directory`. That's great, but the problem is there's
no visibility in that directory from notmuch's perspective.

Even worse, message-mode leaves stray messages in there. While cleaning
it up, I even found an old message from 2018 that had a *password*
(*gasp*!) in it that I carelessly sent to other people. Interestingly,
that message was sent encrypted (with OpenPGP) but was still stored as a
tempfile in cleartext there.

Obviously, none of that so far is directly notmuch's fault: those are
all problems with message-mode.

*But* I would argue that notmuch should at least allow me to recover
from a power failure like this, as a MUA. It should "know" that
message-mode stores those messages there, and, why not, also store its
tempfiles there. And indeed, if I hit [control-x control-s] on this
message, it *does* get saved as a "draft" in that it gets written in:

~/Maildir/drafts/cur/1573596264.M156307P32312.curie:2,DF

That's a great improvement already. I don't remember exactly when or how
this happened, but that's great and I thank whoever did that for us
here.

I do remember hacking something like this together before that happened
however. I made message-mode write temporary messages directly in that
folder (by setting the auto-save-directory to ~/Maildir/draft/new),
which notmuch would somewhat pickup at the next automated `notmuch new`
run. But that made notmuch unhappy for various reasons:

 1. the autosave files don't have message IDs which would confuse notmuch

 2. the files wouldn't automatically be removed from notmuch's database
    even when (or if!) message-mode would actually clean them up

I had to write hacks like this to cleanup those files:

https://gitlab.com/anarcat/scripts/blob/master/notmuch-clear

It was a mess, so I reverted message-auto-save-directory back to its
default and totally forgot about it.

Which led me to writing a second long email to dkg.

Which made me kind of obstinately sad.

Which is very weird.

But that's probably standard IETF behavior by this point. Not sure which
draft, RFC or BCP that is, but that's probably irrelevant. ;)

To make a long story short, I think the following should happen:

 1. message-mode should automatically cleanup after itself a little
    better (not notmuch's job? yay double-negative, that means it's much
    job!)

 2. encrypted emails should *never* be written in cleartext on the
    filesystem (not notmuch job, which also means it's much job!)

 3. notmuch's draft subsystem should know about Emacs' autosave files
    and somehow show them in the UI

Sorry for the long email. My attempts at comedy in this one are probably
far from the previous one, but the topic was less happy and I was less
motivated. I promise to try again next time though, and thank you for
flying nutty anarcat.

Cheers,

A.
-- 
The problem is not a lack of highly educated workers, the problem is a
lack of highly educated workers willing to work for the minimum wage or
lower in the U.S. Costs are driving outsourcing, not the quality of
American schools.       - Scott Kirwin, IT Professionals Association