RFC: notmuch powered (personal) (end-to-end) e-mail system

[Ben Gamari]

On Sun, 20 Mar 2011 16:07:50 +0200, Ciprian Dorin Craciun <ciprian.craciun at gmail.com> wrote:
>     Hello all! (Sorry for the long email.)
> 
> [snip]
> 
>     * the most feature full UI is the Emacs one -- thus limited remote
> access (I mean from an arbitrary computer with only a web-browser);
> (and I'm not a very big fan of Emacs;)
> 
There have been a few attempts to put together an HTML front-end to
notmuch[1]. None have made it very far though. It would be nice to see
this space filled.

>     * most are still dependent on external IMAP systems -- this is not
> a problem with notmuch itself, but for the integrating clients;
> 
Not entirely sure what you mean by this. You could easily use
e.g. notmuch-deliver as the local delivery agent with a SMTP server and
you'd have no need for IMAP.

>     * SPAM -- as above -- is not integrated;
> 
Nor should it be. Mail indexing, viewing, composing, and
filtering are all orthogonal parts of a mail system. It takes all of
ten lines to invoke a spam filter in your filter script.

>     * filtering (tag applying) is not automatic (as in integrated in
> notmuch itself or the client), but triggered through external scripts;
> 
Again, there is no reason why this should be incorporated into your
mail indexer.

>     As such I'm thinking on implementing a custom end-to-end email
> system and I would like to hear your feedback before embarking on such
> a task.
> 
Notmuch works so well for its audience because it adheres to the UNIX
philosophy of "do one thing and do it well." The goal of an "integrated
end-to-end" mail system might sound nice, but IMHO it's a recipe for a
kludgey, unmaintainable nightmare which is mediocre at performing its
task, on a good day. Perhaps I'm misunderstanding your proposal but it
seems to me like you are taking an easy, already solved problem and
turning it into a difficult one.

>     I'm targeting the following features:
>     * (inbound) SMTP integration, thus once an email is received it is
> automatically pushed through the system; (I'm primarily targeting
> those users that afford to run their own SMTP server; but the solution
> could still be adapted for those that only want the other features;)
> 
Is there something wrong with Postfix with notmuch-deliver as a LDA?

>     * automatic spam filtering, and tag applying;
>
A traditional sorting script with bogofilter/spamassassin?

>     * automatic email triggers based on tags (such as user
> notifications, forwarding, etc.)
>
Again, a sorting script?

>     * remote RPC-like access to the whole system;
> 
What's wrong with SSH?

>     * remote Web user interface;
> 
Nothing fills this need currently. Feel free to write up something but
please don't couple it to some all-inclusive beheamoth of a project.

Personally, I would think more carefully about this project before
proceding. It sounds like you intend on reinventing various portions of
the wheel several times. Nothing you have listed is difficult to do with
a few scripts, notmuch, and an SMTP server. 

>     About the overall architecture I'm thinking on adopting the following:
>     * in general the whole system is decomposed in independent
> components (long-lived OS daemons) that each one does a particular job
> (see below);
>     * all the components communicate between each-other through a
> message queue system (for example ZeroMQ or RabbitMQ);
>     * all the communication is JSON based;
> 
>     The components would be:
>     * SMTP inbound gateway -- for example I could take qmail or
> Postfix and replace the delivery agent with a custom process that
> pushes the email into the system; (any other solution suggestions?);
>     * email store -- as the name suggests it is a simple
> key-value-like store that should persist raw email-messages; it should
> be as robust as possible, and its contents should be the only thing
> needed to reconstruct all the other derived data; (I could use here a
> simple process that maintains a maildir, I could go also with a
> BerkeleyDB wrapper, or even something more sophisticated;)
>     * spam filter -- which either classifies the email or trains the
> spam filter; (for example I would use bogofilter;)
>     * email index -- this is where notmuch would come into play; it
> would be fed with emails, which it would automatically apply tags and
> issue trigger notifications based on tags; it also maintains a set of
> filters and tags to automatically apply;
>     * (maybe) a coordinator that should delegate and monitor requests
> to the above components; but if I'm using RabbitMQ and carefully
> designing the above components, they could drive each other;
>     * restful web service that would intermediate access to all the
> above components;
> 
>     For now I have the following uncertainties:
>     * how should I handle multiple users? I think each user should
> have it's own store / notmuch / bogofilter instance (at least in terms
> of storage if not even in terms of separate daemon);
>     * should I keep the emails is a file-system, or a key-value store?
> (the file-system is more bug-free, but I'm confident that a BerkeleyDB
> instance would be more efficient);
>     * should I use libnotmuch or for starters just make a notmuch tool
>     wrapper;
>
>     * and the most pressing one, transactions: I would like that at no
> point does a message get half processed or lost; as such I need
> notmuch to behave transactionally -- indexing the message and tagging
> it should be atomic and durable; (is there a way with libnotmuch to
> control the underlaying BerkeleyDB database?)
>
It tries to be as reliable as possible. I believe before a patch a few
months ago it was possible to kill notmuch during "notmuch new" and lose
messages (they would never be indexed). It's possible there are more
bugs such as this. If they do exist they should be dealt with.

Also, notmuch uses xapian, not bdb.

Anyways, that is my two cents. Good luck.

Cheers,

- Ben


[1] https://github.com/dme/noneatall