notmuch's idea of concurrency / failing an invocation

[Thomas Schwinge]

Hallo!

On Fri, 28 Jan 2011 15:10:01 +1000, Carl Worth <cworth at cworth.org> wrote:
> On Thu, 27 Jan 2011 17:20:21 -0500, Austin Clements <amdragon at mit.edu> wrote:
> > I'm looking into breaking notmuch new up into small transactions.  It
> > wouldn't be much a leap from there to simply close and reopen the database
> > between transactions if another task wants to use it, which would release
> > the lock and let the queued notmuch task have the database for a bit.
> 
> That sounds like something very useful to pursue. Please continue!

Ack!  And actually -- I just wondered about that: what happens if
``notmuch new'' has executed notmuch_database_add_message for a new
message M, but then is killed before adding any tags and finishing up
(and supposing that the DB isn't in an invalid state now).  This process
of adding M to the DB and applying any tags isn't one single transaction
currently, right?  (And this is exactly what you're working on
chainging?)  Am I right that what currently happens is that upon the next
``notmuch new'' run, notmuch will not reconsider M (given that it already
is present in the DB), but continue with the next messages -- thus
leaving M without any tags?  This isn't a very likely scenario, but still
a possible one.

> > It seems silly to have a daemon when all of notmuch's state is already on disk
> > and queue on a lock is as good as a queue in a daemon, but without the
> > accompanying architectural shenanigans.

Ack, too.  A daemon seems one abstraction layer too much.  (But I'm not
actively opposed either, if someone has a valid use for such a scheme.)

> It would definitely be nice to avoid the complexity inherent in having a
> daemon, but how do you imagine "queue on a lock" to work? We don't have
> anything like that in place now.

I suppose what he means is trying to get the lock, and if that fails wait
a bit / wait until it is available again.

Actually, as a next step, wouldn't it also be possible to add some
heuristic to avoid ``notmuch new'' (being a low-priority task) blocking
some interactive user (UI; high-priority task)?  But we can pursue such
schemes as soon as the basic infrastructure is in place.

> Another advantage that can happen with queueing (wherever it occurs) is
> to allow a client to be very responsive without waiting for an operation
> to complete. Though that can of course be band if the operation isn't
> reliably committed.

(Obviously this can only work as long as we don't immediatelly need the
operation's result; think ``notmuch show''.)

So, if the DB has the functionality to internally queue and immediatelly
acknowledge transactions, and only later (reliably) commit them, wouldn't
that be fine indeed?  For example, ``notmuch tag'' then wouldn't have to
wait for the DB to be writable.  (And note that I have no idea whether
Xapian supports such things.)  But on the other hand we would like to
immediatelly display the requested change in the UI, right?

What notmuch-show.el:notmuch-show-remove-tag currently does is *not*
re-asking the DB for a message's current tags after having removed a
specific one, but instead it interprets the tag removal command itself --
which is easy enough in this case, and rather unlikely to ever yield
different results, at least unless there's another process operating on
the DB concurrently.

Otherwise, the other way round, the client could maintain a list of to-do
items, to which actions are added if the DB is currently busy, and this
list is periodically worked on in order to get it empty.  For example,
tag changes that are in this list, but not yet committed in the DB could
be displayed in another color in the UI.  But doing so would shift the
responsibility to the UI, which should be in the DB, in my humble
opinion.  (Actually, this issue feels similar to the one who should be
doing the re-trying in case the DB is busy: the UI, or the notmuch
process itself, which we're discussing in another thread.)


As you can guess I'm not very much into DBs, and neither too much into
concurrent systems, so if my ideas don't make sense, please feel free to
refer me to literature.


Grüße,
 Thomas
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20110128/a3d67691/attachment.pgp>