[notmuch] [PATCH] Calls to notmuch get queued and executed asynchronously.

[Aaron Ecay]

On Fri, 30 Dec 2011 10:52:17 +0000, David Edmondson <dme at dme.org> wrote:
> > I discovered this patch a while ago and it almost applies cleanly (the
> > conflicts are easy to fix though) and it has made the emacs UI for
> > notmuch _much_ more responsive and enjoyable for me.

This is definitely true for me as well.  So much so that I reinvented
this patch from scratch a couple of weeks ago.  (As Picasso said, good
artists copy, great artists steal.)

> > As discussed on IRC, issues such as where to handle retries in case
> > the database is locked should probably be handled in the notmuch
> > binary instead of each UI.

Hmm.  When my implementation detects that it cannot get the DB lock
after trying several times (up to ~2 mins of waiting), it stops
trying but continues to queue up operations.  The user can then
manually restart the queue.  This isn’t a principled solution, but
it avoids losing tagging operations from emacs while a long-running
process has the DB locked.  (My original motivation for writing the
patch was actually that my new-mail script sometimes holds the DB
lock when I am trying to read mail, leading to emacs errors and
dropped tagging operations.  The speedup was a welcome side effect.)

I haven’t had a chance to carefully look at the patch you found, so IDK
what its behavior would be in this case.

(The other thing that I dislike about the patch you found is that it
uses a call to the “sleep” command to wait, rather than using built-in
emacs functionality.)

> It would be good to have an updated version of this patch if it is still
> considered useful, though I don't suffer particularly from lag in the
> emacs UI.

The emacs UI has small delays, which dropped away when I applied my
version of this patch.  After being conditioned to expect them, I was
surprised to find how much faster everything seemed.  My hardware is old
but not ancient (5.5 year old Macbook; I suspect the bottleneck for
notmuch is the 5400rpm HDD).  Depending on your setup, you too might be
pleasantly surprised.

The test suite is borked on OS X and I don’t have access to a linux
machine while traveling.  I was planning on sending my patch in early
Jan. when I had a chance to verify it under the test suite (probably
requiring some changes to the emacs test library to make it async-safe).
I also haven’t tested the patch on Emacs versions older than 24 – but I
don’t think there are any impediments to compatibility with v.23 (not
sure about earlier versions).

Since there is interest, I’ll go ahead and send it now with all the
usual caveats about code under development.  I have been using the patch
for a couple of weeks without problems, though.  Daniel, if you want to
un-conflict (and squash) the patches from James that might be useful, at
least to compare the two approaches.

Aaron

-----cut-here-----

>From f0a0fe04254d9b5e17c873b293c6a5a270cb909a Mon Sep 17 00:00:00 2001
From: Aaron Ecay <aaronecay at gmail.com>
Date: Mon, 19 Dec 2011 12:14:31 -0500
Subject: [PATCH] [emacs] add async tagging support

still a WIP
---
 emacs/notmuch.el |   55 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 files changed, 54 insertions(+), 1 deletions(-)

diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index fde2377..ca077c2 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -446,6 +446,59 @@ Complete list of currently available key bindings:
   (let ((message-id (notmuch-search-find-thread-id)))
     (notmuch-mua-new-reply message-id prompt-for-sender)))
 
+(defvar notmuch-process-queue nil
+  "Queue of pending notmuch tag operations.
+
+Each entry in the queue is a list of strings, which are arguments
+to be passed to the notmuch command.")
+
+(defvar notmuch-process-current-command nil
+  "The currently executing notmuch command arguments")
+
+(defvar notmuch-process-wait-time 1.0
+  "How long to wait for the db lock, in seconds.")
+
+(defvar notmuch-process-error nil)
+
+(defun notmuch-process-restart-queue ()
+  (interactive)
+  (setq notmuch-process-error nil)
+  (notmuch-process-kick-queue))
+
+(defun notmuch-process-kick-queue ()
+  (when notmuch-process-error
+    ;; TODO: better error msg, tell how to manually kick
+    (error "Notmuch couldn't get the DB lock after trying for more than a minute."))
+  (when (and notmuch-process-queue
+	     (memq (process-status "notmuch") '(exit nil)))
+    (let ((args (car notmuch-process-queue))
+	  proc)
+      (setq notmuch-process-queue (cdr notmuch-process-queue)
+	    notmuch-process-current-command args
+	    proc (apply #'start-process "notmuch"
+			(get-buffer-create "*Notmuch output*")
+			notmuch-command args))
+      (set-process-sentinel proc #'notmuch-process-sentinel))))
+
+(defun notmuch-process-sentinel (proc status)
+  (if (equal status "finished\n")
+      (progn
+	(setq notmuch-process-wait-time 1.0
+	      notmuch-process-current-command nil)
+	(notmuch-process-kick-queue))
+    (setq notmuch-process-wait-time (* 2 notmuch-process-wait-time)
+	  notmuch-process-queue (cons notmuch-process-current-command
+				      notmuch-process-queue))
+    (if (<= notmuch-process-wait-time 64)
+	(progn
+	  (run-at-time notmuch-process-wait-time nil
+		       #'notmuch-process-kick-queue))
+      (setq notmuch-process-error t))))
+
+(defun notmuch-process-queue-command (&rest args)
+  (setq notmuch-process-queue (append notmuch-process-queue (list args)))
+  (notmuch-process-kick-queue))
+
 (defun notmuch-call-notmuch-process (&rest args)
   "Synchronously invoke \"notmuch\" with the given list of arguments.
 
@@ -474,7 +527,7 @@ messages instead of running (notmuch-call-notmuch-process \"tag\" ..)
 directly, so that hooks specified in notmuch-before-tag-hook and
 notmuch-after-tag-hook will be run."
   (run-hooks 'notmuch-before-tag-hook)
-  (apply 'notmuch-call-notmuch-process
+  (apply 'notmuch-process-queue-command
 	 (append (list "tag") tags (list "--" query)))
   (run-hooks 'notmuch-after-tag-hook))
 
-- 
1.7.8.2

-----cut-here-----

-- 
Aaron Ecay