DatabaseModifiedErrors causing troubles

[Gaute Hope]

Gaute Hope <eg at gaute.vetsj.com> wrote on Mon, 11 Aug 2014 14:17:54 +0200:
> Hi,
>
> I've been working on an application that keeps a read-only handle on
> the notmuch database open for a long time. In some cases when a new
> message is added along with some renames of other messages using
> 'notmuch new' while the application is running I get an Xapian
> exception: DatabaseModifiedError:
>
>   A Xapian exception occurred performing query: The revision being
> read has been discarded - you
>   should call Xapian::Database::reopen() and retry the operation.
>
> Which seems to be printed from: notmuch_query_search_threads ->
> notmuch_query_search_messages:294.
>
> I have not been able to make a smaller test case at the moment (this
> happens with offlineimap updating an maildir and notmuch new run
> afterwards + some tagging).
>
> I can work around this by checking for a NULL pointer returned from
> notmuch_query_search_threads () and re-open the database
> (notmuch_database_close () -> notmuch_database_open ()). But I have no
> way of knowing programatically if this really is the error that has
> happened. There should be some way of propagating the error
> information or (even better for my case; for notmuch to reopen the
> database), one option is the Gmime way of passing an pointer to an
> error structure that is filled up by the notmuch interface function.

Hi again,

I have not yet been able to create a test for this with a fresh db (I
must be missing out on some of the parts triggering this).

I need to keep a query open for a while (because running it takes a
while), and this keeps coming back and biting me. I currently do a test
every time I need to access the query further (load more threads..). The
test is as follows, which is the easiest and quickest test I've found to
work reliably:


    /* tries to provoke an Xapian::DatabaseModifiedError
     *
     * returns true if db is invalid. queries will be invalid
     * afterwards, but might still respond true to notmuch_valid
     * calls.
     *
     */

    time_t start = clock ();
    bool invalid = false;

    /* testing */
    notmuch_query_t * q = notmuch_query_create (nm_db, "thread:fail");

    /* this will produce an error, but no way to ensure failure */
    notmuch_query_count_threads (q);

    /* this returns NULL when error */
    notmuch_threads_t * threads = notmuch_query_search_threads (q);

    if (threads == NULL) invalid = true;

    notmuch_query_destroy (q);

    float diff = (clock () - start) * 1000.0 / CLOCKS_PER_SEC;
    cout << "db: test query time: " << diff << " ms." << endl;

    if (invalid) {
      cout << "db: no longer valid, reopen required." << endl;
    } else {
      cout << "db: valid." << endl;
    }

As mentioned before, I am basically guessing that the reason I get
threads == NULL is the DatabaseModifiedError. There is no way to know
for sure.

For portability I would suggest starting to move towards the GError
scheme provided by glib (also used by gmime). This is a somewhat major
effort though since ensuring that error propagation is done right [0] is
somewhat tricky. It provides more or less the same functionality as
exceptions do in C++.

There is also the problem of having to change the API - one way to avoid
that is to create wrappers that behave like the old API, but don't
handle errors that good. This requires the addition of addiontal _error
variations of the current set of functions. It will be a mess.

Cheers, Gaute

[0] https://developer.gnome.org/glib/stable/glib-Error-Reporting.html