automatic database upgrades (was: Re: [PATCH 0/7] cli: notmuch new improvements)

[Mark Walters]

Thanks for posting this. You are quite right about it being orthogonal
to this series so a clear +1 from me for the series.

What about a config option? Something like
database_auto_upgrade=true/false? I wouldn't have a strong preference
which was the default (though I would choose "false" in my own
config). I guess we would need a command line --upgrade to allow people
with database_auto_upgrade=false to do force/allow the upgrade.

Best wishes

Mark








On Sat, 25 Jan 2014, Jani Nikula <jani at nikula.org> wrote:
> On Sat, 25 Jan 2014, Mark Walters <markwalters1009 at gmail.com> wrote:
>> This series LGTM.
>
> Hi Mark, thanks for the review!
>
>> I do now recall there was some discussion on irc about the automatic
>> database upgrade: it would be good to have that documented but the
>> consensus was to do it, so +1 from me.
>
> Here's some summary, as promised. Please bear in mind that the
> discussion is pretty much irrelevant to this particular patch
> series. (We might discuss whether a warning about upgrade should be
> printed to stderr also with --quiet, but IMHO that can be a follow-up
> patch.)
>
> A database upgrade is required when the user upgrades to a new version
> of notmuch that has a modified database schema. See
> id:cover.1389304779.git.jani at nikula.org for an example of a proposed
> database change.
>
> A database upgrade is a rare event. Most of the time, it's okay to go
> back and forth with notmuch versions on the same database, but a
> database upgrade is an irreversible process after which the user must
> use the new version of notmuch. To go back requires a full rebuild of
> the database.
>
> We don't have recent experience with the database upgrades. The last
> time it happened was before notmuch 0.1 (yes, 0.1) was released, when
> the whole upgrade mechanism was introduced:
>
> commit 909f52bd8c4bdfa11cd3e75e3d0959e0293689bd
> Author: Carl Worth <cworth at cworth.org>
> Date:   Thu Jan 7 18:26:31 2010 -0800
>
>     lib: Implement versioning in the database and provide upgrade function.
>
> Some of the points in favor of requiring manual intervention (such as
> running 'notmuch new --upgrade' or a new command 'notmuch upgrade')
> before upgrading the database:
>
> * The database upgrade is an irreversible process. The user should have
>   a chance to decide whether to go ahead with that. You can't go back to
>   the old notmuch and database version without rebuilding the database.
>
> * The user should be given the chance to make backups first in case
>   something goes wrong.
>
> * The database upgrade might take a long time for large databases. The
>   user should be able to choose when to go ahead with that.
>
> Some of the points in favor of upgrading automatically:
>
> * cworth: "One potential concern is that [requiring manual intervention]
>   effectively breaks notmuch until the user intervenese and runs this
>   new command. So that can complicate things for any interface that sits
>   on top of notmuch."
>
> * cworth: "In general, I'm often frustrated with programs that say "I
>   cannot continue until you run command <foo>.". If command <foo> needs
>   to be run, and the software knows it, why doesn't it just run it
>   itself? [...] So a message like "Run 'notmuch upgrade'" seems it could
>   corrode the user's trust in notmuch to maintain its own state."
>
> * There are people who run notmuch new non-interactively. There's no
>   easy answer to handling that if manual intervention is required.
>
> * It should always be okay to kill the upgrade, and continue at a later
>   time, in case it takes too long.
>
> Reading the logs again, I'm not so confident about us reaching a
> concensus. Maybe it was just me changing my mind during the course of
> the discussion... so we can try to reach concensus here.
>
>
> BR,
> Jani.