Proposal: Features instead of versions

[Austin Clements]

I believe our current approach to database schema changes is
inhibiting the evolution of Notmuch's features and would like to
propose what I think is lighter-weight solution.

Currently, our database schema is versioned and each database schema
change must occur "atomically" in Notmuch's development history:
before some commit, Notmuch uses version N, after that commit, it uses
version N+1.  Hence, each new schema version can introduce only one
change, the task of developing a schema change falls on a single
person, and it must all happen and be perfect in a single commit
series.  This makes introducing a new schema version hard.  We've seen
only two schema changes in the history of Notmuch.

I'd like to propose that we switch to a "feature set", recorded in a
database field.  The recent introduction of boolean folder terms would
be a "feature" (and the lack of that feature would imply probabilistic
folder terms).  Likewise, the file terms added in version 1 would be a
"feature".

The upgrade process would be structured around the delta between the
database's feature set and the desired feature set.  For many things,
it would be easy to support databases both with and without a feature,
which would enable "unstable" features that can be developed and
tested over time, and different features could be under development in
parallel.  We can also mark features as required or optional for
opening the database in read mode and replace our current unknown
version warning (which the user can't act on in any useful way) with
either no warning or a straight failure.

I've done some of this for my ghost messages support (restructuring
upgrade, unstable features, and supporting both the current schema and
the new schema), and it was simple and worked nicely.  This is also very
similar to how ext4 works [1].

Thoughts?


[1] https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#The_Super_Block
fields 0x5C through 0x64.