Update on python-cffi bindings

[Daniel Kahn Gillmor]

On Thu 2017-12-21 12:30:39 +0100, Floris Bruynooghe wrote:
> The API changes a lot and there is no easy migration.  And history has
> shown that's a terrible way to get something new adopted.  Last time I
> suggested a possible multi-tiered approach (maybe not as explicit):
>
> 1 I think it's possible to move the memory management technique to the
>   existing API without API breakage.  It would still allow users to call
>   functions in the wrong order etc, but that's not any regression.
>
> 2 It's probably possible to either switch the existing API to use CFFI
>   or create a drop-in replacement for it based on CFFI.  The benefit
>   here is two-fold: users get better PyPy performance and I believe it
>   becomes easier to maintain the bindings, e.g. all you need to do to
>   call notmuch_database_get_path is
>   capi.ffi.string(capi.lib.notmuch_database_get_path(dbptr)) (see
>   https://github.com/flub/notmuch/blob/cffi/bindings/python-cffi/notdb/_database.py#L263)
>   for an actual example).  But this really depends on what everyone else
>   here that maintains the Python bindings here thinks.  I'd encourage to
>   have a look at the CFFI implementation to get an idea of this.

these two steps on their own seem like they give us:

  * better and safer memory management

  * better performance on PyPy

  * arguably, easier maintenance of the bindings.

These seem like unambiguous wins, and there is no downside -- people
using the notmuch module can just upgrade to the new version and it's
done.  I'd love to see these changes happen.

The only thing it doesn't do is provide more idiomatic bindings for new
users, which you describe as:

> 3 As last step I still think providing the more idiomatic bindings is
>   useful, especially for new users.  It does take the burden of
>   correctly calling the C functions somewhat more.  This could then
>   either treat a notmuch_cffi as a lower level API which more directly
>   maps the C API or it could call C directly as it does now.  I'm not
>   currently sure on which is more feasible or better here.
>
>   An additional thing that could be done here to ease migration is to
>   allow creating the new objects from the old ones allowing a codebase
>   to gradually adopt the newer API where appropriate.  E.g.:
>
>      db = notmuch.Database(...)
>      msg = db.find_message(...)
>      new_db = notdb.Database.from_notmuch(db)
>      new_msg = notdb.Message.from_notmuch(msg)
>      print('Tags not on msg: {}'.format(new_db.tags - new_msg.tags))
>
>   This would rely on the existing API to migrate to CFFI, otherwise it
>   could still be possible but would be very hairy.

I'm not sure i understand this last sentence.  Are you saying that step
3 depends on step 2 happening?

I'm not sure about the name "notdb" but i don't mind the idea of there
being some other "more-pythonic" notmuch bindings.  New users would
likely prefer it, and that'd be fine.

> - For exposing completely new APIs, sure building the
>   notdb.ImmutableTagSet and MutableTagSet was not straight forward,
>   likewise for the PropertiesMap.  Many other things are much easier
>   though.  One possible nice way to alleviate this with the idea of the
>   existing notmuch API being the lower-level layer nearly mimicking the
>   C-API directly.  That way adding new APIs there is more or less
>   straight forward and there is less time pressure to add them to the
>   higher level API.  Especially if mixing the APIs is supported.

I think this is in line with the approach described above.  I like it.

> I must admit I didn't look too much at the existing tests untill just
> now.  If I understand correctly the existing tests are in
> T390-python.sh.  In this case I'd say the tests I added are a lot more
> thorough.  The reason I added tox.ini is to easily test against multiple
> python versions, i.e. CPython 3.5, 3.6 & PyPy 3.5.  If I had to guess at
> the best way to integrate I'd say it's probably best to create a
> TXXX-python-pytest-pyXY.sh for each supported python version and lose
> tox.ini.  I'd suggest for those tests to be a simplified version of what
> tox does: create a virtualenv and run pytest inside of it.

I personally would rather not deal with virtualenvs during development
if we don't need them.

Take a look at test_python() in test/test-lib.sh to see how the python
tests are currently handled -- it's pretty simple, relies on the
configured version of notmuch for testing, and it would be great to keep
the simplicity.

I think the approach you've outlined above sounds really good, and i
would be happy to see it happen for notmuch.  It strikes me as much more
useful and less scary than a hard cutover or overhaul that requires
adoption of a new API from the ground up.

Thanks for continuing to push on this, Floris!

         --dkg
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20171221/b063a00b/attachment-0001.sig>