pytest integration for the notmuch test suite

[David Bremner]

Floris Bruynooghe <flub at devork.be> writes:

> On Sun, Mar 25 2018, David Bremner wrote:
>
>> Here's one approach. A given pytest "file" can be embedded in a normal
>> (for us) test script.  As I write this, it occurs to me you might be
>> thinking of embedding unit tests in the bindings source files; that
>> would be easy to add, something along the lines of
>>
>> test_begin_subtest "python bindings embedded unit tests"
>> test_expect_success "${NOTMUCH_PYTEST} ${NOTMUCH_SRCDIR}/bindings/python/notmuch"
>
> I was trying to construct something where a full pytest run on one
> python version was one subtest.  For granularity I think treating an
> entire pytest run as a subtest with just checking the return code should
> be sufficient,
> e.g. `python2.7 -m pytest ${NOTMUCH_SRCDIR}/bindings/python/notmuch`.
>
> But the whole test in this case would be this same subtest but once with
> python2.7, python3.5, python3.6 etc.
>
> What do you think of this?

The notmuch build system currently looks for a single version of
python. I'm pretty sure we don't want to require multiple python
versions to build notmuch, although I guess it wouldn't be that hard to
detect whatever versions exist in the environment and run pytest with
each of them. I just need a list of names to look for (or a better way
to find all the pythons installed). Debian has the handy comand
"py3versions --installed", but I suppose that's completely unportable.

Then each one can be checked to see if it finds the pytest module.

It sounds doable to me; it needs a bit more work in configure, but I can
drop the stuff in test-lib.sh

d