pytest integration for the notmuch test suite

[David Bremner]

Tomi Ollila <tomi.ollila at iki.fi> writes:

> On Sun, Mar 25 2018, David Bremner wrote:
>
>> Here's one approach. A given pytest "file" can be embedded in a normal
>> (for us) test script.  As I write this, it occurs to me you might be
>> thinking of embedding unit tests in the bindings source files; that
>> would be easy to add, something along the lines of
>>
>> test_begin_subtest "python bindings embedded unit tests"
>> test_expect_success "${NOTMUCH_PYTEST} ${NOTMUCH_SRCDIR}/bindings/python/notmuch"
>
> Hmm.
>
> Looks a bit strange to embed the pytest snippets into shell script and then
> execute each of these individiually. The only thing py.test seems to do here is
> "visualizing" assert output. We could just use normal python otherwise, and
> just not (necessarily) drop things into functions.

Yes, I had pretty much the same thought. It's also true that pytest
allows mixing tests with regular definition, if desired (i.e. it can
search for all the functions starting test_ or some other marker).
It's also true this doesn't really cost more at run time than the
current test_python; they both exec the python interpreter.

> If we had pytest, I'd suggest the tests were written and executed
> separately (from one test script) and then results collected somehow
> to the final aggregator.

Sure, that sounds nicer. I just don't want to invest a lot in writing
glue before it's clear what is needed. If start by running the pytest
files the crude way, we can always improve the glue later. That does
argue against doing many embedded snippets, I agree.

d