experimenting with pytest tests

[Floris Bruynooghe]

Hi,

>From another conversation on this list I've dug up my earlier attempts
at integrating a pytest run into the notmuch testing suite.  I'd be
grateful for some guidance on whether this is the right way to go about
things.

Here's the diff of configure:

diff --git a/configure b/configure
index c5e2ffed..8aa57b83 100755
--- a/configure
+++ b/configure
@@ -567,6 +567,28 @@ if [ $have_python -eq 0 ]; then
     errors=$((errors + 1))
 fi
 
+check_python() {
+    local bin=$1
+    local var=$(echo $bin | tr -d '.')
+    printf "Checking for $bin (with: pytest)... "
+    if command -v $bin > /dev/null; then
+        if $bin -c 'import pytest' >/dev/null 2>&1; then
+            eval have_$var=1
+            eval $var=$bin
+            printf "Yes.\n"
+        else
+            printf "No (skipping $bin tests).\n"
+        fi
+    else
+        printf "No (skipping $bin tests).\n"
+    fi
+}
+
+check_python python2.7
+check_python python3.5
+check_python python3.6
+check_python pypy3.5
+
 printf "Checking for valgrind development files... "
 if pkg-config --exists valgrind; then
     printf "Yes.\n"
@@ -1209,6 +1231,10 @@ NOTMUCH_HAVE_MAN=$((have_sphinx))
 
 # Name of python interpreter
 NOTMUCH_PYTHON=${python}
+NOTMUCH_PYTHON27=${python27-}
+NOTMUCH_PYTHON35=${python35-}
+NOTMUCH_PYTHON36=${python36-}
+NOTMUCH_PYPY35=${pypy35-}


And I was then also trying to introduce a test/T391-pytest.sh file.
What I'm trying to do in this file, but it doesn't currently work, is to
have one test, with 4 subtests where each subtest is a pytest run with a
particular python version.  But if the NOTMUCH_PYTHON27 (etc) is not
found in sh.config then the subtest should be skipped.  I'm not really
familiar enough with test-lib.sh to do this correctly without a bunch of
more looking into it, but maybe someone else knows how to do this?
Anyway, here my failing attempt:

#!/usr/bin/env bash
test_description="python unittests"
. ./test-lib.sh || exit 1


test_require_external_prereq "${NOTMUCH_PYTHON27}" && {
    test_begin_subtest "${NOTMUCH_PYTHON27}"
    (
        cd "$NOTMUCH_SRCDIR/bindings/python"
        PYTHONPATH=".${PYTHONPATH:+:$PYTHONPATH}" \
	$NOTMUCH_PYTHON27 -m pytest
    )
    test_expect_equal $? 0
}


test_require_external_prereq ${NOTMUCH_PYTHON35} && {
    test_begin_subtest "${NOTMUCH_PYTHON35}"
    (
        cd "$NOTMUCH_SRCDIR/bindings/python"
        PYTHONPATH=".${PYTHONPATH:+:$PYTHONPATH}" \
	$NOTMUCH_PYTHON35 -m pytest
    )
    test_expect_equal $? 0
}


test_done


Any tips on whether this is the right direction?

Many thanks,
Floris