experimenting with pytest tests

[Tomi Ollila]

On Sat, Apr 07 2018, Floris Bruynooghe wrote:

> Hi,
>
> From another conversation on this list I've dug up my earlier attempts
> at integrating a pytest run into the notmuch testing suite.  I'd be
> grateful for some guidance on whether this is the right way to go about
> things.

good stuff, some comments inline

>
> Here's the diff of configure:
>
> diff --git a/configure b/configure
> index c5e2ffed..8aa57b83 100755
> --- a/configure
> +++ b/configure
> @@ -567,6 +567,28 @@ if [ $have_python -eq 0 ]; then
>      errors=$((errors + 1))
>  fi
>  
> +check_python() {

perhaps check_pytest()

> +    local bin=$1
> +    local var=$(echo $bin | tr -d '.')

I'm not so happy of the above, but good enough in this context,
alternarives are somewhat overkill

(e.g. calling like `check_python -v python27 python2.7`
 or even `check_python python2 7` or  `check_python python 2 7` or
 `check_python python 2 . 7` ;)

> +    printf "Checking for $bin (with: pytest)... "
> +    if command -v $bin > /dev/null; then
> +        if $bin -c 'import pytest' >/dev/null 2>&1; then
> +            eval have_$var=1
> +            eval $var=$bin
> +            printf "Yes.\n"
> +        else
> +            printf "No (skipping $bin tests).\n"
> +        fi
> +    else
> +        printf "No (skipping $bin tests).\n"
> +    fi
> +}
> +
> +check_python python2.7

perhaps check_python python3.4 -- that is what EPEL for RHEL/centos7
provides (hmm, it seems python 3.6 is coming to EPEL, see
http://www.nic.funet.fi/pub/mirrors/fedora.redhat.com/pub/epel/7/x86_64/Packages/p/
(or any other mirror), anyway that looks a bit incomplete so far)

> +check_python python3.5
> +check_python python3.6
> +check_python pypy3.5
> +
>  printf "Checking for valgrind development files... "
>  if pkg-config --exists valgrind; then
>      printf "Yes.\n"
> @@ -1209,6 +1231,10 @@ NOTMUCH_HAVE_MAN=$((have_sphinx))
>  
>  # Name of python interpreter
>  NOTMUCH_PYTHON=${python}
> +NOTMUCH_PYTHON27=${python27-}
> +NOTMUCH_PYTHON35=${python35-}
> +NOTMUCH_PYTHON36=${python36-}
> +NOTMUCH_PYPY35=${pypy35-}

perhaps NOTMUCH_PYTEST_PYPY35=${pytest_pypy35-} ...

> And I was then also trying to introduce a test/T391-pytest.sh file.
> What I'm trying to do in this file, but it doesn't currently work, is to
> have one test, with 4 subtests where each subtest is a pytest run with a
> particular python version.  But if the NOTMUCH_PYTHON27 (etc) is not
> found in sh.config then the subtest should be skipped.  I'm not really
> familiar enough with test-lib.sh to do this correctly without a bunch of
> more looking into it, but maybe someone else knows how to do this?
> Anyway, here my failing attempt:

We could start just with if test -n "$NOTMUCH_PYTEST_PYTHON27"
and iterate from that...


> #!/usr/bin/env bash
> test_description="python unittests"
> . ./test-lib.sh || exit 1
>
>
> test_require_external_prereq "${NOTMUCH_PYTHON27}" && {
>     test_begin_subtest "${NOTMUCH_PYTHON27}"
>     (
>         cd "$NOTMUCH_SRCDIR/bindings/python"
>         PYTHONPATH=".${PYTHONPATH:+:$PYTHONPATH}" \
> 	$NOTMUCH_PYTHON27 -m pytest
>     )
>     test_expect_equal $? 0
> }

The above could look like...

run_pytest() {
     test_begin_subtest "$1"

     PYTHONPATH="$NOTMUCH_SRCDIR/bindings/python${PYTHONPATH:+:$PYTHONPATH}" \
 	"$1" -m pytest $NOTMUCH_SRCDIR/bindings/python

     test_expect_equal $? 0
}

if test -n "$NOTMUCH_PYTEST_PYTHON27"; then
     run_pytest "$NOTMUCH_PYTEST_PYTHON27"
fi

(dropped short-circuit &&, since if we ever added `set -e` the
 unhandled || -case would make execution stop there)

(we might want to add export PYTHONDONTWRITEBYTECODE=donot so 
$NOTMUCH_SRCDIR/bindings/python is not polluted with *.pyc files,
and we might want to think how to handle all the output files
pytest spits out...)

if the whole test script is for pytest run, PYTHONPATH could
also be put into environment variable "global" to this 
particular script.

> test_require_external_prereq "${NOTMUCH_PYTHON27}" && {
>     test_begin_subtest "${NOTMUCH_PYTHON27}"
>     (
>         cd "$NOTMUCH_SRCDIR/bindings/python"
>         PYTHONPATH=".${PYTHONPATH:+:$PYTHONPATH}" \
> 	$NOTMUCH_PYTHON27 -m pytest
>     )
>     test_expect_equal $? 0
> }

> test_require_external_prereq ${NOTMUCH_PYTHON35} && {
>     test_begin_subtest "${NOTMUCH_PYTHON35}"
>     (
>         cd "$NOTMUCH_SRCDIR/bindings/python"
>         PYTHONPATH=".${PYTHONPATH:+:$PYTHONPATH}" \
> 	$NOTMUCH_PYTHON35 -m pytest
>     )
>     test_expect_equal $? 0
> }
>
>
> test_done
>
>
> Any tips on whether this is the right direction?
>
> Many thanks,
> Floris