parallelize test suite

[Daniel Kahn Gillmor]

On Sat 2019-05-04 19:53:23 -0300, David Bremner wrote:
> Last time we discussed parallel test running, there we concerns about
> multiple versions of certain servers colliding with each other. This
> still seems to be at least a theoretical issue with smtp-dummy, although
> a glance suggests that it might only currently be used in T310-emacs.sh.
> I'm not sure what a robust solution is here. 
>
> - gpg-agent
> - emacs
> - dtach
>
> Did I miss any other background processes run by the test suite?
>
> I can imagine gpg-agent is managed OK these days since it's started
> automagically by gpg.

gpg-agent is fine, because it's isolated by $GNUPGHOME, and each test
uses a distinct $GNUPGHOME (see GNUPGHOME="${TEST_TMPDIR}/gnupg in
test-lib.sh)

> emacs seems to use the current process id in the socket name, so that
> also should be OK, although it should maybe be replaced with something
> more robust to avoid problems with pid rollover. I _think_ including the
> test name in the emacs server would do the trick

I would have no objection to this improvement in isolation of the emacs
server processes, but i think the pid rollover race condition is so
minor that i don't think it sholud be a blocker for the adoption of this
series.

> The dtach socket is in the tmp.T* directory, so that should be OK.
>
> I wonder if a good solution would be to make running the test suite in
> parallel be opt-in (e.g. by configuration option). Or at least have a
> way to disable it for situations like CI and autobuilders.

I agree that making it possible to force serialized tests would be good.

I'd prefer that running tests in parallel be the default, though i
wouldn't object to a ./configure --serialize-tests option if someone
wants to implement it.

I actually think that CI and autobuilders *should* exercise the parallel
tests, as annoying as that might be initially, because it seems likely
to catch any other potential entanglements.

thanks for the review!

   --dkg
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20190505/924d2f36/attachment.sig>