parallelize test suite

[Tomi Ollila]

On Sun, May 05 2019, Daniel Kahn Gillmor wrote:

> On Sat 2019-05-04 19:53:23 -0300, David Bremner wrote:
>> Last time we discussed parallel test running, there we concerns about
>> multiple versions of certain servers colliding with each other. This
>> still seems to be at least a theoretical issue with smtp-dummy, although
>> a glance suggests that it might only currently be used in T310-emacs.sh.

we,d need to change 25025 with something else -- configured in the test
script ? -- same could be used as part of emacs server socket ?


>> I'm not sure what a robust solution is here. 
>>
>> - gpg-agent
>> - emacs
>> - dtach
>>
>> Did I miss any other background processes run by the test suite?
>>
>> I can imagine gpg-agent is managed OK these days since it's started
>> automagically by gpg.
>
> gpg-agent is fine, because it's isolated by $GNUPGHOME, and each test
> uses a distinct $GNUPGHOME (see GNUPGHOME="${TEST_TMPDIR}/gnupg in
> test-lib.sh)
>
>> emacs seems to use the current process id in the socket name, so that
>> also should be OK, although it should maybe be replaced with something
>> more robust to avoid problems with pid rollover. I _think_ including the
>> test name in the emacs server would do the trick
>
> I would have no objection to this improvement in isolation of the emacs
> server processes, but i think the pid rollover race condition is so
> minor that i don't think it sholud be a blocker for the adoption of this
> series.

when test code done correctly (test_done ends it), the shell launching
emacs holds the pid $$ until emacs exits, so the pid is not reused
in name for any emacs process (but if bash died leaving emacs running
then we'd have chance for collisions...)

>
>> The dtach socket is in the tmp.T* directory, so that should be OK.
>>
>> I wonder if a good solution would be to make running the test suite in
>> parallel be opt-in (e.g. by configuration option). Or at least have a
>> way to disable it for situations like CI and autobuilders.
>
> I agree that making it possible to force serialized tests would be good.
>
> I'd prefer that running tests in parallel be the default, though i
> wouldn't object to a ./configure --serialize-tests option if someone
> wants to implement it.

no need for configure option, one can just remove parallel(1) from system ;)
... well, we could use ${PARALLEL:=parallel} in script for users to shadow
it out in some other way...

> I actually think that CI and autobuilders *should* exercise the parallel
> tests, as annoying as that might be initially, because it seems likely
> to catch any other potential entanglements.

... after it has been proven a bit more to work...

One thing more, the "perverse" pre-caching done in add_email_corpus
can be retained -- just do that step *before* going to parallelism...

>
> thanks for the review!
>
>    --dkg

Tomi