[PATCH] test: Better handling of stdout and stderr

[Carl Worth]

On Wed, 10 Nov 2010 00:40:35 +0100, Michal Sojka <sojkam1 at fel.cvut.cz> wrote:
> Git-style tests suppress stdout and stderr unless -v is given.
> cworth-style tests (created by test_begin_subtest, test_expect_equal)
> do not have this behavior so implement it the same.
> 
> Additionally, for both test styles, the test-lib.sh is changed so that
> the content of suppressed stdout and stderr is shown in case of failed
> test.
> 
> Finally a test for this functionality is added to basic tests

This looks like some great stuff, Michal! I especially like the way you
are testing the test-suite framework itself[*].

I did have one local commit to test-lib.sh that broke this slightly, (I
changed it so that you will now get intermediate files like
test-verbose.4.out rather than test.4.out). That would have been easy
enough to fix, but I also noticed output like the following from "make
test":

...
	 PASS   Search by to: (address)
	 PASS   Search by to: (name)
	 PASS   Search by subject: (phrase)
	json: Testing --format=json output
	 PASS   Show message: json
	 PASS   Search message: json
	 PASS   Search by subject (utf-8):
	 PASS   Show message: json, utf-8
	 PASS   Search message: json, utf-8
         
	thread-naming: Testing naming of threads with changing subject
	raw: Testing notmuch show --format=raw
	 PASS   Generate some messages
	 PASS   Attempt to show multiple raw messages
...

At first I thought the change just caused some newlines to be dropped,
(such as the missing newline just before the "json" line). But then I
noticed that all output from all of the thread-naming tests is entirely
missing.

Do you want to look into what's going on there?

Meanwhile, I've just now pushed out my little test-lib.sh change, so you
can update the patch for that as well.

Thanks,

-Carl

[*] The current git test suite tries to do some things like this, but
not as cleverly. So it actually reports a FAIL and a BROKEN test when in
fact it's only trying to exercise the parts of the test suite that
report FAIL and BROKEN. I found that objectionable, so simply removed
those tests. But calling to a subordinate test that would report FAIL
and then reporting PASS after checking that would be just fine. That's a
change we could implement and even contribute upstream to the git
project itself.

-- 
carl.d.worth at intel.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20101110/2e0330b6/attachment.pgp>