[notmuch] A functional (but rudimentary) test suite for notmuch

[Michal Sojka]

On Thursday 04 of February 2010 21:50:18 Carl Worth wrote:
> The test suite is still extremely rudimentary. Here are some things I'd
> like to improve about it:

I converted the actual version of notmuch-test to git test framework.
The result is in the followup patches.

I'd like to know opinion of others. If Carl agrees that it could be
merged I can do additional work as I describe below.

The conversion was not as straightforward as I expected mainly because
of problems with quoting. There are several sources of quotation problems
one being Carl's hashed array parameters. I thing it would be
sufficient to use plain variables for passing additional parameters.
Instead of:
    add_message [from]="\"Sender <sender at example.com>\"" \
                [to]=test_suite at notmuchmail.org \
                [subject]=notmuch-reply-test \
                [date]="\"Tue, 05 Jan 2010 15:43:56 -0800\"" \
                [body]="\"basic reply test\"" &&
I'd do:
    (
    msg_from="Sender <sender at example.com>"
    msg_to=test_suite at notmuchmail.org 
    msg_subject=notmuch-reply-test
    msg_date="Tue, 05 Jan 2010 15:43:56 -0800"
    msg_body="basic reply test"
    add_message
    )

A possible additional improvement is elimination of
execute_expecting(). Combination of action (running notmuch) and
testing of a result in a single function makes it hard to distinguish
whether the problem is in the action or in the output. For example, if
notmuch is killed because of segfault, it would look like that no
output was produced. So instead of:

    execute_expecting new "No new mail. Removed 3 messages."

I'd write something like:

    echo "No new mail. Removed 3 messages." > expected

  test_expect_success 'Run notmuch'    
    'notmuch_filter_output new > actual'
  test_expect_success 'Compare actual and correct output'    
    'test_cmp expected actual'

where test_cmp is basically diff -u. This has also the advantage that
longer output (e.g. as in t0003-reply.sh) is more readable if there is
a difference.

On Thursday 04 of February 2010 22:27:52 Oliver Charles wrote:
> Carl, have you considered outputting the test suite in the same format
> as the test anything protocol? [1] I only mention this because it
> might be a nice way to easily do some reporting (or perhaps even
> continuous integration) notmuch, with trivial effort.

According to http://testanything.org/wiki/index.php/TAP_Producers#Git_Project 
Git test output does not conform to TAP, but from my quick look it could be 
easily changed to conform.


On Friday 05 of February 2010 00:29:27 Carl Worth wrote:
> Looking at TAP, one thing I don't like is that it prints the
> success/failure of the test first, before the description of the
> test. That's not so nice in the case of a long-running (perhaps
> infinitely running) test where you might need to interrupt it, but you'd
> still want to know *what* was running for so long.

Git test framework also outputs first the result and then the test. It can be 
easily changed by -v switch.

To conclude, the output of running make in test/ directory now looks like 
this:
*** t0000-basic.sh ***
*   ok 1: test that mail store was created
*   ok 2: mail store should be empty
*   ok 3: NOTMUCH_CONFIG is set and points to an existing file
*   ok 4: PATH is set to this repository
*   ok 5: success is reported like this
*   still broken 6: pretend we have a known breakage
*   FIXED 7: pretend we have fixed a known breakage
*   ok 8: test runs if prerequisite is satisfied
* skip 9: unmet prerequisite causes test to be skipped
* fixed 1 known breakage(s)
* still have 1 known breakage(s)
* passed all remaining 8 test(s)
*** t0001-new.sh ***
*   ok 1: No new messages
*   ok 2: Single new message
*   ok 3: Multiple new messages
*   ok 4: No new messages (non-empty DB)
*   ok 5: New directories
*   ok 6: Alternate inode order
*   ok 7: Message moved in
*   ok 8: Renamed message
*   ok 9: Deleted message
*   ok 10: Renamed directory
*   ok 11: Deleted directory
*   ok 12: New directory (at end of list)
*   ok 13: Deleted directory (end of list)
*   ok 14: New symlink to directory
*   ok 15: New symlink to a file
*   ok 16: New two-level directory
*   ok 17: Deleted two-level directory
* passed all 17 test(s)
*** t0002-search.sh ***
*   ok 1: Search body
*   ok 2: Search by from:
*   ok 3: Search by to:
*   ok 4: Search by subject:
*   ok 5: Search by id:
*   ok 6: Search by tag:
*   ok 7: Search by thread:
*   ok 8: Search body (phrase)
*   ok 9: Search by from: (address)
*   ok 10: Search by from: (name)
*   ok 11: Search by to: (address)
*   ok 12: Search by to: (name)
*   ok 13: Search by subject: (phrase)
* passed all 13 test(s)
*** t0003-reply.sh ***
*   ok 1: Basic reply
*   ok 2: Multiple recipients
*   ok 3: Reply with CC
*   ok 4: Reply from alternate address
*   ok 5: Support for Reply-To
*   ok 6: Un-munging Reply-To
* passed all 6 test(s)
*** t0004-uuencoded-data.sh ***
*   ok 1: Generate message
*   ok 2: Ensure content before uu data is indexed
*   ok 3: Ensure uu data is not indexed
*   ok 4: Ensure content after uu data is indexed
* passed all 4 test(s)
*** t0005-dump-restore.sh ***
*   ok 1: Generate some message
*   ok 2: Dumping all tags
*   ok 3: Clearing all tags
*   ok 4: Restoring original tags
*   ok 5: Restore with nothing to do
* passed all 5 test(s)
fixed   1
success 51
failed  0
broken  1
total   54

Michal