[PATCH] cli: notmuch-show with framing newlines between threads in JSON.

[Mark Walters]

On Sun, 01 Jul 2012, Tomi Ollila <tomi.ollila at iki.fi> wrote:
> On Sat, Jun 30 2012, Mark Walters <markwalters1009 at gmail.com> wrote:
>
>> Add newlines between complete threads to make asynchronous parsing
>> of the JSON easier.
>> ---
>>
>> notmuch-pick uses the JSON output of notmuch show but, in many cases,
>> for many threads. This can take quite a long time when displaying a
>> large number of messages (say 20 seconds for the 10,000 messages in
>> the notmuch archive). Thus it is desirable to display results
>> incrementally in the same way that search currently does.
>>
>> To make this easier this patch adds newlines between each toplevel
>> thread. So the ouput becomes
>>
>> [
>> thread1
>> , thread2
>> , thread3
>> ...
>> , last_thread
>> ]
>>
>> Thus the parser can easily tell if it has enough data to do some more
>> parsing.
>>
>> Obviously, this changes the JSON output. This should not break any
>> consumer as the JSON parsers should not mind. However, it does break
>> several tests. Obviously, I will fix these but I wanted to check if
>> people were basically happy with the change first.
>
> To provide this feature rather than relying on newlines the parser should
> use it's state to notice when one thread ends. 
>
> Such a change could be used (privately) for human consumption -- allowing 
> free change of whitespace during inspection (in a debugging session or so).
> Computer software should not rely (or suffer) from any additional
> (or lack thereof) whitespace there is...
>
> ... or at least a really convicing argument for the chance needs to
> be presented (before "restricting" the json output notmuch spits out).
>
> Btw: AFAIC (json-read) parses the whole json object (ignoring whitespace,
> including newlines outside strings). So I quess notmuch-pick uses something
> slightly different (probably using json.el subroutines)..

I was following Austin's suggestion (on irc and
id:"20120214152114.GQ27039 at mit.edu"). The idea is that each thread in
the JSON output is an entire JSON object. Thus pick skips the first [
and the waits until there are two \n's in the incoming stream. Then it
knows that the complete first thread has been received and it parses
that with json-read as normal. The important thing is that it is trivial
to tell when a complete (and so parsable) JSON object has arrived.

It seems to work, but I am definitely open to other approaches.

> Btw2: I'm very interested to see notmuch-pick in action -- I just don't
> see this a way to do this particular support properly.
>
> Btw3: is search is ever going to use json we'll face the same problem -- 
> unless writing each line as a separate json object (and starting to use 
> s-expressions for speed)
>
>> Also, should devel/schemata be updated? It seems a little unclear as
>> this is not really a "JSON" change as the JSON does not care about the
>> newlines.
>>
>> Best wishes


> and best luck with your notmuch-pick work.

Thanks!

Mark

>>
>>  notmuch-show.c |    5 +++++
>>  1 files changed, 5 insertions(+), 0 deletions(-)
>>
>> diff --git a/notmuch-show.c b/notmuch-show.c
>> index 195e318..4a1d699 100644
>> --- a/notmuch-show.c
>> +++ b/notmuch-show.c
>> @@ -942,6 +942,8 @@ do_show (void *ctx,
>>  
>>      if (format->message_set_start)
>>  	fputs (format->message_set_start, stdout);
>> +    if (format == &format_json)
>> +	fputs ("\n", stdout);
>>  
>>      for (threads = notmuch_query_search_threads (query);
>>  	 notmuch_threads_valid (threads);
>> @@ -963,6 +965,9 @@ do_show (void *ctx,
>>  	if (status && !res)
>>  	    res = status;
>>  
>> +	if (format == &format_json)
>> +	    fputs ("\n", stdout);
>> +
>>  	notmuch_thread_destroy (thread);
>>  
>>      }
>> -- 
>> 1.7.9.1
>>
>> _______________________________________________
>> notmuch mailing list
>> notmuch at notmuchmail.org
>> http://notmuchmail.org/mailman/listinfo/notmuch