[PATCH 0/3] Outline fix for emacs tagging race

[Tomi Ollila]

On Fri, Nov 09 2012, Mark Walters <markwalters1009 at gmail.com> wrote:

>
> This patch series implements the above mentioned solution and seems to
> work except for one problem.
>
> Since emacs now tags each message in a thread in the search buffer it
> is easy to ask it to tag a lot of messages. This could be done
> individually which would be ridiculously slow so instead they are all
> done in one batch. But now it is relatively easy to take notmuch over
> the threshold for ARG_MAX.
>
> In [3] Tomi did some experiments and found on a standard Debian system
> with getconf ARG_MAX =131072 that command lines with 10000 200 byte
> arguments worked. I am a little puzzled by that as I get the same
> results and I getconf ARG_MAX gives 2097152 for me.
>
> More importantly though, when trying to execute a command from emacs I
> am finding that 131072 is the limit on the command length in bytes and
> we can hit this with something around 1500 messages (see end for a
> very hacky emacs test script). This is probably more than we can
> expect in a single thread so tagging from show is probably safe but it
> does cause a problem when tagging from search.

That's because your test script below constructs one very loooong
string (and used just one arg). Like you said, [3] used 10000 200-char args.

This test script is more like what the perl(1) script did in [3]

(progn
  (setq arglist '("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"))
  (setq n 10000)
  (while (> n 0)
    (setq n (1- n))
    (setq arglist (cons "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" arglist)))

  (apply 'call-process "/bin/echo" nil t nil arglist))

(except that the args here are just 80 chars but you should get the point :)


I did not check your patches yet.. buf if you concatenate there you 
could try to provide those as separate args...

... then we can experiment how notmuch (and xapian) copes with thosew
IIRC notmuch will concatenate one long string out of those and do
xapian stuff using that (but then we're not limited by command line
arguments restrictions).

>
> Best wishes 
>
> Mark

Tomi

>
> [3] id:m2liody7av.fsf at guru.guru-group.fi

> TEST SCRIPT

> (progn
>   (setq bigstring
> "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa")
>   (setq n 1310)
>   (setq big nil)
>   (while (> n 0)
>    (setq n (1- n))
>     (setq big (concat big (format "%s" n) " " bigstring)))
>   (call-process "echo" nil t nil big))