Alternative (raw) message store (i.e. instead of maildir)

[Ciprian Dorin Craciun]

On Sat, Aug 11, 2012 at 11:50 PM, Jameson Graef Rollins
<jrollins at finestructure.net> wrote:
> On Sat, Aug 11 2012, Ciprian Dorin Craciun <ciprian.craciun at gmail.com> wrote:
>>     My problem with it is that it doesn't scale... And I don't mean
>> this in a theoretical sense, I mean it in the concrete one: I have
>> about 661k emails... And a single `notmuch sync` takes a few tens of
>> seconds...
>
> Hey, Ciprian.  That sounds really slow, which makes me wonder if there
> are other things going on here.
> I have 155k messages, but notmuch new
> takes a fraction of a second for me.  This initial indexing certainly
> takes a long time (hours potentially), but additions after that should
> be really fast.  What version of notmuch are you using?  What version of
> xapian?


    Don't think there is anything wrong here... Its just drags with
the file system...

    So just to give a complete info:
    * hardware: Core i5, 8GiB RAM (7.5GiB of which is the FS cache),
SSD (about 175MiB raw disk access);
    * `notmuch --version`: 0.13 (built from sources on latest ArchLinux);
    * `notmuch count`: 701820;
    * `notmuch new` (after adding 5925 new emails, at touching others):
~~~~
Processed 7017 total files in 3m 19s (35 files/sec.).
Added 6061 new messages to the database. Detected 1116 file renames.
~~~~
    * actually the entire thing took almost 5 minutes, but the first
two it didn't display anything just acesing the disk;
    * `notmuch new` (another go, but this time I've `time`-d it):
~~~~
No new mail.
real    0m40.546s
user    0m4.523s
sys     0m17.506s
~~~~
    * `notmuch new` (yet another go, no change):
~~~~
No new mail.
real    0m39.190s
user    0m4.229s
sys     0m17.697s
~~~~
    * just to `du` the maildir (there are also 40k other files in
other maildirs not included in this count):
~~~~
8.7G    ......
real    0m22.229s
user    0m1.023s
sys     0m7.890s
~~~~
    * on `new` no hooks are run;
    * the file system in cause is JFS;


    As such I doubt the problem is with notmuch itself, and I guess
it's the file system interaction...

    Now I know I have a really obscure corner case, and I'm positively
amazed on how good notmuch handles this situation. I just wandered if
I could have fixed my problem by moving to an embedded DB, thus
skipping all that syscall overhead...

    Ciprian.