Notmuch indexing 21 million emails

[Tom Bulli]

I have been able to speed that up with the code below - basically increase "XAPIAN_FLUSH_THRESHOLD" based on the total virtual memory divided by the avg. size of an email times 2 (just to be safe).  It seems to be faster since it does less xapian updates.  However, I have a nagging feeling that ""XAPIAN_FLUSH_THRESHOLD" could even be higher since I don't see any increase in used memory (via "top -d 1").  The server in question has eight CPU cores and 8GB RAM, running Debian squeeze on a 32bit architecture (I know - but it is what it is :) ).


# Assume an average size of 120KB per email
#  and use at most half the virtual memory
XFT=$(($(free -otk  | awk '/^Total/ {print $2}') / 240))
# Keep more index info in memory before flushing to disk
[ $XFT -lt 10000 ] && XFT=10000
su - archive -c "export XAPIAN_FLUSH_THRESHOLD=$XFT; notmuch new --verbose"




----- Original Message -----
> From: Felipe Contreras <felipe.contreras at gmail.com>
> To: Tom Bulli <mrbulli at yahoo.com>
> Cc: "notmuch at notmuchmail.org" <notmuch at notmuchmail.org>
> Sent: Wednesday, November 23, 2011 10:40 AM
> Subject: Re: Notmuch indexing 21 million emails
> 
> On Tue, Nov 22, 2011 at 5:02 AM, Tom Bulli <mrbulli at yahoo.com> wrote:
>>  I have a project where I need to search about 21 emails - and decided to 
> use "notmuch" for it.  The system is a Debian Squeeze, the notmuch 
> version is "0.8-1~bpo60+1" from "kyria's" private 
> repository.
>> 
>>  I am running the "notmuch new" for approx. 4 days now - and 
> according to "not,uch count" it has indexed about 4.5 million emails.
>> 
>>  Is this expected performance?  Is there any way to speed that up?
> 
> It would be nice to run something like this with OProfile (or perf)
> and see if there's some obvious fixes.
> 
> -- 
> Felipe Contreras
>