muchsync runs out of memory

[Antoine Beaupré]

I have been impressed with the features of muchsync when I recently
found out about it. muchsync seems to do everything i want: i have my
server where i usually take my mail, but then my laptop that i want to
sync and run notmuch + emacs on...

however, when i tried it, it failed:

[1011]anarcat at angela:~$ muchsync -v --noup --init Maildir
marcos.anarc.at
[SERVER] [notmuch] Note: Ignoring non-mail file:
/home/anarcat/Maildir//.Archives/dovecot-uidlist
[SERVER] [notmuch] Note: Ignoring non-mail file:
/home/anarcat/Maildir//.Archives/dovecot.index.log
[SERVER] [notmuch] Note: Ignoring non-mail file:
/home/anarcat/Maildir//.Drafts/dovecot-uidlist
[...]
[SERVER] [notmuch] Note: Ignoring non-mail file:
/home/anarcat/Maildir//courierimaphieracl/notmuch
[SERVER] [notmuch] Note: Ignoring non-mail file:
/home/anarcat/Maildir//courierimapkeywords/:list
[SERVER] [notmuch] Processed 53 total files in almost no time.
[SERVER] [notmuch] Added 4 new messages to the database.
[SERVER] [notmuch] Tagging spam (still incomplete)
[SERVER] [notmuch] tagging folders
[SERVER] [notmuch] tagging feeds
[SERVER] [notmuch] Tagging the rest
[notmuch] No new mail.
synchronizing muchsync database with Xapian... 4.525728 (+4.525728)
starting scan of Xapian database... 4.526494 (+0.000766)
opened Xapian... 4.530846 (+0.004351)
scanned message IDs... 4.531486 (+0.000640)
scanned tags... 4.532962 (+0.001477)
scanned directories in xapian... 4.533484 (+0.000521)
scanned filenames in xapian... 4.534243 (+0.000760)
adjusted link counts... 4.534790 (+0.000546)
finished synchronizing muchsync database with Xapian... 4.585857
(+0.051067)
received server's version vector... 4.673185 (+0.087328)
received hashes of new files... 13.704394 (+9.031209)
created directory /home/anarcat/Maildir/.Archives.2010
created directory /home/anarcat/Maildir/.Archives.2010/cur
created directory /home/anarcat/Maildir/.Archives.2010/new
created directory /home/anarcat/Maildir/.Archives.2010/tmp
created directory /home/anarcat/Maildir/.ham
created directory /home/anarcat/Maildir/.ham/cur
created directory /home/anarcat/Maildir/.ham/new
created directory /home/anarcat/Maildir/.ham/tmp
created directory /home/anarcat/Maildir/.Archives.2011
created directory /home/anarcat/Maildir/.Archives.2011/cur
created directory /home/anarcat/Maildir/.Archives.2011/new
created directory /home/anarcat/Maildir/.Archives.2011/tmp
Warning:
/home/anarcat/Maildir/.Archives.2011/cur/1447185197.M640040073P4037Q12085Rc1c3db4c.angela:2,
is an mbox containing a single message,
likely caused by misconfigured mail delivery.  Support for
single-message
mboxes is deprecated and may be removed in the future.
A Xapian exception occurred opening database: Unable to get write lock
on /home/anarcat/Maildir/.notmuch/xapian: Couldn't fork: Cannot allocate
memory
/home/anarcat/Maildir: A Xapian exception occurred
Processus arrêté

when i run it again, it thinks everything is fine:

[1012]anarcat at angela:~137$ muchsync -v --noup Maildir marcos.anarc.at
[notmuch] Processed 1 file in almost no time.
[notmuch] Added 1 new message to the database.
synchronizing muchsync database with Xapian... 1.304332 (+1.304332)
starting scan of Xapian database... 1.306140 (+0.001807)
opened Xapian... 1.310448 (+0.004308)
scanned message IDs... 1.656520 (+0.346072)
scanned tags... 1.744574 (+0.088054)
scanned directories in xapian... 1.746279 (+0.001704)
scanned filenames in xapian... 20.900691 (+19.154412)
adjusted link counts... 20.939274 (+0.038584)
finished synchronizing muchsync database with Xapian... 21.248633
(+0.309359)

but it's not fine:

[1019]anarcat at angela:~$ notmuch count
6289
[1021]anarcat at angela:~$ ssh anarc.at notmuch count
202094

at first, it did not look like it used up all the RAM, because
I was looking at Munin graphs which are updated every 5 minutes..

after talking with the author privately (the above is a copy of a
previous conversation), I was told to try the following patch

--- a/muchsync.cc
+++ b/muchsync.cc
@@ -15,7 +15,7 @@

 using namespace std;

-#if 0
+#if 1
 // This gives core dumps to make it easier to debug
 struct no_such_exception_t {
   const char *what() noexcept { return "no such exception"; }

Then I could confirm muchsync was killed by the OOM killer:

In the end, it did run out of memory:

[47386.773767] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[...]
[47386.774287] [15162]  1000 15162  1280802   835232    2478       8 405017             0 muchsync
[...]
[47386.774303] Out of memory: Kill process 15162 (muchsync) score 866 or sacrifice child

... so around 1.2GB in use... I don't think it dumped core - if so, I
couldn't find the core file.

Note that only 500MB of mails were transfered, out of around 11GB of
notmuch db + mail in my remote maildir.

Anyone using muchsync with similar data sizes?

I'd love to get this working - I had trouble reseting my offlineimap +
local notmuch synchronisation the last time I tried... Notmuch tags
would not get assigned properly, for some reason...

Thanks for any advice,

a.

-- 
Nothing incites to money-crimes like great poverty or great wealth.
                        - Mark Twain