Mail archives in Git using ssoma

[W. Trevor King]

On Sun, Aug 21, 2016 at 06:37:04PM +0000, Eric Wong wrote:
> "W. Trevor King" <wking at tremily.us> wrote:
> > On Fri, Nov 07, 2014 at 11:03:21AM -0800, W. Trevor King wrote:
> > > Eric Wong has been working on some tools to store email in a Git
> > > repository, and his client-side code is ssoma [1].  I wanted a bit
> > > more metadata than the stock ssoma-mda [2], and ended up just
> > > writing a ssoma-mda in Python [3]…
>
> Btw, for public-inbox, I'm using git-fast-import now, so imports are
> a bit faster and $GIT_DIR/ssoma.index is no longer used.  This was
> crucial for getting git at vger archives imported in a reasonable time.
>
> public-inbox-* still keeps ssoma.index up-to-date for backwards
> compatibility with ssoma, and will probably do so until 2020 or
> later (there'll be a few years of deprecation notices)
>
> So I or someone else needs to update Perl ssoma to use fast-import
> at some point, too; and I suggest your python version do the same.

ssoma-mda imports 22k notmuch messages in around 15 minutes (with
profiling enabled), and:

  $ python -m cProfile -o profile import.py notmuch.mbox
  $ python -c "import pstats; p=pstats.Stats('profile'); p.sort_stats('cumulative').print_stats(10)"
  Sun Aug 21 12:56:49 2016    profile

           101823722 function calls (99078415 primitive calls) in 885.069 seconds

     Ordered by: cumulative time
     List reduced from 1145 to 10 due to restriction <10>

     ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       70/1    0.002    0.000  885.069  885.069 {built-in method exec}
          1    0.111    0.111  885.069  885.069 /home/wking/src/notmuch/notmuch-archives.git/import.py:9(<module>)
          1    0.400    0.400  884.915  884.915 /home/wking/src/notmuch/notmuch-archives.git/import.py:17(import_mbox)
      22875    0.601    0.000  863.371    0.038 /home/wking/src/notmuch/notmuch-archives.git/ssoma_mda.py:362(deliver)
      22875    8.943    0.000  810.459    0.035 /home/wking/src/notmuch/notmuch-archives.git/ssoma_mda.py:207(append)
      22875    0.418    0.000  308.353    0.013 /home/wking/.local/lib64/python3.4/site-packages/pygit2/index.py:146(write_tree)
      22875  307.855    0.013  307.855    0.013 {built-in method git_index_write_tree}
      22874    0.575    0.000  279.293    0.012 /home/wking/.local/lib64/python3.4/site-packages/pygit2/index.py:238(diff_to_tree)
      22874  278.501    0.012  278.501    0.012 {built-in method git_diff_tree_to_index}
      22875    0.088    0.000   80.413    0.004 /home/wking/.local/lib64/python3.4/site-packages/pygit2/index.py:99(read)

38 ms per ssoma delivery is probably fast enough, especially if you
are invoking ssoma-mda once per message, since process setup will take a similar amount of time:

  $ time python -c 'print("hello")'
  hello

  real    0m0.016s
  user    0m0.013s
  sys     0m0.003s

It's possible that fast-import would shave a few ms off the pygit2
addition (I'm not sure, and maybe pygit2 is faster than fast-import).
But I doubt it matters enough either way to be worth changing unless
you are dealing with a really large corpus.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20160821/92b69f11/attachment.sig>