performance problems with notmuch new

[Don Zickus]

On Wed, Apr 15, 2020 at 05:29:13PM -0300, David Bremner wrote:
> Don Zickus <dzickus at redhat.com> writes:
> 
> >> runs in about 30s here (i7 4770 / SSD).  Replacing --small with --medium
> >> takes about 10M (so a superlinear slowdown in wall clock time, since
> >> that represents a 10x scale-up in the corpus size.).
> >
> > Hmm, for me --small was 35s and --medium was 32 minutes.  This is on a
> > i7-9750H / nvme.  I would expect numbers similar to yours.
> 
> I did another few tests test for --medium and they all take 7-9 minutes,
> depending what else is going on on the machine.
> Here's my breakdown of times (unfortunately a bit of hand editing is
> needed to clean up the warnings)

Sorry for the delay.  I re-ran the test today and see similar numbers to
yours now. Roughly 6m30s per run (3 runs).

T00-new.sh: Testing notmuch new                         [0.4 medium]
			Wall(s)	Usr(s)	Sys(s)	Res(K)	In/Out(512B)
  Initial notmuch new   61.86	59.13	2.53	228124	2024/1087672
  notmuch new #2        0.01	0.00	0.00	9136	0/160
  notmuch new #3        0.00	0.00	0.00	9068	0/8
  notmuch new #4        0.00	0.00	0.00	8940	0/8
  notmuch new #5        0.00	0.00	0.00	9060	0/8
  notmuch new #6        0.00	0.00	0.00	8932	0/8
  new (7500 mv)         44.76	32.67	12.02	186156	0/541024
  new (7500 mv back)    47.82	34.81	12.91	187240	0/500000
  new (7500 cp)         26.41	20.06	6.28	183448	0/378192

T01-dump-restore.sh: Testing dump and restore           [0.4 medium]
			Wall(s)	Usr(s)	Sys(s)	Res(K)	In/Out(512B)
3.35	1.82	1.31	11992	624/31976
  dump *                0.60	0.57	0.02	11356	0/4344
  restore *             0.69	0.62	0.06	9064	0/0

T02-tag.sh: Testing tagging                             [0.4 medium]
			Wall(s)	Usr(s)	Sys(s)	Res(K)	In/Out(512B)
  tag * +new_tag        39.96	24.99	14.74	80048	16/282944
  tag * +existing_tag   0.00	0.00	0.00	8968	0/0
  tag * -existing_tag   29.98	18.18	11.70	20040	0/208264
  tag * -missing_tag    0.00	0.00	0.00	9036	0/0

T03-reindex.sh: Testing reindexing                      [0.4 medium]
			Wall(s)	Usr(s)	Sys(s)	Res(K)	In/Out(512B)
  reindex *             40.56	33.46	7.02	228908	0/459240
  reindex *             39.91	33.10	6.75	223768	112/378824
  reindex *             40.04	33.10	6.89	223724	0/334824

T04-thread-subquery.sh: Testing thread subqueries       [0.4 medium]
			Wall(s)	Usr(s)	Sys(s)	Res(K)	In/Out(512B)
  search thread:{} ...  0.29	0.26	0.02	23256	0/24
  search thread:{} ...  0.27	0.25	0.02	23296	0/24
  search thread:{} ...  0.29	0.27	0.02	23192	0/24


The only thing I can think of is my fstrim service runs 1x / week on Monday
at midnight and maybe that helped clean things up??  Perhaps I should
increase that frequency or run it manually when things go bad.

Thanks for the feedback!

Cheers,
Don