performance problems with notmuch new

[David Bremner]

Don Zickus <dzickus at redhat.com> writes:

>> runs in about 30s here (i7 4770 / SSD).  Replacing --small with --medium
>> takes about 10M (so a superlinear slowdown in wall clock time, since
>> that represents a 10x scale-up in the corpus size.).
>
> Hmm, for me --small was 35s and --medium was 32 minutes.  This is on a
> i7-9750H / nvme.  I would expect numbers similar to yours.

I did another few tests test for --medium and they all take 7-9 minutes,
depending what else is going on on the machine.
Here's my breakdown of times (unfortunately a bit of hand editing is
needed to clean up the warnings)


performance-test/notmuch-time-test --medium

T00-new.sh: Testing notmuch new                         [0.4 medium]
			Wall(s)	Usr(s)	Sys(s)	Res(K)	In/Out(512B)
  Initial notmuch new   66.29	62.22	2.82	241148	0/1089784
  notmuch new #2        0.03	0.00	0.00	9864	0/160
  notmuch new #3        0.00	0.00	0.00	9292	0/8
  notmuch new #4        0.00	0.00	0.00	9556	0/8
  notmuch new #5        0.00	0.00	0.00	9396	0/8
  notmuch new #6        0.00	0.00	0.00	9316	0/8
  new (7500 mv)         49.02	35.88	12.59	185152	0/392720
  new (7500 mv back)    59.41	43.04	15.88	185888	0/413824
  new (7500 cp)         36.09	26.55	9.02	182160	0/411840

T01-dump-restore.sh: Testing dump and restore           [0.4 medium]
			Wall(s)	Usr(s)	Sys(s)	Res(K)	In/Out(512B)
  load nmbug tags       5.37	2.09	1.63	12172	0/31864
  dump *                0.63	0.58	0.04	11756	0/4344
  restore *             0.72	0.65	0.06	9572	0/0

T02-tag.sh: Testing tagging                             [0.4 medium]
			Wall(s)	Usr(s)	Sys(s)	Res(K)	In/Out(512B)
  tag * +new_tag        54.45	31.93	20.26	86380	8/250512
  tag * +existing_tag   0.00	0.00	0.00	9396	0/0
  tag * -existing_tag   46.73	26.07	20.08	20580	0/284248
  tag * -missing_tag    0.00	0.00	0.00	9316	0/0

T03-reindex.sh: Testing reindexing                      [0.4 medium]
			Wall(s)	Usr(s)	Sys(s)	Res(K)	In/Out(512B)
  reindex *             78.39	63.06	14.31	229204	0/546136
  reindex *             70.52	56.68	13.12	223980	0/333608
  reindex *             78.02	62.61	14.92	225160	0/374424

T04-thread-subquery.sh: Testing thread subqueries       [0.4 medium]
			Wall(s)	Usr(s)	Sys(s)	Res(K)	In/Out(512B)
  search thread:{} ...  0.37	0.33	0.04	26508	0/24
  search thread:{} ...  0.38	0.33	0.04	23592	0/24
  search thread:{} ...  0.37	0.34	0.03	26612	0/24

415.28user 128.24system 9:13.07elapsed 98%CPU (0avgtext+0avgdata 241148maxresident)k
8inputs+7294896outputs (0major+458590minor)pagefaults 0swaps