notmuch ignoring alot of emails

[Tomi Ollila]

On Sun, Mar 31 2019, Alexei Gilchrist wrote:

>>> When I run notmuch I get a bunch (hundreds) of emails that are 
>>> ignored
>>> with:
>>>
>>> Note: Ignoring non-mail file: ...
>>>
>>> The files are valid maildir files but have a paragraph somewhere in 
>>> the
>>> body where someone has written "From ".
>>>
>>
>> And do they also have have a line starting with "From " as the first
>> line? This makes them mbox files. The second "From " makes them mbox
>> files with multiple messages. Notmuch thinks your MDA (the thing that
>> made those files) is misconfigured, assuming my guess about the format
>> is correct.
>
> Every message file begins with “From “. This is true of all messages 
> downloaded by both offlineimap (with type = Maildir) and mbsync.
> neomutt has no issues dealing with these files as maildir and mu has no 
> issues indexing them either. I’m assuming that stating with “From 
> “ is part of the maildir spec.
>
> The problem occurs specifically with notmuch. If someone sends a message 
> with a line that begins with “From “ in the *body* then it confuses 
> notmuch.
>
> mu can correctly index these messages but my mu is linked against 
> libgmime-2.6, my notmuch (0.28.3) is linked against libgmime-3.0.
>
>
>>> Is there a fix to force the recognition of maildir files in this 
>>> case? I
>>> thought this was a solved problem with gmime since 2.6.7.
>>
>> There is not currently a way to do that. It's not a GMime problem, 
>> it's
>> a design choice of notmuch to avoid parsing multiple message
>> mbox's. That was originally added as a safety feature, and I think it
>> should probably stay the default. If someone wants work on adding a
>> configuration switch I can point them in the right direction.
>
> This is a poor design decision. It means anyone on the internet can 
> break your mail setup simply by sending a message with a line starting 
> with “From “.
> (and using usual quoted-printable Content-Transfer-Encoding).

There are few things to remember in notmuch development:

- notmuch is more of an evolution than intelligent design. it is hard to 
  do any long-planned design when writing email software...

- we all do welcome people do SMOP with notmuch and tolerate patches with
  good commit messages and elegant content.

- it may take some time to get changes reviewed...

In this particular case it would be nice if someone(tm) investigated how 
mu and neomutt handles these email -- and how broken (if at all) those go
if those are given large mbox file... was it so that both of those can
read mbox files... 
(which notmuch doesn't (but one can always use mboxvievfs! >;)))?

> Try it. Send yourself a message with the line “From bad parsing comes 
> chaos” and see if your notmuch can find it. My version can’t.

My MDA (md5mda.sh) does not add 'From ' as beginning of first
line in my delivered emails (i.e. works similarly in this respect as
David's MDA).

Tomi