[PATCH 00/11] Add ghost messages and fix thread linking

[Austin Clements]

On Sat, 04 Oct 2014, Tomi Ollila <tomi.ollila at iki.fi> wrote:
> On Fri, Oct 03 2014, Austin Clements <aclements at csail.mit.edu> wrote:
>
>> This series modifies our database representation of messages that have
>> been referenced by other messages, but for which we don't have the
>> message itself.  Currently, we store this information as Xapian
>> metadata, but this has several downsides for performance and
>> complexity and results in hard-to-fix thread linking bugs.  This patch
>> series implements "ghost messages", which replace this Xapian metadata
>> with Xapian documents that look and act very much like regular message
>> documents, but simply have no content.  This simplifies and speeds up
>> our thread linking algorithm and fixes the currently broken thread
>> linking test.
>>
>> Ghost messages also open up interesting future possibilities, such as
>> "pre-seeding" tags for messages that are not yet indexed.  This could
>> be used to make notmuch insert simpler and more robust, as part of tag
>> synchronization, and to improve nmbug's behavior when tags arrive
>> before messages.
>
> The code looks OK to me -- there are IMO some strange things but those
> don't change the status quo -- I did look a little past the diffs into
> the code to understand something...
>
> I am now having these patches applied to my 'home' notmuch and haven't yet
> seen anything strange there. In this setup I have 27 emails missing that
> nmbug expects there to be -- let's see if I can get ghost messages there.

In its current form, it won't help with these missing messages from
nmbug.  In the future, nmbug could use ghost messages to record the tags
for those messages anyway and then it becomes an interface question of
whether nmbug should report such missing messages.

> ... and tests pass, ran while writing the above part...
>
> I'll put these in use next week on one of my 'work' notmuch. There I have
> seen a problem that while I am on one thread, 
> `notmuch-poll-and-refresh-buffer' picks new mail but suddenly this thread
> now has changed it's thread id :O -- making the search buffer go blank.
> I'll see whether it still happens with these (may be totally unrelated)
> and perhaps investigate further...

I have this same problem with some people I communicate with.  I haven't
looked in to it, but I assume their messages reference only the
immediate parent, so if it's a reply to another unindexed message then
the thread ID may change depending on indexing order and other arbitrary
factors.  I may be wrong, but if this is the case, I don't think ghost
messages will change this behavior.  Solving this may require more
intelligently picking the "winning" thread ID when merging threads;
e.g., pick the thread ID with the most messages, rather than picking
arbitrarily.