[PATCH v2 3/4] index: repair "Mixed Up" messages before indexing.

[Daniel Kahn Gillmor]

When encountering a message that has been mangled in the "mixed up"
way by an intermediate MTA, notmuch should instead repair it and index
the repaired form.

When it does this, it also associates the index.repaired=mixedup
property with the message.  If a problem is found with this repair
process, or an improved repair process is proposed later, this should
make it easy for people to reindex the relevant message.  The property
will also hopefully make it easier to diagnose this particular problem
in the future.

Signed-off-by: Daniel Kahn Gillmor <dkg at fifthhorseman.net>
---
 doc/man7/notmuch-properties.rst | 13 +++++++++++++
 lib/index.cc                    | 22 +++++++++++++++++-----
 lib/notmuch-private.h           |  1 +
 test/T351-pgpmime-mangling.sh   |  2 --
 4 files changed, 31 insertions(+), 7 deletions(-)

diff --git a/doc/man7/notmuch-properties.rst b/doc/man7/notmuch-properties.rst
index 802e6763..31de576e 100644
--- a/doc/man7/notmuch-properties.rst
+++ b/doc/man7/notmuch-properties.rst
@@ -109,6 +109,19 @@ of its normal activity.
     example, an AES-128 key might be stashed in a notmuch property as:
     ``session-key=7:14B16AF65536C28AF209828DFE34C9E0``.
 
+**index.repaired**
+
+    Some mail transport agents mangle messages in transit in ways that
+    are both detectable and reversible.  If notmuch encounters such a
+    mangling during indexing, it will try to index the repaired form
+    of the message (while still leaving the message on disk
+    untouched).  If successful, it will use the ``index.repaired``
+    property to note the kind of mangling that was repaired.
+    Currently, only one form of repairable mangling is detected and
+    repaired, which is denoted with ``index.repaired=mixedup``.  See
+    https://tools.ietf.org/html/draft-dkg-openpgp-pgpmime-message-mangling
+    for more information.
+
 SEE ALSO
 ========
 
diff --git a/lib/index.cc b/lib/index.cc
index 1fd9e67e..44a42deb 100644
--- a/lib/index.cc
+++ b/lib/index.cc
@@ -385,11 +385,20 @@ _index_mime_part (notmuch_message_t *message,
     GMimeContentType *content_type;
     char *body;
     const char *charset;
+    GMimeObject *repaired_part = NULL;
 
     if (! part) {
 	_notmuch_database_log (notmuch_message_get_database (message),
 			      "Warning: Not indexing empty mime part.\n");
-	return;
+	goto DONE;
+    }
+
+    repaired_part = _notmuch_repair_mixed_up_mangled (part);
+    if (repaired_part) {
+	/* This was likely "Mixed Up" in transit!  We will instead use
+	 * the more likely-to-be-correct variant. */
+	notmuch_message_add_property (message, "index.repaired", "mixedup");
+	part = repaired_part;
     }
 
     _index_content_type (message, part);
@@ -441,7 +450,7 @@ _index_mime_part (notmuch_message_t *message,
 				       notmuch_status_to_string (status));
 	    _index_mime_part (message, indexopts, child, msg_crypto);
 	}
-	return;
+	goto DONE;
     }
 
     if (GMIME_IS_MESSAGE_PART (part)) {
@@ -451,14 +460,14 @@ _index_mime_part (notmuch_message_t *message,
 
 	_index_mime_part (message, indexopts, g_mime_message_get_mime_part (mime_message), msg_crypto);
 
-	return;
+	goto DONE;
     }
 
     if (! (GMIME_IS_PART (part))) {
 	_notmuch_database_log (notmuch_message_get_database (message),
 			      "Warning: Not indexing unknown mime part: %s.\n",
 			      g_type_name (G_OBJECT_TYPE (part)));
-	return;
+	goto DONE;
     }
 
     disposition = g_mime_object_get_content_disposition (part);
@@ -473,7 +482,7 @@ _index_mime_part (notmuch_message_t *message,
 
 	/* XXX: Would be nice to call out to something here to parse
 	 * the attachment into text and then index that. */
-	return;
+	goto DONE;
     }
 
     byte_array = g_byte_array_new ();
@@ -519,6 +528,9 @@ _index_mime_part (notmuch_message_t *message,
 
 	free (body);
     }
+ DONE:
+    if (repaired_part)
+	g_object_unref (repaired_part);
 }
 
 /* descend (if desired) into the cleartext part of an encrypted MIME
diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index 6fc5b366..ff3c6bbe 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -53,6 +53,7 @@ NOTMUCH_BEGIN_DECLS
 #include "error_util.h"
 #include "string-util.h"
 #include "crypto.h"
+#include "repair.h"
 
 #ifdef DEBUG
 # define DEBUG_DATABASE_SANITY 1
diff --git a/test/T351-pgpmime-mangling.sh b/test/T351-pgpmime-mangling.sh
index f65b8a24..4555f937 100755
--- a/test/T351-pgpmime-mangling.sh
+++ b/test/T351-pgpmime-mangling.sh
@@ -21,7 +21,6 @@ test_json_nodes <<<"$output" \
                 'body:["original"]'"$bodytext"
 
 test_begin_subtest "repaired 'Mixed-up' messages can be found with index.repaired=mixedup"
-test_subtest_known_broken
 output=$(notmuch search --output=messages property:index.repaired=mixedup)
 test_expect_equal "$output" id:mixed-up at mangling.notmuchmail.org
 
@@ -29,7 +28,6 @@ test_begin_subtest "index cleartext of 'Mixed-Up' mangled PGP/MIME message"
 test_expect_success 'notmuch reindex --decrypt=true id:mixed-up at mangling.notmuchmail.org'
 
 test_begin_subtest "search cleartext of 'Mixed-Up' mangled PGP/MIME message"
-test_subtest_known_broken
 output=$(notmuch search --output=messages body:password)
 test_expect_equal "$output" id:mixed-up at mangling.notmuchmail.org
 
-- 
2.20.1