[PATCH 1/1] Store and search for canonical Unicode text [WIP]

[David Bremner]

Rob Browning <rlb at defaultvalue.org> writes:

>
> Before this change, notmuch would index two strings that differ only
> with respect to canonicalization, like tóken and tóken, as separate
> terms, even though they may be visually indistinguishable, and do (for
> most purposes) represent the same text.  After indexing, searching for
> one would not find the other, and which one you present to notmuch
> when you search depends on your tools.  See test/T570-normalization.sh
> for a working example.

One way to break this up into more bite sized pieces would be to first
create one or more tests that fail with current notmuch, and mark those
as broken.

> Up to now, notmuch has let Xapian handle converting the incoming bytes
> to UTF-8.  Xapian treats any byte sequence as UTF-8, and interprets
> any invalid UTF-8 bytes as Latin-1.  This patch maintains the existing
> behavior (excepting the new canonicalization) by using Xapian's
> Utf8Iterator to handle the initial Unicode character parsing.

Can you explain why notmuch is the right place to do this, and not
Xapian? I know we talked back and forth about this, but I never really
got a solid sense of what the conclusion was. Is it just dependencies?

> And because when the input is already UTF-8, it just blindly converts
> from UTF-8 to Unicode code points, and then back to UTF-8 (after
> canonicalization), during each pass.  There are certainly
> opportunities to optimize, though it may be worth discussing the
> detection of data encodings more broadly first.

It seems plausible to specify UTF-8 input for the library, but what
about the CLI? It seems like the canonicalization operation increases
the chance of mangling user input in non-UTF-8 locales.

> FIXME: what about existing indexed text?

I suppose some upgrade code to canonicalize all the terms? That sounds
pretty slow.

> ---
>
>  Posted for preliminary discussion, and as a milestone (it appears to
>  mostly work now).  Though I doubt I'm handling things correctly
>  everywhere notmuch-wise, wrt talloc, etc.

I really didn't look at the code very closely, but there were a
surprising number of calls to talloc_free. But those kind of details can
wait.