[PATCH] Fix shared library loading in Python bindings.

[Justus Winter]

Hi Julian :)

Quoting Julian Berman (2013-05-26 05:53:38)
> Specifically, fixes loading on OS X, where libnotmuch will be
> a dylib.:
> ---
>  bindings/python/notmuch/globals.py | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/bindings/python/notmuch/globals.py b/bindings/python/notmuch/globals.py
> index c7632c3..5e08e73 100644
> --- a/bindings/python/notmuch/globals.py
> +++ b/bindings/python/notmuch/globals.py
> @@ -18,11 +18,12 @@ Copyright 2010 Sebastian Spaeth <Sebastian at SSpaeth.de>
>  """
>  
>  from ctypes import CDLL, Structure, POINTER
> +from ctypes.util import find_library
>  
>  #-----------------------------------------------------------------------------
>  #package-global instance of the notmuch library
>  try:
> -    nmlib = CDLL("libnotmuch.so.3")
> +    nmlib = CDLL(find_library("libnotmuch"))

Does this work for you on Darwin? On my box (Debian/Linux) I have to
use "notmuch" instead of "libnotmuch" to get anything from
find_library:

In [5]: print ctypes.util.find_library("notmuch")
libnotmuch.so.3

In [6]: print ctypes.util.find_library("libnotmuch")
None

Then again, find_library is different for every architecture under the
sun...

>  except:
>      raise ImportError("Could not find shared 'notmuch' library.")

I'm leaning towards declining the patch in it's current form. If the
bindings do not work on OS X, we need to find another solution. There
are two reasons for this:

1. find_library was once used, but was removed since it is (has?) been
problematic wrt to LD_LIBRARY_PATH usage:

% git show f378f458
commit f378f45893bb4263402008b2abd8aab522901fb6
Author: Cedric Cabessa <ced at ryick.net>
Date:   Mon Apr 5 03:03:51 2010 +0200

    find_library does not read LD_LIBRARY_PATH, but CDLL does.

diff --git a/cnotmuch/globals.py b/cnotmuch/globals.py
index ef2686f..fa20ae8 100644
--- a/cnotmuch/globals.py
+++ b/cnotmuch/globals.py
@@ -3,17 +3,17 @@ from ctypes.util import find_library
 
 #-----------------------------------------------------------------------------
 #package-global instance of the notmuch library
-#TODO: lazy load this on first access?
-so = find_library('notmuch')
-if so is None:
-  raise ImportError("Could not find shared 'notmuch' library.")
-nmlib = CDLL(so)
+try:
+    nmlib = CDLL("libnotmuch.so.1")
+except:
+    raise ImportError("Could not find shared 'notmuch' library.")
[...]

As a heavy LD_LIBRARY_PATH user I don't want to break this. So I tried
to test whether find_library now respects LD_LIBRARY_PATH or not:

teythoon at thinkbox ~ % echo $LD_LIBRARY_PATH
/home/teythoon/.local/lib
teythoon at thinkbox ~ % strace -e trace=open python -c 'import ctypes; ctypes.CDLL("libnotmuch.so.3")' 2>&1 | grep notmuch
open("/home/teythoon/.local/lib/libnotmuch.so.3", O_RDONLY) = 3

That's how it's done now and indeed it finds my libnotmuch.

teythoon at thinkbox ~ % strace -f -e trace=stat,open python -c 'import ctypes.util; ctypes.util.find_library("notmuch")' 2>&1 | grep notmuch

Nothing. Funny. Let's see:

teythoon at thinkbox ~ % strace -f -e trace=fork,execve,clone python -c 'import ctypes.util; print ctypes.util.find_library("notmuch")'
execve("/usr/bin/python", ["python", "-c", "import ctypes.util; print ctypes"...], [/* 63 vars */]) = 0
clone(Process 12000 attached
child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f0734a649d0) = 12000
[pid 12000] execve("/bin/sh", ["sh", "-c", "/sbin/ldconfig -p 2>/dev/null"], [/* 63 vars */]) = 0
[pid 12000] clone(Process 12001 attached
child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f119d5479d0) = 12001
[pid 12001] execve("/sbin/ldconfig", ["/sbin/ldconfig", "-p"], [/* 63 vars */]) = 0
Process 12001 detached
[pid 12000] --- SIGCHLD (Child exited) @ 0 (0) ---
Process 12000 detached
--- SIGCHLD (Child exited) @ 0 (0) ---
libnotmuch.so.3

So it also prints libnotmuch.so.3, but only because the version
installed from the Debian archive is also libnotmuch.so.3:

teythoon at thinkbox ~ % /sbin/ldconfig -p | grep notmuch
        libnotmuch.so.3 (libc6,x86-64) => /usr/lib/libnotmuch.so.3

So I guess *if* I had a libnotmuch.so.2 in my ldconfig(8) cache (-p
prints the cache), find_library would have returned libnotmuch.so.2
and not my libnotmuch.so.3. So your patch would most likely break this
kind of setup.

2. Uh. I actually looked at /usr/lib/python2.7/ctypes/util.py and I
almost got eye cancer from just looking briefly at it... The
implementation of find_library differs for various operating systems
and most (all?) of them use fork/exec some program (even gcc in some
cases) to look up the libraries. I'd like to avoid this if possible,
maybe even if this means handling OS X as a special case in our
bindings. For reference, our current solution does not require any
fork/exec calls:

teythoon at thinkbox ~ % strace -f -e trace=execve,fork,clone python -c 'import ctypes; ctypes.CDLL("libnotmuch.so.3")' 
execve("/usr/bin/python", ["python", "-c", "import ctypes; ctypes.CDLL(\"libn"...], [/* 63 vars */]) = 0

Thoughts anyone?

Cheers,
Justus
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: signature
URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20130530/bcdeb16e/attachment-0001.pgp>