one-time-iterators

[Patrick Totzke]

Excerpts from Austin Clements's message of Thu May 26 22:43:02 +0100 2011:
> http://notmuch.198994.n3.nabble.com/notmuch-s-idea-of-concurrency-failing-an-invocation-tp2373468p2565731.html
ah, good old peterson :P thanks.

> > > Though, Patrick, that solution doesn't address your problem.  On the
> > > other hand, it's not clear to me what concurrent access semantics
> > > you're actually expecting.  I suspect you don't want the remaining
> > > iteration to reflect the changes, since your changes could equally
> > > well have affected earlier iteration results. 
> > That's right. 
> > > But if you want a
> > > consistent view of your query results, something's going to have to
> > > materialize that iterator, and it might as well be you (or Xapian
> > > would need more sophisticated concurrency control than it has).  But
> > > this shouldn't be expensive because all you need to materialize are
> > > the document ids; you shouldn't need to eagerly fetch the per-thread
> > > information.  
> > I thought so, but it seems that Query.search_threads() already
> > caches more than the id of each item. Which is as expected
> > because it is designed to return thread objects, not their ids.
> > As you can see above, this _is_ too expensive for me.
> 
> I'd forgotten that constructing threads on the C side was eager about
> the thread tags, author list and subject (which, without Istvan's
> proposed patch, even requires opening and parsing the message file).
> This is probably what's killing you.
> 
> Out of curiosity, what is your situation that you won't wind up paying
> the cost of this iteration one way or the other and that the latency
> of doing these tag changes matters?

I'm trying to implement a terminal interface for notmuch in python
that resembles sup.
For the search results view, i read an initial portion from a Threads iterator 
to fill my teminal window with threadline-widgets. Obviously, for a
large number of results I don't want to go through all of them.
The problem arises if you toggle a tag on the selected threadline and afterwards
continue to scroll down.

> > > Have you tried simply calling list() on your thread
> > > iterator to see how expensive it is?  My bet is that it's quite cheap,
> > > both memory-wise and CPU-wise.
> > Funny thing:
> >  q=Database().create_query('*')
> >  time tlist = list(q.search_threads())
> > raises a NotmuchError(STATUS.NOT_INITIALIZED) exception. For some reason
> > the list constructor must read mere than once from the iterator.
> > So this is not an option, but even if it worked, it would show
> > the same behaviour as my above test..
> 
> Interesting.  Looks like the Threads class implements __len__ and that
> its implementation exhausts the iterator.  Which isn't a great idea in
> itself, but it turns out that Python's implementation of list() calls
> __len__ if it's available (presumably to pre-size the list) before
> iterating over the object, so it exhausts the iterator before even
> using it.
> 
> That said, if list(q.search_threads()) did work, it wouldn't give you
> better performance than your experiment above.
> 
> > would it be very hard to implement a Query.search_thread_ids() ?
> > This name is a bit off because it had to be done on a lower level.
> 
> Lazily fetching the thread metadata on the C side would probably
> address your problem automatically.  But what are you doing that
> doesn't require any information about the threads you're manipulating?
Agreed. Unfortunately, there seems to be no way to get a list of thread
ids or a reliable iterator thereof by using the current python bindings.
It would be enough for me to have the ids because then I could
search for the few threads I actually need individually on demand.

Here is the branch in which I'm trying out these things. Sorry for the
messy code, its late :P
https://github.com/pazz/notmuch-gui/tree/toggletags

/p
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20110526/a8045534/attachment.pgp>