The attached script, when edited to fill in valid LiveJournal
credentials (or InsaneJournal, if you change the URL and user list)
fails for me in a very odd way. It fetches the first two URLs quite
happily, and returns a 401 on the third after querying my password
manager about 6 times (always the third, no matter what the user list
is). Rebuilding the opener each time around the loop doesn't fix it.
Recreating the HTTPDigestAuthHandler (and then rebuilding the opener)
*does* fix it. Seems like any given instance of HTTPDigestAuthHandler
fails on the third hostname.

Python package version 2.5.1-5ubuntu5.2. Haven't tried it on newer versions.

Chris,
Is there a chance that this is some sort of protection on LJ's side?
Does a given instance mean the same connection being reused? What
happens with longer sleeps?

When I say a given instance, I mean an HTTPDigestAuthHandler object. If
I take the three lines before my loop (authhandler=, opener=,
install_opener) and move them into the loop, the problem goes away,
whereas moving only the latter two of those doesn't help. So, creating a
new HTTPDigestAuthHandler each time fixes the problem, where surely it
shouldn't make any difference? Also, I've tried sleeps of 30 seconds,
with the same results (reuse handler -> fail on the 3rd request, new
handler each time -> works for all requests).

Hmm, notice that AbstractDigestAuthHandler handles retries: 

class AbstractDigestAuthHandler:
    def __init__(self, passwd=None):
        ...
        self.retried = 0
        ...
    def reset_retry_count(self):
        self.retried = 0

    def http_error_auth_reqed(self, auth_header, host, req, headers):
        authreq = headers.get(auth_header, None)
        if self.retried > 5:
            # Don't fail endlessly - if we failed once, we'll probably
            # fail a second time. Hm. Unless the Password Manager is
            # prompting for the information. Crap. This isn't great
            # but it's better than the current 'repeat until recursion
            # depth exceeded' approach <wink>
            raise HTTPError(req.get_full_url(), 401, "digest auth failed",
                            headers, None)
        else:
            self.retried += 1
        if authreq:
            scheme = authreq.split()[0]
            if scheme.lower() == 'digest':
                return self.retry_http_digest_auth(req, authreq)


The fact that your password manager is queried six times suggests you're
seeing the "if self.retried > 5:" above. 


Could you make a local copy of /usr/lib/python2.5/urllib2.py
 and add some prints (specially around get_authorization) to see what is
going on?

Reasonable idea, Daniel, but changing self.retried makes no difference.
What does solve Chris' problem (which I can reproduce, btw) is
preventing the nonce_count from incrementing.

i.e. comment out the line :
 self.nonce_count += 1
in AbstractDigestAuthHandler.get_authorization

This makes me wonder if all this is being caused by LJ, intentionally or
not, rejecting repeated requests.

Actually, this does look more like an urllib2 bug.
According to RFC 2617, the nonce count should represent the number of
requests sent with a particular nonce. But we don't reset the nonce
count when we start using a new nonce. That discrepancy in nonce counts
causes LJ to reject the connection.

Why does it fail the third time, rather than the second? See the LJ code
from http://code.sixapart.com/svn/livejournal/trunk/cgi-bin/ljlib.pl
(which I *think* applies here, but could easily be wrong):
    # check the nonce count
    # be lenient, allowing for error of magnitude 1

Attaching a patch. I don't have a test to go with it, except for the one
submitted with the bug report.

Interesting issue. RFC 2617 supports the claim. In RFC 2617, section
3.2.2 The Authorization Request Header, we see that nonce-count is
maintained for each particular nonce value and it can used by the server
to verify the replays.

"""
The nc-value is the hexadecimal count of the number of requests
(including the current request) that the client has sent with the nonce
value in this request.  For  example, in the first request sent in
response to a given nonce value, the client sends "nc=00000001".  The
purpose of this  directive is to allow the server to detect request
replays by  maintaining its own copy of this count - if the same
nc-value is  seen twice, then the request is a replay. 
"""

resetting the nonce_count for each new nonce is the correct thing to do.
Patch seems okay and I going ahead with that.

We don't have test for HTTPDigestAuth yet, the nonce tests should be a
part of that.

Fixed in r76288 (trunk), r76289 (release26-maint), r76290 (py3k) and
r76291(release31-maint). The issue is fixed. I am keeping it open to
remind of the pending tests (HTTPAuthDigest and nonce value check testcase).

Ping?

Okay, This vaguely got out of my mind. Shall come with the tests for HTTPAuthDigest.

Are the tests for HTTPAuthDigest still outstanding?