Yeah, this is weird.

I can no longer sign in on Stack Overflow, and trying a couple of other services didn't work either, e.g. I couldn't sign up to OpenStreetMap or vox.com, as they said "invalid credentials" or something along those lines.

Though this test site does seem to work fine: https://openid-consumer.appspot.com/

Also happening for me. Stack Overflow won't let me on at all any more ("Message signature was incorrect.") Several times in a row.

I tried several times in a row because a site that I built myself (using perl's Net::OpenID::Consumer) has recently been failing intermittently. I might get a few failures in a row but then on one try I get in. The message is a little more helpful:

Error: OpenID failure: naive_verify_failed_return: Direct contact invalidated ID provider response.

That seems like it might be enough to dig into the issue, which I haven't done yet, but it does seem to have something to do with IndieAuth ... most other users of the site I made are using pip.verisign.com and they are not reporting any trouble to me.

I wouldn't be surprised if it had to do with changes in signature algorithms offered/accepted that might have been made recently (or something really simple like the expiration of a cert?). Only guessing at this point.

I have also been delegating to github. I wonder if they have a load sharing arrangement where, for some time this summer, some servers had been updated with new certs or crypto policies and others had not? That would explain how I was having intermittent failures before. Today it seems to be a hard failure. I signed out of my own site in order to copy/paste the error message above, and now it doesn't look like I'm going to get back in easily.... :(

-Chap

Hmm, no, it's still intermittent at my own (Net::OpenID::Consumer-based) site ... after n failed attempts, I have succeeded again. Hmm.

The consumer code that reports naive_verify_failed_return is in Consumer.pm around lines 921 to 995, where it does roughly this:

if ($assoc) {
    $self->_debug("verified_identity: verifying with found association");
    ....
} else {
    $self->_debug("verified_identity: verifying using HTTP (dumb mode)");
    # didn't find an association. have to do dumb consumer mode
    # and check it with a POST}
    ... # does a POST back to indieauth echoing the request contents and signature
    ... # parses the response
    return $self->_fail("naive_verify_failed_return") unless
        $args{'is_valid'} eq "true" || # protocol 1.1
        $args{'lifetime'} > 0; # DEPRECATED protocol 1.0
}

I need to get some logging going to see what's failing (and what is succeeding, when it succeeds). Until then, I won't know if the lucky requests succeed because the association is found and the whole POST check is skipped, or because the check is always done but sometimes succeeds. I can see in the http server logs that there is an assoc_handle among the request parameters, so why the second code branch is sometimes taken, I still have to find out.

It appears to be indieauth's own load sharing arrangement!

It turns out my app is always taking the second ("dumb") branch and re-contacting indieauth to confirm the signature, because it wasn't configured with an association cache. The cache would probably make it faster and more robust, and I'll try that out and send it upstream. But, the dumb method ought to also work.

So far, I've tried signing in to my app a whole bunch of times, and when it recontacts indieauth for the verification, it has been getting responses from 3 different servers:

Client-Peer: 45.79.83.22:443
Client-SSL-Cert-Subject: /C=US/CN=p3k.indieauth.com/[email protected]

Client-Peer: 104.236.101.11:443
Client-SSL-Cert-Subject: /C=US/CN=yeti.indieauth.com/[email protected]

Client-Peer: 173.230.155.197:443
Client-SSL-Cert-Subject: /OU=Domain Control Validated/OU=PositiveSSL/CN=indieauth.com

The responses from yeti come back is_valid:true while from the other two servers I get is_valid:false.

It took me a great many attempts just now before I hit yeti again and got into my own app.

Maybe that means I can eventually get in at stackoverflow too, if they don't count attempts and lock me out first.

I haven't delved deeply enough into the protocol to know whether your load-sharing machines just need to have some common secret they share so that any one of them can handle an openid.mode=check_authentication request and confirm the signature was good, or if you would actually have to keep a cache of recently issued auths in sync between them, or proxy a request back to the server that can verify it. With any luck maybe it's just the first case, and the servers somehow got set up with different secrets when they need to be the same. I probably can't help much more with that part. :)

One other tangential nit:

These servers seem to be returning certificate chains that include the trusted root. For example, p3k is sending back this chain:

$ bin/openssl s_client -connect 45.79.83.22:443 -servername indieauth.com
...
Certificate chain
 0 s:/C=US/CN=p3k.indieauth.com/[email protected]
   i:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class 1 Primary Intermediate Server CA
 1 s:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Class 1 Primary Intermediate Server CA
   i:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Certification Authority
 2 s:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Certification Authority
   i:/C=IL/O=StartCom Ltd./OU=Secure Digital Certificate Signing/CN=StartCom Certification Authority

where cert 0 is the end-entity cert for p3k, which the client doesn't already have, and is signed by cert 1, which the client also doesn't already have so it's naturally included in the chain, and it is signed by cert 2, the StartCom CA, which clients already know about and trust, so cert 2 doesn't have to be sent in the chain.

It serves no purpose to send the root cert in the chain: if it's not already in the client's known-trusted certs, the client's not going to trust it any more because you sent it. For most clients it's considered a harmless redundancy, but openssl 1.0.1i (at least) actually flags it as an error:

verify error:num=19:self signed certificate in certificate chain

so to keep all clients happy it would be better to have your servers only send the server's own end-entity cert plus the intermediate cert, but not the root one.

I currently have a special-case in my DNS resolver to make indieauth.com always resolve to yeti, and signins are consistently working now in my own app. (Too bad I can't go hack stackoverflow's resolver the same way...) I hope this ends up being easy to fix and I can take that special case back out before I forget I put it there! :)

oh boy, thanks for tracking this down everyone. I'll have to figure out the best way to handle this. I'm not sure I have the ability/time/desire to modify how the OpenID Ruby gem stores state to work in this decentralized way, but I might be able to do something where I force the main indieauth.com server to handle all OpenID verifications.

The load balancing method that indieauth.com uses now works by having the DNS for indieauth.com resolve to multiple different servers, none of which share any common backend. Any information that needs to be shared between requests is sent encrypted to the client so that the client passes it off to the next server. The nice thing is it makes it easy to add and remove backend servers, and they don't rely on a shared service.

Thanks for looking into it. I guess what I (thought I) understood about the protocol was as you describe: if there's an assertion from indieauth being passed around, it's been signed, and the openid.mode=check_authentication request would just be a "hey, is this your signature?" request (because it's done in the absence of any cached association to make me confident the signing key was really yours). And if it's just a "hey, is this your signature?" request, then I'd suppose any of your servers could answer the question on sight, as long as they all know the same secret keys.

So I might have to read further into the protocol to see if it really is something more complicated, like a "hey, do you remember signing this recently?" question. That would be the tricky sort.

if it's just a "hey, is this your signature?" request, then I'd suppose any of your servers could answer the question on sight, as long as they all know the same secret keys.

Heck, if it's just an "is this your signature" question, they should be able to answer it as long as they all agree what their public keys are.

But maybe I'm on the wrong track thinking in terms of keypairs ... I remember Diffie-Hellman crops up in the protocol somewhere, so we may be talking about a brand new key that was just agreed to between consumer and provider, and so that's what would have to be available to the other load-sharing servers?

a brand new key that was just agreed to between consumer and provider

and with the beauty of DH being that consumer and provider just agreed on a new key without it ever appearing on a wire between them, 'twould seem a shame to immediately stick it on a wire between that server and its peers....

Now yeti is also returning is_valid:false and I am dead in the water.

ok, so I took my special case out of the DNS resolver, and now p3k is letting me in.

... not consistently, though ... demo this evening was less awesome than it could have been ...

@jcflack
Do you think using a not load-balanced, separate subdomain like openid.indieauth.com makes sense to fix this?

Couldn't hoit. Unless one server would be swamped by the volume of traffic from all the bazillions of indieauth openid users....

heh "bazillions". But actually now that almost all the other OpenID services shut down, I'm seeing a lot more people using indieauth.com

plus if I have to try 40 times in a row to get into an app (that happened the other day) and it isn't only happening to me, that could be exaggerating the normal load a bit....

@aaronpk
Have you rolled out a fix in this regard already? Which?
It it working for me now!

Still no joy for me getting in at stackoverflow.

I thought it was fixed a few days ago, because I got in at SO. Now I can't, again.

I suppose I just got lucky once.

I'm pretty sure I know what the problem is now. What's happening here is if your DNS is cached or resolves to the same server in between requests then it works fine. If your DNS returns one of the other servers then it fails. So it's definitely the kind of thing that works some of the time and not others, apparently at random. I'm still debating which is the better fix, either using the session cookie to store state or forcing all OpenID requests to a specific server. Will post back soon!

I just set up a non-load balanced DNS for this, https://openid.indieauth.com/

The easiest solution (for now) is if you change your openid delegate to https://openid.indieauth.com/openid. I'm not super happy about this because it means everyone using this needs to make the change. However, it is a fix and it works right now.

I don't think client-side cookies are going to be an option after all, since apparently the openid server needs to keep track of whether a nonce has already been used.

My one other thought is to somehow have all the IndieAuth.com servers reverse proxy all /openid requests back to the openid server, but that will take some more time to figure out. In the mean time, this fix works!

Hmm ... my results aren't so good.

Over at stackexchange, I'm getting the same message as always: "Message signature was incorrect." Perhaps I don't know for certain that the problem at stack.* was exactly the same problem I could trace in my own app ... since I can't trace theirs. It seemed likely it was the same, but maybe something else is also not quite right.

Meanwhile, back at my own app that I can trace, I now have a new problem:

[DEBUG Net::OpenID::Consumer] semantic info (http://anastigmatix.net/) = openid2.provider => https://openid.indieauth.com/openid, openid2.local_id => http://anastigmatix.net/
[DEBUG Net::OpenID::Consumer] Server is https://indieauth.com/openid
[DEBUG Net::OpenID::Consumer] fail(server_not_allowed) None of the discovered endpoints matches op_endpoint.

So ... it goes to my homepage, from which it clearly reads the updated URL. There isn't any other occurrence of indieauth in that file anywhere. But somewhere in the process of contacting https://openid.indieauth.com/openid it seems to get the idea that it has reached https://indieauth.com/openid instead, and then complains about the mismatch.

From what I quickly grep, op_endpoint is an attribute in a reply from the OP, identifying itself. Could there be a place where you didn't change that to match the new domain name?

You're totally right, there is a config setting for it. I just updated it to tell it about openid.indieauth.com, so can you try again? I guess my one test last night at stackoverflow was not representative.

Cool, I just got in both places. (Once ... I'm sure the trauma will still be with me for my next several logins, I'l be clicking the mouse and thinking will it won't it will it won't it will it won't it ... but once is good, for a start. :)

haha well that's a good start! I'll leave this open for a while longer and see how things go.

@aaronpk
Thanks a lot for handling this! 🌹

This works! Thank you!

One thing that took me a moment to figure out is when @aaronpk says "change your openid delegate to openid.indieauth.com/openid" in the comment above, it means to change your openid.server, not openid.delegate. So, something like:

<link rel="openid.server" href="https://openid.indieauth.com/openid" />
<link rel="openid.delegate" href="http://yoursite.com" />