We came, we saw, we did, we got spanked, we did it right.

Last week Mark Pilgrim and I released an implementation of the AtomAPI, both a client and a server. That implementation included a new authorization scheme that we came up with. Now we would have liked to used HTTP Digest authentication, and the AtomAPI should support Digest authentication, but for many users setting up Digest just isn't possible. Many users, like myself, are using a server that does not have Digest authentication turned on. Similarly not everyone has the ability to use .htaccess files, which you need to be able to modify to setup Digest autentication, that or modify your servers httpd.conf which is even rarer still. However we heard from CMS vendors that they want at least the level of password security that HTTP Digest offers. So we needed to come up with a scheme that:

1. Is a challenge-response Digest authentication scheme.
2. Is able to be handled by a CGI program with no .htaccess, httpd.conf tricks or requirements to be running as an Apache module, ala mod_perl or mod_python.
3. Gracefully extends current authentication schemes.
4. Is the simplest thing that could possibly work.

So what we did was a simple transposition of Digest authentication into custom HTTP headers:

That authorization scheme was a variant of the RFC 2617 Digest authentication with the following changes:
1. Used sha1 instead of md5.
2. Triggered authentication by rejecting a request with an HTTP status code of 447 instead of 401.
3. The server response when rejecting a request with a 447 also included an Atom-Authenticate: header that included the 'nonce'. This parallels the role of the WWW-Authenticate:  header in HTTP Digest.
4. The client sends an Atom-Authorization: header with all the Digest authentication information in it.
5. The server sends back an Atom-Authentication-Info: header with the 'nextnonce'.

Now this was a good first pass but we did several things wrong as Ken Coar pointed out on his site and on the atom-syntax mailing list. Basically it boils down to the fact that we ignored the built-in extensiblity of the HTTP authenciation mechanism and went around it starting with returning an HTTP status code of 447.

What we should have done:

1. Triggered an auth by rejecting a request with an HTTP status code of 401.
2. The server response includes an Authenticate: header that includes Atom as an authentication scheme.
3. The client then sends an Authorization: header with the scheme of Atom with all the Digest authentication information going into X-Atom-Authorization: header.
4. With every request the server sends back an X-Atom-Authentication-Info: header with the 'nextnonce'.

Note that this now uses the extensibility of the HTTP authentication scheme. When the server responds with a WWW-Authenticate: header, that header lists, in order of preference, the authentication schemes that the server supports. The client then selects the most preferred scheme that it also knows. Note that the whole authentication mechanism is designed to be robust to the addition of new/unknown schemes. We are leveraging this extensibilty by passing in an authentication scheme to Apache that it doesn't know about. The Apache server reacts appropriately by passing the request onto the CGI program to give it a chance to handle the authentication.

That is exactly what we did, and now have released new versions of the client and server that use the new authentication scheme. Just to reiterate that this scheme is another HTTP authentication scheme that works alongside Basic and Digest authentication. Servers can support one or more of Atom, Basic and Digest authentication. Note that the server side has changed so the old client will no longer be able to create or update entries. Sorry about that, such is life on the bleeding edge.

Here the details of how the Atom authentication scheme works:

1. The client tries to do something that requires authentication, for instance, POSTing a new entry to http://diveintomark.org/cgi-bin/atom.cgi/blog_id=14. The server sends back and HTTP error code of "401 Unauthorized", and a WWW-Authenticate header like this:

WWW-Authenticate: Atom realm="some server realm name", qop="atom-auth", algorithm="SHA", nonce="some unique server-specific value"

2. The client takes the username, the realm given by the server, and the password, and concatenates them to create an intermediate value which we will call A1:

A1 = username + ":" + realm + ":" + password

3. The client takes the HTTP verb it wants to use (in this case "POST") and the path part of the URL it wants to post to (in this case "/cgi-bin/atom.cgi/blog_id=14"), and concatenates them into an intermediate value which we will call A2:

A2 = verb + ":" + uri

4. The client creates a unique client-specific value, which we will call "cnonce". How this happens is completely client-specific, but it should change on every request, and future values should not be guessable.

5. The client takes A1, A2, the qop given by the server, the nonce given by the server, and the cnonce created by the client, and creates a digest, which we will call "response":

response = sha(sha(A1) + ":" + nonce + ":" + "00000001" + ":" + cnonce + ":" + qop + ":" + sha(A2))

6. The client resends its original request, with the addition of two headers. First the X-Atom-Authentication header with all of the following values filled in:

X-Atom-Authentication: Atom username="...", realm="...", nonce="...", uri="...", qop="atom-auth", nc="00000001", cnonce="...", response="..."

N.B.Fixed the above line, which should use Atom instead of Digest.

And then the Authentication header signalling that we are using the Atom authentication scheme:

Authentication: Atom key="value"

Note that the key-value pair after the 'Atom' have no significance for us as Apache will not pass down the Authentication: header to the CGI program even when it is an authentication scheme it doesn't know, which is why all the real useful information was moved into X-Atom-Authentication: which will get passed to a CGI program.

7. If the username/password is not valid, the server will respond with an HTTP error code 403, and a new WWW-Authenticate header, and the client starts all over.

If the client screwed something up (forgot a value, sent a malformed authentication request), the server will respond with an HTTP error code 400, and a new WWW-Authenticate header.

If the client successfully authenticated, the server will do what the client asked (in this case, post a new entry). Every subsequent response from the server may contain an X-Atom-Authentication-Info header that includes a "nextnonce" value. If present, the client must discard the previous nonce value and use the new nonce value to recalculate the digest response on the next request. (This protects against replay attacks.) The client should cache and reuse the other values given by the server (realm, qop, algorithm). Either way, only one extra round trip is required per session (before the first action, to get the initial authentication challenge). The client does not need to do additional round trips once they have successfully authenticated, as long as they stay current with their nonce values.

If the server does not return a new nonce value, the client should continue using the old one, and increment the value of nc as a hexadecimal number, and recalculate the digest response. So on the second request, the client would recalculate the response like this:

response = sha(sha(A1) + ":" + nonce + ":" + "00000002" + ":" + cnonce + ":" + qop + ":" + sha(A2))

Many many thanks to Ken Coar for working patiently with Mark and I on fixing our intial version into the better scheme we have just released. Any errors and omissions lie solely with Mark and I. Thanks again Ken!

Update 2 - 27-Aug-2003: Fixed the response calculation in step 7 which was missing the outer sha() around it.

Posted by Greg Reinacker on 2003-08-25

Could someone please explain to me why this is better than plain old Digest Authentication?  I'm guessing it's something like "CGI programs can read the 'X-Atom-Authentication' but not the 'Authentication' header when not running as an Apache module (for some obscure reason)."  I really would like to know though.

EDIT:  Ok I really should read the article better before commenting, since it just confirms what I just wrote.  Bad Xiven.

Posted by Xiven on 2003-08-25

The RFC 2617 tragedy

I have three times implemented RFC-2617 Digest authentication, and each time ended up unable to use it.

The problem is that web-server authors implement Digest Authentication without providing any way to plug in the user database of the application in question.  With Apache I need to edit the .htaccess files by hand for each user, for Microsoft IIS I must enter all users in to the Windows user database (!).  With IIS it is at least often possible to turn off its authentication features (by claiming disingenuously that all users are anonymous).

The problems with authentication and Atom probably apply to all REST applications -- perhaps some generic solution should be sought (or Apache fixed)?

Posted by Damian Cugley on 2003-08-26

Damian,
It looks like we have both run into the same exact problems. This authentication mechanism was designed to solve these types of problems. You may notice that there is nothing specific to the Atom syntax or syndication in our proposal. Feel free to use it other situations as you see fit.
Of course, my number one preference would be a solution that used Digest authentication as it is defined in RFC 2617 and is usable by everyone from with their CGI programs, without requiring .htaccess or httpd.conf access.

Posted by Joe on 2003-08-26

Is this a typo?

In step 5 you have:

response = sha(sha(A1) + ":" + nonce + ":" + "00000001" + ":" + cnonce + ":" + qop + ":" + sha(A2))

while in step 7 you have:

response = sha(A1) + ":" + nonce + ":" + "00000002" + ":" + cnonce + ":" + qop + ":" + sha(A2)

In step 7, the whole response string should be SHA'ed, right?

Posted by Stewart Johnson on 2003-08-27

Stewart,
Good catch. Yes, the whole response should be sha'd. I have updated the document.

Posted by Joe on 2003-08-27

Damien,

That is hardly a problem with REST. I use custom digest authentication all the time and it works fine. Yes, I don't use Apache but designing protocols and standards around the flaws in one piece of software is silly. Atom really should not invent its own authentication scheme. Authentication is such a tricky problem and there a thousands of custom solutions out there. Requiring atom servers to handle some specific, new, and highly redundant scheme that's not-quite-digest would be very, very wrong in my opinion. I certainly wouldn't implement it. Leave authentication out of the spec and let people do whatever they need to do to get the job done.

Posted by Bo on 2003-08-29

Bo, generally this isn't about servers, it's about clients.  If you have a web server where your weblog software can use server-based Basic or Digest authentiation, you're done.  Weblog server software does not need to implement Atom-Digest if they don't want to.

Certain weblog developers, on the other hand, feel they need to because the market for their software for the most part does not allow their software to use Basic or Digest authentication.  So they will be the ones to implement Atom-Digest server-side.

Clients will need to support the standard Basic and Digest methods (which their libraries probably already do), and also Atom-Digest.

In theory, this might not have needed to be in the core for 1.0, but several weblog developers have stated this is a "showstopping" feature for them.

Posted by Ken MacLeod on 2003-08-29

So don't roll your own custom authentication system.... Roll your own custom Digest authentication system.  Is there a technical reason why you can't just do Digest auth by yourself?

If you can control the HTTP headers you should be able to do this.

Kevin

Posted by Kevin Burton on 2003-09-03

Kevin,
Yes, because you can not handle Digest authentication in a CGI script under Apache. Apache eats all the auth headers and does not pass them to the CGI script. Ever. Well, there are two exceptions to this, one is your are running as a module, ala mod_perl, or the second is you recompile Apache with the right flag set. Niether of which I would classify as a viable solution for the intended audience.

Posted by Joe on 2003-09-03

Damian,

I don't know why you are using a .htaccess file for every use in order to do digest authentication in apache - it is unnecessary. A file may be used instead and using other sources (including databases) is quite easy...

Posted by JoeJoe on 2003-09-04

Just wanted to note some progress on the Atom API support in blojsom. My blog entry about it and Mark's blog entry about it. In short, we're moving along quite well with the current specification, Sandler, and the wxAtomClient.

Posted by David Czarnecki on 2003-09-10

Hi,
I'm implementing a system with Blojsom. But I have a problem with Atom-Authentication: my users are stored on a LDAP server and their password are already hashed with SHA1. With proposal alghoritm (and with standard digest authentication) there is no way to compare them.

response = sha(sha(A1) + ":" + ...

an alternative way can be

response = sha(A1 + ":" + ...

However, with LDAP a reversible algorithm will be better.

Posted by Marco on 2003-10-11

try at http://wanderware.com/ for

free edition uses text file
others use SQL database

Posted by iis digest authentication on 2003-11-24

Apache eats all the auth headers and does not pass them to the CGI script. Ever. Well, there are two exceptions to this, one is your are running as a module, ala mod_perl, or the second is you recompile Apache with the right flag set.

Posted by Michael Mantik on 2003-12-04

Happy New Year!

Posted by Alex on 2004-01-01

Does apache really eat all the auth headers? Is there no way round this? (Not even if I kick the box or something?) What flag can be set to recompile apache to make this work?

http://www.experts-exchange.com/Web/Web_Servers/Apache/Q_21500706.html

Posted by Dave k on 2005-07-22

Dave K,
I have it on good authority that there is a compile time flag in Apache that allows auth headers through, though I  don't have any first hand experience myself.

Posted by Joe on 2005-07-22