Content Negotiation Considered Harmful, Again

Elliotte Rusty Harold

Apparently Google does not recognize XHTML, at least not when served as application/xhtml+xml. Try this search which should return exactly one hit pointing to an XHTML document. Notice that the file format is “unrecognized” and they offer to let you “View it as HTML”.

Ouch. I appear to be suffering from the same fate. Oddly enough, Sam doesn't seem to have that problem. The difference between us may be in content negotiation. Here is my code for determining when to use application/xhml+xml or text/html:

# Only serve XHTML to those clients that can understand it.
if serialization in matching:
        best = mimeparse.best_match(matching.keys(), environ.get('HTTP_ACCEPT', 
        'application/xhtml+xml'))
        (contenttype, serialization) = (best, match[best])
if serialization == 'xhtml' and environ.get('HTTP_USER_AGENT', '').find("MSIE") >= 0:
        (contenttype, serialization) = extensions['html']

Note the great bit of code for detecting and compensating for IE, which sends the equivalent of "*/*" in its Accept: header, yet doesn't handle XHTML.

The mimeparse module follows the algorithm given in RFC 2616 for matching the media type against the Accept: header, and it is covered in detail in my XML.com article, "Just use media types?".

Compare this to Sam's detection algorithm, which is implemented in his .htaccess file:

RewriteCond %{HTTP_ACCEPT} application/xhtml\+xml
RewriteCond %{HTTP_ACCEPT} !application/xhtml\+xml\s*;\s*q=0\.?0*(\s|,|$)
RewriteCond %{REQUEST_URI} \.html$
RewriteRule .* - [T=application/xhtml+xml;charset=utf-8]
RewriteCond %{HTTP_ACCEPT} application/xhtml\+xml
RewriteCond %{HTTP_ACCEPT} !application/xhtml\+xml\s*;\s*q=0\.?0*(\s|,|$)
RewriteCond %{REQUEST_URI} !\.
RewriteRule .* - [T=application/xhtml+xml;charset=utf-8]

The difference being that Sam will serve up his content as application/xhtml+xml only if that media type appears explicitly in the Accept: header, and it's not followed with a zero or fractional quality parameter. This neatly fixes the problem for IE and apparently also for the Google crawler. If I were a gambling man I'd wager that Google also sends Accept: */*, and just like IE, doesn't understand XHTML.

I've warned against using content negotiation before and this is just another example of the kinds of problems it can cause.

At this point I will probably drop back to text/html for all my pages and explore other means of displaying SVG, or convert the SVG into PNGs on the fly.