Do we need WADL?

Everybody's atwitter about WADL, a description file for REST services, and since it's supposed to be RESTful I regularly get questioned about it. For this post I'm going to experiment and adopt Stephen O'Grady's Q&A style.

My thanks to Patrick Mueller and Dave Johnson for volunteering to be guinea pigs for my arguments, and thanks to #redmonk for the use of their channel where we initially held the discussion.

Q: Does REST need a description document?

A: No.

Q: You're new to this Q&A format, aren't you? How about giving some more details.

A: There are, as always, two problems. Those two problems are always the same: technical and cultural.

The technical problems I will get back to later. There are a number of cultural assumptions built into WADL, and those cultural assumptions come from the WS-* arena. The first is that you want, and need, machine generated code. The idea that you could get anything useful from machine generated code is, to me, interesting. If I describe an Atom Syndication Feed in WADL, how close will the generated code be to a feed reader like Bloglines? Or even to a library like Abdera? If I write a really good WADL for (X)HTML how close will the generated code be to a web browser? The point is that generated code stubs are so far from a completed consumer of a web service that the utility seems questionable.

The urge to code generate is particularly strong around XML Schema. You might believe you can talk people out of using WADL that way, but I doubt it; people will see XML Schema and will automatically look for the code generation button. That way of thinking is baked into the culture.

The second cultural hurdle is the implicit assumption that all useful data will be in the form of XML, which ignores the vast amount of non-XML content on the web today, which, outside of RSS and Atom, is all of it (HTML, CSS, JavaScript, podcasts, videos, JSON, etc.). Do I really need to bring up the painful memories of binary content and SOAP?

Q: Ok, so WADL is doomed to repeat all the mistakes of WS-*, only this time with more verbs, and that failure will be based on completely soft issues like 'culture'. That's boring. Can you go into the technical details now?

A: Sure.

Here is the very first example in the WADL specification:

<application xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://research.sun.com/wadl/2006/10 wadl.xsd"

    xmlns:tns="urn:yahoo:yn"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema"
    xmlns:yn="urn:yahoo:yn"
    xmlns:ya="urn:yahoo:api"
    xmlns="http://research.sun.com/wadl/2006/10">
    <grammars>
        <include
            href="NewsSearchResponse.xsd"/>
        <include
            href="Error.xsd"/>
    </grammars>

    <resources base="http://api.search.yahoo.com/NewsSearchService/V1/">
        <resource path="newsSearch">
            <method name="GET" id="search">
                <request>
                    <param name="appid" type="xsd:string"
                        style="query" required="true"/>
                    <param name="query" type="xsd:string"
                        style="query" required="true"/>
                    <param name="type" style="query" default="all">
                        <option value="all"/>
                        <option value="any"/>
                        <option value="phrase"/>
                    </param>
                    <param name="results" style="query" type="xsd:int" default="10"/>
                    <param name="start" style="query" type="xsd:int" default="1"/>
                    <param name="sort" style="query" default="rank">
                        <option value="rank"/>
                        <option value="date"/>
                    </param>
                    <param name="language" style="query" type="xsd:string"/>
                </request>
                <response>
                    <representation mediaType="application/xml"
                        element="yn:ResultSet"/>
                    <fault status="400" mediaType="application/xml"
                        element="ya:Error"/>
                </response>
            </method>
        </resource>
    </resources>
</application>

That WADL file is a description of a search interface. But here is how you should really do it:

<?xml version="1.0" encoding="UTF-8"?>
<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/">
   <ShortName>Web Search</ShortName>
   <Description>Use Example.com to search the Web.</Description>
   <Tags>example web</Tags>
   <Contact>admin@example.com</Contact>
   <Url type="application/rss+xml" 
        template="http://example.com/?q={searchTerms}&amp;pw={startPage?}&amp;format=rss"/>
</OpenSearchDescription>

Q: What's the difference?

A: A mime-type.

Q: That doesn't seem like much, does it make a difference?

A: Yes, it makes a big difference. When you get an OpenSearch document there is a whole data model and a set of interactions you know are possible because you read the OpenSearch specification. By reading that spec you know how to construct search queries. When I get a WADL document it might describe anything, from how to construct a search, to the APP, to JEP, to XML-RPC.

Also, if you are writing an OpenSeach client, you will write something that accepts any OpenSearch document. That is, the server that provides that OpenSearch document can change it at any time and all the consumers of that service will 'keep up'. If it was deployed as a WADL file, there is a strong chance that clients would be compiled against that WADL, and those clients would break every time the WADL changed.

So when I say the difference is a 'mime-type', what I mean is that there is an entire spec somewhere which describes what that document means, and that meaning may include hypertext functionality, ala (X)HTML, XForms, and OpenSearch.

Q: You say people will compile against a WADL and the clients will break when the WADL changes. How do you know that?

A: You're right, I don't know that for a fact, but I do know history, and that WSDL was all about the conversion into code. I also know that WADL is so wide open that you can describe almost anything with it. For example, there is that WADL for a search interface which I just showed you, and in the same specification there is also a WADL the describes the Atom Publishing Protocol. How many clients are going to be written to dynamically consume both of those files? To handle a server switching from one to the other to describe it's interface?

Q: So there's an example of APP being described by WADL in the WADL spec, aren't they equivalent?

A: First, there are some problems with that example. The first problem is that it assumes Atom can be fully described by XML Schema. It can't. The second is the inclusion of the 'app:member-type' element, which is meaningless in WADL, for which you have to go read the APP spec, which, if you are persistent enough and actually read this whole article, you will see that makes my point.

But to answer your question, no, they are not equivalent since APP already has a whole data and service story, and the key point is that that knowledge is triggered by a mime-type. In reality it's a few mime-types: application/atom+xml, application/atomcat+xml, and application/atomsvc+xml. Regardless, those mime-types, defined in RFC 4287 and RFC (whatever APP will be), mean that those messages are 'self-descriptive'. Not 'self-descriptive' in a mind-reading sort of way, but in a 'follows RFC 2616' and 'you can look into RFC 4287 for more details' kind of way. It is 'self-descriptive' in the same way that HTML resources are 'self-descriptive'; they contain links to other resources, and in the case of forms, contain a description of a "next-state", either via POST or GET. In order to use HTML, CSS, and JavaScript you need to read those specs and implement a consumer for them, which in the case of HTML is non-trivial, and doesn't necessarily mean a browser - think search crawlers.

Q: So every new 'type' of data gets its own MIME sub-type? application/bugzilla-bug+xml and application/bugzilla-bug+json, and ...

A: If that data doesn't fit into an already existing scheme, then yes, it should get its own mime-type. Think, for example, of image types. We don't have a generic "image/bin" media-type and force intermediaries to peek inside the binary blob to figure out if you have a JPEG or a PNG. If something fits into the general form of a collection, then you should be using Atom. If something fits the general form of a search then use OpenSearch. If something fits into the general form of a form then use (X)HTML.

Note that APP just describes a general collection, and you don't have to use the whole collection idiom. For example, a full APP implementation is (list members, create member, get member, update member, retrieve member, delete member), but if you restict to just (list members) then you have syndication, which is useful in its own right. If you restrict yourself to (list members, create member), you have a read-only drop box. A collection that just supports (create member) is a blind drop box. Support only (list members, retrieve member, delete member) and you have a reviewing queue. You get the idea.

Q: So no one should be designing their own protocols?

A: I didn't say that, just that you should first look out over the protocols that are available and choose one if it is already close. If none match, then go ahead and design your own, document it well, and make sure you take the additional steps of registering mime-types for any new formats.

Q: Even if I just want to define a protocol for my server to talk to my client, which is just a web browser?

A: That's kind of out of scope. We're talking about things that scale way beyond that. If you are talking about a situation where you control both ends of the wire and no one else is expected to use that interface, then you don't need WADL do you?

Q: But APP has a Service Document, and if APP needs a Service Document, doesn't that imply that other RESTful services need something similar?

A: Other RESTful services, like OpenSearch, may need their own service description doc, but they should have their own separate mime-type, so consumers know what they "mean".

Q: I don't get that, why not define a standard way to express the introspection information?

A: First, not every RESTful service will require it's own service doc, take HTML for example. Second, if you define a 'standard' file format for such descriptions, what does that get you? You give no hint to intermediaries what that file is for, which is the same problem SOAP has, every request is POST, and every request has the media-type 'application/xml'. There's no meat for intermediaries to dig into to be able to help. For a concrete example look at the Atom Syndication Format. RFC 4287 is out there, which defines the media-type 'application/atom+xml' and part of that definition is how to do encryption and signing of Atom documents. That means it's now possible to build an Atom encrypting/decrypting and signing/checking intermediary. You can't do that for generic 'application/xml' messages.

Q: People are going to want to describe interaces, how can that be bad?

A: Yes, people want to describe interfaces, and those descriptions are brittle. If I download a WADL and compile my client today it will break tomorrow when you change the service. If, instead, you use hypertext, link following and request construction based on the hypertext and client state, then the interface won't break when you change servers, or URI structures.

Q: Atom and APP are described interfaces, does that mean that they're brittle?

A: The description of APP and Atom are in RFCs and will never change. Now, those RFCs might get deprecated with newer versions, but the old versions will never go away, and they will never change. WADL allows someone to describe a new service every day of the week, and later that week change the interface.

Q: So brittle == flexible and flexible == carved in stone? That seems odd.

A: Yes, it seems odd when you put it that way though I would point out that "flexible == a minimum base of interop carved in stone" with extensions built on top of that base.

Q: What about the Flickr API? Wouldn't that be better with a WADL?

A: First, you can't expect me to believe that if you had a carefully crafted WADL you could hand it to WADL2Java and out would pop flickrfs. Second, if that had been built on top of the APP then instead of trying to use a WADL generated client I could start with a generic APP client and build from there, and an APP library is going to get you a lot further, be a lot better tested, and better adapted to the task at hand than some generated code.

Q: You don't expect everything to be built with APP, do you?

A: No, there are applications like search for which OpenSearch is a better fit. And I'm sure there are other areas that would require their own protocols. I'm also aware of my own blindspots, so why not take a look at this list and see the kinds of applications that have been given APP interfaces.

Q: So we've talked over the pragmatics, but how does this come from REST?

A: Much of this is driven by two constraints. The first is obviously "hypertext as the engine of application state", but another one even less frequently cited is "a constrained set of media-types". Think of the two extremes. If there was only one media type, 'thing/thing', then you wouldn't be able to do anything useful. The other extreme is just as bad, if there was a huge amount of media-types floating around the web then you could never use it either. Think of trying to build a browser and having an endless list of image media types to implement. This also affects intermediaries, think of those accelerating intermediaries some dial-up provders use to reformat images on the fly so they are smaller. Those would be impossible to write unless there were a constrained set of image media types. Given the history of SOAP and WSDL, I believe WADL will be used to create a whole slew of resources with a single media type. Even worse, when you look inside them, they will probably represent a nearly infinite set of new document types. So the idea is to re-use existing protocols like OpenSearch and APP, that keeps the mime-type manageable.