BitWorking

Do we need WADL?

Everybody's atwitter about WADL, a description file for REST services, and since it's supposed to be RESTful I regularly get questioned about it. For this post I'm going to experiment and adopt Stephen O'Grady's Q&A style.

My thanks to Patrick Mueller and Dave Johnson for volunteering to be guinea pigs for my arguments, and thanks to #redmonk for the use of their channel where we initially held the discussion.

Q: Does REST need a description document?

A: No.

Q: You're new to this Q&A format, aren't you? How about giving some more details.

A: There are, as always, two problems. Those two problems are always the same: technical and cultural.

The technical problems I will get back to later. There are a number of cultural assumptions built into WADL, and those cultural assumptions come from the WS-* arena. The first is that you want, and need, machine generated code. The idea that you could get anything useful from machine generated code is, to me, interesting. If I describe an Atom Syndication Feed in WADL, how close will the generated code be to a feed reader like Bloglines? Or even to a library like Abdera? If I write a really good WADL for (X)HTML how close will the generated code be to a web browser? The point is that generated code stubs are so far from a completed consumer of a web service that the utility seems questionable.

The urge to code generate is particularly strong around XML Schema. You might believe you can talk people out of using WADL that way, but I doubt it; people will see XML Schema and will automatically look for the code generation button. That way of thinking is baked into the culture.

The second cultural hurdle is the implicit assumption that all useful data will be in the form of XML, which ignores the vast amount of non-XML content on the web today, which, outside of RSS and Atom, is all of it (HTML, CSS, JavaScript, podcasts, videos, JSON, etc.). Do I really need to bring up the painful memories of binary content and SOAP?

Q: Ok, so WADL is doomed to repeat all the mistakes of WS-*, only this time with more verbs, and that failure will be based on completely soft issues like 'culture'. That's boring. Can you go into the technical details now?

A: Sure.

Here is the very first example in the WADL specification:

<application xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://research.sun.com/wadl/2006/10 wadl.xsd"

    xmlns:tns="urn:yahoo:yn"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema"
    xmlns:yn="urn:yahoo:yn"
    xmlns:ya="urn:yahoo:api"
    xmlns="http://research.sun.com/wadl/2006/10">
    <grammars>
        <include
            href="NewsSearchResponse.xsd"/>
        <include
            href="Error.xsd"/>
    </grammars>

    <resources base="http://api.search.yahoo.com/NewsSearchService/V1/">
        <resource path="newsSearch">
            <method name="GET" id="search">
                <request>
                    <param name="appid" type="xsd:string"
                        style="query" required="true"/>
                    <param name="query" type="xsd:string"
                        style="query" required="true"/>
                    <param name="type" style="query" default="all">
                        <option value="all"/>
                        <option value="any"/>
                        <option value="phrase"/>
                    </param>
                    <param name="results" style="query" type="xsd:int" default="10"/>
                    <param name="start" style="query" type="xsd:int" default="1"/>
                    <param name="sort" style="query" default="rank">
                        <option value="rank"/>
                        <option value="date"/>
                    </param>
                    <param name="language" style="query" type="xsd:string"/>
                </request>
                <response>
                    <representation mediaType="application/xml"
                        element="yn:ResultSet"/>
                    <fault status="400" mediaType="application/xml"
                        element="ya:Error"/>
                </response>
            </method>
        </resource>
    </resources>
</application>

That WADL file is a description of a search interface. But here is how you should really do it:

<?xml version="1.0" encoding="UTF-8"?>
<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/">
   <ShortName>Web Search</ShortName>
   <Description>Use Example.com to search the Web.</Description>
   <Tags>example web</Tags>
   <Contact>admin@example.com</Contact>
   <Url type="application/rss+xml" 
        template="http://example.com/?q={searchTerms}&amp;pw={startPage?}&amp;format=rss"/>
</OpenSearchDescription>
That's an OpenSearch document, that also describes a search interface.

Q: What's the difference?

A: A mime-type.

Q: That doesn't seem like much, does it make a difference?

A: Yes, it makes a big difference. When you get an OpenSearch document there is a whole data model and a set of interactions you know are possible because you read the OpenSearch specification. By reading that spec you know how to construct search queries. When I get a WADL document it might describe anything, from how to construct a search, to the APP, to JEP, to XML-RPC.

Also, if you are writing an OpenSeach client, you will write something that accepts any OpenSearch document. That is, the server that provides that OpenSearch document can change it at any time and all the consumers of that service will 'keep up'. If it was deployed as a WADL file, there is a strong chance that clients would be compiled against that WADL, and those clients would break every time the WADL changed.

So when I say the difference is a 'mime-type', what I mean is that there is an entire spec somewhere which describes what that document means, and that meaning may include hypertext functionality, ala (X)HTML, XForms, and OpenSearch.

Q: You say people will compile against a WADL and the clients will break when the WADL changes. How do you know that?

A: You're right, I don't know that for a fact, but I do know history, and that WSDL was all about the conversion into code. I also know that WADL is so wide open that you can describe almost anything with it. For example, there is that WADL for a search interface which I just showed you, and in the same specification there is also a WADL the describes the Atom Publishing Protocol. How many clients are going to be written to dynamically consume both of those files? To handle a server switching from one to the other to describe it's interface?

Q: So there's an example of APP being described by WADL in the WADL spec, aren't they equivalent?

A: First, there are some problems with that example. The first problem is that it assumes Atom can be fully described by XML Schema. It can't. The second is the inclusion of the 'app:member-type' element, which is meaningless in WADL, for which you have to go read the APP spec, which, if you are persistent enough and actually read this whole article, you will see that makes my point.

But to answer your question, no, they are not equivalent since APP already has a whole data and service story, and the key point is that that knowledge is triggered by a mime-type. In reality it's a few mime-types: application/atom+xml, application/atomcat+xml, and application/atomsvc+xml. Regardless, those mime-types, defined in RFC 4287 and RFC (whatever APP will be), mean that those messages are 'self-descriptive'. Not 'self-descriptive' in a mind-reading sort of way, but in a 'follows RFC 2616' and 'you can look into RFC 4287 for more details' kind of way. It is 'self-descriptive' in the same way that HTML resources are 'self-descriptive'; they contain links to other resources, and in the case of forms, contain a description of a "next-state", either via POST or GET. In order to use HTML, CSS, and JavaScript you need to read those specs and implement a consumer for them, which in the case of HTML is non-trivial, and doesn't necessarily mean a browser - think search crawlers.

Q: So every new 'type' of data gets its own MIME sub-type? application/bugzilla-bug+xml and application/bugzilla-bug+json, and ...

A: If that data doesn't fit into an already existing scheme, then yes, it should get its own mime-type. Think, for example, of image types. We don't have a generic "image/bin" media-type and force intermediaries to peek inside the binary blob to figure out if you have a JPEG or a PNG. If something fits into the general form of a collection, then you should be using Atom. If something fits the general form of a search then use OpenSearch. If something fits into the general form of a form then use (X)HTML.

Note that APP just describes a general collection, and you don't have to use the whole collection idiom. For example, a full APP implementation is (list members, create member, get member, update member, retrieve member, delete member), but if you restict to just (list members) then you have syndication, which is useful in its own right. If you restrict yourself to (list members, create member), you have a read-only drop box. A collection that just supports (create member) is a blind drop box. Support only (list members, retrieve member, delete member) and you have a reviewing queue. You get the idea.

Q: So no one should be designing their own protocols?

A: I didn't say that, just that you should first look out over the protocols that are available and choose one if it is already close. If none match, then go ahead and design your own, document it well, and make sure you take the additional steps of registering mime-types for any new formats.

Q: Even if I just want to define a protocol for my server to talk to my client, which is just a web browser?

A: That's kind of out of scope. We're talking about things that scale way beyond that. If you are talking about a situation where you control both ends of the wire and no one else is expected to use that interface, then you don't need WADL do you?

Q: But APP has a Service Document, and if APP needs a Service Document, doesn't that imply that other RESTful services need something similar?

A: Other RESTful services, like OpenSearch, may need their own service description doc, but they should have their own separate mime-type, so consumers know what they "mean".

Q: I don't get that, why not define a standard way to express the introspection information?

A: First, not every RESTful service will require it's own service doc, take HTML for example. Second, if you define a 'standard' file format for such descriptions, what does that get you? You give no hint to intermediaries what that file is for, which is the same problem SOAP has, every request is POST, and every request has the media-type 'application/xml'. There's no meat for intermediaries to dig into to be able to help. For a concrete example look at the Atom Syndication Format. RFC 4287 is out there, which defines the media-type 'application/atom+xml' and part of that definition is how to do encryption and signing of Atom documents. That means it's now possible to build an Atom encrypting/decrypting and signing/checking intermediary. You can't do that for generic 'application/xml' messages.

Q: People are going to want to describe interaces, how can that be bad?

A: Yes, people want to describe interfaces, and those descriptions are brittle. If I download a WADL and compile my client today it will break tomorrow when you change the service. If, instead, you use hypertext, link following and request construction based on the hypertext and client state, then the interface won't break when you change servers, or URI structures.

Q: Atom and APP are described interfaces, does that mean that they're brittle?

A: The description of APP and Atom are in RFCs and will never change. Now, those RFCs might get deprecated with newer versions, but the old versions will never go away, and they will never change. WADL allows someone to describe a new service every day of the week, and later that week change the interface.

Q: So brittle == flexible and flexible == carved in stone? That seems odd.

A: Yes, it seems odd when you put it that way though I would point out that "flexible == a minimum base of interop carved in stone" with extensions built on top of that base.

Q: What about the Flickr API? Wouldn't that be better with a WADL?

A: First, you can't expect me to believe that if you had a carefully crafted WADL you could hand it to WADL2Java and out would pop flickrfs. Second, if that had been built on top of the APP then instead of trying to use a WADL generated client I could start with a generic APP client and build from there, and an APP library is going to get you a lot further, be a lot better tested, and better adapted to the task at hand than some generated code.

Q: You don't expect everything to be built with APP, do you?

A: No, there are applications like search for which OpenSearch is a better fit. And I'm sure there are other areas that would require their own protocols. I'm also aware of my own blindspots, so why not take a look at this list and see the kinds of applications that have been given APP interfaces.

Q: So we've talked over the pragmatics, but how does this come from REST?

A: Much of this is driven by two constraints. The first is obviously "hypertext as the engine of application state", but another one even less frequently cited is "a constrained set of media-types". Think of the two extremes. If there was only one media type, 'thing/thing', then you wouldn't be able to do anything useful. The other extreme is just as bad, if there was a huge amount of media-types floating around the web then you could never use it either. Think of trying to build a browser and having an endless list of image media types to implement. This also affects intermediaries, think of those accelerating intermediaries some dial-up provders use to reformat images on the fly so they are smaller. Those would be impossible to write unless there were a constrained set of image media types. Given the history of SOAP and WSDL, I believe WADL will be used to create a whole slew of resources with a single media type. Even worse, when you look inside them, they will probably represent a nearly infinite set of new document types. So the idea is to re-use existing protocols like OpenSearch and APP, that keeps the mime-type manageable.

Nicely put. This clearly makes the point that self-descriptive data is more important than IDL. The treatment of brittle/flexible/carved in stone didn't quite work for me. From Robert Martin's "Design Principles and Design Patterns"[1] there is a definition of Stability that I think could apply here as well. "Stability is related to the amount of work required to make a change." In that paper a software module is "stable" not because of anything inherent to itself (high quality, well proven design, ...), but simply because other things depend on it. Depending on APP is a safe thing to do because the standard is going to be very stable (because many servers and clients are going to implement and support it). Also, APP is designed with flexibility in mind; clear extension points and semantics for how to handle unknown data. [1] http://www.objectmentor.com/resources/articles/Principles_and_Patterns.pdf

Posted by john on 2007-06-05

Some comments on my blog.

Posted by Patrick Mueller on 2007-06-05

See A beeter approach than WADL :)

Posted by zimbatm on 2007-06-06

Hi Joe, excellent write-up. I wrote about the half-baked idea of a dynamic description language a while back. One potentially capable of evolving over time. I never really saw the idea through though. I get where your coming from here, the brittleness of WADL, why I'd never want to use it, but don't see that a WDL has to be that way. If depreciation and upgrade paths can be defined in protocol, providing clients with time to adapt, a persistent URI, least-amount-of-validation(preferably none) and machine/human readable documentation, then I think a protocol that doesn't require registering a new mime-type whenever an interface changes could aid in innovation and adoption. Personally I don't believe seperating the protocol spec from the resource it describes is a good idea. It should all be in one. The entire resource, interface and document should be self-describing. Not require that a client go to another resource that could itself break(Standard disappears), for information on how to interact with it.

Posted by Craig Overend on 2007-06-06

> if you had a carefully crafted WADL you could hand it to WADL2Java Well, for a lot of today's APIs (Digg, Yahoo!, Del.icio.us, Amazon E-Commerce, ...) REST Describe & Compile does at least a not too bad job. It is WADL2Anything, kind of :-) At present it supports Java, PHP 5, Python, and Ruby. However, the application is still in development and far from a 1.0 version. Try it online: http://tomayac.de/rest-describe/latest/RestDescribe.html Source code: http://rest-api-code-gen.googlecode.com/files/rest_describe_v0.3.zip Details: http://blog.tomayac.de/ViewByCategories.php?category=Work Cheers, Tom

Posted by Thomas Steiner on 2007-06-06

Craig Overend:

If […], then I think a protocol that doesn’t require registering a new mime-type whenever an interface changes could aid in innovation and adoption.

Did you read the last paragraph of Joe’s post? That was precisely his point: there should be neither a new MIME type for every single service nor a single MIME type that all services use. Your proposal seems to come so perilously close to the latter that I wonder whether you understood Joe’s point.

Personally I don’t believe seperating the protocol spec from the resource it describes is a good idea. It should all be in one. The entire resource, interface and document should be self-describing.

Can you give any reasons for this? Clients are not going to be able to infer meaning anytime soon (not before AI exists), so what you’re saying amounts to saying that no code should be reusable between services (or even between resources of a single service).

We have proof beyond any doubt that centrally defined, sufficiently specified and widely implemented standards work and that they’re a good idea. (Cf. IP, TCP, HTTP, XML, etc.) What sort of advantage do see in the model of “one media type per resource” that could outweigh both the approach’s own drawbacks and the loss of the benefits of “the internet way” we’ve had so far?

Not require that a client go to another resource that could itself break(Standard disappears), for information on how to interact with it.

You have to be… really paranoid to deploy a service on the internet and then want to protect against the eventuality that an IETF standard would disappear…

Posted by Aristotle Pagaltzis on 2007-06-06

john,

Nicely put. This clearly makes the point that self-descriptive data is more important than IDL.

Thanks!

I would have to disagree about stability though. Atom, and soon the APP, are fixed because they are carved in stone. Even if no one implemented them, those RFCs would remain unchanged. You can't believe how important that is.

Also, APP is designed with flexibility in mind; clear extension points and semantics for how to handle unknown data.

Agreed.

Posted by joe on 2007-06-06

I'm sure you know this Joe .. but it was SOAP 1.1 that had text/xml as the content type. SOAP 1.2, the real standard version, uses application/soap+xml. Bye, Sanjiva.

Posted by Sanjiva Weerawarana on 2007-06-06

I've put up a new blog entry declaring that REST relies on data contracts and not programming contracts.

Let's change the conversation from "REST doesn't need an IDL" to the more positive "REST already has contracts, and here's how...".
The discussion above is part of that, but I also really liked Alan Dean's Shopping RDF example:
To validate that the server supports the "Shop" protocol, the User Agent can test the document for the existence of the basket element, e.g. with the following XPath expression count(//shop:Basket)!=0

Notice, checking the a protocol is a data expression!

Posted by john on 2007-06-06

Joe, you claim:

If I download a WADL and compile my client today it will break tomorrow when you change the service. If, instead, you use hypertext, link following and request construction based on the hypertext and client state, then the interface won't break when you change servers, or URI structures.

If there any evidence for backing that claim up?

I'd like to know of some actual examples where people did the "right thing" (which, here means at least that they did *not* explicitly describe an interface using anything like WSDL, WADL, or, presumably, plain text), then changed servers or URI structures (though, honestly, 'URI structure' is a bit vague) *and* the "interface didn't break".

I'm not entirely sure what "interface didn't break" means, here, but I assume it means, at least, that no client code had to change in any way.

(FWIW, I think there's *value* in clients breaking when interfaces break, so long as they break loudly and obviously. I guess I'm in the Torvalds School of API evolution in that respect. But that point is orthogonal to your claim, which I find quite curious.)

Cheers,
Kendall

Posted by Kendall Clark on 2007-06-08

Kendall,

If there any evidence for backing that claim up?

Yes, this is one of those things that works so well that you will be prone to claim the examples I'm about to give are "too obvious" or "too simple".

  1. The first one is the most obvious, and you just used it. I've changed my comment form many times over the years, yet everyone is still able to use it. Even the spammers. That was until I came up with my latest markup-based captcha technique, which is so far 100% effective.
  2. I've also moved my syndication feed around a couple times, and I could easily re-arrange my URI structure of my blog and people subscribed to my feed would not notice any difference, even when clicking through to an article. (Yes, syndication is a web service, one of the earliest, simplest, and most powerful.)
  3. In the process of writing my APP server implementation for this blog I changed the URI structure a few times. I never needed to update the client. (But this goes to show how ingrained that kind of breakage is in my psyche, I panicked one time when I launched the client right after updating the server, convinced that I forgot to 'fix' the client to work with the updated server. It was a pleasant, "aha, of course it still works" moment when it come up fine.)
  4. Also, very early versions of my APP server code handled only 'content' and not 'summary'. No client code needed to change when I added 'summary' support.
  5. I'm pretty sure you could find similar stories for OpenSearch, but I don't know any first hand.

Posted by joe on 2007-06-08

Kendall,
To add to Joe's examples:
  1. I can use OpenID on any page I own to redirect to my actual authenticator.
  2. The example Alan Dean put up for Shopping
All of these examples use client side knowledge of a rich hypermedia format to loosely couple the client and server (avoiding detailed hard-coding of URIs).

Posted by john on 2007-06-08

2007-06-05