Testing the Atom API

A spec is only as good as the test suite it ships with.

With that in mind, here's a first pass at what I would like to see a test suite test on an Atom API implementation. How about filling up the comments with all the stuff I missed? Thanks.

Introspection/Discovery

• Find a link to the FeedURI (rel="service.feed") on the web page
• Find a link to the FeedURI in the auto-discovered Atom feed on the same page.

• Confirm it is a valid Atom Feed.
• Find it's PostURI (rel="service.post" in \feed)

PostURI

Start with a perfectly valid and complete Atom Entry fit for POSTing to a PostURI. We'll call that perfectly valid and complete entry the Canonical entry. Cycle through each test in the table below. After each step, reset the Atom Entry to the Canonical entry. The 'ex' namespace used below is declared as xmlns:ex='http://example/com'.

For each step in the test, there is an XPath Action. This is a shorthand that uses XPath notation to define how the Canonical entry is to be modified. Here are some examples:

insert \entry\modified="2003-06-31T23:58:00"

Means to insert a modified element as a child of the entry element, and set it's value to be 2003-06-31T23:58:00.

delete \entry\link

Remove the link child of entry.

Status Code is the HTTP status code that the server should respond with when receiving such a request.

Response Constraint defines further tests on the response sent back from the server, beyond just the HTTP status code.

XPath Action Status Code Response Constraints
(none) 201 [1] [4]
\entry\id="" 400 [3]
\entry\id 400
\entry\title="" 201 Confirm \entry\title in response is empty.
delete \entry\title 400
\entry\summary="" 201 Confirm \entry\summary in response is empty.
delete \entry\summary 201 [5]
\entry\content\@type='text/xml' 201 Confirm that attibute is set the same in the response.
\entry\content\@type='gidderish/does-not-exist' 400
\entry\content\@mode='xml' \entry\content\@type='application/xhtml+xml' 201 Confirm both attrs are set in the response.
\entry\content="" 201 \entry\content==""
delete \entry\content 201 \entry\content=="" or there is no \entry\content
\entry\issued="" 201 \entry\issued != ""
delete \entry\issued 400
insert \entry\modified 400
\entry\created="" 201
delete \entry\created 201
\entry\created="" 201
delete \entry\created 201
delete \entry\author 201
\entry\author\name="" 400
\entry\author\name="some-non-existent-author-name" 201
delete \entry\author\name 400
\entry\author\url="" 400
delete \entry\author\url 201
\entry\author\email="" 400
delete \entry\author\email 201
\entry\author\email="http://example.org" 400
delete \entry\contributor 201
\entry\contributor\name="" 400
\entry\contributor\name="some-phony-name" 201
delete \entry\contributor\name 400
\entry\contributor\url="" 400
delete \entry\contributor\url 201
\entry\contributor\email="" 400
delete \entry\contributor\email 201
\entry\contributor\email="http://example.org" 400
\entry\generator="" 201
delete \entry\generator 400
insert \entry\ex:newtag 201 May or may not appear in the response

Notes

1. All 201 responses MUST include a Location: header.
2. Complete means all optional elements are present and have non-empty elements. All attributes are set explicitly and are set to default values.
3. All 400 responses MUST NOT return a Location: header.
4. Confirm that all the 'right' elements are returned with the response body with the correct values. That is: make sure title matches title, etc.
5. Summary may or may not be present in the response, so nothing to test here.
I tried to start something like this over at http://bikini.sf.net

Posted by Nick Chalko on 2003-12-17

That's probably just an aside note, but I have always wondered: how do you test an HTTP client without being bounded to the need of having an HTTP server running in a particular context? Is there something like a mock HTTP server, or HTTP connection, somewhere on the net? Are you testing against a "real" or "complete" Atom API server implementation, or do your client implementation need not such an "entire" context to be tested?

Posted by Giulio Piancastelli on 2003-12-18

Great stuff, this looks like it should automate well.

Giuilio's comment points to something that might be an issue - it's alright the tools passing the tests in one environment (e.g. Apache on Linux at 127.0.0.1:80) but what about across environments? (IIS on Win2k on 123.212.45.3:8181...behind a firewall...on the 2nd full moon in the Guatamalan year of the Yak).

Posted by Danny on 2003-12-18

There's things like Cactus for Java and Cassini in .NET to allow you to run an embedded server.  As far as Danny's comment, one of the hard things in a test is going to be separating what's Atom from what's HTTP.  You don't want to drop into testing the underlying HTTP client implementation.

Posted by Gordon Weakliem on 2003-12-19

Assuming the first row of tests (none) is test case 1 - I think testcase 18 and 19 are duplicated in 20 and 21

18: \entry\created=""
19: delete \entry\created
20: \entry\created=""
21: delete \entry\created

all return 201.

Good list of tests. Do we need to test for June 31st?

Posted by Isofarro on 2004-01-12

It was for me great pleasure to visit and enjoy this site.

Posted by violla on 2004-06-05