HTTP Conditional GET in APIs – A Forgotten Art?

HTTP protocol has this cool feature called “Conditional GET“.  Let us understand this with an example of twitter API.

Here is an API request to receive timeline of a twitter user in json representation:

GET /1/statuses/home_timeline.json HTTP/1.1

Authorization:  OAuth oauth_consumer_key=”XXXXXXXXXXXXX”,oauth_signature_method=”HMAC-SHA1″ …
Host: api.twitter.com
Connection:Keep-Alive
The response looks something like this:

HTTP/1.1 200 OK

ETag: “f50e33f5b45783a3cf81d3c76e50f065”-gzip
Content-Length: 26832
Expires: Tue, 31 Mar 1981 05:00:00 GMT
Last-Modified: Sat, 16 Apr 2011 18:29:15 GMT
Connection: close
Cache-Control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0
Pragma: no-cache
Content-Type: application/json; charset=utf-8

[ { “favorited” : false, “text” : “Design Patterns – Progressive actions: http://wp.me/pEZOQ-3t #myblog http://wp.me/pEZOQ-3t”, “retweet_count” : 0, “in_reply_to_screen_name” : null, “in_reply_to_status_id_str” : null, “place” : null, …

Assuming that the timeline of a user is not changed, If I make this request repeatedly, I end up getting 26832 bytes transferred every time.  The request could have been to any other twitter resource such as tweets, users, lists etc that probably donot change very frequently.  In fact, with every service, there will most likely be certain resources that donot change very frequently.  And if a client already has a representation of this resource, downloading the same resource again is a wasteful exercise for clients, network and server. This is particularly important for mobile device based clients where the network bandwidth is limited.

As the name implies, Conditional Get makes a GET method conditional. That is, fetching of a resource happens only if certain conditions are met. Let us see what these conditions are by retrying the above request with slight modifications:

GET /1/statuses/home_timeline.json HTTP/1.1

Authorization:  OAuth oauth_consumer_key=”XXXXXXXXXXXXX”,oauth_signature_method=”HMAC-SHA1″ …
Host: api.twitter.com
Connection:Keep-Alive
If-None-Match: “f50e33f5b45783a3cf81d3c76e50f065”-gzip
If-Modified-Since: Sat, 16 Apr 2011 18:29:15 GMT
I made two changes this time. See the highlighted headers.
  • Taken the “ETag” header value from previous response and added it as “If-None-Match” header in the new request.
  • Taken the “Last-Modified” header value from previous response and added it as “If-Modified-Since” header in the new request.

“If-Modified-Since” header tells the server to send the resource representation only if the resource is modified since the date given in the header value.

ETags(entity tags) are server provided  opaque values associated with the resource. ETags are useful strong validator mechanisms. That is, ETags are expected to change when the resource is modified. A simple implementation could represent ETag as a hash-value of resource representation. Given that a server could compare resource’s current ETag value and the one presented in the request to decide if the client holds a stale representation or not.

By sending previously received ETag value in “If-None-Match” header, client is indicating to the server that it should send the response only if the ETags donot match.

Effectively, with these two new headers, client is indicating to the server that it holds a copy of a resource and that it would like to receive a resource representation in response only if server determines that client is holding a stale copy.

Here is how the new response would look like if the client’s copy is still valid. 304 status code indicates that requested resource is not modified and that the response contains no body.

HTTP/1.1 304 Not Modified

ETag: “f50e33f5b45783a3cf81d3c76e50f065”-gzip
Connection: close

Conditional GET is widely supported by almost all RSS feed servers and RSS clients.  When it comes to APIs, there seem to be complete ignorance. Looking at the wireshark capture of my android mobile device traffic, looks like many apps donot make use of this feature when it is obvious that they should have.

While I have not tried out above requests with real Twitter API, I am glad to see that it does support ETags. I assume it supports Conditional GET as well.  There is no mention of this in any Twitter API docs though. Quick googling indicates that this is the case with pretty much every other API as well.

With API virtualization and API management systems like Apigee, it is possible to implement this feature completely outside the API provider, without changing a single line of their code.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s