Weekly Random links of Interest – ITaaS, Weiner’s downfall, LulzSec and BitCoins

Here are the random links of interest for this week. Have a good weekend.

IT as a Service – A stateless infrastructure architecture model.

Another great post by Lori MacVittie. Cloud is all about shifting to “service” mindset. “Service” is also the foundational concept behind SOA. Every cloud service model (IaaS, PaaS, SaaS) abstracts and decouples certain aspects from the layers below, resulting in a flexibility never seen before. Take the case of IaaS. By abstracting and virtualizing compute, storage and network, we are decoupling applications from underneath physical resources. This ultimately results in the unprecedented flexibility to move applications/workloads (VMWare’s vMotion, Cisco’s OTV etc) across machines in data center and across data centers – for load balancing, availability etc. However, this shouldn’t affect how the services (applications) are consumed. SOA’s way of achieving this is via WSDLs and service registries. At the code level, programmers would immediately recall  Martin Fowler’s Dependency Injection and Inversion of Control principles achieving similar objectives.

By looking at applications as “services” and adopting SOA principles, ITaaS can achieve benefits beyond IaaS.

Twitter & Anthony Weiner’s downfall

Must read. This articles ends with the following quote 🙂

“The details of web product design had led to the pants being pulled down on a promising political career.”

To me, the interesting part of the story is not how a promising politician’s career was put to an abrupt ending but the key product decisions that twitter team made early in the process and how those decisions changed lives of everyone in the last few years.

While twitter has more issues (here are some I face in my daily use of twitter) to solve, clearly the ability to follow someone without requiring his explicit permission has been the winner from the day one.

via @timoreilly

We screw each other over for a jolt of satisfaction

That’s a cheesy title, but one should read it. I have been following the @LulzSec twitter stream for the last one week as they are hacking sites (Sony, CIA, etc) and posting plain text user ids and passwords extracted from site’s internal databases for everyone to see.  In the beginning, I thought hacking sites that ignore basic security mechanisms (SQL Injection, Stronger passwords, Not storing plain text passwords of users in the DB etc) would send a messages to companies and IT organizations across to re-look at their web applications.  And to some extent, @LulzSec may have achieved this purpose. In the recent past, we have not seen such consistent and systematic hacking of sites and it got everyone’s attention about the continued ignorance of basic security practices in web applications.

But the subsequent act of @LulzSec posting the extracted usernames and passwords on public sites for everyone to see and download is a disastrous step. You cannot blame and punish naive internet users for having simple and same passwords across several sites when the so called “expert” application and system developers are not doing a good job in applying basic security practices to begin with. Look at what is happening now: These publicly available usernames and passwords are tempting many normal folks to try and access the same username and passwords on several other sites (facebook, paypal, gmail etc) and see if it just works.

Here is another who user went one step ahead and created a script to automate this process and posted the script itself on the github for everyone to use and try. Too bad.

Given that @LulzSec is so active on twitter, how long is it before they get caught ?

BitCoin, The New Money

If you haven’t heard of BitCoin, here is your hance to mint your own money, virtual money for free 🙂 Read more about here, here,  wikipedia link and of course yours truly quora link, answers for all your questions.

I came across BitCoin on the hackernews.com. After that so many people are posting links to news about bitcoin, one impatient guy couldn’t bear it any longer and wrote a safari extension to hide all bitcoin news on hackernews.com 🙂

 

Understanding the Good in “The Good, the Bad and the Ugly of REST APIs”

With several dozens of APIs getting published every month or so, it is kind of become a routine for a seemingly innocent “How to do REST” or “Guidelines for REST APIs” kind of blog posts become source of controversies around applying REST principles properly.

Last week, it was the turn of George Reese’s blog post titled “The Good, the bad, and the ugly of REST APIs” for the controversy.  There were several reactions and reactions to reactions on all aspects of his post.  Following tweet from George indicates the mood:

One of the things George’s blog advocates under “Good” list is:

Supporting both JSON and XML

I know you love {JSON,XML} and you think everyone should be using {JSON,XML} and that the people who use {XML,JSON} are simply stupid. But you’re not building an API for yourself, you are building it for everyone, including those who think {XML,JSON} rocks and {JSON,XML} sucks. I don’t want to get in the technical merits of either language or even the possibility that there might be distinct use cases for JSON vs. XML. The bottom line is that there are people out there who want to consume APIs in both languages and it’s just not hard or complex to support both.

To which William Vampenepe reacts in his blog post:

I disagree: Two versions of a protocol is one too many (the post behind this link doesn’t specifically discuss the JSON/XML dichotomy but its logic applies to that situation, as Tim Bray pointed out in a comment).

Now that confuses me. Supporting JSON and XML in my mind is no different from a resource supporting multiple media types and serve an appropriate media type using content negotiation.  The post he links to talks about SOAP and REST as multiple protocols. I don’t see the connection.

The other item on George’s “Good” list is:

Providing solid API documentation reduces my need for your help

Solid API documentation enables an individual to both understand the simple examples that help them get started as well as the complex ways in which they can leverage the API to accomplish advanced tasks

That doesn’t seem like a statement to be argued with, isn’t it ?  Apparently not so. Jan Algermissen posted a comment saying:

If you document an API, you API immediately ceases to have anything to do with REST. The contract in RESTful systems is the media types, *not* the API documentation.

I suggest you move that section to “The Bad”

This comment was met with ridicule by many folks including George and William terming it as nothing but silly.

My takeaway from that comment was that the documentation should focus on media types as the rest of the behvaiour of dealing with media types is fairly standard with REST.

Clearly, Jan Algermissen is no newbie to REST. But I am still baffled by the first part of his statement.  How can a little documentation with examples of request and response payloads (even at the risk of duplicating something very obvious in REST way of doing) make it so against REST.

Stu responds with his blog post defending Jan Algermissen’s comment and many other things.

Jan Algermissen’s comment about how when you’ve documented an API, you’re not really doing REST, is actually spot on, fundamentally, but is met with ridicule. I can completely understand how vacuous the comment sounds if you just want to ship an API nownownow, are being short-term practically minded, or are just playing buzzword bingo with architectural styles. But if we actually want to leverage the benefits of the style, we’d work on the core issue of having a generic media type or two for these sorts of use cases.

I am still trying to digest parts of Stu’s post. Most developers learn the best practices by looking at what experts in that domain recommend or see mimic real-world APIs from popular web sites (twitter.com, facebook.com etc). Unfortunately, of late, these are the wrong sources to learn the best practices from (Remember the hashbang controversy).

Here is something even more basic. Try and get a bookmark-able link to a specific tweet on twitter.com site.

 

(Updated 14th June 2011)

I would like to add one more the list of  “good”  of REST APIs.  I am sure all REST purists would now cringe at this. Try and publish WADL for your API. Goal is not to be able to do all weird stuff that tools force you to do with WSDL while consuming the service.  But it definitely helps your API consumers to leverage some tools that would further help them to understand the API better.  For example, check out this cool API console tool from apigee. Apigee API console takes a WADL and provides a nice way to navigate the API, exercise the API (including OAuth), look at the request/responses  and learn iteratively – all with zero coding and based completely on WADL.

It already supports several public REST APIs as an example.

HTTP Conditional GET in APIs – A Forgotten Art?

HTTP protocol has this cool feature called “Conditional GET“.  Let us understand this with an example of twitter API.

Here is an API request to receive timeline of a twitter user in json representation:

GET /1/statuses/home_timeline.json HTTP/1.1

Authorization:  OAuth oauth_consumer_key=”XXXXXXXXXXXXX”,oauth_signature_method=”HMAC-SHA1″ …

Host: api.twitter.com

Connection:Keep-Alive

The response looks something like this:

HTTP/1.1 200 OK

ETag: “f50e33f5b45783a3cf81d3c76e50f065”-gzip

Content-Length: 26832

Expires: Tue, 31 Mar 1981 05:00:00 GMT

Last-Modified: Sat, 16 Apr 2011 18:29:15 GMT

Connection: close

Cache-Control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0

Pragma: no-cache

Content-Type: application/json; charset=utf-8

[ { “favorited” : false, “text” : “Design Patterns – Progressive actions: http://wp.me/pEZOQ-3t #myblog http://wp.me/pEZOQ-3t”, “retweet_count” : 0, “in_reply_to_screen_name” : null, “in_reply_to_status_id_str” : null, “place” : null, …

Assuming that the timeline of a user is not changed, If I make this request repeatedly, I end up getting 26832 bytes transferred every time.  The request could have been to any other twitter resource such as tweets, users, lists etc that probably donot change very frequently.  In fact, with every service, there will most likely be certain resources that donot change very frequently.  And if a client already has a representation of this resource, downloading the same resource again is a wasteful exercise for clients, network and server. This is particularly important for mobile device based clients where the network bandwidth is limited.

As the name implies, Conditional Get makes a GET method conditional. That is, fetching of a resource happens only if certain conditions are met. Let us see what these conditions are by retrying the above request with slight modifications:

GET /1/statuses/home_timeline.json HTTP/1.1

Authorization:  OAuth oauth_consumer_key=”XXXXXXXXXXXXX”,oauth_signature_method=”HMAC-SHA1″ …

Host: api.twitter.com

Connection:Keep-Alive

If-None-Match: “f50e33f5b45783a3cf81d3c76e50f065”-gzip

If-Modified-Since: Sat, 16 Apr 2011 18:29:15 GMT

I made two changes this time. See the highlighted headers.

• Taken the “ETag” header value from previous response and added it as “If-None-Match” header in the new request.
• Taken the “Last-Modified” header value from previous response and added it as “If-Modified-Since” header in the new request.

“If-Modified-Since” header tells the server to send the resource representation only if the resource is modified since the date given in the header value.

ETags(entity tags) are server provided  opaque values associated with the resource. ETags are useful strong validator mechanisms. That is, ETags are expected to change when the resource is modified. A simple implementation could represent ETag as a hash-value of resource representation. Given that a server could compare resource’s current ETag value and the one presented in the request to decide if the client holds a stale representation or not.

By sending previously received ETag value in “If-None-Match” header, client is indicating to the server that it should send the response only if the ETags donot match.

Effectively, with these two new headers, client is indicating to the server that it holds a copy of a resource and that it would like to receive a resource representation in response only if server determines that client is holding a stale copy.

Here is how the new response would look like if the client’s copy is still valid. 304 status code indicates that requested resource is not modified and that the response contains no body.

HTTP/1.1 304 Not Modified

ETag: “f50e33f5b45783a3cf81d3c76e50f065”-gzip

Connection: close

Conditional GET is widely supported by almost all RSS feed servers and RSS clients.  When it comes to APIs, there seem to be complete ignorance. Looking at the wireshark capture of my android mobile device traffic, looks like many apps donot make use of this feature when it is obvious that they should have.

While I have not tried out above requests with real Twitter API, I am glad to see that it does support ETags. I assume it supports Conditional GET as well.  There is no mention of this in any Twitter API docs though. Quick googling indicates that this is the case with pretty much every other API as well.

With API virtualization and API management systems like Apigee, it is possible to implement this feature completely outside the API provider, without changing a single line of their code.

Design Patterns – Progressive actions

Performing an action progressively in increments is a generic design pattern that we can see in many contexts. The goals of the pattern vary depending on the context.

• Reduce perceived latency
• Improve the usability
• Avoid the information clutter
• Optimize resource usage (CPU, Memory etc)
• On-demand resource usage (CPU, Memory etc)

Here are few examples of this pattern

Progressive Disclosure

This is a UI design pattern. The goals typically here are to avoid the information clutter and improve the usability. General approach is to break up the content into smaller chunks/blocks, display one or two chunks to begin with and show the other blocks progressively as required. The techniques to hide/show content and the events that trigger progressive display are presentation technology dependent.

The pattern not only makes the digestion of content easy for novice users but also makes it possible for the advanced users to explore further – in a way helping the transition of novice user to advanced user. A popular avoid-the-clutter example that you may be familiar with is twitter.com site that displays controls (Reply, Retweet etc) when the mouse is hovered on the tweet.

Some relevant blog posts that talk about this technique in detail

Progressive Disclosure in User Interfaces

Wikipedia page

http://www.interaction-design.org/encyclopedia/progressive_disclosure.html

http://www.useit.com/alertbox/progressive-disclosure.html

Progressive Rendering

This is another UI design pattern. The goal is to reduce the perceived latency. Like the previous technique, the general approach is to break up the information into smaller chunks (if not already broken in the original content), process each chunk and display before proceeding with another chunk.

Browser is a popular example where the various pieces of information in a web page (html, java script pages, CSS stylesheets, images etc) are processed in incremental and parallel fashion. The processing includes downloading the entity, parsing the entity, updating DOM. The processed entities are then displayed without waiting for other non-dependent entities.  You would also see this pattern at work when a browser is rendering an image where the resolution of the image is improved progressively as the image data is getting downloaded.

maps.google.com is another example where the tiles comprising the map are separately downloaded, processed and displayed progressively. The end result is that you see parts of the map getting displayed instantly and the rest being filled in progressively.

Progressive Collection of Information

This is again a UI pattern where the goal is to ask the user for information incrementally as required based on the context. The traditional approach of displaying a form with dozens of fields and expecting the user to fill-in all in one shot runs the risk of user loosing interest and not providing the desired information.

A typical user registration form in many sites asks for the following information:

• Email address
• Mailing address
• Payment Details
• Areas of Interest

Instead, one can just ask for Email address to begin with. When the user navigates the site and purchases an item,  then ask for Payment details. If the user buys an item that needs to be shipped, ask for mailing address at that point.

If you think about it, this is no different from how it works in real-life. When you got a brick-and-mortar shop, you are not disclosing any of these details up front. Only when you purchase an item, one is asked about payment details.

Progressive Processing

This is a software design pattern. Goal is to process a stream of information as the stream is downloaded as opposed to waiting to process until the end of the stream happens. In practice, with certain large or never ending streams, it mayn’t be possible to wait for the end of the stream and the only option left is this pattern.  Use of this pattern not only helps in reducing the perceived latency, it would result in optimal use of resources (memory, CPU etc) in some cases.

A familiar example is XML Parsing.  In a server environment where the server is handling multiple requests containing XML documents asynchronously in non-blocking manner, not all request data will be available in one shot due to the inherent nature of network delays and sharing across clients.

If the requirement is to build a DOM of this XML document, a progressive XML parser would fit the scenario better than a traditional non-progressive XML parser. A non-progressive parser either would block on the stream because the data is not available yet (most Java parsers fall into this category) or expect that entire document data be provided in one call. A progressive parser, on the other hand, would accept the input in chunks/increments, parse the chunk and then return the control back to the caller. When more chunks of data are available, a progressive parser would be able to continue parsing.

If the requirement is instead to use a SAX parser to search and extract certain data from the stream, a progressive XML parser would not only reduce the latency, it would also reduce the memory usage (one can discard intermittent data structures and previously received chunks of stream data).

In fact, the only publicly available commercial/open source C language XML Parser that I know is capable of doing progressive parsing is libXML.  At apigee.com, my previous company, we have built couple of progressive XML parsers (one based on lex/yacc and the other based on hand-coded state machine) that not only could do wire-speed XML parsing but also could handle never-ending XML streams (used in financial world).

Use of this pattern influences the API contract of the component using the pattern. In the example of the XML parser above, the progressive parser would likely have appropriate methods to accept chunks of input and be called repeatedly until the last chunk.

Progressive building of data structures

This is a design pattern typically used in software design. Goal is to delay the resource usage to the point of actual requirement.

An example of this is again popular in XML parser world. A delayed DOM parser.  Apache Xerces implements this pattern. In the delayed DOM building mode, When the Xerces parser is given an input XML document, it doesn’t materialize all the DOM nodes in the beginning itself. Instead, only the document and probably root node is materialized. As the methods are called on the root node, subsequent nodes are materialized.

What can you learn from Twitter blocking UberTwitter, Twidroyd etc

Recently, twitter blocked several applications citing violation of terms & agreements and  copyright infringements. Well, this post is not to discuss if twitter did the right thing or wrong.  I would like to highlight a subtle point to other API providers instead.

Thanks to the recent mobile revolution, apps are the defacto mechanism to make your content/service available on the mobile. And apps need to talk to your APIs to make this happen.  APIs are the integral part of this revolution. If your company has a website and is providing some service/content and is not already providing an API, following are some resources that may help you to understand the ecosystem. Google should help you in finding lot more resources…

• API Evengelist – Wiki on the APIs and the ecosystem.
• Programmable Web – A director of APIs
• apigee.com, mashery.com – check out the whitepapers and blogs.  Note that much of this is not vendor biased content.

These days, exposing a REST API is fairly a simple task.  Whether you are a java developer, ruby developer, python developer, php developer or a javascript developer, there are lot of frameworks and libraries that will help you get started with API development and go to production in no time (May be topic for another blog post).  Good thing is that you can focus on your key business aspect and  reduce your time to production significantly.

Unless you are a pro and have done this sort of thing in the past, you will soon realize that you need to do lot more cross-cutting stuff on top of taking the API to production.   For example, you find that few app developers are violating the terms and contracts they agreed to and you need to stop them from using the API – much like what happened in the case of twitter.  A simple IP based access control is not going to help you because unlike website consumers, apps are installed on mobiles and traffic could be coming directly off of these devices. But you cannot block this IP because it is not the end user(consumer) who is violating the terms (in this example), but the app developer – so you need to block the app, no matter which consumer is using it.

Many questions will be coming across your way and you were not prepared to answer. For example, some usage related questions are:

• Which are the top API calls in my service ?

• Can we look at the usage on per-app and per-consumer basis ?

• Can we limit the usage of an app or consumer instead of blocking them altogether when terms are violated ?
• How do we ensure that a single app or consumer traffic oesn’t load the server ?
• Can we expose the usage information to app developers ?

There will be several other questions relatad to security, versioning, monitoring, visibility and analytics.  You can read about them in the links above. The usual choices are  build Vs buy.  Clearly, there is lot of stuff to build if you choose to go that way.  API Management systems are perfect  if you are instead looking at a 3rd party solution.  Good thing about many of these API management systems is that you donot need to make any changes to your API or code. They sit in front of your API server much like a load-balancer and give you all the control and flexibility we talked about.

Whether your API is on-premise,  hosted in cloud or you are just looking for a SaaS like solution, you will find a vendor providing that solution.  So, do some home work, choose the right solution matching your needs.

Here are some API Management vendors that I am aware of:

• www.apigee.com
• www.mashery.com
• www.3scale.com

Disclaimer: I worked for apigee at one point and the intention of this post is not to promote any specific vendor.

Samsung TV hacking & Snooping on WiFi Traffic in Windows 7

When working with a new software product/system, I am one of those guys who is not satisfied just knowing the external interface. I need to know what is going on “under the hood” as well.

Couple of months back, I bought a Samsung LED TV that has cool Internet features. With couple of fun TV app ideas in mind, I wanted to get down to do some coding during the weekends. So, I finally spent some time reading up the documentation and started coding.

In the meantime, I was playing around with some apps that come with TV. It has some nice apps like Skype, Google Maps, Twitter, Facebook etc. The functionality offered in twitter and facebook apps is so limited that one cannot even navigate to the links.  While there is no app store for Samsung India, there seem to be one for Samsung USA. It is so disappointing to see that there is lot more content and apps for USA customers and Samsung India has no clue when they will be made available here. Samsung India customer support is not even aware of an app store or tv apps – I had fun time talking to them though 🙂

I didn’t think that this logic of what apps to be made available is embedded in the firmware. It must be some pre-set region code or something like that set in the TV that the software  further uses to restrict or extend the features. Since the list of apps are not hard-wired into the TV, the software must be talking to some Samsung servers out there. So, I thought I will find out what is going on by looking into the traffic as I turn on the TV.

I had my TV connected to Internet via WiFi to the Linksys WiFi router. Since my windows laptop is also connected to the WiFi in the same network, I thought I could easily snoop on the traffic by firing up Wireshark in promiscuous mode. Apparently not so easy in windows. Unlike linux, it seems that Windows doesn’t allow or has limited support for promiscuous mode with WiFi adaptors. I learnt this hard way though.  Thanks to my former colleagues on LinkedIn Abhijit and MikeB, it seems that  this apparently works on linux and that there are some commercial solutions like AirPcap, I didn’t want to go that far. Plus, I didn’t have access to a linux machine readily.

But with some more time and help from google, I came to know that windows 7 has this cool feature called virtual WiFi adapter using which the windows laptop itself could act as an AccessPoint. That means, TV could now connect to the access point provided by laptop instead of the router. My laptop instead connected to the router via Ethernet.

With this setup, I was able to combine another of windows feature called ICS (Internet Connection Sharing) and configure internet sharing between Ethernet and virtual WiFi adapters. Now, I could snoop on the TV traffic by using promiscuous mode on the Ethernet adapter.  I also found that snooping works perfectly fine on virtual WiFi adapter too.

Next few hours was pure hacking delight looking at all the API calls the TV was making. TV is running on Linux. Luckily, only the initial authentication calls are on HTTPS. Once an auth token is retrieved, subsequent calls are made over HTTP itself.

• authenticating to samsung servers in Korea
• getting details of capabilities(like paid apps) to be exposed,
• catalog calls to retrieve list of apps,
• get list of installed apps,
• retrieve service notifications etc.

Not sure if it is a serious privacy issue, TV also logs several user actions to the server including how many times an app is launched etc.

While doing this, I also came across a project on sourceforge called SamyGo. These guys have been hacking Samsung TVs for a while – including enabling telnet/ftp support, hidden PVR functionality and even modifying the firmware in some cases. Check it out here. While I haven’t rooted my TV yet, I was able to gain lot more insights using their tools.

Anyway, here are the instructions on how to snoop on WiFi traffic on windows, if you ever want to do. You can  use this approach to snoop on your mobile phone app traffic too.

• Start a command window in administrative mode. You can do this by right-clicking on the “Command prompt” menu item in Start | All Programs | Accessories

• Run the following two commands to setup a virtual WiFi adpater

c:>netsh wlan set hostednetwork mode=allow ssid=ssid1 key=password

This creates a virtual WiFi adapter acting as an AccessPoint. Replace ssid1 and password with your own. You can get more help on this command by running “netsh wlan set hostednetwork ?”

c:>netsh wlan start hostednetwork

This actually starts the access point. You may need to re-run this command if you reboot the laptop.

• Enable internet connection sharing

Go to Control Panel | Network & Sharing Center | Change Adapter Settings and right-click on your Ethernet adapter (or any adapter with internet connection) and bring up properties. Change to “Sharing” tab and enable the check box “Allow other network users …”. In the combo box below, select the newly created virtual WiFi adapter.

• Now, go back to the device (TV or mobile) and refresh WiFi access points. You should see the newly created access point ssid1 that the device can connect to.
• That’s it. Have fun with the hacking.

PS: btw, there are some free tools like Vritual Router and Connectify that simplify the steps above into simple UI, but I had little success in snooping on WiFi traffic with them.