Real-world SOA at Amazon

Posted by F 08/06/2006 at 20h41

A great interview with Amazon’s CTO Werner Vogels appeared in ACM’s Queue magazine. A must read if you care about how to run a modern IT organization.

The SOA acronym has been abused to the point of becoming a dirty buzzword for many. The interview shows how Amazon applies service-orientation concepts with a very pragmatic approach.

WV: …The services model has been a key enabler in creating teams that can innovate quickly with a strong customer focus. Each service has a team associated with it, and that team is completely responsible for the service from scoping out the functionality, to architecting it, to building it, and operating it.

Excellent! That’s what I ment with “Project-Oriented-IT”. I’m glad Amazon has been doing that successfully.

Here’s why it works:

WV: …Giving developers operational responsibilities has greatly enhanced the quality of the services, both from a customer and a technology point of view. The traditional model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. Not at Amazon. You build it, you run it. This brings developers into contact with the day-to-day operation of their software. It also brings them into day-to-day contact with the customer. This customer feedback loop is essential for improving the quality of the service.

He obviously explained it much better than I did, and with real-world experience to support it.

It shows that the cultural and organizational aspect of a SOA initiative is more important that the technologies used. SOA is not just about building “software architectures”. If done right, it is a better way of organizing IT operations.

You don’t even need to standardize on specific technologies/tools to run an organization with a service model:

WV: I think part of the chaotic nature the emerging nature of Amazon’s platform is that there are many tools available, and we try not to impose too many constraints on our engineers. […] Developers are like artists; they produce their best work if they have the freedom to do so, but they need good tools. As a result of this principle, we have many support tools that are of a self-help nature. The support environment around the service development should never get in the way of the development itself.

Don’t force tools down the developers throat. (Any manager reading?)

Project-Oriented IT 3

Posted by F 30/05/2006 at 23h13

Over the last years I had the opportunity to work on projects for really big companies. My initial observations led me to think that the bigger the company the more inefficient its IT operations are. Economies of scale seem to work against Information Technology and software: the bigger the company the more inefficient, bureaucratic and slower it gets.

But it would be too simplistic to say that it’s just the “size” of the company what makes it inefficient. It’s not. There are big companies that are efficient. True, the bigger the company the harder it gets, but I think it has more to do with the company’s age, culture, and the simple fact that they are inadequately structured.

One important problem I’ve seen in big enterprises is that people are departmentalized by specific roles (Business Analysts, Software Engineering, QA, Operations and Support, etc) and they do not collaborate as they should: they actually fight. They see each other as the enemy. Their jobs are so “silo-ed” that it’s hard for them to see the work of others.

A guy in charge of IT Operations for a huge financial firm described it to me with examples:

“We get a request to put an application into production with little idea of what it does or how it behaves. The application might have been in development for over a year, but we (Operations) do not get to see it until it goes to QA… which is roughly a few weeks before going to production.”

A big problem: developers do not understand how their solutions are really used in production, and Operations has no input into the development process. A lot of non-functional requirements that are essential for keeping Operation costs down may not be taken into account (like proper logging and monitoring, management of the dependencies).

The Business people blame Operations, which in turn blames the Development team for building unmaintainable software, and the Developers blame it all back to the Business people who never gave them the full requirements in the first place, nor enough time to build it right. But since nobody is responsible for the whole thing, no one fixes it. And every new project is run in the same way.

In small companies (think “start-ups”), individuals wear many hats: from “business analyst” to “operations support”. They are all accountable. I bet they are much more efficient. They may produce better solutions, in less time, with shorter release cycles, with less money, leaving happier customers and happier employees.

If I were a chief, I’d try to use a different approach from that of a traditional company.