What Do You Care What Other People Think?

Open Source Cloud Projects

I had the privilege of meeting some of the developers of open source cloud projects today at CloudConnect.  Adrian Cole of jClouds, Mitch Garnaat of Boto, and Mike Mayo of CloudTouch were on a panel called "Writing Code for Many Clouds." 

We're heading rapidly toward a world where there will be a small number of very big providers of cloud infrastructure - Amazon and Rackspace are currently the top in volume (see Guy Rosen's State of the Cloud for March 2010), with a huge gap to the next providers.  Google and Microsoft will surely account for a large number of sites as they get their adoption engines and developer evangelist teams going.

Basic economics and business strategy dictate that these providers will differentiate from each other based on features and APIs, regardless of standards.  We need to accept this as fact.  We don't have to accept that we have no choice but to pick one or the other, however.  We can embrace open source projects which limit the impact of different clouds, making portability easier. 

This will only happen if developers get involved.  To that end, here's a short (not exhaustive) list of some open source projects that are trying to make a difference.  They need your usage and participation.  You need them to keep the cloud open. 

In no particular order, and including some very early stage projects (like cloudtouch for iPhone and iPad based management of clouds):

(Click to enlarge)

In the slide above (which we presented on the panel) lists the type of abstractions managed by the project.  Compute means managing instances (like EC2), storage means managing storage (like S3 or EBS), IAAS means the works - including load balancing, networking, etc.

Here are links to the projects:

• Fog - http://github.com/geemus/fog
• CloudTouch - http://github.com/greenisus/cloudtouch
• jClouds - http://www.jclouds.org/
• Boto - http://code.google.com/p/boto/
• Dasein Cloud - http://dasein.org/
• libcloud - http://incubator.apache.org/libcloud/
• Deltacloud - http://deltacloud.org/
• Eucalyptus - http://open.eucalyptus.com/
  

Building Apigee for Multiple Clouds

TL;DR warning – this article is about 1500 words.  I prepared this for Shlomo Swidler's panel "Writing Code for Many Clouds" at CloudConnect 2010.

At Sonoa, we have an enterprise product which we turned into a service called Apigee.  From the first, we needed to move beyond just being packaged as a VM and “deployable anywhere” to really living in the cloud.

This is what we’ve learned so far – some of which we anticipated and some of which we reacted to.  Build, deploy, and manage – of the three basic parts of running a service only deploy and manage really change.  The big difference is in operationalization of the system.

Most recently we realized that we needed to be HA across providers and get total control of our latency, so we are building a new datacenter on Rackspace as well.  This is a work in progress so I’ll be reporting from the front lines.

Finally, we’ve helped implement a multi-cloud architecture for ING which has taught us something about where multi-cloud services may be headed.

The first cloud: EC2

1.    Build:

The first and biggest step for any system will be building it as a VM if you haven’t done so before.  Once you have done this, you can practically drop it onto any box.  You’ve become independent of the hardware and other aspects of the operating environment. 

Beyond build, you have to focus on: setting up the network topology, configuring the virtual boxes once they’re up, and managing the result.

2.    Deploy:

 The next phase is figuring out how you bring up instances in your cloud platform.  EC2 has its own interfaces for this, and Rackspace has different ones.  Rightscale normalizes these interfaces and provides a UI.  There’s an open source package with no UI that we evaluated but aren’t using called libcloud. 

Now that you’re hardware independent, you can run as many instances of your service’s components as you can afford.  The main solutions here are Chef and Puppet, both open source.  We use Capistrano for scripting automation. 

Then you need to configure the topology of the different subsystems you’ve built.  Here things get interesting.  EC2 does not support multicasting across your default virtual network; this was tough for us and would be for anyone relying on clustering.  VPN-Cubed from CohesiveFT let us build a private network within our EC2 environment and let us do the multicasting we needed.

Once your network is up and you can push software, it’s just the same as having your own private datacenter.  You can connect from anywhere, manage instances, and get alerts and reports.

3.    Manage:

That brings us to management.  We use Nagios for monitoring our virtual boxes.  We learned that we needed to have a separate machine outside of EC2 as a “monitor monitor” – a Nagios instance that monitored the health and responsiveness of the Nagios box in the cloud environment  We use RightScale for managing all of our accounts and instance creation.  With this setup we’ve had zero downtime since our launch in late August of last year.

We realized at the outset that we wanted to build a service that would be portable, so we chose not to use the least portable features of AWS, such as S3.  While it would have made our life simpler for some of the assets we were managing, there was no corollary (and Walrus, the Eucalyptus storage subsystem that mimics S3, does not count as a corollary, even though it really works).  We did use EBS (Elastic Block Storage) which is so close to a SAN that we felt it was reasonably standard; and forcing our hand was the fact that we needed to solve for persistence and performance. 

But the evils of cloud computing were present as well as the the good.  EC2 does not guarantee the availability of an instance, but the availability of a zone.  As a result we found that the latency of our service had a high degree of jitter (between 5 and 15ms), which was acceptable but not ideal.  The lack of control in this environment means that we’ve been buying instances ahead of our need in order to guarantee not just availability but performance.  This is one of the headaches that cloud computing is supposed to transcend.

In a nutshell – “it’s elastic but you have to manage it.”

So in order to manage the network performance issues (achieve constant performance AND availability) we realized that we needed to go multi-cloud.  We also realized that our core service principle – we’re a cloud service gateway and active proxy for people's API traffic – meant that we had to have a “strongest link” architecture so that no set of failures at a single cloud provider could take down our service.

We’re now building on what we’d anticipated and developing a new instance of our service at Rackspace.  The big changes here are the level of control we have out of the gate for network topology, process isolation, CPU performance… and price, which is higher. 

The second cloud: Rackspace

1.    Build

Architecturally the big differences are database replication and cross-provider load balancing.  This places really specific requirements on your networking design and technology as well as your database design. 

One of the things our service does is store all of our customers’ cloud API traffic for their later use in analytics.  Thinking about data modularly helps with replication.  In a replicated world we need to break out types of datasets – such as customer information and service configuration – into smaller chunks that can meet higher-speed replication requirements cost-effectively, and break them away from all the historical traffic data.  Even the traffic data needs to be handled differently in this world. 

We are now sharding the database into circular tables, where the incoming data is always written to a write-only area, and revolves to the next area every five minutes.  In our user base a 5-minute delay on analytics is more than acceptable (compare this with the SLA for Google Analytics), and the working set of data used for traffic management is handled separately in realtime.  All of this means that we can have either a hot standby or live-live dual-cloud configuration without breaking our customer promise that they can tweak their service at any time at all, and that their analytics are consistently available.  This will also let us evolve both sides of the service as it grows.

2.    Deploy

Deployment tooling stays the same – our old friends RightScale and Capistrano are used to spin and configure instances.

On the networking side, obviously you need to connect your clouds securely in order to replicate between them as well as to exchange performance data which can be used for load balancing.  We found again that VPN-Cubed helps us establish a trusted connection between our heterogeneous cloud environments. 

3.    Manage

Since we are using standard monitoring and management tools – Nagios, RightScale, and Capistrano – these all work in both environments, and our approach of using a “monitor monitor” doesn’t change.. although now we need to monitor monitors in each cloud.

Is there an easier way?

For an infrastructure play like Apigee, we don’t think so.  Given our customer promise of near-zero and predictable latency we need as much control as possible.  For an application-level service play though, we think some parts can be easier.  We’re built on Sonoa technology that manages all of our cloud API traffic processing, as is ING, a financial service company that’s moving to cloud. Their challenge is elasticity in financial modeling – specifically the Monte Carlo simulation workload which is compute-intensive and highly intermittent in use of resource.  When you’re running the simulation, you need all the compute resource you can get.  When you’re not running a simulation, you need almost none.

Cloud infrastructure like EC2 and Rackspace take care of the racking & stacking problem associated with scaling up for Monte Carlo.  You still need to manage that with a  tool like RightScale or libcloud plus your configuration and deployment tool of choice.  But at the higher level where you’re load-balancing between clouds you don’t necessarily need a VPN, as there’s no data replication requirement.  At this layer they’ve implemented a secure API which is called by internal clients, and then this API request is load-balanced by Sonoa’s API gateway.  The gateway then calls the right cloud based on policies set by the monitoring and scheduling software.  So in this situation you are monitoring your cloud instances and letting the API gateway handle the dirty work of dispatching and securing the calls.

10 Lessons Learned From Building to Multiple Clouds:

1.    Get everyone comfortable with virtualization fundamentals, from developers to admins.

2.    Limit your dependency on provider-specific APIs by using 3rd party tools that manage this for you.

3.    There may be SLAs on your cloud instances but there are no SLAs on the APIs your cloud providers give you.

4.    Refuse to use services that have no corollary in other clouds.  It will cost you more in rearchitecture than you gain by using it.

5.    Understand the cost trade-offs for your business of the different clouds’ strengths – especially in the dimension of availability, price, and performance.

6.    Anticipate your needs for data replication and design your databases accordingly.

7.    Pay attention to your networking requirements and network topology.

8.    Consider the granularity of the requests that you need to load balance – is it at the service or API layer or is it finer-grained than that?

9.    You’ll still buy more than you need but the waste ratio is much less in the cloud.

10.    Monitor the monitors!

Mobile is the future of cloud; cloud is the future of mobile

Note - this post is an expansion on an earlier draft posted at Sonoa's blog in early December 2009.

Mary Meeker’s Web 2.0 presentation made a strong case for the imminent boom on the mobile internet.  Some statistics that caught my attention:

• Mobile internet users will exceed 500M human beings in early 2010
• The mobile consumer device market will exceed 10B in the next 5 years
• The current iPhone + iTouch user base is larger than Netscape’s base in 1999
• More than 20% of the world will be on 3G networks by the close of 2010

This is a serious change in how people are using the Internet.  I don’t think of the Internet as synonymous with the Web; the Web is one pattern of applications, based on HTML and HTTP, that uses a common infrastructure – the Internet.  Both the iPhone and Android phones have a much higher share of Web and Internet usage than previous generations of mobile devices.  But are they really using the internet?  Or are they using the Cloud?

From my point of view, the Cloud is another pattern of applications – like the Web – that use a common infrastructure – the internet.  But just as Web application patterns are different from other Internet applications (like email, for example), Cloud application patterns have their own rich array of styles and requirements.  And what we’re seeing with mobile consumer devices is an explosion of applications that reside on the device and use the Cloud for access to remote logic and data.  

 

 (click to enlarge)

APIs are a major part of this - they are user interfaces for machines.  RSS and ATOM are effectively simple, standard APIs for machines to get at content that hasn't been marked up for a human interface.  APIs do this for business logic and data, rather than simple content. Mobile app and web app developers can use a site's APIs to build very cool new applications that look and behave totally unlike the original website.

There are many implications of this pattern, including the need for elastic computing capabilities, commonly thought of as "cloud infrastructure".  Elasticity is an attractive capability in its own right, but when coupled with API-driven applications running on many devices and machines in many places, it becomes crucial.

 (click to enlarge)

While there is plenty of usage of WAP vs. APIs today (the yellow bars vs. the green bars) to power limited functionality versions of websites and deliver these to mobile devices, this is proving both brittle and unappealing by comparison to the vibrant experiences of native applications on iPhone and Android devices.  It seems to me that future of the mobile internet is not the mobile web – resized, limited web pages rendered by a WAP server and discovered via Google and Bing – but the cloud.  

What we’re starting to see across multiple industries and business models (from Amazon and Netflix to Sears and Blockbuster) are the first generation of sites rebuilding their backends to be accessible via cloud APIs in order to give access to mobile applications that will drive their business.  Some sites are on their second and third generation, having rebuilt their web presence to rely on the same cloud APIs that their mobile applications do.  

With the launch of the Apple iPad and the Motorola Droid we are already seeing Ms. Meeker's predictions about mobile devices coming true.  There is more to come and in my mind it's guaranteed that the mobile boom will be accompanied by a massive expansion of APIs and cloud infrastructure along with it.

In the cloud, scale means concurrency

In enterprise computing, scale has traditionally meant "lots of transactions per second."  On Wall Street for many years, "20,000 TPS" was the magic number as it was the rate of a typical market data feed.  Infrastructure like TIBCO's UDP-based information bus and then IBM's MQSeries became the base platforms for much of this scale of computing, and are still heavily used alongside modern JMS and MSMQ implementations.
 
Relatively little attention was paid to concurrent connections.  Enterprise environments tend to be well-regulated, and most applications will have under 1000 simultaneous users (whether human or machine driven).  As a result, application servers and related technologies evolved to support high transaction throughput at limited concurrency.
 
The web on the other hand brought in much higher concurrency requirements, and platforms like WebLogic became default components of web computing environments for sites serving 1,000s people at the same time.  This was a breakthrough and led to significant market success in a short time period.
 
With the rise of cloud computing, two things change.  First, mobile applications and the API economy are driving an order of magnitude increase in the number of simultaneous users.  Second, these users are often machines rather than people, and therefore aren't limited to the demand patterns of humans users clicking links or refreshing their pages.
 
This produces a new set of demand patterns which increase both total throughput and peak concurrency.  As an example, travel sites like Kayak.com and Bing.com/travel issue hundreds of API requests to airline reservation system backends as a result of a single human-driven query.  Furthermore, these requests are being made not just by desktop or web applications but by mobile applications – especially iPhone applications.  As most people are aware, the next 10 billion devices that come online will not be PCs or netbooks but will be mobile devices (phones, MIDs, GPS, game units, media players).  Each of these is prized for its native application experiences.  Each of these devices will be making user-driven and automated calls to cloud services in order to deliver those experiences.
 
Where backend systems are not protected from this demand, they are being penalized in performance and load management.  This causes either outright outages, "web brownouts" where the core website that uses the same backend slows down, or erratic performance across both the web and cloud properties.  Again, mobile access exacerbates the issue due to the intermittent nature of mobile internet connectivity, which multiplies the number of connections that need to be set up and torn down as the device comes on and off the network.
 
So the explosion of concurrent usage is already beginning, as the traffic and backend impact is expanding.  To manage this and maintain stability of existing infrastructure, a new layer of infrastructure is emerging, much as HTTP load balancers have evolved to serve the needs of web computing.  What we're seeing is the rise of cloud service controllers, a category of infrastructure that works well with existing systems and builds on top of the strengths of application servers, enterprise messaging systems, and application delivery controllers.

Business Logic vs. Business Policy in Cloud Services and APIs

As companies move from websites to cloud APIs - we're seeing them separating business logic from business policy.

When you think about business logic, you probably imagine an application server or stored procedures, or custom code that accesses stored data and exposes it in a useful and manageable way – such as “show me all of a given customer’s purchases in this timeframe”.  

These pieces of code reflect the understanding of the fundamentals of the business - like customers, orders, tickets, requests – and how they interact.  In any business you build skill over time in understanding your domain and then build on top of that understanding to take the business further.  Business logic is generally stable, consistent across requests, and subject to careful modification.  After all, you wouldn’t want to put partly tested logic about bank balance transfers into production.

When you think about business policy, what comes to mind? 

Maybe a stack of papers documenting HR or accounting guidelines, or maybe the elements of Sarbanes-Oxley requirements.  Policy is not about how you compute a specific result – like the customer’s purchase history – but about other factors like who can access it, how frequently, from what locations, how it’s rendered, how long the result is considered valid, and other “meta-logic” issues.  In any business you make changes to your policies frequently, as a way to meet the changing face of the market environment.  The nature of business policy is therefore to be dynamic, varied according to the request context, and subject to frequent experimentation.  For example, product managers need to continually test the value of different offers to given segments without having to redevelop the product.

When policies like security (“who can access this and what credentials do they have to show?”) and service level (“how many requests can be made in a given time interval?”) are collapsed into code responsible for business logic, the result is loss of agility, high cost to adopt new policy implementations (e.g. for identity and access schemes or to keep pace with increased hardware capacity), and confusion of concerns (rather than separation of concerns).

In the current era of cloud APIs, where these interfaces are being built as a direct link to what started as the backend for the website, many developers are realizing that what was really policy was built into their business logic layer.  They’re also seeing that adding new clients – going from enabling partners to enabling rich clients to enabling mobile device (such as iPhone or Android) applications – require many adjustments to how the request and response of an API are rendered, but should require no changes to the stable core of business logic.  

Supporting an API with a set of policies that activate based on the user (such as preventing or enabling access to different parts of the API – for example, only the development & testing methods but not production methods, or query methods only, or limiting requests to a certain transaction volume or financial value) and type of client (such as providing pagination and format translation for a mobile device) means that the risk of delivering the API is reduced, and the risk of negative impact to existing users of the API (typically the original website) are reduced.   Also,  the computational load for the policy processing can be moved to a low-cost, scale-out tier and away from the high-value application servers that host the business logic.

So what this seems to indicate is a movement towards a separation of concerns between logic and policy.  Responsibility for business logic processing lies in a stable and slowly changing layer, and policy is processed in a tier that allows for agile modification and enables the total computational result to meet the shifting needs of the business.

Why modern applications need an API proxy

Structures of control are spontaneously generated in every environment and every wave of computing.

Today on the web we have a model where browsers are the single point of control for much of what happens, not just at the level of applications, but at the meta-application level as well.  Not simply usage (“point-click-type”), but things about usage – who is the user (browser cookie), what are they using the app through (user agent), where did they come from (referrer), what can we infer about their behavioral state, and so on – as well as modifications of usage (browser add-ins, content filters, security modes, local caching for performance).  To be sure, some of these things can be and are performed using infrastructure between the browser and the website (such as content filtering, security, and caching), but the guaranteed component is the browser.

This is one of the reasons that Google Analytics is so popular and useful – you can rely on it to tell you useful things about your traffic because it can rely on the browser as a predictable point of control.  Including an invisible piece of content on your web page makes the browser fetch data from Google, implicitly sending information that enables Google to report on your usage.

For web and cloud APIs, what is the equivalent structure of control?

Currently there is no one point like the browser.  This is for great reasons – APIs are all about reusing application or service logic and rendering it to different form factors: pure logic (built into an internal application computation), web UIs (part of a mashup), and most notably, client applications on a wide range of devices (from PCs to mobile phones, set-top boxes, and tablets like the iPad).  These devices are in the early part of a boom that will see over 10 billion individual units in use, representing at least hundreds of unique hardware/software designs.  The sheer utility of these internet-connected devices predicts that their usage will drive high demand for APIs rather than standard websites.  There are initial specifications like BONDI that suggest a standard contract across all of these for “mobile web applications” that include interaction with the features of the local device (such as a camera or GPS) but they are years from broad adoption and don’t attempt to unify all API access down to a common control point.

Given that APIs are to application logic what RSS is for content, we know they will be very important; at least as important as the visible web that we use today and possibly more important.  This suggests that the other things that are spontaneously generated in value-exchange environments like user/customer management, behavior analysis, content filtering, caching, and security – will show up for APIs as well.

The web API equivalent of the browser’s control structure is an API proxy.

This is a control point which unlike a web proxy is fully aware of API content, communications patterns, and able to drive the meta-application controls discussed above.  An architecture like Google Analytics which is founded on a browser’s predictable algorithms cannot work in an API setting.  The same rule applies to add-ons that modify usage – they can’t do so relying on the local device if they are to be widely adopted.  But an API proxy – a server or service on the internet, sitting between the client (regardless of type) – is able to be that point of control.  As traffic runs through it, meaningful data can be captured for immediate outcomes (block access, change the message, or respond from a cache) and later used for behavior analysis and business planning.  Add-ons that modify usage of the API can be installed at this point (content filtering, adding new information such as advertising, or identity management).  All of this can be done while adhering to the contracts of the APIs and supporting the web architecture and rules of HTTP-based applications, and without attempting to solve the logarithmically complex problem of modifications to all the world’s clients.

So API proxies are likely to be necessary for the sustained growth of web and cloud API usage.  There are likely to be several nuances that end up differentiating the different implementations and providers of API proxies.  The key is to start experimenting with them now in order to build better apps and stay ahead of the competition.

Reflecting on 120 days of the CodePlex Foundation

Andy Updegrove wrote an article this week on the CodePlex Foundation's progress to date and where he'd like to see it go.  It put me in a reflective mindset, so I thought I'd take a minute to publish those thoughts. 120 days, in, what are the accomplishments of the CodePlex Foundation?

Back on September 10th, we set a pretty high bar for what the CodePlex Foundation would accomplish in its first 100 days of operations.

I said at the launch press conference:

“What are our plans for the first 100 days? We will recruit an executive director, replace the interim board with a permanent board, establish a Board of Advisors, and begin to solicit projects.”

So what did we actually get done?

Executive Director: "B" We have been working to recruit permanent staff, particularly an Executive Director and a Technical Director. We announced that search in December but we’ve been looking at candidates all along. Here I'd give us a B, because we are very close to making offers to outstanding candidates.  We realized that the technical leadership and mentoring role would be very important, so we elevated the priority of hiring a Technical Director.

Permanent Board: "C"  We’re almost at the point where we can announce a permanent Board of Directors. These will be names that resonate in open source, academic, and corporate software worlds. Look for an announcement this month. It was disappointing to miss this goal in the first 100 days, but the interim Board made a decision early to focus on the project acceptance process, and there we can claim real success. Driven by feedback from our advisors and directors, two of the current six directors will be returning to board, with three newcomers.  For the "replacing the Board of Directors" goal, I give us a C with strong effort. 

Board of Advisors: "A" We were fortunate to assemble a great Board of Advisors, who we learn from in every Advisors' meeting.  A very solid core of people has been involved right through and has been present for every issue.  People like Bob Gobeille, the head of the FOSSology project and a key contributor to FOSSbazaar, Aaron Fulkerson of Mindtouch, Larry Augustin of SugarCRM, and Stephen Walli provided extraordinary advice very early on. We also had great input from our Microsoft advisors Phil Haack, Scott Hanselman and Rob Conery, who are deeply invested in the open source ecosystem for .NET.  Sara Ford, Monty O'Kelley, John Lam, and Monty Widenius have contributed their expertise at critical points.  Also, we recently added Advisors Michael Bechauf, VP of Developer Network and Standards Strategy at SAP as well as Stuart Cohen, CEO of Collaborative Software Initiative and past Executive Director of the OSDL (now Linux Foundation).  As interim board members from the Board of Directors transition off as we move to a permanent board, we hope they will join the Board of Advisors. It’s a significant time commitment, so I suspect a couple of the current Board of Advisors will select out and move into a friends-of-the-foundation role.  So for the Board of Advisors, I'd give the Foundation an A.

Solicit Projects: "A-" Our biggest stretch accomplishment has been the acceptance of a technology gallery, the ASP.Net Open Source Gallery, and two projects, the ASP.Net Ajax Library Project and the Orchard Project. To get that result we first had to develop a project acceptance process, described in the Project Acceptance and Operations Guidelines document, which was released in October. With these guidelines in place we were able to accept a gallery donation and Board members Bill Staples and Shaun Walker drove that process. The Project Acceptance and Operations Guidelines document also described very strong guiding principles for the Foundation.  We also minted our first Gallery Manager (Dmitry Robsman).  Ideally I would have liked to have seen us also accept a community-led project and a non-Microsoft technology stack project.  We're making progress on both but didn't achieve these in the first 100 days, so I'd grade our progress at an A-.

Additional achievements:

"+" I think we did well at a technical level in introducing innovations in contribution agreements, legal structure, licensing agreements and other functional documents that will make it easier for corporations to participate in open source, and easier for developers to contribute code and projects without getting in the weeds with licensing and legal issues. While we do need to change the licensing agreement based on the experience we have had with community contributions, I think the assignment agreement is strong and working as intended.  So far it looks like the patent aspects of our agreements work as well. 

"-" We did not do so well in broadly communicating the structure and value of these innovations - people who we explained these to in small group settings or one-on-one understood pretty quickly, but we didn't figure out how to articulate our work in a way that it made sense at scale.

"√" We had another internal stretch goal of signing a second corporate sponsor in the first 100 days, which we didn't achieve but made good progress on.

Unsung labor:

This doesn't include all of the mundane work that the Directors and Advisors, our Deputy Director Mark Stone, our legal teams at K&L Gates and Greenberg Traurig, our operations team from Virtual Inc., our site hosting work from Engage Software, and our marketing team from Topaz Partners have done in raising a startup off the ground from scratch.  When we launched in September we had no infrastructure, no management practices, and no permanent staff.  We were a raw startup in every way.

One thing that many people that I've met don't realize is that the board members of a non-profit are non-paid volunteers.  Titles like Treasurer and President just mean that you are doing that much more work for free.  So I appreciate the support that each of our companies have given us to spend so much time on this - Microsoft for Bill, Britt, and Stephanie, DotNetNuke Corp for Shaun, Novell for Miguel, and my employer Sonoa all get our thanks.

Closing Thoughts:

I think we will continue to learn more as we go along, and our governance and legal documents may change as we get more advice. What we have to do better is explain the value of the Foundation. I think we have explained a lot of the intent and ideals, and we have connected on a detailed level with some community projects as well as some Microsoft sponsored projects, but we really need to step up to a new level of effort and be very clear and specific about what our agreements say, what results they are intended to produce and the process that wraps them up.

So the next few months will see changes on the Board, permanent staff, and progress in how we communicate our value propositions. Making progress there will let us attract new sponsors, especially when people begin to appreciate the value of the legal innovations we offer to corporate sponsors.  Longer term I'd like to transition to being a membership-based organization.

We have learned a great deal so far; we're grateful to those who have helped us get this far; and we’re excited about the next period of growth. Let us know what you think on our CodePlex Foundation Google group, or comment here.

Startup Adjustments

The patterns of day-to-day life are very different in a startup. 

At Microsoft, the sheer size of the organization builds a set of nearly automatic patterns, with constant demands for attention to specific projects and specific teams, and recurring requirements like quarterly financial statements, preparation for quarterly business reviews, monitoring and guidance of execution both locally at corporate headquarters and in worldwide subsidiaries.  The challenge was not in determining what will get done but how to compress my schedule and manage my team more effectively by teaching and encouraging, rather than directing and demanding.  With a full-time administrative assistant and a business manager (Microsoft’s equivalent of a chief-of-staff), more than 70% of my time was effectively directed by the undeniable needs of the business. 

Running a team of 120 people in 80 countries in a company of 91,000 requires not just worldwide timezone awareness, but connection to the corporate and local constituencies that those 80 people need to have on their side.  At Microsoft, 120 people is a lean team that must rely on many other organizations in order to execute anything from marketing campaigns to discounts to customer intelligence.  These dependencies include corporate and field marketing and sales for multiple channels (direct/enterprise, indirect/enterprise, indirect/midmarket, indirect/small business; direct/government, indirect/telecommunications… the list goes on but you get the point). 

Add to this the corporate governance team and process and you begin to see the layers of clockwork required to operate such a massive machine.  Political influence and personal connection is the lubricant that lets these gears run, so to be an effective leader you need to be available and involved in the workings of each of these adjacent gears.  This translates to understanding their challenges, finding ways to be relevant and helpful, and integrate your needs into their plans so that your goals are truly shared – you could say “adopted” in the context of adopting a child – by these other teams.

This is not much more complex than required – overall the structure at Microsoft is fairly lean for a company of its revenue and profit, and is constantly tuned by an executive who ran operations for Wal-Mart and his outstanding team.  It’s just that the problem itself is really hard, and the clock is always running.

Switching to a startup of 65 people (mostly engineers) with a shallow hierarchy and a flat leadership team structure as an individual contributor leading strategy is about as different of a day-to-day experience as I can imagine (apart from not working at all, of course).  No forced direction, no gaps between departments (including the fact that some departments are fully represented in a single person) and no mandatory schedule.  Seemingly endless expanses of open time on my schedule.  Titular responsibility for the company’s strategy. 

And what is strategy, really?  To me, strategy is a conversation – a conversation shared by all the operational leaders of an organization that accurately reflects the market, a belief in its future shape, and the appropriate actions required to match the company to that future shape.  If the conversation is managed correctly, the appropriate actions will show up almost automatically based on how each leader is thinking.  So to be effective in a role like this you have to be engaged with the operational leaders, understand and add value to their day-to-day work, and then reflect the company’s beliefs back to itself and to the market.  Start with a  blank slate, discarding your preconceptions, and go from there.  But starting from scratch takes much more effort than keeping pace with an existing flow.

Eight weeks later, I am finally (mostly) adjusted to the changes – from the trivial (managing my own schedule and coordinating with other people’s admins) to the substantial (feeling emotionally satisfied at the end of a week where I didn’t have “important meetings” scheduled every 30 minutes) to the crucial (entering the rhythm of the company and adding value without direct authority).

I’m satisfied with the changes and find that for the most part I’m more lighthearted, more flexible, and softer in style than I was at Microsoft.  My family certainly approves.  There is a certain “West Wing” feeling of constantly dealing with issues of world-shaking importance that is missing, but in that space I think I’m finding room for more thinking and writing, and interest in other aspects of what’s happening in the software industry and beyond.

Open vs. Free in Cloud APIs

Open and free are orthogonal in cloud computing APIs, but in this world the free aspect isn't typically interpreted as freedom, but price. Freedom in the cloud is more typically associated with data and your rights to move it from one service to another and this is a very important attribute. Application portability has also been raised as an important attribute but currently the application models are early in their maturity curve and vary from straight web programming with a REST or SOAP front end to prepackaged virtual machines to application frameworks like Microsoft's Azure and Google's AppEngine.

Open and free are both crucial attributes in order for a market economy to grow. There are many aspects of cloud computing but for the developers and users of cloud services, the atomic unit of the cloud is the API. Openness and economic growth is a deep subject that may provide clues for how to build a better cloud – in reading the linked paper, consider that a company's APIs represent the "export goods in which it has a comparative advantage". In the software industry open platforms have typically outperformed closed platforms in the long run due to the economies that develop on top of them, cementing those platforms' place in a range of markets.

Open APIs…

Open APIs in the cloud are:

• Openly documented
• Available via self-service (i.e. developers can sign up on a website, get an API key, with no hassle)
• Using open technologies (SOAP, REST, RSS)

An easily accessible example of this is http://bit.ly. There are enterprise services that meet these criteria as well, but bit.ly is clear and can be used to understand the broader applications of APIs. Bit.ly is a simple URL shortening service that also lets you see how many people have clicked on your shortened version of that URL. It's really useful if you want to both project important articles on the web and understand the reach of your projection.

Test 1: Openly documented

• APIs are documented here: http://code.google.com/p/bitly-api/wiki/ApiDocumentation)

Test 2: Available via self-service

• Go to bit.ly and you'll be able to sign up for an account right away for their direct website
• Click the "account" button at the top right and look at the second section, "API Key"
• You can now access their APIs programmatically by using this key

Test 3: Using open technologies

• You can see this is a REST API from reading the API documentation (granted, it is a mix of verbs and nouns)

… lead to 3^rd party innovation

Tweetdeck has already used this to integrate with them, so Tweetdeck users put their bit.ly API key into Tweetdeck, which then uses the Twitter API to talk with Twitter, and automatically uses their bit.ly account (indicated by the API key) to shorten URLs that are typed into tweets. It made Tweetdeck better and probably increases the traffic to bit.ly.

This could also work on the iPhone application of Tweetdeck but it's not yet implemented in the version I have. Many iPhone applications work by combining the native client capabilities of the iPhone with one or more cloud APIs that provide access to services in a clean, machine-friendly way.

From the efficiency of innovation perspective, keep in mind that Twitter never contacted Tweetdeck to use their API, nor did bit.ly (as far as I know).  The Tweetdeck guys simply built a killer application that uses those services via APIs rather than scraping their web sites (a brittle and slow approach).  In the last month, Tweetdeck also added Facebook and MySpace support via their APIs.  Again, efficiency and reach through designing your core business service to be "remixed" is found through APIs – Tweetdeck users got new value through the app they like and Facebook and MySpace got a new stream of user-driven content, all without sales or business development teams engaging at the outset.

If we take that example and scale that across different services offered by different businesses in different industries - media, financial services, marketing, e-commerce – we see that open APIs against these services are leading to increase usage and revenue due to new innovation by their partners and customers.  The classic example of this is eBay's statement back in 2008 that 60% of their traffic was coming through APIs rather than their website – 6 billion API calls per month. That API has been available since 2001, and what we're seeing in 2009 is an API tipping point as Dion Hinchcliffe articulated last November. 

Coming next: Free vs. Paid APIs.

Free Is Not The Opposite Of Commercial

Free software is not the opposite of commercial software. It is not an attack on commercial software; it can be a way of getting work done that enables software companies to collaborate with community projects for the betterment of both. Users can benefit, because they have access to software that is the product of the efforts of vibrant, diverse communities, dedicated to creating great software, efficiently and cost-effectively.

For those who have followed these topics for some time, this is not new information, but I wanted to take the opportunity to clearly establish the CodePlex Foundation's perspective on these issues.

There are successful software companies whose businesses are based on free software.

The CodePlex Foundation’s Board of Directors and Board of Advisors prove the point. The individuals who are contributing their time and effort to our success have decades of industry experience leading companies that have successfully built revenue and profit solely from free software. Ximian, VA Linux, MySQL, SugarCRM, and DotNetNuke are examples of how to run profitable software companies, built on a free software license base. Ximian (Miguel de Icaza) and VA Linux (Larry Augustin, Mark Stone) were built on GPLv2-licensed software technologies; MySQL (Monty Widenius) is licensed under GPLv2, SugarCRM (Larry Augustin) is based on GPLv3, and DotNetNuke (Shaun Walker) is based on BSD.

Ximian provided Linux-based applications for email (Evolution), messaging (Gaim), and productivity (OpenOffice.org) which were packaged into the Ximian Desktop and were all free software.  The company was acquired in 2003 by Novell.

VA Linux, which began life as one of first commercial businesses focused on distributing and managing Linux-based IT systems, provided companies with an alternative to more expensive UNIX-based workstations; the company has since changed its business model to become a publisher and provider of software development support. Now known as SourceForge, the company operates SourceForge.net, Slashdot, IT Manager's Journal and Freshmeat as well as Ohloh.

MySQL used GPLv2 to establish one of the early dual licensing businesses, offering a free community version under GPLv2. For companies that wanted to embed the database into proprietary products, MySQL was offered under a royalty-based non-GPL license.

SugarCRM in 2007 was re-licensed under GPLv3. It had previously offered its software under a custom license.  Like MySQL, SugarCRM has a dual-license business model: the Sugar Community Edition is available as free software, and the company also sells annual subscriptions to SaaS versions of Sugar Professional and Sugar Enterprise. This model has built value for SugarCRM, its partners, and also for developers looking for a good CRM system.

Shaun Walker of Dot Net Nuke built a great content management system on a BSD license.  Contributors sign a Contributor License Agreement (CLA) that enables the company to defend the project in the event of a legal dispute regarding intellectual property claims. 

There are also business models beyond software companies that can build profitable enterprises based on free software - including development services, hosting, and deployment services companies - which are beyond the scope of this discussion.

So from my perspective and that of the CodePlex Foundation, free software is not the opposite of commercial software.  The two terms are orthogonal and can either apply or not apply to any given piece of code. We also acknowledge the distinctions present in the community between “free software” and “open source software”.  We will tend to use the latter to describe the broad range of software projects that we want to support. We believe in making it easier to get things done, with the best tool for the job, with the best license for the tool; and we believe that by enabling more software companies to contribute to community projects, we can help advance the state of software.

Why It's Called the CodePlex Foundation

It's taken from www.codeplex.com, an open source project repository built by Microsoft a few years ago that has now expanded to host over 10,000 open source projects built by individual developers as well as roughly 500 by Microsoft employees.  The "code" part of the name refers to the source code which is the substance of an open source project.  The idea behind "plex", like metroplex or building complex is that it's a commons which serves a wide range of individuals. 

The CodePlex Foundation (www.codeplex.org) is intended to serve a wide range of projects, regardless of platform or technology base, to enable the exchange of code and understanding among software companies and open source communities.  Many people benefit from open source software, but a small number of people contribute back.  It seems to us that software companies should be in a very good position to contribute back based on employee skills and the benefit that these companies derive from specific open source projects, but in our experience there are a variety of obstacles that prevent or reduce contribution.  These range from development culture to copyright and patent licensing practices.

We are working with the leaders of other foundations, including the Apache Software Foundation and the Eclipse Foundation, to ensure that the CodePlex Foundation is a useful complement and enables community-based open source projects to expand their reach through increased contributions from software companies.  We are also receiving a great deal of thoughtful input (including Andy Updegrove's article and the community input on our Google Group).  There's a long way to go - this really is a brand-new startup and we are clearly in the alpha release - but from what I've seen I'm very optimistic that we can provide value to open source community projects.

Moving on and the CodePlex Foundation

After 5 great years at Microsoft I am moving on.  Yesterday, the creation of the CodePlex Foundation was announced, as well as my departure from Microsoft.  I'll be joining a cloud computing startup later this month in Silicon Valley.

It was a hard decision, as the time I've spent at the company has been both challenging and rewarding.  I remember a sense of disbelief when I first interviewed for the Open Source Technology Strategy role in December 2005 and Bill Hilf told me that one of the core responsibilities of the job would be a quarterly briefing with Bill Gates and Ray Ozzie on open source technology trends.  On the other hand, he said it would involve external public speaking - primarily to fairly polarized audiences.  That was a doubly scary proposition as I was not a public speaker and I knew the audiences quite well, since I'd competed with Microsoft for many years in prior companies and used Slashdot as my homepage. 

I was certain that open source was an industry wave that Microsoft would not be able to ignore, and that it was getting closer to an inflection point.  I had spent over a year at the company in the Silicon Valley-based  venture capital team under Dan'l Lewin and Cliff Reeves, and I could see that the company's approach to disruptive market dynamics was starting to change.  No matter what happened, I was sure that being at the center of the open source team at Microsoft during the coming years would be fascinating and important.

46 months later, I am amazed at the changes that have occurred for the company, for the team I belonged to, and the sentiments of the industry.  There is much work left to be done but the distance we traveled was remarkable.  I have a feeling that I'll be reflecting on this period for the next several months, so I won't try to cover it all here.  Matt Asay wrote a very generous article with a brief reflection of the changes which is here.  I've learned an enormous amount thanks to the intelligent, energetic, and fearless people that worked for me, the bold and honest open source community leaders who engaged with me, and the Microsoft executives who supported me.

I have heard some strange speculation on why I'm leaving and the answer is simple - for personal reasons, my wife and I have decided to move our family back to California.  I had many discussions with executives at Microsoft about how to continue my work from here, and received a warm reception along with a range of opportunities.  Ultimately I decided that I could not do justice to a corporate/worldwide position from afar, and that I could not bear to live away from my family and commute to Seattle five days a week.

So the change is made, and it has been very challenging.  But it's now complete and I am excited to launch the CodePlex Foundation and join my new startup.  As a close friend told me this week, "The best is yet to come" and I do believe that.  To everyone who has helped bring me this far, I say thank you; I am very grateful.

Architecture Strategies for Catching the Long Tail

Fred Chong has completed his work on the overview of the architectural guidance for building Software-as-a-Service (SaaS) applications on Microsoft infrastructure.  If you haven't seen it yet, check it out.

Fred spent several weeks in the field with SaaS providers who have built their applications on Microsoft in order to understand the key issues core principles in building a scalable, multi-tenant on-demand environment.  Many of the companies they worked with are startups managed by the Microsoft Emerging Business Team (EBT), and I had the privilege to connect with some of them.

The EBT is actively reaching out to and working with SaaS providers building on the Microsoft platform in order to improve their market exposure and business relationships.  If you are interested in connecting with the EBT, check out their SaaS page and drop a line to SaaSMS@microsoft.com

Open Source Strategy at Microsoft

I've been quiet on this blog for the last few months because I've taken on a new role at Microsoft: Open Source Technical Strategy.  At this point, some of you are thinking:

a) I'm joking
b) I'm crazy
c) I've joined a dark conspiracy

But in fact the truth is:

d) none of the above

As Director of Platform Technology Strategy (official title), I run the Open Source Software Lab at Microsoft, where we have hundreds of physical and virtual servers running 40+ distributions of Linux, 12+ variant of Unix, and several versions of Windows.  The research projects we do run from testing interoperability of network protocols like IPSEC and IPv6 between Linux and and Windows technology, the user experience and technical capabilities of HPC projects like ROCKS and Ganglia, to the broader attributes like size of developer base and changes in the development model for different Open Source projects.

We're also working with JBoss and SugarCRM on optimizing their open source applications for Microsoft infrastructure like Windows Server and SQL Server.  This has been fun, rewarding work that has helped to demonstrate the truth of our statements about working with Open Source.

Finally, I'm active in the Microsoft Shared Source Initiative, where I am responsible for Technical Strategy.  We are seeing some great work from inside the company - teams from all product groups wanting to contribute to Open Source in some way.  This week, Microsoft launched CodePlex in beta.  CodePlex is a developer community infrastructure hosted by Microsoft on behalf of Open Source developers - a place for code from both Microsoft product teams and the community to reside and for the developers themselves to collaborate.  Currently a dozen projects are there, ranging from IronPython to the Commerce Starter Kit.

I have a new blog at http://port25.technet.com, a site we've built to have a constructive dialog on Open Source, Interoperability, and Microsoft. 

Change is coming and I'm thrilled to be a part of it.  I'm particularly grateful to people like Andrew Aitken, Mary Coleman, and Matt Asay for what they taught me about Open Source and the Open Source community.

SaaS Architecture Guidance

I've had the privilege to work with Fred Chong and Gianpaolo Carraro, a pair of extremely bright guys who work in the Architecture Strategy group.  They have a passion for SaaS and are applying themselves energetically to mapping out the space of SaaS application architectures.  They realized that there is very little guidance or documentation on best practices for SaaS architectures in the community at large - while there is some shared knowledge and a set of conversations, no hard documentation exists in any comprehensive way.

Fred is now on a mission to build definitive guidance on how to grapple with tough issues like multi-tenancy (including defining what that means), security, customization, metadata, and operations readiness.

Fred has posted the outline of the SaaS architecture guidance book that he is developing on his blog, and I'm pasting some of the chapter outlines here as well.  Fred is an expert in Identity Management and Web Services management (he literally wrote the book for WSM at Microsoft), so he knows what he is talking about.

1. Introduction
2. Business Model
3. Application Architecture Overview
4. Scaling 101
5. Data Management
6. Tenant Management
7. Tenant Customization
8. Application and Data Security
9. Programmable Software Services
10. Programmable Software Service Consumption
11. Instrumentation and Monitoring
12. Configuration Management
13. Metering
14. Infrastructure Security
15. Operation Structure

If you're interested in this, drop by Fred's site and take a look - better yet, send him an email and tell him what you think.

Future of Commercial Open Source

I got to take part in a workshop in Santa Clara yesterday put on by SDForum - "The Future of Commercial Open Source" - unlike a standard event in the valley, this started with "opening comments" from the panelists and then broke out the 70+ attendees into 6 working groups, each discussing a question about "What will Commercial Open Source be like in 2010?"

Andrew Aitken of The Olliance Group was the instigator, and brought together a really outstanding set of people - Tim O'Reilly, Rod Smith of IBM, Simon Phipps of Sun were panelists, and the constellation of VCs and open source companies was amazing.  As a bonus, Mark Radcliffe of DLA Piper Rudnick summarized the first draft of the GPL 3.0. I've worked with Andrew Aitken in the past for insights into Open Source and given his knowledge and connections, it was no surprise that the event was a success.

It was no surprise that Microsoft was mentioned many times - but the big surprise to me was hearing Tim O'Reilly say, "If you consider companies that contribute to Open Source projects to be Open Source Companies, then you would have to say Microsoft is one, because they contribute a lot of source code."  He went on to say that his definition of Open Source companies is companies that use OSS, like Google, SugarCRM, etc.

Tim's insights:

• Open Source is about sourcing commodities - we should look at the core this phenomenon as the software supply chain enabled by Sourceforge
• Can SourceLabs, SpikeSource, and OpenLogic become part of the Sourceforge ecosystem?  This would mean that they are truly integrated into the software supply chain.  What about Palamida, Black Duck, and CollabNet?
• There is a linkage between OSS and the Long Tail - find niche markets and use OSS to source niche technologies
• Ultimately, success for OSS will come from applying the Dell model - instrument the front end and back end of your business and find the inefficiencies - this is where the opportunities for OSS ecosystem infrastructure lies
• By 2010, the business frontier and the hacker frontier will move.  The hotspots will be very different and we should expect to be surprised.  OSS will be part of every software business, and every company's strategy - much like the Internet has done.

Simon Phipps (who is the chair of OpenSolaris) commented that "customer progress with open source is much slower than the industry's progress, so we must be careful and not rush in where customers fear to tread."

We were told not to blog about the content of the roundtable "thinking sessions" themselves, and I won't, but I will say that I had the pleasure of working with some real luminaries in the space - John Roberts (CEO of SugarCRM, the most interesting OSS business model today in my view) and Mary Coleman of Walden International.  I learned a lot from the discussion and met some very sharp people.  It was very lively given our table consisted of two commercial OSS companies (SugarCRM and Alfresco), two commercial giants (Microsoft and SAP) and four VCs...

Finally Doc Searls had an amusing and insightful summary of the state of OSS and software in general, drawing heavily on the construction industry as a metaphor for the mature state of the software industry.

Where are the OSS/BSS companies for SaaS?

Most of the companies I've spoken with have had to build their own OSS/BSS systems to run their Software-as-a-Service operation.  Provisioning, metering, and billing are core business functions for service delivery organizations, and while there are custom aspects to them, and hence good reason to build your own, there are probably better reasons to buy them off the shelf… if they were available.

Companies like Portal Software and Convergys have long supplied these systems to the telco industry, yet I've seen no indication that they are moving to engage the SaaS segment; nor have I heard of VC-backed startups targeting this critical infrastructure space.

A related need is that for a robust pricing engine that is capable of handling multiple pricing schemes (i.e. per transaction, per user-month, or per CPU-month).  Again, this is a core business function, and as we saw in the great Telco battle between Verizon and AT&T Wireless, the winner is the one that can combine their difference service assets into the widest variety of packages.  The story behind this (from my perspective) was partly due to integration (Verizon made the investments required to integrate their dozen-plus backend billing systems) and partly due to pricing & packaging - the ability to target a new demographic or customer segment with a custom package that is more attractive than the competition.

One quote I heard in the last few months - and I apologize, I have forgotten who said it - is that in the end, telco margin per customer is probably the same between "friends and family", "small business", and "enterprise" packages… but the success of the business is in crafting the package that feels right to the target customer and incents them to make the purchase.  So Verizon won, because they used their pricing agility to win new customers.

The corollary for SaaS that I'm taking away is that pricing flexibility is going to be key to competitive advantage… which brings me back to "where are the OSS/BSS apps for SaaS?"

Scoble Interview: Enterprise Mash-ups

I got the chance to sit down with Robert Scoble for 20 minutes last Friday.  The thing that has puzzled me recently is whether mash-ups are for real, or just another Web 2.0 buzzword.

So I asked Robert: "Are mash-ups going to make a real difference outside of the digerati - do you think they'll make it into the enterprise?"

His take was: what's to prevent mash-ups from being the main way that departmental apps are built in most enterprises 3-4 years from now?

We've seen composite applications pitched for years, with some very cool companies like Digital Harbor building development environments for constructing them, but in a way mash-ups are an even simpler way to build a composite application - pull together a set of ICCs (still looking for the definitive term for this, as are others, it seems - see Harry Pierson's discussion).

So what would an enterprise mash-up server look like?  How would it have to be packaged in order to be adopted by departmental users?  Would it be hosted like the other new technologies that are breaking into departmental use (salesforce.com, MarketTools, ExactTarget, etc.) or would it need to be a local server?

One of the other issues Robert raised was that of attention, and its necessary evolution to make the computing experience more useful.  Currently, attention is captured and processed only on the server, which makes for some thorny issues about trust (who am I trusting with my attention data? and why?).  To deal with these issues it seems to me that there will need to be a client-side standard for attention, implemented by browsers for inter-ICC communication, as opposed to relying on the server to have good behavior.

Then, in a twist of synchronicity, I got to talk with a very early-stage company building a mash-up server that can run on the client or the server.  It's in stealth so I can't disclose much more yet... but I am becoming convinced that this programming model could be just the thing to build SaaS composites on.

[Updated to add reference to Harry Pierson's entry.]

Contrary Evidence at the SLC

At last week's IDC Software Leadership Council, opinions on Open Source were not the only surprise.  The IT Execs from 4 companies (Manufacturing, Healthcare, Heavy Equipment, and Financial Services) were generally against subscription models for software.  As the conversation progressed, two key statements came out:

1. The CIOs did not want to go with subscriptions because if their business grew beyond expectations, they didn't want to pay the software provider more ("Why should we share the upside"?)
2. The CIOs did want providers to reduce pricing if their business failed to grow to expectations

I think this was a case of two things - first, the natural expression of greed (wanting vendors to share the downside but not the upside) - but more importantly, a need for the industry to provide clearer guidance on standard pricing models.  There was a clear conflation between value-based pricing and usage-based pricing in the mind of the customers.  Additionally, they were concerned that in the long run it would be cheaper to go with perpetual licenses and pay maintenance, dealing with amortization internally, rather than pay a subscription cost which would never go down.

So where do we go from here to advance the understanding of the SaaS industry's offer to CIOs of mainstream companies?  Are the doubters right, and will SaaS fail to penetrate large enterprises because of these issues?

Disruptive Innovation and Darwin

I have great respect for Clayton Christensen, who has done serious work putting real meaning into the term "disruptive innovation".  He's quoted in the latest Newsweek article on Skype, saying:

Still, the novel pay schemes "are hugely interesting," says Harvard Business School professor Clayton Christensen, who cites both Kazaa and Skype as great examples of the "disruptive innovations" he studies. "It's almost never the technology itself that paralyzes the incumbent—it's that the incumbent has a system for making money that's irrelevant to the new guys."

The combination of companies growing up after the bubble burst, dealing with recession and market pressure by using free software and then reducing customers' cost to use through "rightsized" pricing (moving amortization costs of hardware and software from the customer to the vendor, and reducing overall management costs through centralization) has produced some very fit organisms - not unlike biological organisms that have deveoped in impoverished environments, they may be tougher than the incumbents of enterprise software.

Axel Shultze of BlueRoads has said that no one originally set out to get to the current state of SaaS - it was a series of changes to fit the customer environment that led the industry here.  It will be very interesting to see where the intersection of pay-as-you go and ad-funded software business models leads us next.  I think Darwin would be having fun right now watching this industry evolve in real time.