I used an infrastructure site as an example because the value proposition is easy to understand when you use a site that has a clear and simple monetization strategy. Video sharing sites are arguably an even worse example than S3, because the value of uptime is so hard to perceive or compute. It's likely that even Twitter doesn't understand the true value of a customer-hour of Twitter uptime, because the site isn't monetized and so much of the value is concentrated in the brand. Measuring that is like voodoo, only less empirical. ;)
If I have 5 users and the site goes down for 5 minutes, those 5 users will get 5 minutes each of annoyance. If I have a million users, each of those users will get 5 minutes of annoyance each also. There is no difference to the user there.
No, but there is a big difference for you! If a user is worth a dollar per year, the five-user site is worth five bucks per year, but the million-user site is worth a million bucks. If each patch to your code causes 0.1% of users to abandon your product (a number which depends on the odds that a patch will cause a rollback, and on the odds that a rollback will annoy a user enough to make them leave), patching a 5-user site costs you half a cent per year on average (most likely it has no perceptable cost, since odds are no users will leave) but each patch to a million-user site costs you $1000 per year in revenue. And that's just the linear cost. There are nonlinear consequences: one or zero annoyed users is nothing to worry about -- unless that user is Michael Arrington -- but a clique of 1000 annoyed users is potentially a movement: a critical mass of people who will all start complaining about your company on Twitter on the same day, potentially costing you your next 10,000 or 100,000 or 1 million users while simultaneously empowering your competitors, who may begin building the site that will take you down by poaching those dissatisfied users.
This is just the flip side of scalability. As a programmer you enjoy mighty economies of scale: Running a site with a million users is more expensive than running a single-user site, but it is much less than a million times as expensive. But this leverage also applies to your mistakes: a mistake that costs you a dollar when your site is small might cost you $1,000,000 when your site is big. And it's the same mistake! Typos are just as easy to make on big sites as on small ones.
Obviously, this doesn't mean that you shouldn't ever change the site. Presumably each and every one of your patches is valuable, and will bring in revenue to pay for its own insurance premiums. Right? :) But you do need to think about that calculation, because you do occasionally make mistakes. As your userbase grows, you may wish to test each patch on a subset of users to be sure they will really like it, and that the additional revenue is really going to be there. You may wish to institute tests and internal audits that lower the risk of rollbacks, or failover mechanisms to lower the cost of rollbacks. And before long, lo, you will be that which you deplore: A company with a bunch of annoying internal controls! But at least you'll have revenue to console yourself with.
But I think what Dustin is saying (correct me if I'm wrong) is that the multiplier applies both ways. And that the total cost of making a 5 minute downtime mistake, even to a million users, could easily be outweighed by the benefits of releasing a product/feature/site 2 weeks early. In most cases, I think large companies are risk adverse instead of risk neutral to situations like this.
I agree with both of you that it varies considerably based on what the site does (infrastructure, videos, games, etc).
In most cases, I think large companies are risk adverse instead of risk neutral to situations like this.
I'm not going to argue with that. Just because a certain increase of caution is rational doesn't mean that caution isn't being overapplied in many cases, just as PG suggests in his original post.
If I have 5 users and the site goes down for 5 minutes, those 5 users will get 5 minutes each of annoyance. If I have a million users, each of those users will get 5 minutes of annoyance each also. There is no difference to the user there.
No, but there is a big difference for you! If a user is worth a dollar per year, the five-user site is worth five bucks per year, but the million-user site is worth a million bucks. If each patch to your code causes 0.1% of users to abandon your product (a number which depends on the odds that a patch will cause a rollback, and on the odds that a rollback will annoy a user enough to make them leave), patching a 5-user site costs you half a cent per year on average (most likely it has no perceptable cost, since odds are no users will leave) but each patch to a million-user site costs you $1000 per year in revenue. And that's just the linear cost. There are nonlinear consequences: one or zero annoyed users is nothing to worry about -- unless that user is Michael Arrington -- but a clique of 1000 annoyed users is potentially a movement: a critical mass of people who will all start complaining about your company on Twitter on the same day, potentially costing you your next 10,000 or 100,000 or 1 million users while simultaneously empowering your competitors, who may begin building the site that will take you down by poaching those dissatisfied users.
This is just the flip side of scalability. As a programmer you enjoy mighty economies of scale: Running a site with a million users is more expensive than running a single-user site, but it is much less than a million times as expensive. But this leverage also applies to your mistakes: a mistake that costs you a dollar when your site is small might cost you $1,000,000 when your site is big. And it's the same mistake! Typos are just as easy to make on big sites as on small ones.
Obviously, this doesn't mean that you shouldn't ever change the site. Presumably each and every one of your patches is valuable, and will bring in revenue to pay for its own insurance premiums. Right? :) But you do need to think about that calculation, because you do occasionally make mistakes. As your userbase grows, you may wish to test each patch on a subset of users to be sure they will really like it, and that the additional revenue is really going to be there. You may wish to institute tests and internal audits that lower the risk of rollbacks, or failover mechanisms to lower the cost of rollbacks. And before long, lo, you will be that which you deplore: A company with a bunch of annoying internal controls! But at least you'll have revenue to console yourself with.