I ask because I swear A/B and multivariatate tests have been around my head a lot lately, and when I finished reading the article, the first thing I thought was: Why not just deploy to 1% of the users and see if it works?
Then I thought about how hard would be to manage multiple versions of the same software, specially data, amongst different user. Certain features presented to the 1% might be incompatible with the other 99%. But that's a technical problem. Very hard to solve, but manageable. Then I imagined somekind of framework that would make communication between different versions of the data floating around easier, with "how to transform" data from-and-to version 1.1 and 1.2 easily.
Anyway, I am really curious, because it sounds like a good solution :)
As far as data transofmration goes you have two options:
1. Something akin to ActiveMigration is RubyOnRails world. This allows going back and forth different versions of your data's schema.
2. Use a more open data scheme such as Google App Engine uses where adding/removing properties to an object is not as disruptive compared to SQL-based solutions.
It turns out the system works fine with SQL based alters. We do have to do real work to deploy expensive alters (apply them to standbys, fail over, repeat, or worse) but in general it's cheap to change schemas.
Unfortunately, it's very manually intensive to roll back schema changes, so it's one of the few places where we put old school process in place (a DBA who reviews all schema changes prior to deployment)
ActiveMigration really solves a different problem. Our problem is that adding indexes or altering popular tables is impossible to do on a live and in production database. To get those changes out we have to go through quite a bit of extra work. It's really a MySQL limitation, not a process problem.
I ask because I swear A/B and multivariatate tests have been around my head a lot lately, and when I finished reading the article, the first thing I thought was: Why not just deploy to 1% of the users and see if it works?
Then I thought about how hard would be to manage multiple versions of the same software, specially data, amongst different user. Certain features presented to the 1% might be incompatible with the other 99%. But that's a technical problem. Very hard to solve, but manageable. Then I imagined somekind of framework that would make communication between different versions of the data floating around easier, with "how to transform" data from-and-to version 1.1 and 1.2 easily.
Anyway, I am really curious, because it sounds like a good solution :)