It is because I don't want to serve the people that it would be effective on.
If you're marginally buying / not buying my product based on color scheme or (buzz)word order or some other piece of puff, we're all better served if you just moved along.
So you think you and your "target customers" are immune to the effects of subtle changes? You and your customers are the lucky ones with complete, unfiltered, and unbiased access to the conscious and subconscious (well, not to you) processes going on in your brain that influence your day-to-day life?
Just a thought -- this POV implies that there is a single way to explain every piece of information on your site that is optimal for each person who might be a customer. (The corollary is that all your customers are the same.) A person doesn't have to be swayed by a buzzword, color scheme, or fluff to respond to different presentation of facts. You know this is true if you've ever had to explain anything to anyone. Some examples of how different presentations of your content might affect an individual:
- English: first language?
- Country/region of origin?
- What did they read before coming to your site?
A lot of things influence human behavior, and A/B testing isn't just to tease out the fluffy bits you can use to sell to the gullible.
I'm a customer of yours. I'm in your target market (guessing: technical person who wants a proper backup solution).
However, I would never have found or bought your product if I hadn't seen your link on a thread on HN about git-annex a few days ago. If I hadn't followed that link specifically from talking about a project I'm really enthusiastic about I doubt I would have bought.
> It is because I don't want to serve the people that it would be effective on.
If you remove the set of all people it will be effective on from the set of all humans then you will get ∅.
If your A/B testing is about color schemes and buzzword order, you're doing it wrong. Random spaghetti testing is a surefire way to waste your time.
“Green vs orange” is not the essence of A/B testing. It’s about understanding the target audience. This starts with research, and your hypotheses are validated with split testing. Doing research and analysis can be tedious and it’s definitely hard work, but it’s something you need to do.
Serious gains in conversions don’t come from psychological trickery, but from analyzing what your customers really need, the language that resonates with them and how they want to buy it. It’s about relevancy and perceived value of the total offer.
Unless you have the ability to foresee the future, it's impossible to know in advance which language, content and layout will resonate the best with your target audience.
Most of the case studies around A/B testing is very misleading. The value you get from testing color schemes is likely to be insignificant - unless you have a really horrible color scheme as one of the options or you have traffic comparable to Googles and Facebooks of the world.
That does not mean you shouldn't test. You definitely need to test and learn from your customers. There are other better ways of doing it, than just A/B testing as is often advocated.
Is your desire to only serve certain people based in the long run on profit motive (i.e. you think your fewer customers will be more valuable and stick around longer), or is it just out of principle even if it means less profit in the long run?
How do you know your current design is excluding the right people? The people who are buying your product now might just be reacting to your current color scheme (or lack thereof) or word order.
I don't mean this to be snarky so please don't take it that way. I don't see where color scheme, button text or carousel slide order will make a difference if you're selling something someone really needs, is informed about, etc. Maybe the argument is less "red buttons get 0.8% more people to buy" and more "red buttons get people to read our copy 40 seconds longer and that means 0.8% more people will understand what we're selling," but it's always been presented as "convince those on the edge to buy whether they understand/need the product or not."
One thing I learned early on from working at a college computer lab in my late teens is that people very often don't even see certain elements on the screen. The post I responded to made it sound like people are deciding not buy because they consciously evaluated the button color and didn't like it.
Maybe people didn't notice the button when it was a different color. An A/B test couldn't tell you that but having watched a lot of people using unfamiliar software, that seems a lot more plausible.
If the point of a website is to communicate then A/B tests are a good way of making sure that you're communicating effectively.
> I don't see where color scheme, button text or carousel slide order will make a difference if you're selling something someone really needs, is informed about, etc
This is true if you are unique in the market of a lot better than competitors. But if you are only 5% better even people who know quite a lot about your services and niche will not always be able to tell.
Agree that "white hat" conversion rate optimisation should be about getting people to understand the product
If you are a consumer facing site, every single customer of yours will have a point where they know nothing about you and need to decide whether to go with you or a competitor. That's the kind of thing that A/B testing is good for.
However if you've got an "enterprise product" that people have to use because their management tells them that they have to, then this does not apply to you.
So what you say can be true. But is probably true for fewer companies than think it is true about them.
It is because I don't want to serve the people that it would be effective on.
If you're marginally buying / not buying my product based on color scheme or (buzz)word order or some other piece of puff, we're all better served if you just moved along.
[1] You know who we are.