Sign up for our newsletter! You won't regret it.
A/B Testing Etsy Marketing Small business Statistics Uncategorized

What’s the Benefit of Controlled Experimentation?

Science Experiment with Beakers

Hindsight is 20/20.

Looking back on the medical and scientific practices from hundreds of years ago, it’s easy to point out lots of things that were less-than-perfect. Drilling holes in people’s skulls in response to personality disorders seems like overkill. But smart, creative people lived in every age, making huge breakthroughs based on the knowledge of those who came before them. Because of them, some things seem obvious to us now — hundreds of years ago, not so much. What I find really intriguing though are the cases where a technology was proven to be effective but was for some reason slow to catch on. The use of antiseptics to reduce infection rates in surgery was published in 1867, but decades passed before their use became widespread. Jeez, people.

Are there technologies like this today? Ideas that are proven to work but are massively underutilized? Ideas that people hundreds of years from now will look back on and say, what took so long?  Well, I think so. And the kicker is, it’s hundreds of years old but nowhere close to realizing its potential.  This idea is controlled experimentation.

So. Many. Variables.

The idea of a controlled experiment is really simple at its most basic level: if you want to know whether a certain something affects a certain result, you need to hold EVERYTHING equal and only change that particular thing.


Does eating a banana make you run faster? You can’t just eat a banana and then go for a run and see if you’re faster. After all, maybe you eat bananas in the morning, and you always have more energy in the morning, and it has nothing to do with bananas. But if everything is exactly the same – the time of day, your other meals, your sleep, etc. — then you can confidently say your faster run times are because of the banana.  Controlled experiments are (by definition) experiments where statistical methods are used to “hold everything else equal”.



Controlled experiments are more commonplace today than at any time in history — they are the basis of most academic research in medicine, psychology, and economics; they are the standard required by regulatory agencies like the FDA to determine whether a new drug or medical device is effective; and large companies like Google, Amazon and Facebook run thousands of experiments every day to optimize every aspect of their business.

Falling back on guesswork?

Despite this, so many of the decisions that people and businesses make are based on guesswork and anecdote. How should you pick what clothes to wear for an important meeting? What keywords or photos will bring traffic to your site? Will eating some foods make you sleep better? Wouldn’t life be just a little easier if you knew, REALLY knew, the answers to questions like these?

There is — or at least was — a very good reason that experiments are not more commonplace:  controlled experiments can be pretty difficult to execute because need to be designed and set up with a lot of precision, they are logistically complicated to run (imagine trying to figure out every variable and then keep it constant on top of that), and they require statistical knowledge to analyze if you want to actually use the results. But I said was. Now, this whole process can be automated, which means that controlled experimentation is now accessible to anyone and everyone! (Not just major organizations with tons of resources.) And people can take advantage of it even if they have no idea what an experiment is or what problem it solves. We can move from a world where experiments are the exclusive domain of experts who occasionally pass down a nugget of wisdom via news articles, to a world where everyone can make data-driven decisions informed by actual experiments.

We’re making moves.

I wanted to found Whatify because I wanted to see that become a reality. Instead of lying awake at night because you unknowingly ate something that makes it hard to sleep, algorithms will learn what foods keep you awake and help you avoid them. Instead of wondering why your Etsy shop isn’t growing, you can test your listings and find out what will increase your traffic (hint hint, nudge nudge—that’s what we do).  I wanted to create a way that people can automatically set up hundreds or thousands of experiments and on our end we run them, analyze the results, and put those results to work for you.

Sure, there are other analytics services out there, but not like ours. Usually analytics means that you get numbers which help you make informed decisions. You might see trends in your sales of different products, or you might learn what keywords other successful sellers with similar products have used.  This information is helpful, but it can be misleading and sometimes it’s confusing what to even DO with the information in the first place. If sales have been trending downward for your best selling product, should you advertise it more? Or perhaps advertisements won’t help because your customers have found substitutes? If a successful shop uses the tag, “vintage picture frames”, are they successful because of that tag or in spite of it? How can you even tell?



Product Photography


Creating data that people can actually use to help their businesses succeed was a huge motivator for me. By utilizing controlled experimentation, we can translate analytics into action: if in a controlled experiment, photo B generates more sales than photo A, you should use photo B and your sales will increase. If using the tag “vintage picture frames” drives traffic in an experiment relative to your existing tags, you should use it. It’s that simple! Because we do controlled experiments, Whatify doesn’t just give you a bunch of numbers and trends. We give you actionable recommendations, and we’ll implement those recommendations if you let us.

If ya got it, use it.

If you’re saying, “but I already make evidence-based decisions,” ask yourself about the kind of evidence that you are using. Maybe you haven’t done an experiment yourself, but you read a magazine article about what foods keep you awake, and it was based on scientific research. You didn’t just guess when you chose your product photo, you thoroughly researched what photos work best. That’s a good start! But even so, you’re only scratching the surface of what a data-driven approach can truly tell us. A website with “Etsy photo tips” might tell you to feature your product in a “lifestyle” shot as it would be used in practice, but does that really work? Is it a lifestyle shot better than a photo which features a more direct photo of your product? The truth is, it depends. Having analyzed hundreds of thousands of photos, we’ve learned that even the best rules do only a bit better than chance. Instead of using simplistic (even evidence-based) rules, why not find out what ACTUALLY works best in every specific instance?


Reading a Magazine

Let’s do this.

When people hundreds of years from now look back on the 20th and 21st century, they will wonder: why did they do so much guessing? I really believe that the idea of using controlled experiments to make decisions will seem as obvious as cars having round wheels rather than square ones. Even what we normally consider a well-researched and informed decision might depend on unreliable assumptions and extrapolations. With Whatify, you can stop guessing and start using data to figure out what works.

0 comments on “What’s the Benefit of Controlled Experimentation?

Leave a Reply

Sign up for our newsletter! You won't regret it.
%d bloggers like this: