Assumption Testing

• 01:06

  

  A real-life, willingness to pay story.

  Why we can't simple ask, "What would you pay for this?"

• 00:54

  

  A/B tests are not the best discovery tools.

  But they are great for measuring impact.

• 00:50

  

  Assumption tests simulate a specific moment.

  They don’t test the whole idea.

• 01:02

  

  Assumption Tests vs. Experiments

  One is for discovery, one is for measuring impact.

• 00:43

  

  Design your assumption tests for the best case scenario.

  It’s not cheating.

• 01:10

  

  Does assumption testing change with product maturity?

  Not really.

• 00:48

  

  Don't ask customers what they are willing to pay.

  3 things you can do instead.

• 01:06

  

  Don’t get stuck assumption testing.

  Make decisions based on what you are learning.

• 01:03

  

  Don’t use percentages in your success criteria.

  They are ambiguous.

• 00:59

  

  Don’t waste time categorizing your assumptions.

  Use the categories to generate assumptions.

• 01:30

  

  How do you make decisions based on the results of your assumption tests?

  Remember to compare and contrast.

• 00:57

  

  How to evaluate your assumption test results.

  Compare and contrast.

• 01:14

  

  How to set success criteria for assumption tests.

  Our goal is to get so specific that someone else can call the win or loss

• 01:14

  

  My organization won’t let me run assumption tests.

  Don’t ask for permission.

• 01:22

  

  Stop validating your ideas

  You are inviting confirmation bias in.

• 01:09

  

  Test assumptions across your solutions.

  Not one solution at a time.

• 01:11

  

  The value of early signals.

  And why we don’t want to start with large-scale tests.

• 01:08

  

  Two tools every product team should have access to.

  No, I'm not endorsing specific tools.

• 01:31

  

  Understanding false positives and false negatives.

  They will happen in assumption testing (and large-scale A/B testing too).

• 02:15

  

  We are not scientists.

  We don’t have time for that.

• 01:17

  

  What do you do if an assumption test fails?

  Get used to it.

• 00:45

  

  What do you do if an assumption test passes?

  It depends.

• 01:28

  

  What is an assumption test?

  A structured activity designed to help us evaluate risk.

• 00:38

  

  When should product teams run assumption tests?

  All the time.