Testing a startup idea before writing code: problem interviews and smoke tests

TUREFF• •0

The most expensive way to find out nobody wants your product is to build it first. Founders, especially technical ones, are tempted to start coding because it feels like progress. A few weeks of disciplined testing can tell you whether the problem is real, who has it most and whether they will act, before you commit months of work. This guide focuses on techniques suited to tech startups; general business validation such as pre-sales and competitor research is covered elsewhere on the site.

Write your riskiest assumptions down

Most startup ideas rest on a chain of beliefs: this group has this problem, it hurts enough that they look for a fix, they can be reached affordably, and they will pay or change habits. List them, then rank them by how badly the company fails if each is wrong and how little evidence you have. Test the top two or three first.

Problem interviews done properly

Customer conversations are the cheapest research you can do, but they are easy to get wrong because people are polite.

Talk about their life, not your idea. Ask about the last time the problem happened, what they did, what it cost and what they tried.
Prefer facts about the past to opinions about the future. "Would you use this?" invites a kind yes. "How did you handle it last month?" gets evidence.
Look for existing spending and workarounds. Spreadsheets, hired help and duct-taped tools mean the pain is real.
Do not pitch until the end, if at all. Once you describe your solution, people start being nice to you.
Ask for a commitment. An introduction to a colleague, a follow-up meeting or agreement to try a prototype costs them something. Compliments cost nothing.
Talk to enough people in one segment to see patterns, and write up notes the same day.

Rob Fitzpatrick's book The Mom Test is a popular short guide to this style of questioning.

Smoke tests

A smoke test measures whether people act on a promise of the product before it exists.

Landing page: describe the product and its benefit, with one clear action such as joining a waiting list or requesting access. Drive targeted traffic to it and measure the share who act.
Fake door: inside an existing product or site, add a button for a feature you have not built. When people click, explain it is coming and ask if they want to be told. Be honest and do not take money for something that does not exist.
Explainer video: show how the product would work. Dropbox is the classic example: a short demo video shared in 2008, while the product was still in development, helped its beta waiting list grow from about 5,000 to 75,000 people overnight, according to the company's own account.
Pricing test: show a price, or several plans, before the sign-up step, and see whether interest holds up once cost is visible.

Manual first

For many services you can deliver the outcome by hand to a few customers before automating anything. Zappos began, as its founder has described, by photographing shoes in local shops and buying a pair only when someone ordered online. Doing it manually shows you what customers value, what they ignore and what the real workflow is.

Set the bar before you run the test

Decide in advance what result would count as success, what would count as failure and how long the test runs. Otherwise every result looks encouraging after the fact. Track where traffic came from: friends and family will sign up out of loyalty, so their numbers are not evidence.

Respect people and the law

Collect only the data you need, tell people what you will do with it and follow privacy rules where you operate. If you take any payment before launch, be clear about delivery and refunds. A lawyer can tell you what applies to your situation.

What to do with the results

Strong action from a clear segment: build the smallest product that serves them.
Interest but no action: the problem may be real but not urgent, or your message is wrong. Change one thing and test again.
Little interest anywhere: be glad you found out early, and move on to the next idea or the next segment.

What is the smartest test you have run before building? Share it in the comments.