Test the Feature Before You Build It
Validate a risky feature with SQL queries and emails instead of feature code.
- Difficulty
- Easy
- Time to result
- ~weeks to results
- Steps
- 5
- Confidence
- 83%
Before building a streaks and achievements feature that risked undermining users' intrinsic motivation, Superhuman ran the idea through old-fashioned product management with no feature code at all. The team wrote SQL scripts against existing usage data, analysed the metrics, then created what Vohra calls a little email feature: they took a sample of the user base, emailed those users insights about their own behaviour in Superhuman, and analysed what came back. They ran roughly three rounds, switching up which statistics were sent each time, analysing sentiment and digging in with customers. The finding was that some statistics are not particularly compelling while others are — streak statistics in particular were something people loved to know about. Only then did the team gain enough conviction to build.
Origin
Extracted from Acquired: Vohra describes how Superhuman validated its streaks feature after arguing that reward mechanics usually backfire.
Core principles
- 01The cheapest version of a feature is a query and an email.
- 02Conviction comes from customer sentiment, not from a plausible argument.
- 03Most candidate content is not compelling; you have to find the one that is.
- 04Iterate the test, not the implementation.
- 05Ship only once you can articulate why it will not backfire.
How to run it
- 1
Name the risk, not just the idea
Write down what specifically could go wrong if the feature is built as imagined — here, that a reward would undermine the motivation users already had.
Pro tip A named risk tells you what the test has to measure.
- 2
Query the data you already have
Write scripts against existing usage data to produce the insight the feature would have surfaced, and look at the metrics yourself first.
Pro tip Existing behavioural data usually contains the whole feature in raw form.
Watch out Do not write any feature code at this stage.
- 3
Deliver it by hand to a sample
Email a sample of users their own insights, so the value proposition is tested without any interface being built.
Pro tip Manual delivery also lets you vary the framing per round with no engineering cost.
- 4
Analyse sentiment, then iterate the test
Read what comes back for sentiment rather than counting clicks, swap the statistics you send, and run several rounds until the compelling variant separates from the noise.
Pro tip Three rounds was enough at Superhuman to distinguish loved statistics from ignored ones.
Watch out One round tells you about your first guess, not about the feature.
- 5
Build only on explicit conviction
Write down what you now believe and why, including the conditions under which the feature works, then build to those conditions.
Pro tip The conditions usually constrain the design — for streaks, that it stay off centre stage and celebrate rather than nag.
In the wild
Superhuman wanted to know whether achievement statistics would help or hurt. Rather than building, the team wrote SQL scripts, analysed the metrics, then emailed a sample of users insights about their own behaviour, running about three rounds while switching up which statistics were included. Sentiment analysis and customer conversations showed that most statistics were unremarkable but streak statistics were something people genuinely loved to know about, which is what justified building the feature at all.
→ Conviction to ship a specific mechanic, and evidence about which statistics to include.
Common mistakes
Testing with a single round
One batch of statistics only tests your initial guess. Multiple rounds with varied content are what separate the compelling insight from the forgettable ones.
Measuring engagement instead of sentiment
Opens and clicks say the email arrived. What the team needed was how people felt about being shown the statistic, which only comes from reading and discussing replies.
Is it for you?
Best for
Founders and product managers testing a data-driven or insight-driven feature where the content itself is the risk.
Not ideal for
Features whose value depends on real-time interaction that cannot be simulated outside the product.
From the transcript
“We didn't write any feature code. What we did was we wrote a bunch of SQL scripts. We then analyzed the metrics. We then created…”
“We did about three rounds of this, switching up the stats that we would email, analyzing the sentiments”
From the episode
Special: Superhuman Part II - Designing Software to Feel like a Game (with Rahul Vohra)