Early Q4, I presented a thought experiment to the team. If a CEO barged into the office of a product owner who’s responsible for buying our product and said ‘Hey, we need to stop paying for this Local Logic stuff’, what would they answer?
Our clients buy our products because they believe we help their website better match people and places, increasing the number and quality of leads going to their agents. I intentionally use the word believe because until now, it was just that, faith. Their intuition was telling them that our products have an impact on their business but we had yet to arm them with data supporting it.
So in Q4, we set out to quantify the value of our products for all stakeholders involved: our clients (brokerages), our clients’ clients (agents) and our clients’ clients’ client (homebuyers). This piece is about how we quantified the value for that last stakeholder, homebuyers.
I don’t want to bury the lede: 66% of homebuyers would be very disappointed to see our Local Content product go away. But without context and some background, it’s difficult to understand what that means.
To NPS or not to NPS
Net Promoter Score is the gold standard when it comes to quantifying the value of a product in the B2B space. You’ve seen it before, it’s the annoying pop-up that asks, “On a scale of 1 to 10, how likely are you to recommend this product to a colleague or friend?”
Unfortunately, NPS doesn’t really work for us because our users have no idea what “Local Logic” is. They don’t differentiate between our clients and us, a third party provider. We feared that asking users to complete an NPS survey about our product would yield results about how likely users would be to recommend our clients. So we needed something else.
While going through First Round Review’s amazing newsletter1, I read about Superhuman’s framework for calculating whether they’d hit product-market fit. Similar to us, Superhuman felt like they were working on something important with raving reviews here and there, but they lacked a leading indicator of how valuable they were to their users.
So, instead of NPS, they asked how disappointed their users would be if Superhuman were to suddenly stop existing. Like the old adage “You don’t know what you got til it’s gone”, Superhuman’s frameworks asks, if we were gone, how disappointed would you be: very, somewhat or not at all?
We thought this framework would be a great fit for us. First, it clearly differentiated us—neighborhood information—from our clients while remaining concise. Second, it gave it us a glimpse into why users found our products valuable (if they would be very disappointed to see us go) and how we can improve (if they would only be somewhat disappointed) thereby informing our roadmap. Third, by asking the user to self-identify as an agent, investor or a homebuyer, we can understand our value for every stakeholder.
Here’s what it looks like:
Real good & caveats
Now the meaty part. Here are the topline results:
“How disappointed would you be if you didn’t have this neighborhood information during your home search?”
66% of users would be very disappointed
23% would be somewhat disappointed
11% wouldn’t be disappointed
This is overwhelmingly positive news for us and our clients! How positive? Research into this methodology shows that traction is strongest for companies above the 40% bar. Slack had clocked in at 51% very disappointed early on.
Our results do come with a few caveats. The users of our products do not pay for them.It’s easier to let go of something when it’s not making a hole in your wallet. Also, flipped on its head, we still have a fifth of users2 out there who expect more out of our products—who wouldn’t be disappointed if we were gone. We only want disappointed users! So going forward, 66% is our baseline. We’ll look at every new feature and say would this disappoint more people if it didn’t exist?
That being said, this baseline is enough for a great conversation between any product owner and CEO:
CEO: “Hey, let’s stop paying for this Local Logic stuff!”
Product owner: “Wait, no. 66% of our users would be very disappointed to see Local Logic go. Only 10% of users agree with that decision. ”
1 10/10 would recommend to a friend (which I have!) but would only be somewhat disappointed if they no longer existed (there are a lot of newsletters out there!)2 The "not disapointed" users (11% in our case) are usually considered a write-off since they are too far from your use case to ever be convinced.