October 3, 2026
AI made building cheap. It didn't make knowing cheap.
The estimate came back this week: 84 hours.
The feature is an integration checklist. Our integration plugs into a big ecommerce platform that merchants can customize in almost any way they want. That flexibility is great for them, and a headache for us, because their customizations can quietly break parts of our integration. The idea is simple: give merchants one place, inside their own admin, that shows what's working in the integration and what's missing, so they stop wondering what's wrong.
A few years ago, 84 hours would have started a debate. That's two weeks of a developer's time. Is it worth it? What does it push back? Someone would write a doc, someone would push back, and we'd spend a week deciding whether to spend two weeks.
Today, with a coding agent, a first version could probably exist in a few days. So the debate feels silly. Just build it.
Except that "can we build it?" was never the hard question. It just used to be the expensive one, so it got all the attention. Building got cheap. Knowing didn't. Here's what AI didn't change: finding out whether the thing you built actually worked.
And for most products, that's still slow and expensive. Three reasons.
You don't have enough users. This checklist would be seen by 300 to 400 merchants. That's it. When people say "AI lets you run hundreds of experiments," they're picturing an app with millions of users. With 400 merchants, I can't test five versions of a dashboard and pick a winner. I can barely test two. And integration problems don't happen every day, so even a single test needs weeks before it tells me anything.
I've started calling this the two-test problem: you can build a hundred versions this week, but you only have the users to test two.
Some results are hard to see. The goal of this checklist is trust: merchants who stop wondering whether the integration is working. Trust doesn't show up on a dashboard next week. It shows up slowly, in fewer anxious emails and calmer conversations, over months.
What you ship stays. This dashboard would live inside the merchant's admin, not ours. Once merchants rely on it, removing it is a conversation with every one of them. Every new check needs maintaining. And here's the part that worries me most: a tool built for trust can destroy trust. If the checklist ever flags something as "missing" when it's actually fine, merchants panic, open tickets, and stop believing it. A wrong checklist is worse than no checklist.
Code is cheap. Living with what you shipped isn't. The trap: skipping the "why" An integration checklist sounds obviously useful. Who would argue against showing merchants what's broken?
That's exactly what makes it dangerous. When something sounds obviously useful and takes a day to build, nobody stops to ask what problem it solves. And the answer changes what you build:
If the goal is fewer support tickets, the checklist should focus on the issues that generate the most tickets today. If the goal is faster onboarding, it should guide new merchants step by step, and matter much less after setup. If the goal is trust, so merchants stop wondering whether the integration is working, it needs to be right every single time, even if it checks fewer things.
Those are three different dashboards. Building "a checklist" without picking one means building none of them well.
For us, the answer is trust and transparency. That one decision changes the design more than any feature idea: fewer checks, all of them reliable, each one explained in plain language. A checklist with 40 items that's wrong 5% of the time loses to one with 12 items that's never wrong.
Asking "why" isn't new. Good PMs always did it. What's new is how easy it has become to skip it. What I'm doing instead Before anyone spends 84 hours, or even one, here's the plan.
Decide what "trust" looks like in numbers. Trust is fuzzy, so I need signals I can count. Today, how many support tickets are really "something seems wrong and I don't know what"? How long does support spend figuring out what's broken before they can even start fixing it? If merchants trust the checklist, both should drop. And there's a nice third signal: tickets that start with the answer. "My checklist says the product feed is missing, can you help?" is a merchant who already knows what's wrong. We agree on the targets now, before seeing any results, because once results come in, it's very tempting to call anything a win.
Test the idea before testing the dashboard. The cheapest version of this isn't a dashboard at all. It's a script that runs the checks for 20 merchants, and an email to each one saying "here's what's working and what's missing in your setup." An agent can write that script in an afternoon. Then two questions. Was the report right? Support can compare it with what they actually find. And did it help? Did those merchants stop asking "is it working?" and fix things on their own? If the report is wrong or ignored, a nicer screen won't change that, and we just saved 84 hours.
Agree on when to stop. If the reports aren't accurate, or merchants keep asking the same questions after six weeks, we stop and rethink. Saying that now, while nobody is attached to the feature, is much easier than saying it later. So what is the PM for? When building was expensive, the PM's job was deciding what to build.
Now that building is cheap, the job is deciding what's worth knowing, and making sure the team doesn't ship things it can't learn from.
The 84 hours used to be the hard part. Now it's the easy part. The hard part is the question that comes after: how will we know it worked?
So here's mine for you: what did your team ship last month that you still can't say worked?