My first assignment out of law school was monitoring M&A deals.

I sat in front of a subscription tool that aggregated announced mergers, acquisitions, financings, and the rest. You could sort by deal size, by type, by buyer. You could set alerts. And the strategy my team ran on top of it was simple enough to explain in one sentence: a big deal gets announced, big deals mean lots of contracts to review, lots of contracts mean someone needs help, so we call them and win the work.

It sounds reasonable.

It’s also wrong in two structural ways that took me years to fully appreciate. If you’re building an origination motion for contract-review work today — whether you’re a vendor, a services firm, or a legal-ops team trying to staff ahead of demand — those two failures are worth as much as any feature comparison.

Failure one: by the time it’s news, the work is gone

The first thing that breaks is timing.

A deal that’s been announced is a deal that’s already been planned. The diligence scope is set. The advisors are picked. Often the contract-review approach — internal team, outside counsel, a vendor, some combination — was decided weeks or months before the press release went out. You are reading about a decision, not catching one in flight.

So the announcement, which feels like the start of an opportunity, is usually the end of one. The companies that show up in your alert feed are disproportionately the ones you can no longer help on this deal. The genuinely addressable work was upstream, in the quiet period when nobody outside the principals knew it was happening.

This is the part that impacts everything else. If announced deals are mostly too late, then reactive monitoring isn’t an origination strategy — it’s a lagging indicator dressed up as a lead source. The work that actually converts comes from being known before the deal exists: proactive relationships, so the company comes to you when the deal is still a secret. You can complement this — but not replace — with news monitoring.

Failure two: volume doesn’t mean need

The second failure is subtler and, I think, the more useful one, because it survives even if you fix the timing problem.

The whole premise — more deal volume means more contract work — assumes a correlation that just isn’t reliable. Deals are structured, and the structure determines whether contracts matter at all. A few examples that look similar in a volume column and behave nothing alike:

The acqui-hire. A handful of people command an enormous package — you can get to several hundred million, even half a billion, on a few names in the current AI talent market. It will light up any deal-size screen you build. The contract review attached to it is close to zero. Nobody’s reviewing a customer book; there barely is one. Huge number, no work.

The asset purchase. A buyer takes a discrete set of assets — say, a fleet of trucks. The assets are owned outright; there’s not much contractual tail to diligence on the buy side, and any related agreements may be getting retired or terminated as part of the structure anyway. So buy-side need is low. But flip it: the seller may have real work understanding how hard those contracts are to exit. The same deal generates demand on one side and almost none on the other, and a volume metric can’t see the difference.

Business-as-usual masquerading as nothing. Meanwhile, the company quietly enters into fifty thousand new agreements a year in the ordinary course of business. That’s a completely separate pool of potential work from anything M&A-driven — larger and steadier — and it never shows up in a deal feed at all, because it never gets announced.

Put those together and the lesson is uncomfortable for anyone who loves a clean dashboard: high deal volume is weakly correlated with contract-review demand, and the cases where it’s most wrong are often the flashy ones that dominate the news.

What actually predicts demand

None of this means prediction is hopeless. It means the naive screen is the floor, not the ceiling.

The floor — and I mean “naive” as a term of art, not an insult — is heuristic. Big, diversified companies with the cash flow or borrowing capacity to keep doing deals are likely serial transactors; their historical deal cadence tells you something. Working with physical goods helps: a business with a manufacturing or supply-chain footprint tends to carry more contractual surface area than a lighter software-first business, though large enough software companies blow that rule up with sheer customer and vendor count. Fortune 500, deal history, industry density — that’s a defensible level-one list, and it’s cheap to build.

The ceiling is where you stop treating those as the answer and start treating them as inputs. Layer in the things a screen can’t see: your relationship history, your own pipeline, deal-structure patterns, the buy-side/sell-side asymmetry above. Feed that into a model and you’re no longer asking “who’s big?” — you’re asking “who’s likely to transact in a way that actually generates contract work, and where are we well-positioned to win it?” That’s a prediction worth making.

One honest caveat, because it’s the thing most product pitches skip: the highest-value version of this is driven by your internal context — your relationships, your data, your read on structure. An outside-in tool can get you the level one list. It can’t replicate the part that makes the prediction good, which is the proprietary context only the firm doing the work has. Proceed accordingly and be wary of anyone selling you the ceiling as if it came in a box.

If you’re working on origination or capacity planning around contract work and any of this maps to (or contradicts) what you’re seeing, I’d love to discuss.

The context mattered. We were a nascent contracts data business operating inside a larger alternative legal services firm. Our business — turning contracts into structured data for M&A due diligence and integration — was maybe ten percent of the company’s revenue. The other ninety percent ran a different model entirely: placing attorneys with in-house legal departments on a consulting basis. That larger business was our best source of referrals. When one of their senior account executives recognized that a client’s problem was better suited to what we did, they’d send it our way — and they’d still get credit toward their goals, so incentives were aligned.

The head of our business wanted to double down on that referral engine. His solution was internal marketing — get the other teams excited about what we were building, make them feel connected to it, make them more likely to think of us when the right client situation came up. He tasked me, roughly a year and a half into the job, with making that happen through a newsletter compiling the latest cross-functional updates.

I’ve been thinking about that assignment lately because it’s a useful test case for something that comes up constantly: where do agents actually fit into knowledge work, and where does the person still have to drive?

What the Assignment Actually Involved

Before getting into the AI question, it’s worth being specific about what the work required, step by step.

The first thing I had to do was understand the business problem behind the assignment. The newsletter wasn’t the goal — increasing sales revenue was, through increased origination from the referral channel. That distinction mattered because it shaped every subsequent decision about what to include and how to frame it.

The second thing was figuring out what to compile. I started with our CRM to pull sales history, stakeholder context, deal status, and anything useful I could surface without taking up anyone’s time. Then I reached out across sales, product, marketing, and client service for anything the CRM didn’t capture — their read on what was worth highlighting, recent wins, delivery milestones, collateral. This produced a pile of good raw material: deals closed, new clients landed, product updates shipped, branding work completed.

The third thing — and this is where most of the judgment lived — was deciding what actually mattered and why, and how to communicate that to our audience.

Winning a deal with a new enterprise client was significant not just for its revenue, but for what it signaled about our ability to win that class of business and what it meant for the senior person on the other side of the house who originated it. A major renewal and upsell was worth highlighting not just as a financial outcome but as a recognition of the delivery team whose work made it possible. The connections between how collaboration across different parts of the business enabled our sales motions — and giving specific credit to the people involved — that’s what made the newsletter land, not the facts themselves.

Finally, the fourth thing was packaging it into a format that salespeople, executives, and other business stakeholders would actually read, let alone enjoy and be inspired by.

Running It Back With Agents

The interesting question isn’t whether an agent could do this today. Parts of it clearly could be addressed. The interesting question is which parts, and what that leaves for the person.

Compilation is largely addressable. The mechanical work of gathering updates across departments — if that information lives in a shared system with appropriate access — is exactly the kind of multi-step retrieval task agents handle well. Connect to the CRM, pull recent updates by department, surface them in a structured format. That’s real and useful. In my case I was doing much of this manually through CRM data research, email and conversations. Some of that friction goes away.

That said, the compilation step has its own constraints worth naming. The relevant information isn’t always well-structured or correctly tagged. Ambiguity in underlying data produces ambiguous outputs. And there are legitimate access and privacy boundaries — not every relevant conversation is logged, not every system is connected, and not every organization should want them to be. These aren’t temporary engineering problems. They’re features of how organizations actually function.

Summarization is addressable, and honestly it was already addressable before agents. Given a set of inputs, an LLM produces coherent summaries. This isn’t a new capability agents unlock — it’s baseline LLM behavior. Useful to have in the workflow, not a step-change.

Formatting and packaging is partially addressable. A first draft of the newsletter structure, given clear inputs and a well-defined template, is something an LLM handles comfortably. The blank page problem gets easier. The judgment in the final version is still a human job.

What doesn’t move, at least not yet: the judgment calls at the beginning and middle of the process.

Understanding why the assignment existed — that the real goal was referral activation, not newsletter production — required knowing the business, the person making the ask, and the organizational dynamics well enough to read between the lines. That context wasn’t written down anywhere. It came from being embedded in the organization.

Deciding what actually mattered in a pile of raw updates required knowing which outcomes were genuinely significant versus routine, which relationships between teams were worth reinforcing, and what would actually resonate with a specific audience of attorneys and senior salespeople being asked to care about something adjacent to their primary work. An agent surfacing everything equally would have produced a longer document, not a better one.

A More Recent Illustration

There’s a more current version of the same dynamic that I’ve been thinking about while using Claude Code for my own projects.

When Claude Code runs, it periodically pauses and asks whether you want to auto-accept its next actions or review them manually. It’s a small UX detail, but it captures something important: the model itself doesn’t know when human review is warranted. It can execute well across a wide range of tasks. What it can’t do is determine, in your specific context with your specific constraints, which outputs are safe to accept without review and which ones need your eyes on them first.

That decision — when to trust the output and when to verify — is a judgment call. And it’s not incidental that the tool surfaces it explicitly as a user choice rather than resolving it internally. The people who built it understand that this is a structurally human decision, not a capability gap to be closed in the next model release.

The newsletter assignment had the same structure. An agent could have helped me compile, draft, and format. What it couldn’t have done is tell me which deal was worth leading with, or how to frame a delivery team’s contribution in a way that would actually motivate a sales floor to send us more referrals. Those calls required context that lived in relationships and organizational dynamics that no system was tracking.

What This Suggests Practically

The newsletter assignment was never going to be fully automated, and that’s not a failure condition. The right way to think about it is as a workflow with a natural division of labor: the person drives the judgment-heavy front end, the tools handle the execution-heavy middle, and the person reviews and refines the back end.

In practice that looks something like this. You identify the business problem clearly enough that it can guide downstream decisions — human work. You use the tools to pull and structure relevant inputs from the systems that have them — agent territory. You evaluate what came back against the judgment you brought in at the start — human again. You use the tools to draft and format — mixed, depending on how clearly you’ve defined what good looks like. You review and refine with your audience in mind — human, and not optional.

The thing I’d push back on is the framing that treats this division as temporary — that eventually the agent handles the judgment calls too as models improve. Maybe. But the judgment calls in the newsletter assignment depended on context that is structurally difficult to provide to any system, not just ones with current capability limits. The organizational dynamics, relationship histories, and political considerations that informed my decisions weren’t in any system. Getting them there fully would require a level of organizational surveillance that most environments wouldn’t and shouldn’t accept.

Until that changes, the person isn’t a bottleneck to be engineered around. They’re the thing that makes the output worth producing in the first place.

Intro

There’s been a lot of noise lately about AI agents — most recently OpenClaw, before that OpenAI’s agents, Anthropic’s, Google’s, and the growing ecosystem of tools that let you automate workflows, string tasks together, and run complex operations with minimal intervention. The pitch is compelling: more getting done, less effort spent doing it.

I’ve found most of that pitch to be true in a narrow but important sense. These tools genuinely do help you get more done. I use them regularly for exactly that purpose.

But I keep noticing a gap between the claim and what’s actually happening — between getting more done and creating value. They’re not the same thing. The confusion between them, I think, is one of the more consequential misunderstandings in how these tools are being talked about right now.

Accompanying audio note:

The Productivity Claim

When people talk about AI productivity gains, they’re usually describing something real: tasks that previously took an hour now take ten minutes. Analyses that required manual effort can be automated. Research that involved dozens of searches can be compressed into a few prompts.

This is genuinely useful. I don’t want to dismiss it. But productivity, strictly speaking, just means output per unit of input. Getting more things done in less time. And productivity is only valuable to the degree that the things being done are worth doing.

That sounds obvious…but watch what happens in practice. Someone deploys an agent that automates a multi-step process. They count the hours saved. They extrapolate to the team, multiply by average hourly rate, and arrive at a dollar figure. The number sounds large. The announcement goes out: $X million in value created.

In most cases, this conflates two very different things: the cost of the activity and the value of the outcome.

Getting more done faster is only as valuable as what’s getting done. If an AI agent helps you accelerate a process that was already misaligned with what your customers needed, you’ve become more efficient at something that shouldn’t have existed in the first place. That’s not value creation — it’s waste, just at a higher throughput.

The More Interesting Question

None of this means productivity gains are worthless. It means the question worth asking is: what do you do with the time you get back?

There are two very different answers. One is that the freed capacity gets absorbed by more tasks of the same kind — a never-ending queue of lower-priority work that expands to fill available bandwidth. More things get done. The ratio of important things to unimportant things stays roughly the same. The volume increases.

The other answer is that the freed capacity gets redirected toward genuinely higher-leverage work — the things that actually move outcomes, that create something worth having, that compound over time. If that happens, then the productivity gain is real in the deeper sense: it bought you time to work on the things that matter.

Which answer you get depends almost entirely on whether you have the judgment and the agency to make that call. Judgment to identify what the high-leverage work actually is. Agency to actually redirect toward it — which, in an organizational setting, is far from guaranteed.

In a previous article I wrote about the difference between where you build in the stack and where your time actually compounds. The same logic applies here. Productivity tools don’t answer the allocation question. They just widen the gap between people who are asking it well and people who aren’t.

A Third Category

There’s one more thing I’ve been sitting with, and it’s harder to articulate cleanly.

Most discussions of AI productivity focus on one of two categories: either you were doing the task before and now do it faster, or you weren’t doing it and now you can. What I’ve started to notice is a third category — things you technically could have done before, but that were just annoying and friction-laden enough that you didn’t, in practice, ever do them.

A specific example: I’ve had FSA accounts for years, with funds that expire if unused — the classic use-it-or-lose-it situation. Submitting claims, especially for borderline or unfamiliar expense types, has always involved navigating opaque requirements, decoding denial reasons, and figuring out exactly what documentation was needed. Annoying enough that I’d let some money expire rather than waste time fighting the process.

Earlier this year I used ChatGPT to work through a claim denial. Not just as a search engine — as a conversational partner. I described the situation, asked about common denial reasons for that category, worked through what documentation would address each one, and drafted a resubmission. The claim was approved. Several hundred dollars I would have left on the table.

The productivity framing undersells what happened there. I didn’t do that task faster — I did a task I wasn’t going to do at all. The LLM didn’t just reduce time cost. It reduced friction below the threshold where I’d actually act.

There’s probably a psychological dimension to this as well. Something about the conversational format — the ability to ask follow-ups, to get a response that acknowledges your specific situation rather than returning generic search results — lowers the activation energy in a way that a list of web links doesn’t. I’m not sure whether that’s because the medium maps to how people have evolved to interact with and process information, their mental model or something else. But the effect is real.

This matters because it suggests a category of value that doesn’t show up in time-savings calculations. Not hours recaptured. Not existing tasks accelerated. But things that now happen that otherwise wouldn’t — claims filed, questions answered, decisions made with better information. That’s not productivity. It’s enablement.

What This Means in Practice

The honest version of the AI agent story is more nuanced than the headlines suggest. These tools genuinely do help you get more done. In the right conditions — when there’s judgment about what’s worth doing and agency to act on it — that translates to real value. When those conditions are absent, you get faster hamster wheels.

For anyone building with or using these tools, I think the most important questions aren’t about the tools themselves. They’re about the human side of the equation: What are you actually trying to accomplish? Where does the allocation of your time currently mismatch what you’d choose if you thought clearly about it? And what category of things — the ones you’re not doing because the friction cost is too high — might be worth reconsidering?

Getting more done is the easy part. These tools are genuinely good at it. The harder part — which the tools don’t touch — is knowing what to do with the time.

This is Article 7 in a series on building with and using AI tools as a non-native technologist. Previous articles covered financial planning with AI tools, a framework for city selection, evaluating LLM tools, building the Seattle Nature Access Map, why judgment was always the bottleneck, and where in the stack to build.

This was back before LLM tools became mainstream.

The queries weren’t sophisticated, but they handled my use cases well. They were also easy to understand and troubleshoot if something didn’t seem to be working.

I learned to write these – passably. Enough to pull data, join tables, answer a question without waiting for an engineer.

Once I reached a good enough level - I deliberately never got much better at it than that.

Every hour I could have spent becoming a stronger SQL writer was an hour I wasn’t doing the thing I was actually better at: figuring out which query mattered, what the data meant, and what it implied for the product. Where the models seemed to be doing well, where they needed to improve, and where we had no idea.

I could always find someone to write better queries. I couldn’t always find someone who knew what to ask.

That wasn’t laziness. It was a judgment about where my time compounded.

I’ve been thinking about that decision a lot since starting to build with AI tools in earnest — first the financial planning work I wrote about in Article 1, then the nature access map in Article 4. The more I build, the more I notice that the most consequential decisions aren’t about how to build something. They’re about where in the process your leverage actually lives.

The Stack Has Layers — And They’re Not Equal

When people talk about “building with AI,” they’re usually collapsing several genuinely different things into one.

At the foundation, there’s training foundational models — the work Anthropic, OpenAI, and Google do. Enormous compute, proprietary data, specialized research teams. Almost no one reading this should be operating here.

One level up: fine-tuning and domain adaptation. Taking an existing model and adjusting it for a specific task or dataset. Still technical, but accessible to a much wider group — though the relevant question increasingly is whether the base model already does what you need, because often it does.

Above that: building AI-powered applications. Connecting LLMs to data, interfaces, and logic to make something useful. This is where most of the interesting work is happening right now, and where the barrier to entry has dropped most dramatically.

And above that: using AI-powered tools to create things that aren’t primarily engineering at all — writing, analysis, structured research, products where the judgment and domain knowledge are the core value, and the code is just the vehicle.

These are genuinely different layers. Which one you should work at isn’t obvious, and “I can build this now” doesn’t answer it.

The Right Level of Abstraction Is the Same Question

In a previous article, I wrote about how judgment was always the bottleneck — that vibe coding made it visible, not new. This piece is about a related but distinct question: given that you have judgment, and given that you can now build things you couldn’t before, where should you be building?

This is a question I’ve been trying to answer for myself.

There’s a useful frame from computer science education that I found clarifying here. When I first learned to code, I started with a site that abstracted away almost everything — you typed commands, things ran, but you had no idea why. Then I took a more rigorous introductory class that went deeper. Then, eventually, a course that started just above the transistor level — actual hardware, zeros and ones, how you get from electricity to software. Each level of abstraction revealed something the previous one had hidden.

The lesson wasn’t “always go to the lowest level.” Going down to the physics of transistors was helpful for understanding how computers work. It wasn’t necessary for building a web application. The insight was that the right level of abstraction depends on what you’re trying to do — and choosing wrong in either direction creates problems.

Too high: you can’t diagnose what breaks. This is what happened when I hit the rendering bug during the nature access map build. Claude kept proposing fixes that introduced new problems, and we went in circles until I suggested using browser dev tools to surface actual error information. Abstracting away all understanding of the underlying system meant I was stuck the moment something went wrong.

Too low: you’re spending your time on infrastructure that isn’t your actual problem. If I’d tried to build a more capable machine learning pipeline from scratch for the nature access scoring, instead of using public data and simpler distance calculations, I’d have spent months on something that wasn’t what the app was focused on.

Choosing where in the stack to build is the same decision as choosing the right level of abstraction. The mistake is treating it as fixed.

The Chef Doesn’t Make the Spatula

Here’s the underlying principle: being capable of doing something and that being a good use of your time are different things.

A skilled chef could, in principle, forge their own knives, mill their own flour, build their own oven. They might have enough mechanical intuition to figure it out. But that’s not where their differentiation lives, and spending time there means spending less time on the cooking — which is where they’re actually hard to replace.

The question for anyone building now is the same one I was asking about SQL years ago: where does my time compound? Where is the gap between what I can do and what most people can do actually widest?

During the ten years I’ve spent in legaltech, the honest answer was never “write better ML code.” It was understanding what types of contract data matter to attorneys, salespeople, and finance teams — and how to structure and surface it in a way that was most useful to them.

That meant understanding what clause types mattered to an M&A attorney under what conditions, how to translate a 91% accuracy rate into something meaningful for a lawyer managing risk, which edge cases would surface in exactly the situations that mattered most. Those were judgment calls that required domain knowledge I’d spent years building. Better tooling wouldn’t have made them easier — it would have made the engineering around them cheaper, which is a different thing.

That calculus hasn’t changed. What’s changed is that the floor of “good enough building” is now much higher, which means more people can reach it — and that makes the question of where you’re actually differentiated more urgent to answer explicitly. Before LLMs, if you couldn’t write the code, the question was moot. Now it isn’t, and that’s mostly a good thing. It just means the “where should I build?” question is no longer optional.

So Where Does Your Time Compound?

Building toward something of my own has sharpened this question more than anything else. If I can build more easily than before, what should I actually build? At which layer?

The honest version of the question isn’t “can I do this?” It’s: where is my specific combination of domain knowledge, judgment, and taste producing something that isn’t easily replicated by someone working adjacent to me?

For some people, that points toward a lower layer — they have genuine ML depth, or systems architecture intuition, or infrastructure knowledge that’s still genuinely hard to replicate. For most of us building on top of these models, the answer points up: toward the quality of the problem being solved, the specificity of the insight, the understanding of the user, the design of the experience.

This doesn’t mean the engineering disappears. As I wrote in Judgment Was Always the Bottleneck, engineering judgment — knowing when a latency issue will create a UX crisis under load, knowing when a context window limitation will cap what’s possible, knowing when something technically works but creates maintenance debt that isn’t worth it — that’s still judgment, just expressed through technical decisions. The stack doesn’t have a clean separation between “engineering” and “everything else.”

What it means is that for most people who came to technology laterally — through law, finance, a domain specialty, a problem they needed to solve — the differentiation probably isn’t at the foundational layer. It’s in knowing what to build that’s actually worth building.

The spatula is cheaper now. The chef is still the chef.

This is Article 6 in a series on building with AI tools as a non-native technologist — covering what actually changes when someone without an engineering background starts building products. Previous articles covered financial planning with Claude Code, a framework for city selection, evaluating LLM tools, building the Seattle Nature Access Map, and why judgment was always the bottleneck.

Highlights:

• All laws are are actually principles applied under certain conditions or assumptions

• The strength of a principle is a function of how tightly you define the conditions under which it applies — narrow the scope, strengthen the law

• Complex systems are hard to predict because multiple counteracting principles operate simultaneously

• This is why macroeconomics gets predictions wrong repeatedly: too many counteracting principles operating simultaneously

• The same pattern shows up in machine learning: a model trained on all contracts performs worse than one scoped to a specific contract type

• And in UX: a design principle that worked for one user group often fails when applied to a different one without re-examining the underlying assumptions

There’s a popular narrative being told right now about vibe coding, and it goes roughly like this: coding used to be the bottleneck. Now it isn’t. The new bottleneck is judgment and taste. If you can think clearly about what to build, the tools will handle the rest.

It’s a compelling narrative. But it only tells half the story.

Not because judgment doesn’t matter — it does, enormously, and I’ll spend most of this article on why. But because the framing implies that judgment is new to the equation. That before vibe coding, the separating factor between great products and mediocre ones was mainly technical execution.

My re-frame is simpler: judgment was always the bottleneck. Vibe coding just made it visible.

What Actually Changed

Things did change, substantially. Claude Code and similar tools collapsed the barrier to entry for certain categories of development work in a way that prior no-code tools never quite managed. The prototype-to-production gap is narrower than it was. The floor of what a non-engineer can ship went up substantially.

But here’s the disconnect: we’re confusing a change in what we can see as a change in what was present.

Before widespread vibe coding, poor product judgment was largely contained inside institutions. It existed — and produced plenty of mediocre features and failed products — but it was wrapped inside teams at companies, hidden behind proprietary systems, filtered through organizational process. You didn’t see it at scale in the open.

Now you do. Thousands of people are shipping publicly, often free, often building in the open. So a much higher volume of things to look at means a much higher volume of poorly crafted things to look at. The ratio of strong products to weak ones was probably always similar. What changed is that you now have a massive lens on all of it, versus a microscope pointed mostly at what large companies chose to release.

There’s a marketing layer on top of this too. “I’m a builder” has become the new “I’m a disruptor” — a buzzword with similar dynamics to disruption a decade ago. When everyone’s shipping, people pay attention to what gets shipped. The scrutiny goes up alongside the volume. Both effects make judgment failures more visible without changing the underlying rate.

The story isn’t “judgment matters now.” It’s “judgment failures are harder to hide now.”

Judgment Isn’t New — It Was Always Load-Bearing

To understand why, it helps to break down what building something actually requires. At the highest level, three things: knowing what to build, knowing how to build it, and getting it to market.

The first category — what to build — means understanding the problem you’re solving, for whom, and conceptually how. When I built the nature access map, I was my own first user. I knew the problem in depth because I’d lived it: I wanted to understand, by neighborhood, which areas of Seattle had parks and trails within walking distance before committing to a move. That gave me a clear problem, a clear user, and a clear mental model of what a useful solution would look like.

The second category — how to build — means going from that concept to something tangible: mock-ups, prototypes, and eventually working code. This is where the LLM tools have changed things most dramatically. Historically, you couldn’t get to a working prototype at all without substantial technical fluency. Now, for many categories of work, you can describe what you want in plain English and get something functional back.

The third category — distribution and commercialization — means actually getting the product in front of users and, if you’re building something commercial, making money from it.

What changed is the second category. What didn’t change is the first and the third. And the first category — judgment about what to build, for whom, and why — was always the harder problem. It’s just that when the second category was blocking most people entirely, the first category’s difficulty was invisible. You can’t expose a judgment problem in something that never gets built.

I’ve spent a decade in legaltech, turning contracts into structured data for enterprise customers. The engineering was real — machine learning models, complex document processing pipelines, large-scale data extraction and classification, and increasingly GenAI-powered tools. But the decisions that actually determined whether the product worked were almost never purely engineering decisions. Was the model accuracy on this clause type sufficient for the risk appetite of an in-house counsel or M&A attorney? Which edge cases would appear infrequently enough that we could accept them, versus which would surface in exactly the situations that mattered most? How did you explain a 92% accuracy rate to a lawyer in a way that mapped to how they actually processed risk?

Those were judgment calls. They required domain knowledge, pattern recognition, and the ability to translate between technical realities and human contexts. No amount of better tooling would have made them easier. The people who made them well were valuable long before vibe coding existed.

The Complexity Matrix No One Is Talking About

Part of why the popular narrative feels limiting is that it collapses a genuinely complex set of distinctions into a single claim.

Whether vibe coding meaningfully changes the engineering bottleneck depends entirely on where you are across multiple dimensions at once:

Internal tool vs. external product. An internal tool built for yourself has radically different tolerances — for rough edges, for edge cases, for performance under load. When I built the nature access map, it ran locally, served one user, and if it broke I fixed it. That’s a completely different problem than a SaaS product with uptime requirements, paying customers and users who aren’t you.

Prototype vs. production. The gap between something that works in a demo and something that works under real conditions — concurrent users, unexpected inputs, evolving requirements — remains enormous, and LLMs haven’t closed it. Prior no-code tools produced a high ratio of prototypes to production-ready products for real reasons. There’s an honest open question about whether this generation finally crosses that threshold. I think the answer is “maybe, in limited contexts, but we don’t fully know yet.”

Simple vs. complex use case. Some design patterns have been solved thoroughly enough that you can implement them at a raised level of abstraction and they’ll work reliably. Others haven’t. New interfaces — AR, voice, novel interaction paradigms — haven’t accumulated the solved patterns that web and mobile have built up over decades. What worked on desktop didn’t always translate to mobile. What works now won’t automatically translate to whatever comes next. Every new interface introduces substantial new variables and trade-offs.

Regulated vs. unregulated. Financial services, legal, healthcare — these domains introduce constraints that compound complexity in ways that aren’t just about technical difficulty. The cost of certain failure modes is qualitatively different. The judgment required to navigate those trade-offs is domain-specific in ways that can’t be substituted by general technical fluency.

Scale. A few users is categorically different from a few thousand or a few million. Engineering judgments that produce acceptable outcomes at small scale often produce catastrophic ones at large scale. Latency barely noticeable to one user becomes a UX crisis under load. Data pipelines that work fine on small datasets break in ways that are hard to predict without depth.

The claim that “coding is no longer the bottleneck” may be true in the simplest corner of this matrix — internal tools, prototype-grade, unregulated, small scale. It becomes progressively less true as you move in any direction.

Engineering Judgment Is Still Judgment

There’s a subtler problem buried in how “judgment” is being used right now: it’s being treated as a single thing when it’s actually several.

Product judgment — understanding user problems, prioritizing correctly, knowing when good enough is good enough — that’s one kind. Design judgment — taste in visual hierarchy, interaction patterns, information density — that’s another. Engineering judgment is another — and it doesn’t disappear because LLMs can write code.

Engineering judgment is knowing that a given API’s rate limits will create a bad user experience under realistic load, before you find out by watching it fail. It’s recognizing that a context window limitation will cap what’s possible for your use case, and structuring your approach accordingly. It’s knowing when the cost of using an LLM makes it the wrong tool for a particular subtask, even if it would technically work.

When I was building the nature access map and hit a frontend rendering bug that sent Claude into circles — each fix creating a new problem — what broke the loop wasn’t better prompting. It was recognizing that we needed browser developer tools to surface actual error information, rather than continuing to diagnose from assumptions. That was a judgment call, made by a human, that the LLM hadn’t arrived at on its own. And this was a simple, single-user, locally-run application. The engineering complexity scales from here fast.

You can’t decouple engineering decisions from user experience decisions, or from business decisions. The people who understood this — who made proactive choices about downstream needs before being asked, who proposed building test infrastructure before it became obviously necessary, who recognized when a technically feasible solution would create maintenance debt that wasn’t worth it — were always valuable because of those judgment behaviors. They just happened to express them through engineering work.

Vibe coding doesn’t make that less relevant. It reduces the coordination costs of separating judgment from execution across different people. That’s genuinely useful. It doesn’t synthesize the judgment.

The Economics of It

When the cost of development decreases, the value of complementary capabilities increases. This is basic economics — cheaper peanut butter raises demand for jelly — and it explains why judgment, taste, and domain expertise are getting more attention right now.

But the implied corollary — that the previous bottleneck disappears — doesn’t follow. It becomes less constraining for simpler cases. It remains fully present for complex ones. And new bottlenecks emerge in places previously masked by the old one.

When I built the nature access map, part of what made it worth building was that the search cost of finding a tool that answered my specific question exceeded the development cost of building it myself. That math used to reliably go the other direction. Now it sometimes flips. That’s a real shift — but one that made domain knowledge more load-bearing, not less. The development cost dropped. The judgment requirements didn’t.

There’s a second economic dynamic worth naming. When anyone can build, the supply of apps proliferates rapidly. At some point supply exceeds demand. And in a buyer’s market, buyers have two levers: pay less for the same thing, or demand higher quality. These are in tension — as quality requirements increase, fewer people can meet them, which changes pricing again. But the net effect is that “good enough” stops being sufficient in the same way it was when supply was scarce. When you were one of a few people who could ship anything at all, the bar was low by necessity. When thousands of people are shipping similar things, the baseline shifts. The judgment required to clear it doesn’t go away. It goes up.

The Open Question

We’ve had “low-code” and “no-code” tools for years. Their history is consistent: useful for prototypes, insufficient for production, high ratio of demos to shipped products. They lowered the floor without raising the ceiling. People built more things, most of which didn’t make it.

The open question is whether this generation of tools — Claude Code and its successors — finally crosses that threshold. Whether the production gap will actually be closed for a meaningful range of use cases in a way it wasn’t before.

My honest answer is: maybe. Given enough time, probably, for some categories of use cases and some levels of complexity. But we don’t fully know yet, and the people making the strongest claims — in either direction — are probably overstating their case.

What I do know is that the judgment question isn’t new. It was always the hard part. The fact that more people can now see it — because more people can now build things and see the results — doesn’t mean it recently emerged from new tools.

It was always there, hidden in the gap between shipping and shipping something worth using.

This is Article 5 in a series on building with AI tools as a non-native technologist — covering my real experiences and observations building products as someone without a formal engineering background. Previous articles covered financial planning with Claude Code, a framework for city selection, using LLM tools to help with a cross-country move, and building the Seattle Nature Access Map.

No one could answer it well.

Google Maps can route you between two points. LLMs can tell you that Seattle “has great parks.” Walk Score can give you a general walkability number. But none of them answered my actual question: for a given block within a given neighborhood, what’s the walking distance to a meaningful green space?

So I built it myself.

This is the story of how a personal frustration during my move from New York to Seattle became my first public app—built with Claude as a coding partner, shipped as an MVP over a handful of nights and weekends, and designed to answer one specific question that no existing tool answered well.

Why This Existed as a Problem

In my previous articles, I wrote about the framework I used to pick a city and how I used LLMs during the planning process. One of my core criteria was convenient walking access to nature—not “nature exists somewhere in the metro area,” but “I can walk to a real park in under 15 minutes from my front door.”

This mattered to me because of lived experience, not aspiration. Living in New York, I’d discovered that regular morning walks—first through Carl Schurz Park, then Central Park—were one of the highlights of my daily life. I felt measurably better on days I walked. Over time, I realized that a city with great restaurants and nightlife but mediocre park access would be a worse fit for my actual life than a city with fewer amenities but a large park within walking distance.

The problem was that once I had this criterion, I couldn’t evaluate it efficiently.

I did rounds of Google searches. I asked LLMs. I checked existing nature and hiking apps. Nothing quite fit. There are plenty of apps for finding trails or planning hikes—but I wasn’t looking for a weekend adventure. I wanted to know: if I live on this block, can I walk to a good park easily? And how does that compare to the next block over, or the next neighborhood?

That question turns out to be surprisingly hard to answer without building something.

When Finding Is Harder Than Building

This experience led me to a realization I hadn’t expected: for a sufficiently specific personal need, the search costs of finding the right tool can actually exceed the development costs of building one yourself.

Historically, that wouldn’t have been true. Building a map-based web app required professional software engineering skills and weeks or months of development time. The rational move was always to search harder, compromise on fit, or just do the research manually.

But something has shifted. Between vibe coding with LLM tools like Claude, publicly available geographic data, and open-source mapping libraries, the cost of building a simple, personalized tool has dropped dramatically. Meanwhile, the cost of searching—scrolling through apps that almost-but-don’t-quite solve your problem, configuring tools not designed for your use case, cobbling together manual research across multiple sources—hasn’t changed much.

For me, the math flipped. I’d already spent hours on manual research and still didn’t have what I wanted. Building the thing I actually needed took less additional time than continuing to search for it would have.

This has broader implications for anyone building with AI tools. As development costs decrease, the value of complementary capabilities increases—things like judgment about what to build, taste in how it should work and look, and access to the right data. The bottleneck isn’t “can I code this?” anymore. It’s “do I understand the problem well enough to build the right thing?”

What I Built

The concept was deliberately simple: a color-coded map of Seattle showing walking distance to parks, broken down by neighborhood and then by census block.

That’s it. One question, answered visually, at a glance.

Why a map and not a report? I actually tried the text-based approach first. I ran LLM queries and compiled natural language descriptions of each neighborhood’s park access. It wasn’t very useful. Geographic and spatial relationships are fundamentally hard to convey in text. “Wallingford is 0.8 miles from Woodland Park” is less informative than seeing Wallingford on a map, colored green, with a park icon nearby. The visual format communicates distance, relative position, and comparison across neighborhoods simultaneously—something that would take paragraphs to express in words and still wouldn’t land as intuitively.

This connects to a broader principle I’ve been thinking about: the right interface depends on the type of information. Natural language works well for conceptual questions and subjective reasoning. But for spatial data, structured comparisons, or anything where relationships between data points matter as much as the data points themselves—a visual interface is almost always better.

The flow works like this:

Start with the city overview. Every neighborhood is color-coded from green (excellent park access) to red (poor access), scored 0–100. You can see at a glance which areas of Seattle have the best walking access to nature. Hover over a neighborhood and you see its score.

Click into a neighborhood. Now you see the individual census blocks within it, each with its own color and score. This matters because neighborhood averages can be misleading—a neighborhood might score 65 overall, but the blocks closest to a park could be 85+ while blocks on the far edge might be 35.

Click a specific block. You see the address, the nature access score, the nearest park name, the distance, the block area, and a link to view it on Google Maps.

The whole thing follows a principle called progressive disclosure: show the minimum useful information first, then let people drill deeper based on what they care about. For someone just exploring neighborhoods, the city overview is enough. For someone evaluating a specific apartment, the block-level detail is what matters.

How I Built It (And What Was Harder Than Expected)

I used Claude as my coding partner for the entire build. The overall process followed what most people describe as vibe coding: provide a high-level plan, iterate step by step, and make trade-off decisions along the way.

The plan was conceptually simple. I needed three things: data (parks, neighborhoods, geographic boundaries), a frontend (the map interface), and backend logic (calculating distances, generating scores, connecting everything). This is essentially the model-view-controller pattern—data, interface, and logic working together.

Where it got interesting was the trade-offs.

The data layer was the biggest source of real decisions—not because getting data was impossible, but because every data choice had downstream consequences.

First, how do you define a “park”? Is a 2-acre playground sufficient? What about a botanical garden? A greenway? For MVP, I kept it simple: parks above a minimum size threshold, focused on the kind of green space you’d actually walk through for 20+ minutes. That meant leaving out some edge cases, but it kept the scope manageable.

Second, where does the data come from? Publicly available geographic datasets exist, but they come with trade-offs: API rate limits, licensing restrictions, varying data quality, and inconsistent naming conventions across sources. Park data uses one taxonomy. Neighborhood boundary data uses another. Census block data uses yet another. Making these work together required judgment calls—how to define neighborhood boundaries, how to handle edge cases where a block spans two neighborhoods, what to do when data sources disagree.

Third, how do you serve the data? Making live API requests for every user interaction would be slow and fragile—and could hit rate limits fast if multiple people use the app simultaneously. So I downloaded the data upfront and serve it locally. For an MVP with a small user base, this works well and keeps the experience fast. Scaling would require a different approach, but that’s a future problem.

The frontend was where taste mattered most. The color scheme, the scoring scale, the level of detail in the popup, the decision to show census blocks rather than arbitrary grid squares—these are all judgment calls that determine whether the app feels useful or confusing. No amount of coding skill replaces knowing what the user (in this case, initially me) actually wants to see.

The backend logic surfaced subtle complexity. Distance calculations between geographic points aren’t as straightforward as they appear—coordinate systems, projection methods, and the difference between straight-line distance and walking distance all matter. For MVP, I used a reasonable approximation that’s good enough for “is this a 5-minute walk or a 20-minute walk?” If I need higher precision later, I can refine it.

Where Vibe Coding Helped—and Where It Didn’t

Building with Claude as a coding partner was significantly faster than building alone would have been. The LLM was particularly good at generating boilerplate code for map rendering and data processing, suggesting library choices and architectural patterns, helping debug issues when I could describe the problem clearly, and handling repetitive tasks like data formatting and conversion.

Where it struggled was when problems required context that’s hard to express in a prompt. For example, I hit a rendering bug where the frontend wasn’t displaying blocks correctly. Claude tried multiple fixes, each of which introduced a new problem—we went in circles. What eventually broke the loop was me suggesting we use browser developer tools to inspect the actual errors, which gave us concrete diagnostic information to work with. In another case, a more capable model release solved a problem the previous model couldn’t crack.

This reinforced something I’ve come to believe through this process: understanding computer science fundamentals matters, even in a vibe coding world. Not because you need to write every line yourself, but because when something breaks—and it will—you need enough understanding to diagnose the problem, ask the right questions, and evaluate whether the LLM’s proposed fix makes sense. If you abstract away all understanding of how the system works, you’re stuck the moment something goes wrong.

That said, I want to be honest about the tradeoff: for a simple, single-user application like this, the fundamentals matter less than they would for a production system serving thousands of users. The stakes of a bug are low. The complexity is manageable. If you’re building something for yourself and learning as you go, vibe coding is a remarkably effective way to do both simultaneously. Just know that the further you scale—more users, more data, more features—the more that underlying understanding pays off.

Lessons Learned

Build for yourself first, then share. I built this because I genuinely wanted it. That meant I understood the problem deeply, had strong opinions about what the solution should look like, and could evaluate quality without user testing. Solving your own problem is the fastest path to a useful product.

Start with one question. The app answers one question: how walkable is the nearest park from this location? Every feature decision flowed from that. When I was tempted to add hiking trail data, transit overlays, or restaurant proximity, I asked: does this help answer the core question? If not, it’s post-MVP.

Data is the real bottleneck, not code. The most time-consuming part wasn’t writing code—it was finding reliable data sources, understanding their limitations, reconciling different taxonomies, and making judgment calls about quality. As vibe coding makes development easier, data access and data quality become key differentiators for applications.

The right interface matters more than the right model. I tried answering this question with LLMs (natural language), with manual research (spreadsheets and notes), and with a visual map. The map won—not because it has more information, but because it presents spatial information in the format humans process most naturally. Choosing the right interface for your information type is a design decision that no amount of better AI can substitute for.

Ship the MVP, then iterate. The current version has rough edges. The scoring could be more sophisticated. The data could be more comprehensive. The styling could be more polished. None of that matters as much as having a working tool that answers the core question. Everything else is refinement.

What’s Next

The immediate plan is to continue refining the app—improving the interface, exploring additional cities, and seeing whether other people find it as useful as I do.

But the bigger takeaway from this experience is about a shift that’s happening in who can build useful tools. A few years ago, building a map-based web application required a team of engineers. Today, a product manager with some CS background and an LLM coding partner can ship a functional MVP in weeks.

That doesn’t mean the engineering is trivial. It means the barrier to entry has moved. The scarce resource is no longer “can you code it” but “do you understand the problem well enough, and do you have the judgment to make the right trade-offs?”

For me, this started as solving my own problem. It turned into my first public app. And it taught me more about building products than years of managing them from the other side of the table.

The app is available here: nature-access-map.

This is the fourth article in a series. Article 1 covered using Claude Code for financial planning. Article 2 covered the framework I used to evaluate potential cities to move to. Article 3 covered where LLMs helped and failed during the planning process.

I asked ChatGPT for the best cities to live in matching my criteria.

It gave me Portugal.

I asked for mild weather.

It gave me South Carolina.

That’s when I realized: LLMs are powerful research tools, but they’re solving for the average person—not for you.

Over three months of planning a cross-country move, I learned exactly where AI tools help and where they fail. The key wasn’t finding the “right prompt.” It was understanding which parts of the problem are objective—and which are subjective—and treating them completely differently.

Here’s what I found.

The Real Problem: Your Criteria Aren’t as Clear as You Think

When I started this process, I had what I thought were well-defined criteria: mild weather, walkability, access to nature, reasonable cost of living.

But here’s the thing—most of those words don’t actually mean anything specific.

“Mild weather” means something different if you walk everywhere carrying groceries versus if you drive. I’ve done both. I’ve walked through Atlanta summers carrying grocery bags, and I can tell you that 85°F with humidity when you’re on foot for 20 minutes is a fundamentally different experience than 85°F when you’re stepping from an air-conditioned car into an air-conditioned store.

“Walkability” means something different if you work remotely and mainly care about parks and errands versus if you commute downtown daily.

“Access to nature” means something different if you want weekend alpine hikes versus a daily walk through a 50+ acre park within 15 minutes of your front door.

The hard part isn’t that these criteria are wrong. It’s that they contain hidden assumptions—and you often don’t realize it until you get results that feel off.

Part of the reason is that many preferences are internalized. You don’t actively think about them. They’re visceral. You know the Atlanta summer grocery run feels bad, but you might not think to tell an LLM, “I walk to the grocery store three times a week and I carry bags, so your temperature recommendation needs to account for extended pedestrian exposure in humidity.” That level of context doesn’t come naturally.

It’s similar to how a UX researcher has to coax out a user’s real workflow—people don’t volunteer the details that matter most because those details feel obvious to them.

The Insight That Changed Everything: Objective vs. Subjective

After weeks of getting well-intentioned but unhelpful results, I noticed a pattern.

LLMs performed very differently depending on whether my question was objective or subjective—and much of the frustration came from treating subjective questions as if they were objective.

Objective questions have verifiable, well-structured answers. Historical temperature data. Cost-of-living indices. Transit system maps. Average precipitation by month. These are previously solved, commonly asked, publicly available, and quantifiable. LLMs are excellent at these—often as good as or better than manual research because they can synthesize across sources quickly.

Subjective questions involve personal interpretation, context-dependent meaning, or ambiguous language. “Is it mild?” “Is this neighborhood walkable?” “Is there good nature access?” These feel like they should have clear answers, but they don’t—because the answer depends on who’s asking and why.

The mistake I kept making was asking subjective questions and expecting objective answers.

When I asked for cities with “mild weather,” the LLM had to interpret “mild.” Its interpretation was reasonable—but it wasn’t mine. South Carolina has mild winters. It also has summers that would make my grocery walks miserable. The LLM didn’t know that because I hadn’t connected the dots between “mild” and “I walk carrying bags in the heat.”

This distinction—objective vs. subjective—became the foundation for everything else I did.

Working With Objective Criteria: Let the LLM Do the Heavy Lifting

For objective criteria, LLMs were genuinely excellent.

Once I learned to ask for data instead of adjectives, the quality of answers jumped dramatically.

Instead of: “Which of these cities has mild weather?”

I asked: “For each of these 15 cities, what is the average, median, and range of monthly high temperatures over the last 5 years? Include humidity and precipitation.”

Instead of: “Is Raleigh walkable?”

I asked: “What is Raleigh’s Walk Score? What percentage of residents commute without a car?”

The principle is simple: if a question can be answered with publicly available, well-structured data, ask for the data directly. Don’t ask the LLM to interpret it for you.

This worked well for criteria like temperature ranges, cost-of-living comparisons, transit infrastructure, general park and green space data, and airport access and logistics.

The results weren’t always perfect—LLMs can be confidently wrong on specifics, and data can be outdated—but they were directionally reliable. Good enough to build a shortlist. Good enough to eliminate obvious misfits.

Working With Subjective Criteria: That’s Your Job (But LLMs Can Help You Think)

The subjective criteria were where things got interesting—and where I had to change my approach entirely.

For these, the LLM’s role shifted from “researcher” to “thought partner.” Instead of asking it to give me answers, I used it to help me figure out what my questions actually meant.

Technique 1: Benchmark against cities you know.

Rather than trying to define “mild” in the abstract, I used cities I’d actually lived in as reference points.

“I’ve lived in New York and Atlanta. Compare the summer pedestrian experience in Raleigh, Seattle, and Pittsburgh to those two cities—specifically for someone who walks daily and carries groceries.”

This forced specificity. The LLM couldn’t hide behind “mild climate.” It had to compare against my actual lived experience.

Technique 2: Ask yourself why—and tell the LLM.

When I caught myself using a vague word, I’d ask: why do I actually care about this?

“Mild weather”—why? Because I walk for exercise and errands daily. Because I don’t want to dread going outside for four months of the year. Because I don’t want to rely on AC or heating as a coping mechanism for a climate I fundamentally don’t enjoy.

Each of those “becauses” translates into a more specific, testable question. And once I fed that context to the LLM, the recommendations improved noticeably.

Technique 3: Pressure-test with specific examples.

When an LLM told me a city had “good nature access,” I’d follow up: “What specific parks are within 20 minutes walking distance of downtown? How large are they? Do they have trails or is it a playground?”

The specifics often revealed that “good nature access” meant something very different from what I needed. A city park with a playground and a walking loop is not the same as a 300-acre urban forest with trail networks. Both are technically “nature access.”

This is actually a pattern I recognized from a previous career chapter. Years ago, I managed machine learning annotation projects—labeling training data for AI models. Even with seemingly well-defined categories, annotators would apply labels subjectively. The fix was always the same: go through specific examples, compare them, explain your reasoning, surface the disagreements, and refine the criteria. The same principle applies when you’re “annotating” your own preferences for an LLM.

The Division of Labor

After a few weeks, a clear division of labor emerged:

I used LLMs for thought partnership to pressure-test my criteria, researching and summarizing candidates against those criteria, helping me narrow down on specific dimensions, drafting checklists (apartment tours, moving logistics, donation steps), and estimating logistics and surfacing things I might be missing.

I kept for myself defining what I actually want and how to prioritize it, calibrating subjective labels (”mild,” “walkable,” “quiet”), verifying anything that could cost money or time, and running real-world tests (walking neighborhoods, simulating daily life, trying errands).

The frame that made it click: LLMs are great at research, organization, and estimation. They’re not built for judgment—and personal decisions are mostly judgment.

Once I internalized this, the tool stopped being a frustrating “decision maker” and became a genuine force multiplier.

Top-Down First, Then Bottom-Up

A mistake I made early was trying to get the perfect answer in one shot.

What worked was a two-phase approach:

Top-down (cast a wide net): I asked for 20–40 candidate cities given my criteria. I deliberately wanted a long list with false positives, because false negatives are more costly—if you never consider a city, you never discover it’s a fit.

At this stage, I accepted messy, imperfect results. The goal wasn’t precision. It was coverage—and using the results to refine my own thinking about what mattered.

Bottom-up (go deep on the shortlist): Once I’d narrowed to a handful of cities, I shifted to targeted, specific queries. Neighborhood-level research. Specific parks, transit routes, grocery stores. Temperature data by month. This is where the objective/subjective framework paid off most—I knew which questions to ask the LLM (data) and which to answer myself (judgment).

The key insight: city-level averages hide neighborhood-level reality.

A city can be “unwalkable” overall and still have pockets that are perfect for a car-free lifestyle. Or the reverse. So I treated each promising city’s neighborhoods as a separate research problem—and that’s actually where the decision happened.

Where LLMs Consistently Fell Short

A few patterns emerged:

Ambiguous terms without operationalization. Every time I used a word like “mild,” “safe,” “quiet,” or “diverse” without defining it, I got generic results. The fix was always the same: replace adjectives with numbers, ranges, or specific examples.

Hidden assumptions. The LLM would assume I had a car, or that “affordable” meant a certain range, or that “city” meant metro area rather than a specific neighborhood. The fix: ask the model to list its assumptions, or preempt by stating constraints (”assume no car,” “budget is Y”).

Trade-off weighting. If you ask an LLM to optimize across 10 criteria simultaneously, you’re asking it to make value judgments about your life. It can’t do that. The fix: ask for a longer list with trade-offs noted, and do the weighting yourself.

Freshness and specificity. LLMs can be confidently wrong on details that change—pricing, availability, neighborhood dynamics. The fix: treat every output as a lead to verify, not a fact to trust.

The Limits of Research (And Why Real-World Testing Matters)

At some point, no amount of prompting eliminates uncertainty. You have to test.

I found that the best way to reduce uncertainty was simply visiting—or simulating a visit as closely as possible. Walking the actual routes I’d walk. Checking the actual grocery stores I’d use. Riding the actual transit I’d depend on.

This is also where I learned that not everything should be an LLM conversation. Some questions are better answered by a map. Some by a transit app. Some by walking around for an hour.

I actually started building a tool for one of these gaps—a neighborhood park access map that shows walking distance to parks at the block level—because it was a question I kept asking that no existing tool (including LLMs) answered well. I’ll share more about that in the next article.

The principle: LLMs generate hypotheses. Targeted tools and real-world experience verify them.

What I’d Do Differently

If I were starting this process from scratch, I’d do three things earlier:

First, separate objective from subjective criteria on day one. This alone would have saved me weeks of frustration. Objective criteria get delegated to the LLM immediately. Subjective criteria get a “define what I actually mean” session before any research happens.

Second, use benchmark cities from the start. Instead of trying to define preferences in the abstract, ground everything in places you’ve actually experienced. It’s faster and more reliable.

Third, go neighborhood-level sooner. City-level research is useful for elimination, but the decision lives at the neighborhood level. I spent too long comparing cities when I should have been comparing neighborhoods within the top 3–5.

Conclusion

Using LLMs for this move didn’t eliminate uncertainty. Moving is still a leap.

What it did was compress the process—from fuzzy to structured, from overwhelming to manageable, from “I don’t even know where to start” to “I have a clear shortlist and I know what to test.”

The trick was learning that the most important skill isn’t prompting. It’s knowing which parts of the problem are yours to solve—and using the tool to handle everything else.

LLMs are great at helping you think. They’re great at helping you research and organize.

They’re not great at being you.

Next article: I built a neighborhood park access map because no existing tool answered my question. Here’s what I learned about going from an internal tool to a public product.

I know this because I spent ten years living in a New York City – a city that looks great on paper.

It’s globally recognized, full of opportunity, and has every possible convenience. And yet, over time, it just didn’t feel right for me.

It wasn’t a bad experience. It just felt like I was in the wrong place for reasons I couldn’t quite articulate.

I’d had this feeling for a long time.

But given the range of possible cities to choose from, the uncertainty and ambiguity around them, and a wide range of experiences being reported…

Information overload and decision paralysis followed.

Combine that with psychological tendencies towards pain avoidance (the delightful moving process) and doubt avoidance (how do I even know the new city will be better?) – and slight variations of the status quo becomes the norm.

I.e. stay in the same city.

When I started planning a possible move late last year, I tried to address this by getting advice from both people who knew me (and of course, LLMs).

The advice was well-intentioned but mostly unhelpful.

Not that I felt like anyone was trying to mislead me—but because the advice wasn’t personalized to my life. People were giving me recommendations filtered through their definitions of what matters.

Someone says, “Don’t move to Seattle, it rains a lot.”

Someone else says, “You should move to Nashville or Austin.” When pressed, it’s because they took a two-day business trip, went to great restaurants, and enjoyed the vibe.

Even well-vetted advice can be wrong in context. When I lived in Atlanta for a few years, the default line was: you need a car. Maybe that’s true for many lifestyles. It wasn’t true for mine. What mattered more was how I actually spent my day: school, studying, errands, gym, basic routines.

I lived near where I needed to be, walked more than most people assumed, and used taxis and rideshares strategically. In Atlanta, monthly parking was $~150. Ubers averaged less than $20/week.

The city’s reputation didn’t match my lived experience—because my routine didn’t match the “average” routine implied by the reputation.

That’s the fundamental problem:

Most advice about choosing a city fails because it isn’t personalized.

The good news is: this is solvable.

It just requires a framework that starts with your real day-to-day life.

Step 1: Audit your day-to-day (the foundational move)

Before you research cities, ask a simpler question:

What does my life actually look like right now?

For a couple weeks, track your routine at a practical level. Not “I value health.” Not “I love nature.” The actual sequence:

• Do you work remotely or commute? If you do commute, how?

• How often do you leave the house on weekdays?

• Do you cook, buy pre-cooked meals, or mostly eat out?

• What errands happen weekly (groceries, pharmacy, gym, laundry)?

• What do you do on weekends when you have free time?

• What do you do for fun / to blow off steam?

This is where people get tripped up. They choose cities based on identity (“I’m a nature person”), aspiration (“I’m going to hike every weekend”), or aesthetics (“I want charm”).

None of those are inherently wrong.

But they’re unreliable inputs if they aren’t grounded in practical behavior and habits.

One of the biggest insights I had was realizing that some “preferences” were actually requirements for my mood, health, and productivity—and some things I thought mattered were mostly nice-to-haves.

Example: nature access. It didn’t become real for me because I wrote it down as a value. It became real when I started walking in the park regularly—early, even when it was cold, even when the weather wasn’t ideal. That behavior was evidence. It told me: this isn’t a nice-to-have or a fantasy. It’s a proven requirement for my wellbeing.

Step 2: Turn your routine into criteria (make it testable)

Once you understand your day-to-day, convert it into criteria that are specific enough to evaluate.

Start with broad categories (most people share these):

• Cost of living (true cost, not just rent)

• Housing (size, age, noise, amenities)

• Employment

• Food access (diet, convenience)

• Walkability and transportation

• Healthcare access (if relevant)

• Weather (what you enjoy, tolerate or dislike)

• Access to nature

• Safety (as you define it)

• Logistics (airports, shipping, family visits, errands)

• Density / stimulation (do you want energy or calm?)

Then make them concrete. “Access to nature” is not a criterion. It’s a label.

A criterion is something you can test, like:

• “Within a 15-minute walk: a park I’d actually use.”

• “I can do 80% of my weekly errands without a car.”

• “My grocery options support how I actually eat (not how I wish I ate).”

• “Rent + utilities + predictable ‘extras’ fit within my comfort range.”

• “Noise and stress in the home environment stay below a threshold.”

The point is not to create a perfect scoring system. The point is to stop making decisions based on vibes you can’t operationalize.

Step 3: Prioritize (must-haves vs nice-to-haves, reality vs aspiration)

Now the real work: prioritization.

People frequently confuse a preference with a requirement.

A must-have is a constraint you can’t realistically violate without breaking the move:

• budget reality

• commute requirements (if you have them)

• core health needs

• housing necessities (space, noise insulation, safety, etc.)

• logistics you can’t wish away

A nice-to-have is something you want, but could sacrifice if other wins are strong.

A second consideration matters just as much: Daily needs vs episodic needs.

Daily needs dominate your happiness because they repeat. A great restaurant scene matters less if you eat at home 90% of the time. A world-class airport matters less if you fly twice a year. On the flip side, if you travel constantly, the airport becomes a daily-life consideration.

Finally: Current reality vs future aspiration.

If you aren’t exercising now, don’t overweight “strong fitness culture” as the deciding factor. That doesn’t mean you can’t change—but it means you should be honest about the likelihood that a move alone will change you.

A useful test here:

• If this criterion disappeared, would I be mildly disappointed—or miserable?

• If I got “good enough” here, would that free me to prioritize other things?

Step 4: Apply the framework to your current city first

As with any experiment – having a benchmark or starting point is best.

Run the framework on where you currently live:

• What works well?

• What’s constantly annoying?

• What are you compensating for with money, time, or stress?

• Which friction points keep showing up?

This is how you learn what you’re actually trying to change.

For me, this is also helped flesh out interdependencies. The living environment affects stress. Stress affects sleep. Sleep affects diet. Diet affects health. Health affects productivity. Productivity affects earning power and confidence. Suddenly the city choice isn’t just lifestyle—it’s downstream of everything.

Step 5: Research cities—but evaluate at the neighborhood level

Here’s a simple truth that collapses a lot of noise:

Most cities aren’t one experience. They’re a bundle of neighborhoods.

Even in cities with strong overall characteristics, your day-to-day is determined by:

• where you live

• where you work

• what’s within walking distance

• how you move around

• what your immediate environment feels like

Atlanta has walkable areas. It has areas where you can live without a car. It also has areas where that would be miserable. “Atlanta” as a single recommendation is meaningless without the neighborhood.

Same for Seattle. Same for NYC. Same for almost anywhere that isn’t tiny.

A practical research method that helped me:

• List cities you think you’d like

• Also list cities you’re confident you wouldn’t

• Use the contrast to refine your constraints (e.g., “I don’t tolerate extreme heat” or “I don’t want months of sub-freezing temperatures”)

Step 6: Down-select to 3–10 options (then stop browsing)

At some point, “more research” becomes procrastination.

Build a shortlist. For me, it included places like Boulder, Denver, Nashville, Seattle, Santa Clara, San Francisco, Oakland, Chicago, Tampa, Raleigh, Bellevue.

Your list will differ. That’s the whole point.

Use a simple rule:

• Must-haves = pass/fail

• Nice-to-haves = 1-5 score

• Keep notes on trade-offs, not just ratings

Step 7: Experiment (visits prevent expensive mistakes)

If there’s one cheat code here, it’s this:

Visit. Simulate a normal week or at least a few days.

Don’t just do tourist stuff. Do your routine:

• walk to a grocery store you’d actually use

• try the transit route you’d actually take

• go to a gym you’d actually join

• walk in the neighborhood at the time you’d normally walk

• sit in a café and work for an hour if you work remotely

• pay attention to friction: noise, safety vibe, stress level, convenience

This is also where you learn things you can’t learn online.

I almost chose a building/neighborhood combination that looked perfect on paper—until I visited and noticed sketchy behavior from the building manager. The neighborhood and apartments were great. But the “day to day reality” felt off – ranging from spikes in traffic noise, hidden fees and a lot of “it depends” answers to my questions. That’s not a minute detail. That’s the kind of thing that turns a move into a chronic stress generator.

Step 8: Decide (don’t let perfect be the enemy of good)

There will always be trade-offs.

Some people try to optimize to a mythical “perfect city” and end up stuck, endlessly comparing. Instead, aim for:

Significantly better than your current state.

Then commit.

You can refine over time—sometimes within the same region, sometimes within the same city, often just by switching neighborhoods.

This is where diminishing marginal returns matters. Getting to 8/10 on your must-haves usually beats sacrificing multiple criteria to chase 10/10 on one. “Good enough” is not settling; it’s calculated decision.

Lessons learned (what I wish I could tell my past self)

1) Test aspirations vs reality.
If you don’t do it now, don’t overweight it. Validate needs through behavior.

2) Social life is portable.
“Seattle is less friendly” is too generic to be actionable. You can meet people anywhere and be lonely anywhere. Your habits and communities matter more than city stereotypes.

3) Experimentation prevents expensive mistakes.
Visits, neighborhood testing, and realism checks beat internet certainty.

4) Hidden costs matter.
Not just money—time and friction. Filters, locks, insurance, parking, delivery fees, and all the little things that add up. A rent number without lifestyle math is incomplete.

5) Diminishing marginal returns are everywhere.
You don’t need perfection. You need a strong baseline across key variables. Then you can make a decision regarding what to optimize and when/where to fit in nice to haves.

6) “Access to nature” is not a single preference.
A 5am hiker and a casual park walker are describing different lives. Same label, different requirement.

Closing Thoughts

I’ve walked in the Seattle rain. I’ve seen gray days. And I’m still glad I moved—because the decision wasn’t based on a city’s reputation. It was based on whether the city fit my real day-to-day life.

If you’re considering a move, don’t start with “What’s the best city?”

Start with:

1. What does my life actually look like?

2. What would make it meaningfully better?

3. Which places can realistically support that?

Then build your shortlist—and go test it.

I explored a few different approaches in parallel – a top-down type of workstream and a bottom-up workstream.

In this context – a top-down approach means larger scope: effectively I compiled all of the financial statements and transactions from a given period of time, say the last 3-5 years, and then used Claude Code and other tools to actually parse out the transactions, compile them, categorize them, and analyze them or enable my own analysis of them.

I tested a few different approaches that uses this sort of top-down workflow.

The original overall approach or pipeline was:

- Create an overall plan or outline

- Compile / download financial statements (PDFs)

- Drop into a local folder and have claude code work with them end to end:

• Provide initial context and refine plan using Claude Code’s plan mode

• Convert each PDF into Excel or csv

• Parse out the transactions

• Aggregate the transactions into a single Excel or csv

• Categorize the transactions

• Perform follow-on analysis

• Iterate

After a few hours it became clear that parsing out transactions from PDFs to Excel / csv is deceptively challenging (even for Claude), particularly when working with different data sources. So I replaced that specific step with using a third-party PDF to Excel conversion tool that I already had access to and was both much faster and more reliable.

Otherwise, the overall top-down approaches were the same.

My findings from the top-down approaches were as follows:

• First of all, handling large scope to start can be overwhelming in terms of the amount of the amount of necessary code (and of course corresponding context windows)

• Second of all, because of the volume of code and data corresponding to the scope, the human testing costs become much higher

  • In other words, the realm of possibilities of things that can go wrong are much broader, require more rigorous testing and the testing is more time consuming to perform

  • Unless an issue is glaring or there is some clear prioritization that can be done – for example identify what the largest transactions (such as rent payments), the transactions require a fair amount of individual review

  • However even when prioritizing these things there are all sorts of nuances that can get introduced – for example transactions being combined together, or amounts being off, or occasional exceptions that are difficult to spot (e.g. imagine a paycheck and a bonus being combined together or a rent payment being combined with a security deposit or a move from one apartment to another leading to different rent costs, etc.)

  • Oftentimes the context is situation-specific as well which makes more automated tests or handling this via the planning phase more difficult

• Third of all, in terms of the reliability and accuracy of the code – the top-down approach is decent at creating estimates but if exactness or precision is necessary, it is of limited utility as it would require a fair amount of manual quality control and testing. This may be okay as a starting point but usually requires complementing with other approaches to get more precise data

The bottom up approach

Given the limitations of the top-down approach, the question came up: is there a complementary or perhaps alternative approach to the top-down process that focuses on a smaller scope?

For example, starting with a specific category of spend, mastering that and then repeating the process for other categories.

The business analogy I would use here is how Amazon focused on a specific product category to start (books). There are similar best practices when starting a business – specifically to focus on an underserved niche rather than trying to explore too large of a market and risk more competition, expensive customer / user acquisition, etc.

So the bottom up approach has the benefit of targeting specific categories one at a time (e.g. rent, grocery spending). This in turn involves smaller volumes, similarly structured transactions and less corresponding complexity so it involves less code, simpler testing, less cognitive load and greater transparency into both the data and the code.

This has been my experience so far.

To summarize: the key benefit of the bottom-up approach is simplicity. Less categories, smaller transaction volume, less things that can go wrong, and less things to consider at any given time.

Now compare that with the top-down case where you have many statements across multiple categories. So you have a much larger volume of transactions of different types. You have different types of data structures for each. You have different types of ways in which things are expressed and net of those considerations there are many things that can go wrong potentially at each stage of the process, particularly when you’re parsing, aggregating, and analyzing the transactions.

An Open Question – Building Up to Complex Use Cases?

In addition, there is an additional potential benefit for more complex systems with a large volume of categories. My hypothesis is that if you build it up from the bottom step-by-step, category by category and each category is sufficiently constrained, over time you may be able to create an orchestration agent to actually coordinate each of these as if they were separate services.

For example you can have a grocery purchase analyzer as one service.

You can have a rent payment analyzer as another service, a medical appointment analyzer, a transportation payment analyzer, and so on.

You can do something similar for basically every category of items that is sufficiently standardized or constrained. Then, as you (and the system) develop a strong understanding of these you can then potentially automate the system entirely.

I need to pressure test the feasibility of this by running some experiments, but early indications are promising.

Conclusion

Overall, Claude Code has been very helpful in assisting me with financial planning and other use cases. So far – the top-down approach has been preferable to get a decent estimate of overall spend while the bottom-up approach has shown the most promise when exactness and detailed results are necessary.

We can see this tension everywhere. Google must balance its dominance in search with new bets on AI and conversational interfaces like Gemini. Amazon went from selling books to becoming an “Everything Store”, then turned the internal logistics and computing system that powered that store into an entirely new external business — AWS — one of the most successful divisions in corporate history.

But expansion carries risks. Venture too far from your strengths, and you risk competing where you are weakest.

Capital, time, and attention — all scarce resources — get spread thin and diverted from the highest impact areas, leading to underperformance and, in the extreme, existential risk.

Yet focus alone carries its own danger. When customer preferences shift or technology evolves, companies that stay too narrowly focused can be blindsided by simpler, cheaper, or more relevant competitors. Disruption thrives where incumbents refuse to experiment.

Shopify’s Fulfillment Bet

Shopify’s short-lived fulfillment business captures this tension perfectly.

The business – known as the Shopify Fulfillment Network – launched in 2019.

For years, Shopify’s value proposition was clear: provide small and medium-sized merchants with software tools to compete online. The platform handled websites, payments, and inventory management, but left physical logistics to merchants themselves or third-party providers.

This created a gap. Small merchants lacked the scale to negotiate favorable rates with logistics providers or build their own warehouses. Meanwhile, Amazon’s fulfillment network gave its sellers a powerful competitive advantage. If Shopify could offer similar infrastructure, its merchants could compete more effectively.

The idea was bold: extend Shopify’s software platform into the physical world by building fulfillment and logistics infrastructure for small merchants. If successful, it would give small sellers the same capabilities as giants like Amazon — affordable, reliable, fast delivery — while letting Shopify capture a larger share of the merchant ecosystem. It would enable a more seamless, end-to-end experience for merchants and their customers.

On paper, the strategy made sense. Merchants already used Shopify to run their online storefronts. Why not also help them manage inventory and shipping? It was a logical extension of the “all-in-one commerce platform” narrative.

In practice, though, it pulled Shopify into a fundamentally different business.

Running fulfillment centers, managing warehouses, and coordinating last-mile delivery is operationally complex and capital-intensive — the opposite of Shopify’s asset-light software model. Suddenly, Shopify was competing in Amazon’s backyard, a domain where scale and logistics expertise matter far more than software elegance.

Knowing When to Walk Away

With hindsight, it’s easy to label Shopify’s fulfillment expansion a misstep.

Shopify sold the business in 2023, ultimately deciding to partner with existing providers instead.

But that decision shouldn’t be viewed as simple failure — it’s better understood as an experiment that ran its course.

The willingness to run large experiments is a sign of an innovative culture. Shopify bet big on improving its merchants’ experience, mirroring Amazon’s customer-obsessed ethos. It saw an unmet need and tried to fill it.

The more impressive part, though, is the willingness to walk away.
When the economics didn’t justify continued investment — and when it became clear that the effort was drawing focus away from Shopify’s software and payments businesses — leadership made the call to exit. That takes discipline. Many companies hold onto sunk-cost projects far too long.

Changing Context, Changing Calculus

Part of the decision was driven by evolving company and industry dynamics. As Shopify’s merchant base expanded to include larger enterprises, the value proposition for an in-house fulfillment network weakened; big merchants already had established logistics partners, reducing the need for Shopify to build its own.

The context had shifted – the same strategic move that once seemed visionary now risked becoming a distraction.

Lessons Learned

There are three broader lessons from Shopify’s entrance into and exit from the fulfillment space:

1. Experiment boldly, but deliberately.
Growth often requires venturing beyond your comfort zone and running experiments with high-risk and high-reward. Shopify’s decision to make a large bet on a new business model was a testament to its experimental, high-agency builder culture.

2. Know when to quit.
Success isn’t about avoiding mistakes — it’s about reallocating time and capital quickly when experiments don’t pan out. Strategic focus and adaptability are competitive advantages.

3. Adapt as the world changes.
A good strategy at one moment can become a poor fit later. Shopify’s willingness to re-evaluate assumptions as context evolves is part of its disciplined execution and has helped sustain its momentum since.

The Bigger Picture

Whether at the level of a company, a team, or an individual career, the same logic applies. Experimentation is essential for learning and innovation — but so is focus. The goal isn’t to avoid failed experiments; it’s to make smart bets, learn fast, and redirect effort when the evidence points elsewhere.

Shopify’s fulfillment story isn’t just about logistics. It’s about the art of balancing curiosity with clarity: knowing when to build, when to explore, and when to let go.

From 2023 to 2024, Shopify’s revenues have increased 26% to over $8.8 billion while net income climbed to over $2 billion.

They have two main revenue components:

• Subscription Solutions (~26%) – recurring revenue from monthly subscriptions to access the Shopify platform, plus related activities like app sales and domain registrations.

• Merchant Solutions (~74%) – transaction-based revenue from payment processing, currency conversion, and other services linked to merchant success.

Subscription Solutions

Subscription solutions are mostly comprised of platform access, and are priced accordingly (mix of fixed costs and variable platform fees). There are different tiers of platform access with different corresponding pricing. The reasoning being that when entrepreneurs are starting out, simplified / entry-level platform access makes sense but as a business grows and has more complex needs it will likely need access to a broader range of features to navigate the complexity involved (for example managing multiple sales/marketing channels or managing more customer accounts). Shopify Plus is mentioned as their premium tier with significantly higher pricing than the more basic out of box offerings.

It’s noteworthy that for any given reporting period, no single merchant has accounted for more than 5% of their revenue (which speaks to how diversified the business is).

Also, there are some other recurring revenue sources such as, sale of apps, registration of domain names and sale of themes (online store/website templates). These do not appear to be significant contributors to revenue but help provide a one-stop-shop in some respects, enabling customers (merchants) to get their needs met without having to explore other alternatives, which provides convenience and an overall better user experience.

Merchant Solutions

Merchant solutions are priced on a transaction basis – as more transaction volume comes through the Shopify platform, Shopify collects payment processing and currency conversion fees, referral fees from partners, Shopify Capital (lending to / financing of qualified merchants), and Transaction Fees.

In addition, they generate revenue from services and products like sale of shipping labels, sale of point-of-sale (“POS”) hardware, advertising on the Shopify App Store, and Shop Campaigns (buyer acquisition offering).

While individually small, these involve similar dynamics as the subscription solutions segment of their business. Specifically, these offerings reduce friction and keep merchants operating within Shopify’s ecosystem – strengthening the user experience and retention.

Incentives and the Flywheel Effect

From an incentives standpoint – when merchants succeed, Shopify succeeds.

This happens in a few different ways:

• Subscription solutions

  • When a business is starting out – they don’t need access to premium platform features

  • Free trials (particularly during times of market slowdown) help incentivize entrepreneurs and up and comers to start using the platform. If they start gaining traction as a business and view the platform as helpful in doing so, they’ll start paying

  • If the business succeeds over time, it’s likely growing and will be a prime target to upsell/upgrade to the next tier

  • In addition, while most businesses pay month to month, some elect to pay annually or multi-year subscriptions (with longer terms likely incentivized by discounts, akin to volume-based pricing)

• Merchant solutions

  • The more business a customer (merchant) does through the platform, the more Shopify collects in transaction-based fees. This is even better aligned in terms of incentives as Shopify immediately benefits while the merchant collects most of the profit associated with the sale, creating a win-win situation

  • Referral fees from partner fees are more likely to be generated as Shopify captures more of the market in its core offerings and develops increased brand recognition, and allow all parties in the ecosystem - including customers (merchants), partners and Shopify itself - to benefit accordingly from the network effect

  • Through Shopify Capital, lending to qualified merchants with strong business prospects generates revenue in the short term (loan interest) while functioning as a form of customer acquisition for the core offerings. As these businesses grow, they are more likely to upgrade to premium subscription tiers and process higher transaction volumes.

  • The other offerings – while not significant revenue drivers help customers (merchants) focus on their core business by reducing search/switching costs as well as stay in the platform (possibly reducing risk of customer churn over time)

Conclusion

The genius of Shopify’s model lies in its dynamic, stage-appropriate pricing: it meets merchants where they are and grows with them. Free trials convert to paid plans, paid plans upgrade to premium tiers, and transaction fees scale with success—all while ancillary services reduce switching costs. It’s a masterclass in aligning company incentives with customer outcomes, and the 28% YoY revenue growth in 2024 provides strong evidence that it works.

Next time – I’ll explore Shopify’s profitability and cost structure.

Here are some things I learned from reading Shopify’s Recent Quarterly and Annual Filings:

1. Two Main Revenue Streams

Shopify’s revenue is divided into two broad categories:

• Subscription Solutions (~26%) – recurring revenue from monthly subscriptions to access the Shopify platform, plus related activities like app sales and domain registrations.

• Merchant Solutions (~74%) – transaction-based revenue from payment processing, currency conversion, and other services linked to merchant success.

Together, these two segments reflect Shopify’s hybrid model: predictable recurring revenue plus upside from merchants’ transaction volume.

2. Customers

Shopify’s customers are merchants—ranging from solo entrepreneurs to large enterprises.

3. How Shopify Measures Performance

Shopify highlights two key operating metrics in its filings:

• Monthly Recurring Revenue (MRR): the following month’s expected subscription revenue based on currently active subscriptions

  • up from $144 million in 2023 to $178 million in 2024, a ~24% increase.

• Gross Merchandise Volume (GMV): this is a “merchant success” metric corresponding to the total merchant transactions processed over a period of time, including shipping + taxes, net of all cancellations / refunds

  • up from $235.9 billion in 2023 to $292.3 billion in 2024, a ~24% increase.

It’s interesting to note that the growth rates are the same despite corresponding to separate parts of the business. This could be a coincidence but is worth noting when reviewing the results from different periods.

4. Growth and Pricing Strategy

• In 2022, Shopify introduced Paid Trial Incentives—low-cost access periods designed to attract new merchants after a post-COVID slowdown.

• Many of those merchants upgraded in 2023, boosting growth, followed by a more normalized growth rate in 2024.

• This illustrates how Shopify invests in the long term - providing increased value up front at a time when customers are more price-sensitive and then upselling when the time is right

5. Partnerships and the Payment Stack

• Shopify relies on third-party partners such as PayPal and Stripe to process some payments.

• However, most transactions run through Shopify Payments, the company’s own payment processing system.

6. Deferred Revenue Explained

• Because merchants often pay upfront for monthly or annual access, Shopify records those payments as deferred revenue (a liability) and recognizes them gradually over the service / delivery period.

• It’s a classic SaaS accounting principle that provides visibility into future recognized revenue.

7. Other Operational Notes

• App Store: Shopify operates its own App Store, where developers can build and sell plug-ins that extend store functionality.

• Sale of Logistics Business (2023): Simplified operations and improved margins, but makes year-over-year comparisons more complex.

• Seasonality: Q4 (holiday season) is historically the busiest quarter for Shopify merchants, boosting both GMV and Merchant Solutions revenue.

8. Open Questions Worth Exploring

1. How large is the Shopify App Store in terms of merchant adoption or developer revenue?

2. What impact could tariffs or trade policy shifts have on cross-border merchant sales?

3. How does Shopify Payments add value on top of partnerships with PayPal and Stripe?

4. Is Shopify Payments lower-margin per transaction but higher in operating margin as part of the overall business?

5. Why did Shopify switch from filing Form 40-F to Form 10-K? Does this mean it transferred its legal presence or main operating presence to the United States?

6. How does the logistics sale affect year-over-year comparability in Merchant Solutions revenue?

Part of this is driven by general curiosity about how companies operate, particularly if they have influenced my life (for example as a customer).

Part of it is because I want to strengthen my understanding of finance and other business fundamentals.

And part of it is for inspiration as a product manager and builder working at the intersection of contracts, business and technology (mix of Generative AI, traditional machine learning, and deterministic software).

I started with Duolingo — in part because I’m a fan of the products and company, and in part because there are many parallels between contracts and language learning.

With that, here are some things I’ve learned reading Duolingo’s recent 10-Q and 10-K, and some related open questions that I have:

1. AI Use – Duolingo uses AI in a variety of ways, including personalizing lessons (e.g. predicting the likelihood that a user will get a question correct to help drive the optimal level of difficulty) and using generative AI capabilities in its Video Call product feature. Other examples include significantly accelerating content / lesson creation by leveraging LLM capabilities. Separately but relatedly, they were one of the first companies I saw or heard talking about becoming AI-first.

2. Large Data Moat – They’ve collected a massive amount of data corresponding to over 130 million active users. This in turn helps improve the user experience by personalizing lessons and otherwise improving the product, per the above.

3. Financial Metrics – They’ve experienced significant revenue and profitability growth driven by an engaging product experience. It’s interesting that revenue growth for both the quarter and the year is ~41% YoY despite some possible seasonality in the business. It’s also interesting that their DAU growth is 40% YoY — unclear if this is a coincidence or if there’s a more direct connection between these metrics.

4. User acquisition – This is primarily through word of mouth (driven by an engaging product experience that uses gamification and bite-sized lessons) and brand marketing (e.g. they’ve had some pretty epic social media marketing). However, there is an increasingly complementary use of paid user acquisition as well, for example, in high-opportunity markets experiencing limited organic growth.

5. Mobile as a distribution strategy – Early on, they bet that mobile would be a way of getting to as many learners as possible (target users of the product) due to the ubiquity of smartphones. This has paid off significantly as they continue to enjoy significant growth in active users and paid subscriptions.

6. Company culture – They champion their culture as being distinct from a more typical Silicon Valley culture. I’m not 100% sure what this means, but it could refer to being more disciplined about aggressive investment to drive growth (e.g., not hiring aggressively during the COVID pandemic/lockdown), living in a more affordable geography, and/or being more ideologically diverse.

Open questions that I have:

1. New hardware implications on Design & UX – There’s some interesting AI-enabled hardware innovation on the horizon, including Meta’s VR glasses as well as the OpenAI/Jony Ive collaboration. Is there any plan to start designing for that medium? If I had to guess, it would take years of change management and user adoption to shift from mobile (similar to how it took 10+ years to shift from desktop computers to mobile — though a remarkable new user experience could accelerate this).

2. Scalability and Profitability Growth – How much of the profitability growth is driven by one-time or non-recurring events? For example, it looks like costs may have decreased as AI automated some processes, but were any of these one-time decreases? Also, is there a view that features driven by generative AI will drive cost efficiencies as foundation models continue to benefit from performance increases and economies of scale? Finally, given the increasing need for paid user acquisition, is there a concern that profitability will suffer in the future as a result of increased customer acquisition costs?

3. Geographic/cultural advantages – What are specific examples of how the Pittsburgh geography/non-Silicon Valley culture manifests, and what are the pros and cons relative to Silicon Valley or another large city/tech hub?

4. Financial Metrics Breakdown by Region – What do revenue and profitability growth look like by region?