Imagine receiving this request:

We signed an LOI to acquire ABC Corporation. Please start reviewing the top 100 customer contracts. We expect to close quickly and will send a data-room link.

It can be tempting to get access to the repository ASAP and start reviewing contracts. However, it’s usually better to scope out what the request is actually asking for and any available context to help inform execution.

That is what the intake-contract-workstream skill is designed for — helping clarify and understand an inbound request for contracts analysis in sufficient depth to effectively begin planning and execution.

Where intake fits in the broader contract workstream framework

Almost every post-executed enterprise contract workstream, big or small, can be framed as the same underlying six-part process:

1. Trigger — an event occurs that requires contracts work

2. Define criteria — determine what you’re looking for and why

3. Find — apply the criteria to locate the target population of contracts

4. Analyze — go deeper on the population: validate the initial search results, incorporate nuances, exceptions, and additional criteria, and perform a comprehensive assessment

5. Decide — translate the analysis into recommendations, remediation paths, answers to stakeholder questions and other internal deliverables

6. Remediate — take action through counterparty outreach, internal communication, negotiation, and workflow tracking

For a full breakdown of the six stages, see the article: The Post-Executed Contract Intelligence Framework.

This article focuses on the first phase: Trigger / Intake.

One call-out re terminology: the trigger is the event or request (usually outside of our control). Intake is the process of turning that event or request into enough structured context to begin planning the work.

What the skill does

The intake-contract-workstream skill acts as a structured thought partner.

Give it an email, notes, or other background, and it helps produce a structured intake brief that:

• Classifies the trigger or request

• Identifies the decision or deliverable that contract work needs to support.

• Translates business context into relevant scope: expected contract populations and data points.

• Distinguishes known facts from missing inputs, assumptions, and unknowns.

• Identifies stakeholders, approvals, and potential blockers.

• Generates recommended next steps - including what can begin now versus what should wait and scope-change risks.

It supports six common request categories (archetypes):

1. M&A (Corporate Transactions)

2. Regulatory and policy remediation

3. Backlog / legacy repository review

4. Business-as-usual stakeholder questions

5. Commercial Opportunity / Optimization

6. Incident response

Two ways to use the skill

Option 1: Provide the underlying context directly

If your organization’s policies and approved environment permit it, you can provide the triggering request or background directly to the model.

Whether that’s appropriate depends on the information involved, the system you’re using, and your organization’s confidentiality, privilege, security, and data-handling requirements.

Option 2: Use the skill to generate an intake checklist

Where providing the underlying context directly is not appropriate, the skill can instead generate a structured checklist of the information that would be helpful to collect.

You can then work through that checklist separately and return only the information that is appropriate to provide.

This is less direct, but it can still help structure an incomplete request and identify the inputs most likely to affect scope or execution.

Limitations / What it does not do

The skill does not establish the final scope of the work, perform contract analysis, make decisions, or perform remediation actions.

Its job is to create a better starting point for those later steps while preserving the role of human judgment in defining scope, analyzing contracts, making decisions, and taking action.

It may identify expected / provisional contract populations or possible criteria, but those should not be treated as approved search or review criteria without appropriate review.

Paid subscriber resources

Paid subscribers receive:

• The complete intake-contract-workstream skill bundle for each of Claude and Codex.

• Installation instructions for project-scoped and personal use.

• A worked M&A example (hypothetical).

• Future updates to the intake framework and request category modules.

Read more

The clock starts running on the contracts workstream.

You’re juggling a handful of other tasks. You’re being told to use AI. Life outside of work doesn’t stop either.

So how do you proceed?

Almost every post-executed enterprise contract workstream, big or small, can be framed as the same underlying process. It doesn’t matter whether it’s M&A diligence, regulatory remediation, a backlog review or a one-off question from sales.

In this article I’ll break that process into six pieces and walk through each one, using M&A as the core example.

The six pieces are:

1. Trigger — an event occurs that requires contracts work

2. Define criteria — determine what you’re looking for and why

3. Find — apply the criteria to locate the target population of contracts

4. Analyze — go deeper on the population: validate the initial search results, incorporate nuances, exceptions, and additional criteria, and perform a comprehensive assessment

5. Decide — translate the analysis into recommendations, remediation paths, answers to stakeholder questions and other internal deliverables

6. Remediate — take action through counterparty outreach, internal communication, negotiation, and workflow tracking

The details may vary significantly from one project or task to another but the underlying process will remain fairly consistent.

Subscribe now

1. Trigger

There are all sorts of possible triggers, but almost all of them fall into three main buckets:

1. External events - something outside the normal course of business creates the need for contract work.

   1. Two of the biggest examples are:

      • M&A (buy-side, sell-side, reorgs, etc.)

      • Regulatory remediation

2. Internal initiatives - the business identifies an internal need.

   1. The classic example is a backlog or legacy review: getting better visibility into a large historical contract population, identifying risks and opportunities, or solving for one or more specific use cases

   2. These projects can be very valuable, but they are often perceived as nice to have and easier to defer than an acquisition or regulatory deadline

3. Business as usual - the steady stream of recurring and one-off questions from various business stakeholders. For example:

   • Sales is negotiating a renewal and wants to understand the existing agreement

   • Finance is asking about revenue commitments, deliverable timing or pricing terms

   • Product wants to know what AI or data-use rights the company has

   • Legal wants to identify upcoming renewals, termination rights, or unusual obligations

The type of trigger matters because it shapes everything downstream: urgency, stakeholders, deadlines, risk tolerance, and most importantly, the criteria.

2. Define criteria

Before searching for contracts or specific clauses, you need to answer two interrelated but different questions:

1. Which contracts are in scope?

2. What do we need to know about them?

I think of these as search criteria (aka population criteria) and analysis criteria.

Search criteria: which contracts matter?

For an M&A transaction, examples of the relevant population include:

• The top 100 or 200 customer contracts by revenue

• All active vendor agreements above a certain spend threshold

• Agreements involving a particular legal entity

The initial scope or data may come from a variety of places: for example the finance team, the seller, the M&A team or a data room.

This is where metadata that is use-case-agnostic often becomes essential: document title, party names, dates, contract type and similar information.

Analysis criteria: what do we need to know?

The second question is more substantive and depends on the use case. For M&A, common areas of interest include:

• Assignment restrictions

• Change-of-control provisions

• Consent and notice requirements

• Termination rights

• Governing law

• Restrictive covenants

• Liability and indemnification provisions

These provisions help determine the plan for the contract under the transaction, what actions may be required, what risks the acquirer inherits, and what alternatives exist.

For data privacy or regulatory work, the criteria might instead include:

• Does the required privacy language exist?

• Does it satisfy the relevant regulatory standard?

• Where are the parties located?

• Where is data stored or processed?

• Are sub-processing or cross-border transfers permitted?

For business-as-usual questions, the search criteria and analysis criteria may merge. “Find the contract with Customer X and tell me whether they can terminate for convenience” already contains both the search criteria and the substantive question. In addition, sometimes simply finding the relevant agreement is enough.

There are also common secondary filters: these include things like active versus expired contracts, upcoming renewals, geography, contract value, business unit. Note that the relevance of each of these depends on the use case. An expired contract might be irrelevant to an assignability analysis but highly useful when someone asks, “What language did we agree to last time?” or “Do we already have an agreement with this counterparty?”

One final point: flagging missing information is often valuable in and of itself. In an M&A context, an inability to locate critical customer agreements or understand material contractual obligations can become a substantive diligence finding rather than merely an inconvenience.

3. Find

Once the criteria are defined, the next question is: How do you identify the relevant contracts?

The answer depends on both what you’re looking for and what information you have to work with. There are two common starting points: a list or a data dump.

You get a list

For example the M&A team or the seller provides the top customers by name.

In that case, finding the relevant contracts is a matching exercise: compare the list against the party names in the contracts. If there’s a match, it generally means the contract is in scope. The comparison can run at several levels of strictness but the core concept is the same:

• Exact legal-entity match

• Fuzzy or approximate name match (e.g. Acme Corporation vs. Acme Corp.)

• Affiliate or corporate-family match

One reason the specific matching logic used is important is because contracts are entered into by legal entities, while the business often thinks about contracts in terms of broader customer or vendor relationships. So the key is finding the balance between exact enough to avoid too many irrelevant documents (false positives) while flexible enough to avoid missing any legitimate ones (false negatives).

You get a data dump

Alternatively, you may receive a data room full of documents loosely described as “top customer contracts,” with inconsistent naming conventions and organization. In this case, there are multiple paths forward.

You can first identify parties and other metadata to organize the population.

Or you can begin with substantive clause or language-based filtering—for example:

• Which contracts restrict assignment?

• Which require consent?

• Which require notice only?

• Which provide termination rights in connection with the transaction?

In practice, these approaches often work together. The list helps define the initial population. The follow-on searches pressure test and refine the scope of the population of interest and prioritizes them further, as needed.

4. Analyze

The line between finding a contract and analyzing it isn’t always clear. For example, flagging a contract for assignment restrictions is already a form of analysis.

But the analyze stage is where you go deeper, and it has four layers:

1. Validate the search results

2. Incorporate exceptions, carveouts and nuances

3. Expand the criteria

4. Synthesize

Validate

First, confirm that the results actually match your intent — e.g. that every contract flagged with assignment restrictions actually has them.

If a system identifies 75 agreements as containing assignment restrictions, do they really contain assignment restrictions relevant to the transaction?

A clause may contain the word “assignment” but address the counterparty’s assignment rights rather than yours. It may apply only to certain rights or obligations. It may not directly use the word “assignment” or “assign” at all. Or it may use the same keywords but be completely irrelevant – for example referring to a work assignment rather than the legal transfer of an asset.

Key point: finding text is not the same as determining its semantic meaning or significance.

Incorporate exceptions, carveouts and nuances

Next, examine the exceptions, carveouts, and situation-specific facts that can change the initial categorization.

Suppose a contract says consent is required for assignment—unless the agreement is assigned to an affiliate.

If the deal structure involves assignment to an affiliate, the remediation path for that agreement may move from “consent required” to a completely different remediation path such as “courtesy notice”.

Or consider a clause permitting assignment in connection with a merger, sale of substantially all assets, or similar transaction. Whether the exception applies depends on both the contractual language and the structure of the deal.

This is why the contract and the business context cannot be separated.

Expand

Finally, the analysis often expands beyond the original criteria. You may begin by looking for assignment restrictions. But once you understand the contract population and transaction structure, additional questions emerge.

For M&A, examples include:

• Change of control

  • Assignment and change of control belong to the same general umbrella of concepts and are often analyzed together, but they are not identical.

  • The specific transaction structure may trigger a change-of-control provision even where no contractual assignment occurs.

• Termination rights

  • Termination rights (especially for convenience or a result of corporate transactions) can significantly impact the risk/opportunity profile of a contract

    • If a customer can terminate a significant revenue contract at will, that may represent risk to the acquirer.

    • If the acquired company has a broad right to terminate a vendor agreement, that may create flexibility: the acquirer can potentially exit an unfavorable contract, consolidate vendors to capture deal synergies, or simply use the credible possibility of termination as leverage in renegotiating commercial terms.

• Limitation of liability and indemnification

  • Even assuming the same contractual terms, the significance of contractual risk can change after an acquisition. A smaller company may have operated for years under a contract with unusually broad liability exposure without a dispute ever arising. But when a much larger and better-capitalized company (i.e. a better litigation target if something goes wrong) acquires the business or assumes the agreement, the practical risk assessment may change.

  • That does not automatically mean every contract with unfavorable liability terms must be terminated or renegotiated. But it may change the priority assigned to the contract and the remediation decision.

• Restrictive covenants

  • Various commercial restrictions may conflict with the strategic rationale for the transaction. Examples include: exclusivity language, most-favored-nation clauses, non-competes, non-solicitation and subcontracting restrictions.

  • Imagine acquiring a company specifically to enter a new market, only to discover that a material contract limits the company’s ability to compete, distribute products, work with certain customers, or use particular subcontractors.

Synthesis

At the end of the analysis stage, the output should be more than extracted data.

Net of all of considerations, one or more internal deliverables is created: a spreadsheet, table, report or something else showing the analysis and what it means — the suggested remediation path and why. For example, if 100 contracts were in scope:

• 10 may be expired

• 20 may require assignment consent

• 40 may require notice

• 10 may require an amendment or negotiation due to risk concerns

• 20 should be left alone - perhaps because they’re undesirable contracts coming up for expiration before deal close or because they’re with related entities and don’t require additional action

That final assessment informs the decision.

5. Decide

Before anyone reaches out internally or externally, there’s a need for a decision.

For an M&A transaction, that often means classifying contracts into different remediation paths: assign, exit/terminate, renegotiate, renew, or leave alone.

In addition, more significant issues may also affect the transaction itself. Diligence and integration are often gray areas rather than cleanly separated phases. The significance of a contractual issue depends on when it is discovered, the magnitude of the exposure, the importance of the contract, and the broader rationale for the acquisition. A single problematic contract does not necessarily jeopardize a deal, but a population-level pattern might.

For example, perhaps a large percentage of the target’s revenue is tied to contracts that can be terminated for any reason. Or critical agreements cannot be transferred without consent. Perhaps restrictive covenants interfere with the strategic rationale for the acquisition or the company simply does not have enough durable customer contracts to provide the market beachhead the acquirer expected. Perhaps uncapped liability exists in critical places.

All of these considerations are ways that contract data can impact the overall deal.

6. Remediate (Take Action)

Once the decisions are made, it is time to act.

For M&A and other episodic projects, this often means contract remediation and any related counterparty outreach:

• Requesting consents

• Sending legally required notices or courtesy notices where appropriate

• Collecting responses and following up with counterparties as needed

• Negotiating where consent is difficult to obtain

• Amending agreements as needed before assignment

• Terminating and/or entering into new contracts where necessary

All of this requires a significant workflow tracking component as well, including:

• Internal communication - legal, sales, finance, procurement, integration teams, business owners, and senior leadership may each need different views of the same underlying analysis and remediation

• Tracking the remediation results on a contract by contract basis - which consents are outstanding, which notices went out, which contracts are stuck, what’s blocking or impacting the deal close, etc.

Recap

Almost every post-executed enterprise contracts project can be framed as the same underlying 6-part process:

1. Trigger — an event occurs that requires contracts work

2. Define criteria — determine what you’re looking for and why

3. Find — apply the criteria to locate the target population of contracts

4. Analyze — go deeper on the population: validate the initial search results, incorporate nuances, exceptions, and additional criteria, and perform a comprehensive assessment

5. Decide — translate the analysis into recommendations, remediation paths, answers to stakeholder questions and other internal deliverables

6. Remediate — take action through counterparty outreach, internal communication, negotiation, and workflow tracking

Hopefully the above helps illustrate that post-executed contract intelligence is a multi-faceted problem and that each step in the process has its own challenges and nuances.

That leads to an interesting question: how much of each stage can AI actually help with? Or more specifically: at each stage, from trigger to remediation, what can AI reliably do, where does it fail, what inputs does it need, what are the alternatives and trade-offs, and what are the complementary capabilities that need to exist around the model to get the best results?

If you found this article helpful, subscribe to follow along and forward along to anyone who might find this useful.

Subscribe now

Imagine a $1 million customer contract.

Five-year term. Roughly $200,000 per year. The contract includes a renewal option, but the renewal window is missed. The agreement expires. Now the business has to get a new contract in place.

Because this is an enterprise customer, that’s not as simple as sending over a quick email and getting a signature the next day. It means re-entering the contracting cycle: budget approvals, procurement, infosec review, legal review, stakeholder alignment, and all the other internal safeguards that large companies put around vendor relationships.

Let’s suppose the relationship is strong and the customer renews without major issues. But the delay pushes out a quarter of revenue.

That is roughly a $50,000 problem.

Now imagine the root cause was not a bad negotiation, a missed email, or a failure by the account team.

Imagine it was this: the contract system had the wrong renewal date because it relied on the wrong effective date. The effective date was wrong because the OCR misread the scanned contract.

This is hypothetical, but plausible. It is the kind of failure that becomes possible when post-executed contracts are converted into structured data at enterprise scale – hundreds of thousands of contracts spanning many contract types, use cases and internal systems.

For a deeper breakdown of post-executed enterprise contract intelligence and complexity, see the previous article here and video here.

There are three common OCR failure modes. But before getting into them, we need to unpack why OCR matters for enterprise contracts to begin with.

Subscribe now

Why OCR Is Important: The model often does not see your contract

Enterprise contracts are often stored as PDFs. But a PDF is essentially just a container – the contents of the container (and how clean or machine-readable it is) varies widely.

Some PDFs are text-native. Some are image-only scans. Some have OCR layers that are incomplete, misaligned, or simply wrong.

If the PDF is text-native, a system may be able to extract the embedded text directly. If it is a scan, OCR has to create the text layer. Once that text layer exists, many downstream extraction and classification workflows treat it as the relevant contractual text. This is crucial to understand because a large language model usually has no built-in way to know whether the extracted text is consistent with the original page.

In other words: OCR quality can determine what the AI system thinks the contract says. So when the OCR layer is wrong, the model’s input is wrong. And a scanned contract can be perfectly legible to a human reviewer and still be unreliable for a contract AI system.

With all that in mind, there are three OCR failure modes worth separating based on both the practical impact they have and how difficult they are to spot:

1. No text layer exists

2. Text layer exists but is damaged

3. Silent failure

Failure mode 1: there is no text layer

The simplest failure is when the document has no usable text layer at all.

This happens when the contract is an image-only PDF, when OCR was never run, or when some file-quality issue prevented OCR from producing usable text.

To a human reviewer, the document may look perfectly readable:

To a text-based extraction pipeline, it may be a blank page.

No usable text layer means no reliable text input — at least not without additional preprocessing.

This failure mode is annoying, but relatively straightforward. You can detect that there is no text, no extracted content, or no meaningful input for the model to use.

The harder cases come when OCR did run.

Failure mode 2: the text layer exists, but parts of it are damaged

The more common case is that OCR ran and got enough of the document right that the problem is easy to miss.

Two things make this difficult.

First, OCR quality is not always consistent within a single document or even within the same page. Some parts of the document can have clean, easily readable text and others can be damaged.

Second, whether any damage actually matters depends on the use case. A model may be able to classify a clause correctly even when the text contains small errors. But those same errors may be fatal if the task requires exactness. The challenge is that many contract use cases do require exactness, as we’ll see.

Here are excerpts from the same document with significantly different OCR results.

The indemnification language came through cleanly.

Screenshot:

OCR:

(C) Each Party agrees to defend, protect, indemnify and hold harmless each other Party from and against all claims and demands, including any action or proceeding brought thereon, and all costs, losses, expenses and liabilities of any kind relating thereto, including reasonable attorneys fees and cost of suit, arising out of or resulting from any construction activities performed or authorized by such indemnifying Party; provided however, that the foregoing shall not be applicable to either events or circumstances caused by the negligence or willful act or omission of such indemnified Party, its licensees, concessionaires, agents, servants, employees, or anyone claiming by, through, or under any of them, or claims covered by the release set forth in 5.4(D).

But a few pages away, the term language picked up OCR errors:

ARTICLE VII

TERM

Screenshot:

OCR:

7.1 Term of this OEA. This OEA shall be effective as of the date first above written and shan continue in full force and effect until 11:59 p.m. on December 31, 2049; provided, however, that the easements referred to in Article II hereof which are specified as being perpetual or as continuing beyond the term of this OEA shall continue in force and effect as provided therein. Upon termination of this OEA, an rights and privileges derived from and all duties and obligations created and imposed by the provisions of this OEA, except as relates to the easements mentioned above, shan terminate and have no further force or effect; provided, however, that the termination of this OEA shan not limit or affect any remedy at law or in equity that a Party may have against any other Party with respect to any liability or obligation arising or to be performed under this OEA prior to the date of such termination.

“Shan continue in full force and effect” is damaged text, but it is still recognizably term language.

So if the task is clause classification — does this document contain a term provision? — the OCR may still be good enough for the model to classify it correctly. The model has plenty of semantic context to work with.

But contrast that with data points where exactness is crucial: effective dates, expiration dates, signature dates, notice periods, renewal deadlines, monetary amounts and party names.

Here is a preamble where the date was handwritten into a blank:

The OCR output looked like this:

THIS AMENDED AND RESTATED OPERATION AND EASEMENT AGREEMENT (”OEA”) is made and entered into as of the a<g day of (j)Q.\uber, 1999, between DAYTON HUDSON CORPORATION, a Minnesota corporation (”Target”) and Northway Mall Associates, a New York partnership (”Developer”).

The handwriting distorted the OCR, and the date came out as junk. A date that cannot be parsed is not usable.

Interestingly, running the same page through a different OCR engine produced a completely different — and correct — result:

THIS AMENDED AND RESTATED OPERATION AND EASEMENT AGREEMENT (”OEA”) is made and entered into as of the 28 day of October, 1999, between DAYTON HUDSON CORPORATION, a Minnesota corporation (”Target”) and Northway Mall Associates, a New York partnership (”Developer”).

Same page. Same underlying contract. Two different OCR outputs. One incorrect, one correct.

This is not a rare edge case. Post-executed enterprise contract repositories often contain scans, handwritten dates, signatures or notes, and documents of widely varying age and quality. OCR quality varies by engine (the type of OCR software), by file type, by scan quality, by page, and sometimes by individual field.

That is a recurring theme regarding why post-executed enterprise contract intelligence is so hard – the multi-step, document-to-data pipeline. Model selection is just one step.

Still, in this second failure mode, the damage is at least reasonably easy to spot: “Shan” is not a word and “a<g day of (j)Q.\uber” is not a date.

A QC layer, parser diagnostics and observability, OCR confidence scores, or even a careful human glance at the extracted text may flag the problem.

Which brings us to the failure mode that is the most difficult to catch.

Failure mode 3: the text is plausible — and wrong (aka silent failure)

The most dangerous OCR failure is when the output looks clean.

No garbage characters. No obvious misspellings. No broken date format.

Just a value that happens to be incorrect.

A handwritten 7 gets read as a 1. A 3 gets read as an 8. A $100,000 value gets misread as $1,000,000 — or the other way around.

Here is a real example — a signature block with a handwritten date:

The OCR output appears clean:

AGREED AND ACCEPTED:

PLITT THEATRES, INC.

By: [Signature] Henry Plitt, Chairman of the Board

Date: Oct 12 1982

But look closely at the image. The day appears to be 13, not 12.

The likely reason is that the lower part of the handwritten digit overlapped with the signature line, causing the OCR to read the 3 as a 2.

This situation is much more dangerous than junk text.

Junk text tells you something went wrong so you can course correct as needed.

A date that is plausible but wrong does not throw up any red flags. In this case, “Oct 12 1982” is a valid date – it has the right format and is semantically plausible.

In other words, it is a silent failure.

And silent failures are the ones that create the most risk in contract AI systems.

Why small OCR errors become business problems

You might look at the example above and say – it’s only one day off, that won’t have much impact.

Maybe. But I’d call out two things:

1. Enterprise contracts often have more significant silent OCR failures such as a wrong month or wrong year (note that the example above is from a publicly available contract). This becomes more likely as hundreds of thousands of documents present many opportunities for silent OCR failures.

2. The broader point, however, is that a single, seemingly innocuous bad value rarely stays contained to one situation or use case and can have downstream consequences.

For example, a renewal date or expiration date may not be written explicitly in the document. It may be computed from other extracted fields: effective date plus initial term, potentially plus a renewal period minus a notice window.

If OCR silently corrupts the effective date, the computed expiration date may be wrong.

If OCR silently corrupts the term length, the renewal calculation may be wrong.

If OCR silently corrupts the notice period, the counterparty outreach or operational deadline may be wrong.

If OCR silently corrupts the contract value, the agreement may be routed to the wrong workflow, wrongfully excluded from a report, or treated as below a threshold that should have triggered additional review.

And the model may have no way to know, because every input it was handed looked valid on the surface.

That is how a small OCR problem becomes a contract intelligence problem.

This is also why scale changes the risk profile. In a small set of contracts, these errors may be annoying but manageable. In an enterprise repository with tens or hundreds of thousands of documents, even a low error rate can create meaningful downstream exposure.

OCR quality is not just a preprocessing issue. It is a contract data quality issue.

What can be done

There are various approaches to either reducing or mitigating issues that can come up from OCR or file quality issues. This discussion could easily be its own article.

At the OCR layer - some common approaches involve checking for existing OCR, re-running OCR software, testing different engines and capturing OCR confidence where available.

At the extraction layer - high-impact contracts or fields can be routed into more robust QC workflows.

At the QC or validation layer - rules can catch what is catchable: for example values outside plausible ranges.

In addition, evolving AI model capabilities may help with some of these as well.

Recap and the open question

To recap, there are three OCR failure modes worth separating:

1. No text layer at all — the document is readable to a human but may be blank or illegible to a text-based extraction system. This is usually relatively easy to detect by a well-designed system.

2. Visible OCR damage — the text layer exists, but contains obvious problems like non-words, junk dates, or corrupted phrases. This can also often be detected, and whether it matters depends on the use case.

3. Silent OCR damage — the text looks plausible, but is wrong. This is the most dangerous because it can flow through the pipeline undetected.

The open question is empirical: in a real population of post-executed enterprise contracts, how often does each failure mode occur? And how much does each one affect downstream contract AI accuracy?

That is a type of measurement I have not seen yet — and one that matters if contract AI is going to move from impressive demos to trusted enterprise systems.

Because the real issue is not simply whether an AI model is powerful enough.

It is whether the system has given the model the right contractual text to read in the first place.

If you found this useful, subscribe to The Contract Signal — and forward it to anyone who might find it helpful.

Subscribe now

1. Post-executed – these are contracts that have already been signed or executed, otherwise known as post-signature contracts, as compared to pre-execution or pre-signature contracts that have been drafted, negotiated or redlined but not signed by all parties.

2. Enterprise – at least one party to the contract is a very large business, such as a F500 company.

3. Contract Intelligence – enabling people to make legal and business decisions by turning contracts into structured data that can be searched, filtered, reported on, and analyzed at scale.

Let’s dig into this in greater depth – including how these concepts influence one another.

Prefer to watch? Here's the video version:

Subscribe now

Deeper Dive

Post-executed

First, we should dig into the implications of what it actually means for a contract to be post-executed.

Contracts are written agreements containing the terms of an exchange, including legal obligations on both sides.

Post-executed means the contract is already signed. This means we’re past the drafting stage, negotiation, and redlining — the language has been finalized and agreed to. In other words, pending an amendment to the original contract or another legal mechanism, the language is locked, and the rights and obligations in effect.

Enterprise

Being a large enterprise has many significant implications for contracts as well.

Range and volume of contract types

One implication is that the volume of contracts is much, much higher for large enterprises than it is for smaller businesses.

This happens for a few reasons.

First, enterprises are large, diversified companies, often with multiple business units, many customers and vendors, many products and services, and a more complicated supply chain. For example, everything ranging from the need to manufacture things to the need to distribute products and services at scale means that there are many more significant business exchanges and many corresponding opportunities for contracts across a range of contract types.

These may include manufacturing agreements, reseller or distribution agreements, licensing agreements, partnership agreements, leases, and other types of customer and vendor contracts.

Contract Families

Another implication is that because enterprises have been around for a long time, they have business relationships and contracts from many different periods.

This includes older contracts with longer lifecycles, often including multiple rounds of contracting between entities. These relationships may include more than just an initial master agreement. They may also include NDAs, amendments, statements of work, order forms, addenda, and other related documents.

As a result, large enterprises can have very large families of related contracts.

Complexity of individual contracts

Relatedly, the complexity of the contracts themselves tends to be very high for enterprises. Large enterprises have all sorts of complicated requirements built into their contracting practices.

This is partly due to being subject to different regulations as a result of their broad reach of business activity, often spanning geographies and business lines. Also, because they have been in business for a long time, different laws, rules and conventions may have applied over different periods of time.

It’s also because they are at greater risk of litigation, leading to more robust risk allocation measures. This is driven in part by increased opportunities for something to go wrong, given their higher levels of business activity and transaction volume. It is also driven by the fact that large enterprises often have more money and assets to pursue, making them more attractive litigation targets relative to smaller entities.

As a result, they often have to be more aggressive about risk allocation. This can lead to lengthier, more detailed provisions that limit liability exposure in different circumstances.

Net of these considerations, the population of contracts at a given enterprise is highly mixed. It is not homogeneous or standardized.

Even the same type of contract may have many different variables and complexity drivers baked in.

That gets compounded further by other complexity drivers, such as:

1. File quality. Many of these documents include emails, scans, pages with coffee stains, handwriting in the margins, or image-only PDFs. Each of these can reduce the quality and reliability of OCR (optical character recognition). Bad OCR means the text may not be properly recognized by AI models. So before any AI touches the contract, the file often has to go through a chain of preprocessing steps just to become usable. Example here: link

2. Multi-document files. One file will often contain multiple discrete parts. For example, you may have the core contract, cover pages, exhibits, addenda, additional terms and conditions, and other related documents all in one file. Determining which document or clause takes precedence can be a very challenging exercise as a result. Example here: link

3. No system of record. Enterprises are diversified, with many business units, siloed repositories and multiple custodians. As a result, there is often no central place where all contracts live. Companies frequently can’t even find their contracts, let alone analyze them. And when they do find them, they are often dealing with legacy systems that contain partially executed agreements, duplicates or overlapping documents.

Compounding Complexity

Each of these factors would create complexity on its own. Enterprises often deal with all of them at once, across hundreds of thousands of documents.

So there is a range of complexity drivers, and many of them are closely linked to large enterprises. That’s why the enterprise dimension is so important and why it has so many implications for contracts.

Net of these considerations, you do not have one big, centralized, clean dataset. You have many unstructured, complex, messy, difficult-to-find documents — subpopulations inside subpopulations, fragmented across many systems.

Subscribe now

Contract Intelligence

Collectively, this is what creates the need and opportunity for contract intelligence.

Even finding contracts in an enterprise can be very challenging, let alone knowing the substance of what is in those contracts and what obligations they contain.

And this matters because hidden within these already-executed contracts are collectively huge opportunities, both in terms of risk allocation and mitigation, as well as increasing business value.

For example, it is not unheard of for a contract to lack a limitation of liability clause, which could introduce significant liability for the business, particularly if it is an older contract without other safeguards in place.

Payment terms are often suboptimal, leaving value behind or creating misalignment with the company’s standards, internal finance systems, and processes.

There may also be untapped opportunities to increase prices, or rights and restrictions to assignment, termination or exiting agreements.

The goal of contract intelligence is to take all of these contract documents and be able to search, filter, report on, and analyze them at scale.

The data points of interest vary based on the use cases and the business or legal problems people are trying to solve.

For example, in M&A diligence and integration, you might be interested in assignment, change of control, governing law, and related provisions.

For a business-as-usual type of review where a question comes up from the commercial team, you may be interested in a specific clause, such as the indemnification language agreed to in previous similar situations, or how long a contract requires the company to support a product that is being sunset.

This is a very small sample of potential use cases. The relevant questions vary depending on the business line, industry, function, department, specific situation, and place within the larger supply chain.

Contract Intelligence vs. Contract Summarization

For sake of clarity, it is important to understand that the difference between contract intelligence and a contract summary. These are complementary use cases that serve different purposes.

Contract intelligence specifically relates to turning the natural language of contracts into a structured data asset at scale, so users can gain visibility into their contract data through search, filters, reports and analytics.

In contrast, with a contract summary, you are typically doing a deeper dive into one or more contracts, often at smaller volume, to better understand the underlying nuances.

The critical difference is that the output involves qualitative descriptions of text. These are not ideal for running searches, reports and analytics at scale.

However, summaries can be very helpful when creating a diligence memo or other report to support a decision across a contract population, such as whether to move forward with an acquisition or how to prioritize regulatory remediation.

Summary

To summarize, post-executed enterprise contract intelligence involves three main components:

1. Post-executed – contracts that have already been signed or executed, otherwise known as post-signature contracts, as compared to pre-execution or pre-signature contracts that have been drafted, negotiated or redlined but not signed by all parties.

2. Enterprise – at least one party to the contract is a very large business, such as a Fortune 500 company.

3. Contract Intelligence – enabling people to make legal and business decisions by turning contracts into structured data that can be searched, filtered, reported on, and analyzed at scale.

   Subscribe now

If you found this article helpful, subscribe to follow along and forward it to anyone you think might find it useful.

I’ll take the overlap as validation. The framing works in part because contracts are becoming more than documents to store, negotiate, and search. Once you parse through the dense language, contracts start to look like signals: of risk, leverage, governance, market dynamics, opportunity, and operational behavior.

But the naming overlap is also a small symptom of a larger market pattern.

The volume of contract AI news keeps accelerating. Every week brings another product announcement, research release, workflow claim, or market report. And in that stream of news, one word keeps coming up: benchmark.

Subscribe now

Crosby published a contract-redlining benchmark. Harvey has published benchmarks for contract understanding and legal agents. TermScout, LexisNexis, and others benchmark contract terms against market norms. CLM vendors benchmark cycle time and operational performance.

One word — benchmark — is taking on multiple meanings.

Sometimes “benchmark” means a test. Sometimes it means a dataset. Sometimes it means the results from a market survey or a dashboard metric summarizing the same to compare against industry peers.

It has quietly become a word that sounds precise but is often focused on marketing.

So when (or before) the next benchmark lands in your inbox, the first move is simple: ask what kind of benchmark you are actually looking at.

Benchmark of what? Against what? Scored by whom? Using whose data? And for whose benefit?

What the word actually means

Strip away the jargon and a benchmark is just a reference point. A fixed thing you measure against — a baseline, a starting line — so that any other number means something relative to that initial reference point.

You set the reference, then you compare: better than the benchmark, worse than the benchmark, by how much.

Contracts professionals already had a specific version of this long before AI showed up. In outsourcing and other long-term agreements, a benchmarking clause lets a customer bring in an independent third party to compare a supplier’s pricing and service levels against the wider market — and adjust the deal if the supplier has drifted off-market.

Note what made that mechanism trustworthy: an independent party, an agreed method, and a defined market to compare against.

In the AI era, the word has stretched to cover at least four genuinely different activities. They get announced with the same or similar vocabulary, which is exactly why they are hard to read.

The first evaluates models. The second evaluates contract positions (i.e. the values of given clauses or other data points). The third evaluates operational efficiency. The fourth evaluates AI or system improvement over time.

Separating these concepts can help provide clarity.

The four benchmarks hiding under one word

1. Performance benchmarks — “How good is the AI?”

This is the one generating the loudest headlines: take a task, have AI models do it, and score them against a human expert or a model answer.

One recent example is Crosby’s Redline Bench, which scored frontier models on realistic SaaS-contract redlining against attorney-authored “golden” responses. The reported results put the top model around 50% and the rest clustered below it — a narrow spread, with humans still better at finding new routes to resolution while models tended to anchor on their opening positions.

Conceptually, there is nothing wrong with this. Measuring model performance against domain experts in realistic conditions is a reasonable thing to do. To their credit, the better recent benchmarks in this category disclose a lot: the use cases, the scoring dimensions, the rubric, and in some cases the whole thing is open for other labs to run.

That is valuable as it helps practitioners assess actual performance.

The trouble starts with how the numbers get used downstream.

“Model X scored 50.5%” gets repeated far from the methodology that produced it. Without that context, the number is almost impossible to interpret.

It may be directionally interesting. It may tell you something about frontier-model behavior in a simulated negotiation. It may help compare models under one defined test design.

But it does not automatically tell you which product to buy, which workflow to automate, or whether the same system will work inside your contract population.

That distinction matters.

A multi-turn negotiation benchmark is not the same thing as a data extraction benchmark. A redlining benchmark is not the same thing as a playbook-compliance benchmark. A test of SaaS agreement negotiation is not the same thing as a test of high-volume vendor onboarding, procurement triage, or post-signature obligation extraction.

And in many contract AI workflows, the advanced task depends on the foundational layer being right first.

Before a system can negotiate from a contract, it often has to identify the agreement, extract the relevant provisions, understand the parties, map the clause to the right playbook position, and preserve the context across the workflow. If those earlier steps are noisy, the later benchmarked task may inherit the errors. Sometimes the problem is not only model performance. It is error propagation – the snowballing or compounding of small mistakes at different stages in the process until the end result is a far cry from the original performance findings.

That is why the headline score is not enough.

Did the model miss obvious issues? Did it identify the right concern but propose a weaker redline? Did it produce something commercially reasonable but different from the attorney-authored answer? Was it penalized for not matching the exact path the human took? Would two senior attorneys have agreed on the same “perfect” answer?

A performance benchmark can be useful. But the headline score is not the benchmark. The benchmark is a combination of the task, the dataset, the scoring method, and the judgment calls underneath the score.

2. Norm benchmarks — “Is my contract typical?”

A completely different activity wears the same label.

Here, you assemble a large body of contract data, derive what is standard for a given type of deal, and then compare a new agreement against that norm or standard.

LexisNexis’s Market Standards, TermScout’s reports, and various “state of contracts” reports live in this category. The output is not a model score. It is a reference set.

For example: for vendor agreements of this type, in this industry, at this deal size, mutual indemnification is standard; one-way indemnification is aggressive; this limitation of liability formulation is common; that AI-use restriction is becoming more frequent.

This is genuinely useful and very doable.

Most of it reduces to structured data. A clause like indemnification can be reduced to a small set of positions — both parties indemnified, one party indemnified, neither party indemnified — and once you have coded enough contracts and broken down each clause type sufficiently, you can say how common each position is. Control for industry, deal size, template source, whether the agreement is customer paper or vendor paper, and other criteria and you have a real reference point for negotiation and risk.

But norm benchmarks have a problem that rarely makes the announcement: representativeness.

Much of this analysis is built on public filings, and public contracts are a biased sample by construction. A contract gets filed with the SEC because it crosses a materiality threshold. That means the corpus skews toward large, heavily negotiated agreements that may look very different from the thousands of ordinary commercial contracts a business actually runs on.

Even the filed agreements are frequently redacted at the commercially sensitive points you would most want to benchmark.

I used SEC contracts myself in last week’s article on creating a contract data extraction benchmark, and the same caveat applied there: they are a convenient, legitimate starting point. They are not a stand-in for your private contract population.

A norm built on a skewed sample is a useful hint, not ground truth.

3. Process benchmarks — “How fast is my operation?”

A third meaning has nothing to do directly with model quality or contract terms at all.

It is operational: cycle time, turnaround, throughput, percentage of contracts on standard templates, time-to-signature, number of negotiation rounds, fallback frequency, legal touch rate, approval bottlenecks.

CLM vendors increasingly sell “benchmarking” dashboards that compare your contracting operation against peers.

This is often the management-consulting meaning of benchmark, applied to legal ops. It may also be the most actionable version for many teams.

If your average NDA takes 12 days and comparable teams complete theirs in three, that is useful information to have. By the same token, if 80% of your low-risk agreements still require legal review while similar companies route most of these through self-service or your contracting cycle time spikes in one region or business unit, or by contract type, that is useful too.

But it is worth separating this cleanly from the other meanings.

A vendor saying it can “benchmark your contracts” might mean it can compare clause language to market norms. It might mean it can compare your contracting process to peer operations. It might mean it can measure model performance. Those are different use cases with different products, solutions, and datasets.

Process benchmarks are valuable when the metric maps to a real operational decision.

They are less valuable when the metric becomes theater: a dashboard showing you are slower than “peers” without explaining who the peers are, what work was included, or whether the comparison reflects your risk profile, company stage, industry, deal size, or contracting model.

Subscribe now

4. Baseline benchmarks — “Is my system getting better?”

The fourth meaning is the one practitioners building AI systems use most, and the one that almost never makes the press release.

Here, a benchmark is an internal baseline.

You fix a starting level of performance — an out-of-the-box model, a first prompt, a v1 pipeline, an existing review process — and then measure every later change against it.

Did the new prompt help? Did the more expensive model actually earn its cost? Did retrieval improve accuracy or just add latency? Did fine-tuning move the metric or just the vibes? Did the new clause taxonomy improve consistency? Did the latest model version break something that used to work?

This is how serious AI and data teams actually work.

And it carries two critical ideas the headline benchmarks tend to gloss over but often have a huge impact on assessing the benchmarks and making them actionable.

1. The first is generalization. A model trained or tuned on one contract population can fall apart on a new population due to different drafting, file quality, document structures and conventions used.

2. The second is data drift. A system that scored well last quarter can quietly degrade as the documents flowing through it change.

A single published score is a photograph: one snapshot in time, taken under a defined set of assumptions.

Truly effective benchmarking is more like a video – multiple points in time, multiple snapshots, multiple angles, and enough continuity to tell whether the system is improving, degrading or staying flat.

And the video is what tells you whether something works in production.

Four meanings. Four different questions. One word.

The moment you hear “benchmark,” your first move should be to figure out which one you are actually being sold.

How to read any benchmark

Sorting the type gets you halfway there. The rest is actually performing the analysis.

There are three layers that rarely get discussed and almost always decide whether a number means anything: the use cases, the scoring and the incentives.

The use cases

First, what was actually tested?

“Contract review” can mean redlining a standard NDA against a clear playbook. It can also mean a no-playbook judgment call on a bespoke, heavily negotiated agreement with limited commercial context.

Those are wildly different tasks.

Then, the specific scenarios corresponding to the use cases.

Before you trust a performance number, you want to know the distribution of difficulty behind it. Was the benchmark mostly easy cases? Edge cases? Common issues? Rare but high-risk provisions? First-pass issue spotting? Full redlining? Negotiation strategy? Escalation judgment?

That is the difference between “the AI is good” and “the AI is good at the easy version.”

The scoring

How was “right” decided?

This is where legal benchmarks are genuinely hard, because you are scoring dense, qualitative prose, not simple multiple-choice.

Did they grade it as a simple classification problem — risk flagged correctly or not, one point each over the total possible? Or did they use a rubric that accounts for subjectivity, ambiguity, commercial reasonableness, drafting quality, and the context the model was given?

And what does 100% mean?

Does it mean the model matched the attorney answer exactly? If so, would two senior attorneys have agreed 100% of the time?

Often they would not.

Law is full of gray areas. That is why it needs lawyers. A scoring method that pretends the gray areas are black and white will produce a confident number that does not survive contact with reality.

In addition, watch for false precision.

A score reported to a tenth of a percent — 50.5, 45.1, 44.4 — signals rigor. But the decimal is only as meaningful as the rubric underneath it.

When the rubric is scoring subjective legal judgment, three-significant-figure precision is often decoration. The confident number is the easiest part to produce and the least informative.

The incentives

Who ran the benchmark or experiment, and what do they sell?

This is the layer that matters most and gets discussed least.

Charlie Munger is often quoted as saying: “Show me the incentive and I will show you the outcome.”

A private sector benchmark is almost never published for philanthropy.

A law firm has an interest in a benchmark that shows legal judgment remains hard to automate. A vendor has an interest in a benchmark that makes its workflow look measurable and differentiated. A model lab has an interest in a benchmark that defines progress around the capabilities its model performs well on.

None of that means the benchmark is dishonest.

It means the benchmark has a point of view.

Wherever judgment enters — which tasks to include, how to score gray areas, what to call 100%, which dataset to use, which comparison group to define — the benchmark will reflect the author’s theory of the problem.

Even a scrupulously transparent benchmark is shaped by what its author chose to measure in the first place.

Transparency lets you check the math. It does not remove the incentive that selected the problem.

How to spot a benchmark worth trusting

None of this is a reason to dismiss benchmarks.

They are one of the better things to happen to a field that ran on vibes and demos for years. But they need to be read like evidence, not like slogans.

To summarize, a benchmark you can actually lean on tends to disclose:

• Scope — the tasks tested, how hard they were, and whether the headline average hides meaningful variation.

• Method — how “correct” was scored, who judged, and what a perfect score would mean.

• Data provenance — where the corpus came from and how representative it is of the contracts you care about.

• Authorship and incentive — who built it, who funded it, and what they sell.

• Reproducibility — whether anyone else can run it and get the same answer.

• Business relevance — whether the metric maps to a decision you actually need to make or problem you need to solve.

Use that as a checklist.

A benchmark that discloses its scope, method, and data but is published by a party with an obvious stake in how it is used is not worthless. It just needs to be read with the stake in view.

A benchmark with a strong headline but no corpus, no rubric, no task design, no scoring explanation, and no discussion of limitations is not reliable. It is a claim dressed up as a benchmark, often for marketing purposes.

The through-line

Last week I made the case that contract AI looks magical on one document and gets hard across a population — that the demo and the workflow are different problems.

A benchmark is the same story one level up.

The score is the demo. The methodology is the workflow.

A number that travels without its context is a promise without the receipts.

That is why independent reading of this market matters.

Independent does not mean neutral or detached. I have a point of view, shaped by a decade in enterprise contracts, taxonomy design, structured data and extraction workflows, AI, and legaltech product work.

It means something narrower and more important: I do not have a platform, model, law firm service, or benchmark to sell inside the analysis.

When the company publishing the benchmark also benefits from the way the benchmark is framed, the right response is not cynicism. It is professional skepticism.

What was tested? Against what? By whom? On whose data? And who benefits from the answer?

More signals soon.

If you found this useful, subscribe to follow along, forward it to someone who is about to make a tooling decision off a benchmark headline, and tell me which contract AI terms or claims you want me to unpack next.

Subscribe now

Ask a frontier model to pull the title, parties, and effective date from a single contract and it will probably get the answer right.

That is real progress.

It is also not the same problem as organizing a contract population.

So I ran a first benchmark: 100 real contracts, five metadata fields, three model runs, one fixed prompt, clean text-layer PDFs only.

The result was not a simple leaderboard. The more useful finding was where the workflow started to bend.

At one document, the experience feels almost magical. At 100 documents, the bottlenecks start to show: session limits, inconsistent handling of ambiguous fields, silent wrong answers, and model-specific trade-offs between accuracy, speed, and usage.

That matters because contract AI workflows are increasingly being built on top of specific model versions. And model access itself is now a planning variable. Last week’s export-control disruption affecting Anthropic’s Fable 5 and Mythos 5 models is an extreme example: access can change suddenly for reasons outside the workflow itself.

So this benchmark is partly about accuracy. But it is also about something more practical:

What happens when a capability that works in a demo has to survive a realistic contract-population workflow?

Subscribe now

What I tested

Before you can do anything substantive with a contract population — M&A diligence, renewals management, migration into a CLM, risk analysis — you have to know what you are looking at.

Which documents are contracts at all?

What types of contracts?

What does the document call itself in its own language?

Who are the parties?

When did the agreement take effect?

Organizing contracts is not glamorous, but it is the dependency for everything downstream. Almost every contract intelligence workflow starts here, which is why this is the first benchmark I ran.

The scope was deliberately narrow: extract five metadata fields from each document and return one CSV row per document.

The five fields:

• Document title — as stated in the heading or preamble

• Party names — full legal names as stated in or around the preamble

• Effective date — stated effective date, falling back to other best date available in the preamble

• Preamble — verbatim introductory opening paragraph usually containing the party names and date

• Recitals — verbatim background or procedural history

Nothing downstream. No clause extraction. No obligation tracking. No risk analysis. No product build.

That narrowness is the point. These are the fields many downstream workflows assume you already have, and they are mostly verifiable against the document itself. They also tend to appear near the beginning of the contract, which creates a more controlled extraction task than clause-level fields that may appear anywhere.

This benchmark is not trying to prove that AI can read contracts.

It is trying to test what happens when basic contract reading becomes part of a population-level workflow.

The documents

The sample consisted of 100 material contracts sourced from SEC EDGAR filings.

These are real, executed agreements, not synthetic examples. They are not a perfect analog for enterprise contract populations. They skew toward negotiated, higher-value agreements and generally arrive in cleaner condition than the contracts sitting in a shared drive, inbox archive, or legacy CLM export.

But they are useful for a first benchmark because they are:

• real agreements,

• publicly available at scale,

• structurally similar to enterprise contracts,

• and suitable for manual ground-truth annotation.

For this first run, I used clean text-layer PDFs only.

That means this is not yet a test of scanned contracts, bad OCR, image-only files, or corrupted text layers. Those conditions matter — a lot — but I wanted to isolate model choice first before adding document-quality variation.

The models and prompt

I tested three models against the same 100 documents using the same prompt semantics: Claude’s Sonnet 4.6 and Opus 4.8, and OpenAI’s GPT-5.5. I used Claude Cowork and OpenAI’s Codex to run these.

Model choice was the only variable in this run.

The prompt was fixed across all runs. It was approximately one page of context, written to strike a practical balance: specific enough to define the extraction task and output format, but not so optimized that it became a bespoke engineering exercise.

In other words, this was not a zero-shot “please extract metadata” prompt. But it was also not a heavily tuned production workflow.

It was the kind of first-version prompt a knowledgeable contracts professional could plausibly draft in an hour with a clear understanding of the desired output.

I’m not publishing the full prompt or workflow notes here because the point of this piece is the benchmark result, not a replication guide. But the methodology matters, so I’ve described the corpus, fields, scoring approach, and limitations clearly enough for the findings to be contextualized and evaluated.

How I scored the results

I scored the outputs across three dimensions.

First: field return rate. Did the model return a value for the field at all? I tracked this separately from accuracy because a blank output and a wrong output create different risks. In this run, it was less of a differentiator between models.

Second: accuracy. For free text fields — title, parties, preamble, recitals — I assessed substantive match against the manually annotated ground truth. For effective date – I used exact match after date conversion to YYYY-MM-DD.

Third: time and usage. I tracked wall-clock runtime and session usage consumed per model, per run.

A note on ground truth: contract metadata is often less objective than it looks.

Take document title. Sometimes the cover page says one thing and the opening paragraph says another. Sometimes the heading includes “Executed Version,” “Confidential,” “Exhibit 10.3,” or party names that are not really part of the title. Sometimes the preamble characterizes the document differently from the header.

Effective dates are similar. Across this sample, effective dates appeared in several forms: stated explicitly in the preamble, defined by reference elsewhere in the document, implied by an “as of” convention, or left to the signature block.

So the benchmark uses documented annotation rules applied consistently. Ambiguous cases are flagged rather than forced.

The goal is not a perfect golden set. For contract populations, the term “golden set” often oversells the certainty available. The goal is a reasonably standardized reference set — and the difficulty of building even that is part of the finding.

The headline result

The headline result is not simply that one model performed better than another.

The more useful findings are that 1-the errors were patterned and 2-there are trade-offs between speed, usage and accuracy.

The models did not fail randomly. They struggled on the same kinds of contract complexity:

• ambiguous titles,

• party-name exactness,

• effective dates defined by reference,

• and multi-party agreements.

That matters because it changes how to interpret the benchmark.

If errors were random, the answer would be “use the most accurate model.” But if errors cluster around specific document structures, then knowing your contract population becomes key prior to, or in parallel with model selection.

A better model helps. But a better understanding of the document population may help more.

Speed and usage

In addition, the three models diverged sharply on both accuracy as well as speed and session usage.

A few caveats.

These runs were performed using paid consumer tiers (Anthropic Pro, ChatGPT Plus), not enterprise licenses. That matters because most company use cases would likely require enterprise controls for data security, confidentiality, admin oversight, and related needs. Usage limits and economics may differ materially in those environments.

The GPT-5.5 usage figure also deserves a specific note. Current GPT-5.5 pricing appears unusually favorable and may be subsidized as part of a competitive user-acquisition strategy. That does not mean the figure is meaningless, but it should not be treated as a durable long-term cost benchmark. API usage at scale may tell a different story.

Still, the operational difference was real.

A single Claude Opus 4.8 run on 100 documents consumed roughly 71% of a paid consumer-tier session. That is manageable for one experiment. It is far less likely to be a comfortable foundation for routine extraction work at scale.

At 1,000 documents, consumer-tier session limits would likely become a workflow design constraint well before the task itself is complete.

Accuracy by field

Claude Opus 4.8 performed best in terms of accuracy both overall and for each specific field. GPT-5.5 and Claude Sonnet 4.6 performed similarly overall but varied significantly in terms of field-level performance.

n = 100 documents. Effective date scored on normalized exact match; free-text fields scored on substantive match.

The accuracy gap between models was real, but the spread was not the whole story. The more important question is where each model was wrong, and whether those errors would matter for the workflow.

For example, if one model is five percentage points more accurate but consumes nearly three times the session capacity, that may or may not be worth it. The answer depends on the use case, QC tolerance, document volume, and whether the incremental errors are concentrated in fields that actually matter.

That is why this benchmark is less useful as a leaderboard than as a workflow diagnostic.

As another example, for party names: more lenient criteria (including partial matches – such as where the model found one but not all of the parties) yielded the following results, with substantial improvements for both GPT-5.5 and Sonnet 4.6:

Where the models failed

The failure modes were not random.

Here are some common ones:

1. Document title ambiguity

Title was more ambiguous than it initially appears.

Some documents had a clean title in the heading. Others had a heading that included filing labels, stamps, party names, or version indicators. In some cases, the heading and preamble characterized the document differently.

For example: “FOURTH AMENDMENT TO COMMON SECURITY AND ACCOUNT AGREEMENT” (doc header) vs. “Fourth Amendment” (preamble).

This produced two kinds of issues.

First, models sometimes selected different title sources: header versus preamble. Second, even within the same model run, title-selection behavior was not always consistent.

That inconsistency matters more than a one-off wrong answer. A single bad title can be corrected. An inconsistent title-selection rule creates downstream cleanup work.

2. Party-name exactness

There were some significant party name mistakes that included missing relevant entity names, particularly in multi-party agreements.

However, more often models returned party names that were substantively correct but not exact.

Examples include:

• Including extraneous text in addition to a relevant party name,

• omitting legal suffixes,

• omitting punctuation

In normal reading, these second class of errors are not serious mistakes. A person understands the answer.

But in a structured data workflow, small variations matter. Party names often become merge keys, deduplication inputs, entity-resolution candidates, or search filters. “Close enough” may be fine for a summary but problematic for a migration or analytics workflow.

This is one of the recurring lessons of contract data work: an answer can be substantively right and still operationally messy.

3. Effective-date variation

Effective dates were one of the hardest structured fields.

The reason is not that models cannot read dates. The reason is that contracts state effective dates in various ways.

Some are explicit:

“This Agreement is effective as of January 1, 2024.”

Others are indirect:

“This Agreement is entered into as of the date first written above.”

Others define the date elsewhere, refer to a closing, rely on signature timing, or contain several dates that look plausible but serve different functions.

When models relied too heavily on a standard preamble pattern, they missed dates defined elsewhere. When models searched more broadly, they sometimes surfaced the wrong date: a prior agreement date, notice date, recital date, or signature date.

This is exactly the kind of field where a model can look confident while being wrong.

What did not fail

It is worth saying what worked.

On clean text-layer PDFs with standard preamble structure, all three models were generally consistent. They identified ordinary titles, ordinary party names, and explicitly stated effective dates reasonably well.

That is important. This is not an argument that model-based extraction does not work.

It does work.

The question is where it stops working cleanly, how visibly it fails, and how much workflow you need around it before the output can be trusted.

The failures above are patterned, which means they are addressable. Better prompts can help. Post-processing rules can help. Entity normalization can help. QC sampling can help. A human-in-the-loop review step can help.

But none of that is free.

The model is only one part of the workflow.

The silent failure problem

The most important operational finding was not that the models made mistakes.

It was that the wrong answers looked exactly like the right answers.

Across all three models, every field in every document received a value. No uncertainty flags. No “I could not determine this.” No visible indication in the CSV that a given value required review.

That is convenient until it is dangerous.

At 100 documents, a human QC pass can catch many of these issues.

At 1,000 documents, that is no longer true unless sampling is built into the workflow explicitly. At 10,000 documents, even a small systematic error becomes a data-quality problem. At 100,000 documents, a 2% intervention rate means 2,000 human interventions.

The operational risk is not merely that the model gets something wrong.

It is that the model gets something wrong confidently, and confidence is the only signal available downstream.

What this means

The first lesson: model choice mattered, but document structure mattered more.

Across all three models, the lowest-accuracy documents were not random. They were the documents missing preambles, where effective dates were not clearly defined, where there were more than two named parties, or title/preamble mismatches.

That means knowing your contract population is key prior to model selection. If your population has a high concentration of those structures, prompt design, field definitions, and QC process may matter more than switching models.

The second lesson: speed and usage tell a different story than accuracy.

Claude Opus 4.8 was directionally strongest on accuracy, but it was also the slowest and consumed the most session capacity. Claude Sonnet 4.6 was materially faster and used less session capacity. GPT-5.5 was fastest and cheapest by apparent session usage, but with pricing and durability caveats.

That creates a practical trade-off.

If Claude Opus 4.8’s overall accuracy advantage is 22 percentage points, the decision is not simply “use Claude Opus 4.8.” The decision is whether those extra correct answers are worth the added time, usage constraint, and workflow friction for the specific task.

The third lesson: consumer-tier tools can benchmark the workflow, but they are not the workflow at scale.

At 100 documents, they are usable. At 1,000, session limits and manual handling become serious constraints. At higher volumes, API access, retry logic, QC sampling, structured storage, and review workflows become the relevant comparison.

The fourth lesson: silent failures require explicit QC.

Do not treat a complete CSV as a correct CSV.

A model returning values for every field is not the same thing as a validated extraction workflow. If the output will be used for diligence, migration, analytics, obligation tracking, or system-of-record updates, some QC process has to be designed into the workflow.

Takeaways

These thresholds are directional, based on this first run. I will refine them as the benchmark develops.

10 documents or fewer

Use the chat interface.

At this volume, setup overhead matters more than workflow sophistication. Drag-and-drop, ask for the fields, copy the output into a spreadsheet, and manually check the results.

Model choice is probably less important than clear instructions and human review.

Around 100 documents

The chat interface still works, but the constraints become visible.

You need a repeatable prompt, a consistent output schema, a place to store the results, and a QC pass before using the output downstream. Watch especially for ambiguous titles, effective dates, and multi-party agreements.

This is the tier where many ad hoc legal ops and in-house requests live. It is also where teams can fool themselves into thinking they have a process when they really have a clever manual workaround.

100 to 1,000+ documents

The workflow around the model starts to matter more than the model.

You need consistent prompting, structured output, QC sampling, failure logging, and a storage layer that is not a manually assembled spreadsheet. Consumer-tier session limits become impractical. API access becomes the relevant comparison.

The per-document model cost may be low. The human time cost of managing an ad hoc process at this scale is likely much higher.

10,000+ documents

The question changes.

This is no longer “which model should I use?” It becomes “what operating model do we need?”

Build, buy, outsource, or run a formal internal data operation — but do not mistake model access for workflow readiness. At this scale, reliability, review design, exception handling, and data governance become the work.

The useful distinction

The demo version of contract extraction asks:

Can the model read this contract?

The operational version asks:

Can the workflow produce structured data across this population, with known accuracy, acceptable review burden, manageable failure modes, and outputs that downstream users can trust?

Those are different questions.

The first is increasingly easy to answer.

The second is where most of the real work still lives.

What comes next

This is a first data point, not a conclusion.

I will publish additional cuts as the benchmark produces useful findings: document-quality conditions, additional models, larger scale tiers, and the operational differences between chat-based and API-based workflows.

If you found this useful, the best signal you can send is forwarding it to someone who is about to make a decision about contract extraction tooling based on a demo.

Subscribe now

Last year brought more of the same: LexisNexis announced Protégé™ General AI, Thomson Reuters announced agentic capabilities in CoCounsel Legal, OpenAI published a detailed account of building an internal contract data agent.

That’s not a coincidence.

The market is converging around the contracts space using ever-improving AI capabilities — from creating “contract intelligence” to building end-to-end “workflows.”

But there’s a lot missing in the conversation around these announcements. So let’s unpack what it means and why it matters.

Subscribe now

To start — contract intelligence

I’ve spent about a decade working on this problem.

I started in 2015, inside a small contracts business at Axiom that eventually became Knowable, a LexisNexis company. Back then, the deliverable was a spreadsheet. The engine was people: project managers and analysts running a quality-controlled review process on enterprise contracts, turning dense legal language into structured data someone could actually use.

There was no software to buy. The concept we now call “contract intelligence” predated the term — and predated most of the tools now claiming it.

That is why the current wave is so interesting. The concept is old. The technology is genuinely new and continues to evolve. And the gap between those two things is where the market is hardest to understand and is where the useful analysis lives.

Same space, different layers

Here’s what’s worth noticing: these announcements are solving for different things in different ways, but they’re all filed under similar labels.

Harvey’s Contract Intelligence appears to be a portfolio-level playbook management system — surfacing negotiation patterns, fallback positions, and clause language from executed agreements to make future review faster. The promise is that every signed contract feeds back into and updates the playbook automatically. That’s a workflow-compounding play aimed at in-house teams doing high-volume, BAU contract reviews.

Claude for Legal is a horizontal capability layer: practice-area plugins, first-pass review and redlining through Cowork, and MCP integrations that connect Claude to other systems in your stack. It’s positioned as the intelligence underneath other tools, not a standalone contracts product.

OpenAI’s contract data agent is an internal build for their own finance team — ingesting PDFs and scans, extracting structured data with retrieval-augmented prompting, and serving it up for human-in-the-loop review. The goal is to meet the scale of a growing business with limited headcount.

DocuSign’s agents operate inside its Intelligent Agreement Management platform: checking agreements against company standards, flagging risks, tracking obligations, and allowing teams to build custom agents for deal and renewal workflows through Agent Studio. The use cases overlap with what others serve, but the emphasis is on a platform-ecosystem anchored in their e-signature capabilities.

These are not competitors in the simple way a feature comparison might suggest.

They’re addressing different layers of a workflow that most announcements don’t bother to decompose.

And if you’re an end user — a contracts professional, in-house counsel, legal ops leader, or someone running a deal process — the useful question is not simply “which one is best?”

It is:

Which layer of my actual problem does each one touch, what’s still missing, and what trade-offs come with choosing one approach over another?

The part the announcements skip

That question is harder to answer than it sounds because the announcements are built as much to impress as to inform.

They rarely locate themselves precisely in the workflow. They rarely explain what must already be true for the product to work well. And they rarely dwell on the gap between a staged demo and a system that survives real documents, real users, and real organizational complexity.

Every one of these announcements assumes a set of prerequisites that practitioners know are the actual hard part:

• That the contracts have been collected.

• That you know which contracts are in scope.

• That the entity names in your CRM match the party names in the agreements.

• That users have access to the right documents.

• That someone has decided what data points matter.

• That the output can be reviewed, trusted and used in the company’s legal and business processes.

Very little of that is solved by the announcement itself.

All of it is where most of the time goes.

The decade I spent building in this space taught me one pattern above everything else: each wave of technology solves the layer everyone is staring at, and the value quietly relocates to the layer nobody was watching.

Services gave way to software. Traditional machine learning improved extraction. Generative models changed what could be attempted. Each time, the extraction layer got better — genuinely, materially better.

And each time, the constraint moved somewhere upstream or downstream:

• Which contracts matter?

• Where are they?

• Who has access?

• What should we extract?

• How do we evaluate quality?

• How do we surface the answer clearly?

• How does the output change the way the business actually makes decisions?

That pattern is playing out again now, across every one of these announcements.

It is the thing most worth tracking.

Why I’m writing this

I come at this from a decade of working across ML and LLM product development, contract analysis, solution sales, and end-user research, with cross-functional visibility into how enterprise contracts work actually gets delivered.

I have no vendor to sell and no employer to promote.

The goal of The Contract Signal is to give end users — the people who actually work with contracts — an independent, practitioner-level read on what the capabilities are, what they are not, what is coming, and how to use them to solve real problems.

The next pieces will get hands-on: testing tools against real contract workflows, looking at what they can actually do, where they break, and what they quietly assume you have already solved.

If you work with contracts, buy or evaluate AI tools for legal workflows, or build in this space and see something I’m missing, I’d like to hear from you.

Subscribe to follow along.

Subscribe now

I sat in front of a subscription tool that aggregated announced mergers, acquisitions, financings, and the rest. You could sort by deal size, by type, by buyer. You could set alerts. And the strategy my team ran on top of it was simple enough to explain in one sentence: a big deal gets announced, big deals mean lots of contracts to review, lots of contracts mean someone needs help, so we call them and win the work.

It sounds reasonable.

It’s also wrong in two structural ways that took me years to fully appreciate. If you’re building an origination motion for contract-review work today — whether you’re a vendor, a services firm, or a legal-ops team trying to staff ahead of demand — those two failures are worth as much as any feature comparison.

Subscribe now

Failure one: by the time it’s news, the work is gone

The first thing that breaks is timing.

A deal that’s been announced is a deal that’s already been planned. The diligence scope is set. The advisors are picked. Often the contract-review approach — internal team, outside counsel, a vendor, some combination — was decided weeks or months before the press release went out. You are reading about a decision, not catching one in flight.

So the announcement, which feels like the start of an opportunity, is usually the end of one. The companies that show up in your alert feed are disproportionately the ones you can no longer help on this deal. The genuinely addressable work was upstream, in the quiet period when nobody outside the principals knew it was happening.

This is the part that impacts everything else. If announced deals are mostly too late, then reactive monitoring isn’t an origination strategy — it’s a lagging indicator dressed up as a lead source. The work that actually converts comes from being known before the deal exists: proactive relationships, so the company comes to you when the deal is still a secret. You can complement this — but not replace — with news monitoring.

Failure two: volume doesn’t mean need

The second failure is subtler and, I think, the more useful one, because it survives even if you fix the timing problem.

The whole premise — more deal volume means more contract work — assumes a correlation that just isn’t reliable. Deals are structured, and the structure determines whether contracts matter at all. A few examples that look similar in a volume column and behave nothing alike:

The acqui-hire. A handful of people command an enormous package — you can get to several hundred million, even half a billion, on a few names in the current AI talent market. It will light up any deal-size screen you build. The contract review attached to it is close to zero. Nobody’s reviewing a customer book; there barely is one. Huge number, no work.

The asset purchase. A buyer takes a discrete set of assets — say, a fleet of trucks. The assets are owned outright; there’s not much contractual tail to diligence on the buy side, and any related agreements may be getting retired or terminated as part of the structure anyway. So buy-side need is low. But flip it: the seller may have real work understanding how hard those contracts are to exit. The same deal generates demand on one side and almost none on the other, and a volume metric can’t see the difference.

Business-as-usual masquerading as nothing. Meanwhile, the company quietly enters into fifty thousand new agreements a year in the ordinary course of business. That’s a completely separate pool of potential work from anything M&A-driven — larger and steadier — and it never shows up in a deal feed at all, because it never gets announced.

Put those together and the lesson is uncomfortable for anyone who loves a clean dashboard: high deal volume is weakly correlated with contract-review demand, and the cases where it’s most wrong are often the flashy ones that dominate the news.

What actually predicts demand

None of this means prediction is hopeless. It means the naive screen is the floor, not the ceiling.

The floor — and I mean “naive” as a term of art, not an insult — is heuristic. Big, diversified companies with the cash flow or borrowing capacity to keep doing deals are likely serial transactors; their historical deal cadence tells you something. Working with physical goods helps: a business with a manufacturing or supply-chain footprint tends to carry more contractual surface area than a lighter software-first business, though large enough software companies blow that rule up with sheer customer and vendor count. Fortune 500, deal history, industry density — that’s a defensible level-one list, and it’s cheap to build.

The ceiling is where you stop treating those as the answer and start treating them as inputs. Layer in the things a screen can’t see: your relationship history, your own pipeline, deal-structure patterns, the buy-side/sell-side asymmetry above. Feed that into a model and you’re no longer asking “who’s big?” — you’re asking “who’s likely to transact in a way that actually generates contract work, and where are we well-positioned to win it?” That’s a prediction worth making.

One honest caveat, because it’s the thing most product pitches skip: the highest-value version of this is driven by your internal context — your relationships, your data, your read on structure. An outside-in tool can get you the level one list. It can’t replicate the part that makes the prediction good, which is the proprietary context only the firm doing the work has. Proceed accordingly and be wary of anyone selling you the ceiling as if it came in a box.

Subscribe now

If you’re working on origination or capacity planning around contract work and any of this maps to (or contradicts) what you’re seeing, I’d love to discuss.

The context mattered. We were a nascent contracts data business operating inside a larger alternative legal services firm. Our business — turning contracts into structured data for M&A due diligence and integration — was maybe ten percent of the company’s revenue. The other ninety percent ran a different model entirely: placing attorneys with in-house legal departments on a consulting basis. That larger business was our best source of referrals. When one of their senior account executives recognized that a client’s problem was better suited to what we did, they’d send it our way — and they’d still get credit toward their goals, so incentives were aligned.

The head of our business wanted to double down on that referral engine. His solution was internal marketing — get the other teams excited about what we were building, make them feel connected to it, make them more likely to think of us when the right client situation came up. He tasked me, roughly a year and a half into the job, with making that happen through a newsletter compiling the latest cross-functional updates.

I’ve been thinking about that assignment lately because it’s a useful test case for something that comes up constantly: where do agents actually fit into knowledge work, and where does the person still have to drive?

What the Assignment Actually Involved

Before getting into the AI question, it’s worth being specific about what the work required, step by step.

The first thing I had to do was understand the business problem behind the assignment. The newsletter wasn’t the goal — increasing sales revenue was, through increased origination from the referral channel. That distinction mattered because it shaped every subsequent decision about what to include and how to frame it.

The second thing was figuring out what to compile. I started with our CRM to pull sales history, stakeholder context, deal status, and anything useful I could surface without taking up anyone’s time. Then I reached out across sales, product, marketing, and client service for anything the CRM didn’t capture — their read on what was worth highlighting, recent wins, delivery milestones, collateral. This produced a pile of good raw material: deals closed, new clients landed, product updates shipped, branding work completed.

The third thing — and this is where most of the judgment lived — was deciding what actually mattered and why, and how to communicate that to our audience.

Winning a deal with a new enterprise client was significant not just for its revenue, but for what it signaled about our ability to win that class of business and what it meant for the senior person on the other side of the house who originated it. A major renewal and upsell was worth highlighting not just as a financial outcome but as a recognition of the delivery team whose work made it possible. The connections between how collaboration across different parts of the business enabled our sales motions — and giving specific credit to the people involved — that’s what made the newsletter land, not the facts themselves.

Finally, the fourth thing was packaging it into a format that salespeople, executives, and other business stakeholders would actually read, let alone enjoy and be inspired by.

Running It Back With Agents

The interesting question isn’t whether an agent could do this today. Parts of it clearly could be addressed. The interesting question is which parts, and what that leaves for the person.

Compilation is largely addressable. The mechanical work of gathering updates across departments — if that information lives in a shared system with appropriate access — is exactly the kind of multi-step retrieval task agents handle well. Connect to the CRM, pull recent updates by department, surface them in a structured format. That’s real and useful. In my case I was doing much of this manually through CRM data research, email and conversations. Some of that friction goes away.

That said, the compilation step has its own constraints worth naming. The relevant information isn’t always well-structured or correctly tagged. Ambiguity in underlying data produces ambiguous outputs. And there are legitimate access and privacy boundaries — not every relevant conversation is logged, not every system is connected, and not every organization should want them to be. These aren’t temporary engineering problems. They’re features of how organizations actually function.

Summarization is addressable, and honestly it was already addressable before agents. Given a set of inputs, an LLM produces coherent summaries. This isn’t a new capability agents unlock — it’s baseline LLM behavior. Useful to have in the workflow, not a step-change.

Formatting and packaging is partially addressable. A first draft of the newsletter structure, given clear inputs and a well-defined template, is something an LLM handles comfortably. The blank page problem gets easier. The judgment in the final version is still a human job.

What doesn’t move, at least not yet: the judgment calls at the beginning and middle of the process.

Understanding why the assignment existed — that the real goal was referral activation, not newsletter production — required knowing the business, the person making the ask, and the organizational dynamics well enough to read between the lines. That context wasn’t written down anywhere. It came from being embedded in the organization.

Deciding what actually mattered in a pile of raw updates required knowing which outcomes were genuinely significant versus routine, which relationships between teams were worth reinforcing, and what would actually resonate with a specific audience of attorneys and senior salespeople being asked to care about something adjacent to their primary work. An agent surfacing everything equally would have produced a longer document, not a better one.

A More Recent Illustration

There’s a more current version of the same dynamic that I’ve been thinking about while using Claude Code for my own projects.

When Claude Code runs, it periodically pauses and asks whether you want to auto-accept its next actions or review them manually. It’s a small UX detail, but it captures something important: the model itself doesn’t know when human review is warranted. It can execute well across a wide range of tasks. What it can’t do is determine, in your specific context with your specific constraints, which outputs are safe to accept without review and which ones need your eyes on them first.

That decision — when to trust the output and when to verify — is a judgment call. And it’s not incidental that the tool surfaces it explicitly as a user choice rather than resolving it internally. The people who built it understand that this is a structurally human decision, not a capability gap to be closed in the next model release.

The newsletter assignment had the same structure. An agent could have helped me compile, draft, and format. What it couldn’t have done is tell me which deal was worth leading with, or how to frame a delivery team’s contribution in a way that would actually motivate a sales floor to send us more referrals. Those calls required context that lived in relationships and organizational dynamics that no system was tracking.

What This Suggests Practically

The newsletter assignment was never going to be fully automated, and that’s not a failure condition. The right way to think about it is as a workflow with a natural division of labor: the person drives the judgment-heavy front end, the tools handle the execution-heavy middle, and the person reviews and refines the back end.

In practice that looks something like this. You identify the business problem clearly enough that it can guide downstream decisions — human work. You use the tools to pull and structure relevant inputs from the systems that have them — agent territory. You evaluate what came back against the judgment you brought in at the start — human again. You use the tools to draft and format — mixed, depending on how clearly you’ve defined what good looks like. You review and refine with your audience in mind — human, and not optional.

The thing I’d push back on is the framing that treats this division as temporary — that eventually the agent handles the judgment calls too as models improve. Maybe. But the judgment calls in the newsletter assignment depended on context that is structurally difficult to provide to any system, not just ones with current capability limits. The organizational dynamics, relationship histories, and political considerations that informed my decisions weren’t in any system. Getting them there fully would require a level of organizational surveillance that most environments wouldn’t and shouldn’t accept.

Until that changes, the person isn’t a bottleneck to be engineered around. They’re the thing that makes the output worth producing in the first place.

Intro

There’s been a lot of noise lately about AI agents — most recently OpenClaw, before that OpenAI’s agents, Anthropic’s, Google’s, and the growing ecosystem of tools that let you automate workflows, string tasks together, and run complex operations with minimal intervention. The pitch is compelling: more getting done, less effort spent doing it.

I’ve found most of that pitch to be true in a narrow but important sense. These tools genuinely do help you get more done. I use them regularly for exactly that purpose.

But I keep noticing a gap between the claim and what’s actually happening — between getting more done and creating value. They’re not the same thing. The confusion between them, I think, is one of the more consequential misunderstandings in how these tools are being talked about right now.

Accompanying audio note:

The Productivity Claim

When people talk about AI productivity gains, they’re usually describing something real: tasks that previously took an hour now take ten minutes. Analyses that required manual effort can be automated. Research that involved dozens of searches can be compressed into a few prompts.

This is genuinely useful. I don’t want to dismiss it. But productivity, strictly speaking, just means output per unit of input. Getting more things done in less time. And productivity is only valuable to the degree that the things being done are worth doing.

That sounds obvious…but watch what happens in practice. Someone deploys an agent that automates a multi-step process. They count the hours saved. They extrapolate to the team, multiply by average hourly rate, and arrive at a dollar figure. The number sounds large. The announcement goes out: $X million in value created.

In most cases, this conflates two very different things: the cost of the activity and the value of the outcome.

Getting more done faster is only as valuable as what’s getting done. If an AI agent helps you accelerate a process that was already misaligned with what your customers needed, you’ve become more efficient at something that shouldn’t have existed in the first place. That’s not value creation — it’s waste, just at a higher throughput.

The More Interesting Question

None of this means productivity gains are worthless. It means the question worth asking is: what do you do with the time you get back?

There are two very different answers. One is that the freed capacity gets absorbed by more tasks of the same kind — a never-ending queue of lower-priority work that expands to fill available bandwidth. More things get done. The ratio of important things to unimportant things stays roughly the same. The volume increases.

The other answer is that the freed capacity gets redirected toward genuinely higher-leverage work — the things that actually move outcomes, that create something worth having, that compound over time. If that happens, then the productivity gain is real in the deeper sense: it bought you time to work on the things that matter.

Which answer you get depends almost entirely on whether you have the judgment and the agency to make that call. Judgment to identify what the high-leverage work actually is. Agency to actually redirect toward it — which, in an organizational setting, is far from guaranteed.

In a previous article I wrote about the difference between where you build in the stack and where your time actually compounds. The same logic applies here. Productivity tools don’t answer the allocation question. They just widen the gap between people who are asking it well and people who aren’t.

A Third Category

There’s one more thing I’ve been sitting with, and it’s harder to articulate cleanly.

Most discussions of AI productivity focus on one of two categories: either you were doing the task before and now do it faster, or you weren’t doing it and now you can. What I’ve started to notice is a third category — things you technically could have done before, but that were just annoying and friction-laden enough that you didn’t, in practice, ever do them.

A specific example: I’ve had FSA accounts for years, with funds that expire if unused — the classic use-it-or-lose-it situation. Submitting claims, especially for borderline or unfamiliar expense types, has always involved navigating opaque requirements, decoding denial reasons, and figuring out exactly what documentation was needed. Annoying enough that I’d let some money expire rather than waste time fighting the process.

Earlier this year I used ChatGPT to work through a claim denial. Not just as a search engine — as a conversational partner. I described the situation, asked about common denial reasons for that category, worked through what documentation would address each one, and drafted a resubmission. The claim was approved. Several hundred dollars I would have left on the table.

The productivity framing undersells what happened there. I didn’t do that task faster — I did a task I wasn’t going to do at all. The LLM didn’t just reduce time cost. It reduced friction below the threshold where I’d actually act.

There’s probably a psychological dimension to this as well. Something about the conversational format — the ability to ask follow-ups, to get a response that acknowledges your specific situation rather than returning generic search results — lowers the activation energy in a way that a list of web links doesn’t. I’m not sure whether that’s because the medium maps to how people have evolved to interact with and process information, their mental model or something else. But the effect is real.

This matters because it suggests a category of value that doesn’t show up in time-savings calculations. Not hours recaptured. Not existing tasks accelerated. But things that now happen that otherwise wouldn’t — claims filed, questions answered, decisions made with better information. That’s not productivity. It’s enablement.

What This Means in Practice

The honest version of the AI agent story is more nuanced than the headlines suggest. These tools genuinely do help you get more done. In the right conditions — when there’s judgment about what’s worth doing and agency to act on it — that translates to real value. When those conditions are absent, you get faster hamster wheels.

For anyone building with or using these tools, I think the most important questions aren’t about the tools themselves. They’re about the human side of the equation: What are you actually trying to accomplish? Where does the allocation of your time currently mismatch what you’d choose if you thought clearly about it? And what category of things — the ones you’re not doing because the friction cost is too high — might be worth reconsidering?

Getting more done is the easy part. These tools are genuinely good at it. The harder part — which the tools don’t touch — is knowing what to do with the time.

This is Article 7 in a series on building with and using AI tools as a non-native technologist. Previous articles covered financial planning with AI tools, a framework for city selection, evaluating LLM tools, building the Seattle Nature Access Map, why judgment was always the bottleneck, and where in the stack to build.

This was back before LLM tools became mainstream.

The queries weren’t sophisticated, but they handled my use cases well. They were also easy to understand and troubleshoot if something didn’t seem to be working.

I learned to write these – passably. Enough to pull data, join tables, answer a question without waiting for an engineer.

Once I reached a good enough level - I deliberately never got much better at it than that.

Every hour I could have spent becoming a stronger SQL writer was an hour I wasn’t doing the thing I was actually better at: figuring out which query mattered, what the data meant, and what it implied for the product. Where the models seemed to be doing well, where they needed to improve, and where we had no idea.

I could always find someone to write better queries. I couldn’t always find someone who knew what to ask.

That wasn’t laziness. It was a judgment about where my time compounded.

I’ve been thinking about that decision a lot since starting to build with AI tools in earnest — first the financial planning work I wrote about in Article 1, then the nature access map in Article 4. The more I build, the more I notice that the most consequential decisions aren’t about how to build something. They’re about where in the process your leverage actually lives.

The Stack Has Layers — And They’re Not Equal

When people talk about “building with AI,” they’re usually collapsing several genuinely different things into one.

At the foundation, there’s training foundational models — the work Anthropic, OpenAI, and Google do. Enormous compute, proprietary data, specialized research teams. Almost no one reading this should be operating here.

One level up: fine-tuning and domain adaptation. Taking an existing model and adjusting it for a specific task or dataset. Still technical, but accessible to a much wider group — though the relevant question increasingly is whether the base model already does what you need, because often it does.

Above that: building AI-powered applications. Connecting LLMs to data, interfaces, and logic to make something useful. This is where most of the interesting work is happening right now, and where the barrier to entry has dropped most dramatically.

And above that: using AI-powered tools to create things that aren’t primarily engineering at all — writing, analysis, structured research, products where the judgment and domain knowledge are the core value, and the code is just the vehicle.

These are genuinely different layers. Which one you should work at isn’t obvious, and “I can build this now” doesn’t answer it.

The Right Level of Abstraction Is the Same Question

In a previous article, I wrote about how judgment was always the bottleneck — that vibe coding made it visible, not new. This piece is about a related but distinct question: given that you have judgment, and given that you can now build things you couldn’t before, where should you be building?

This is a question I’ve been trying to answer for myself.

There’s a useful frame from computer science education that I found clarifying here. When I first learned to code, I started with a site that abstracted away almost everything — you typed commands, things ran, but you had no idea why. Then I took a more rigorous introductory class that went deeper. Then, eventually, a course that started just above the transistor level — actual hardware, zeros and ones, how you get from electricity to software. Each level of abstraction revealed something the previous one had hidden.

The lesson wasn’t “always go to the lowest level.” Going down to the physics of transistors was helpful for understanding how computers work. It wasn’t necessary for building a web application. The insight was that the right level of abstraction depends on what you’re trying to do — and choosing wrong in either direction creates problems.

Too high: you can’t diagnose what breaks. This is what happened when I hit the rendering bug during the nature access map build. Claude kept proposing fixes that introduced new problems, and we went in circles until I suggested using browser dev tools to surface actual error information. Abstracting away all understanding of the underlying system meant I was stuck the moment something went wrong.

Too low: you’re spending your time on infrastructure that isn’t your actual problem. If I’d tried to build a more capable machine learning pipeline from scratch for the nature access scoring, instead of using public data and simpler distance calculations, I’d have spent months on something that wasn’t what the app was focused on.

Choosing where in the stack to build is the same decision as choosing the right level of abstraction. The mistake is treating it as fixed.

The Chef Doesn’t Make the Spatula

Here’s the underlying principle: being capable of doing something and that being a good use of your time are different things.

A skilled chef could, in principle, forge their own knives, mill their own flour, build their own oven. They might have enough mechanical intuition to figure it out. But that’s not where their differentiation lives, and spending time there means spending less time on the cooking — which is where they’re actually hard to replace.

The question for anyone building now is the same one I was asking about SQL years ago: where does my time compound? Where is the gap between what I can do and what most people can do actually widest?

During the ten years I’ve spent in legaltech, the honest answer was never “write better ML code.” It was understanding what types of contract data matter to attorneys, salespeople, and finance teams — and how to structure and surface it in a way that was most useful to them.

That meant understanding what clause types mattered to an M&A attorney under what conditions, how to translate a 91% accuracy rate into something meaningful for a lawyer managing risk, which edge cases would surface in exactly the situations that mattered most. Those were judgment calls that required domain knowledge I’d spent years building. Better tooling wouldn’t have made them easier — it would have made the engineering around them cheaper, which is a different thing.

That calculus hasn’t changed. What’s changed is that the floor of “good enough building” is now much higher, which means more people can reach it — and that makes the question of where you’re actually differentiated more urgent to answer explicitly. Before LLMs, if you couldn’t write the code, the question was moot. Now it isn’t, and that’s mostly a good thing. It just means the “where should I build?” question is no longer optional.

So Where Does Your Time Compound?

Building toward something of my own has sharpened this question more than anything else. If I can build more easily than before, what should I actually build? At which layer?

The honest version of the question isn’t “can I do this?” It’s: where is my specific combination of domain knowledge, judgment, and taste producing something that isn’t easily replicated by someone working adjacent to me?

For some people, that points toward a lower layer — they have genuine ML depth, or systems architecture intuition, or infrastructure knowledge that’s still genuinely hard to replicate. For most of us building on top of these models, the answer points up: toward the quality of the problem being solved, the specificity of the insight, the understanding of the user, the design of the experience.

This doesn’t mean the engineering disappears. As I wrote in Judgment Was Always the Bottleneck, engineering judgment — knowing when a latency issue will create a UX crisis under load, knowing when a context window limitation will cap what’s possible, knowing when something technically works but creates maintenance debt that isn’t worth it — that’s still judgment, just expressed through technical decisions. The stack doesn’t have a clean separation between “engineering” and “everything else.”

What it means is that for most people who came to technology laterally — through law, finance, a domain specialty, a problem they needed to solve — the differentiation probably isn’t at the foundational layer. It’s in knowing what to build that’s actually worth building.

The spatula is cheaper now. The chef is still the chef.

This is Article 6 in a series on building with AI tools as a non-native technologist — covering what actually changes when someone without an engineering background starts building products. Previous articles covered financial planning with Claude Code, a framework for city selection, evaluating LLM tools, building the Seattle Nature Access Map, and why judgment was always the bottleneck.

Highlights:

• All laws are are actually principles applied under certain conditions or assumptions

• The strength of a principle is a function of how tightly you define the conditions under which it applies — narrow the scope, strengthen the law

• Complex systems are hard to predict because multiple counteracting principles operate simultaneously

• This is why macroeconomics gets predictions wrong repeatedly: too many counteracting principles operating simultaneously

• The same pattern shows up in machine learning: a model trained on all contracts performs worse than one scoped to a specific contract type

• And in UX: a design principle that worked for one user group often fails when applied to a different one without re-examining the underlying assumptions

There’s a popular narrative being told right now about vibe coding, and it goes roughly like this: coding used to be the bottleneck. Now it isn’t. The new bottleneck is judgment and taste. If you can think clearly about what to build, the tools will handle the rest.

It’s a compelling narrative. But it only tells half the story.

Not because judgment doesn’t matter — it does, enormously, and I’ll spend most of this article on why. But because the framing implies that judgment is new to the equation. That before vibe coding, the separating factor between great products and mediocre ones was mainly technical execution.

My re-frame is simpler: judgment was always the bottleneck. Vibe coding just made it visible.

What Actually Changed

Things did change, substantially. Claude Code and similar tools collapsed the barrier to entry for certain categories of development work in a way that prior no-code tools never quite managed. The prototype-to-production gap is narrower than it was. The floor of what a non-engineer can ship went up substantially.

But here’s the disconnect: we’re confusing a change in what we can see as a change in what was present.

Before widespread vibe coding, poor product judgment was largely contained inside institutions. It existed — and produced plenty of mediocre features and failed products — but it was wrapped inside teams at companies, hidden behind proprietary systems, filtered through organizational process. You didn’t see it at scale in the open.

Now you do. Thousands of people are shipping publicly, often free, often building in the open. So a much higher volume of things to look at means a much higher volume of poorly crafted things to look at. The ratio of strong products to weak ones was probably always similar. What changed is that you now have a massive lens on all of it, versus a microscope pointed mostly at what large companies chose to release.

There’s a marketing layer on top of this too. “I’m a builder” has become the new “I’m a disruptor” — a buzzword with similar dynamics to disruption a decade ago. When everyone’s shipping, people pay attention to what gets shipped. The scrutiny goes up alongside the volume. Both effects make judgment failures more visible without changing the underlying rate.

The story isn’t “judgment matters now.” It’s “judgment failures are harder to hide now.”

Judgment Isn’t New — It Was Always Load-Bearing

To understand why, it helps to break down what building something actually requires. At the highest level, three things: knowing what to build, knowing how to build it, and getting it to market.

The first category — what to build — means understanding the problem you’re solving, for whom, and conceptually how. When I built the nature access map, I was my own first user. I knew the problem in depth because I’d lived it: I wanted to understand, by neighborhood, which areas of Seattle had parks and trails within walking distance before committing to a move. That gave me a clear problem, a clear user, and a clear mental model of what a useful solution would look like.

The second category — how to build — means going from that concept to something tangible: mock-ups, prototypes, and eventually working code. This is where the LLM tools have changed things most dramatically. Historically, you couldn’t get to a working prototype at all without substantial technical fluency. Now, for many categories of work, you can describe what you want in plain English and get something functional back.

The third category — distribution and commercialization — means actually getting the product in front of users and, if you’re building something commercial, making money from it.

What changed is the second category. What didn’t change is the first and the third. And the first category — judgment about what to build, for whom, and why — was always the harder problem. It’s just that when the second category was blocking most people entirely, the first category’s difficulty was invisible. You can’t expose a judgment problem in something that never gets built.

I’ve spent a decade in legaltech, turning contracts into structured data for enterprise customers. The engineering was real — machine learning models, complex document processing pipelines, large-scale data extraction and classification, and increasingly GenAI-powered tools. But the decisions that actually determined whether the product worked were almost never purely engineering decisions. Was the model accuracy on this clause type sufficient for the risk appetite of an in-house counsel or M&A attorney? Which edge cases would appear infrequently enough that we could accept them, versus which would surface in exactly the situations that mattered most? How did you explain a 92% accuracy rate to a lawyer in a way that mapped to how they actually processed risk?

Those were judgment calls. They required domain knowledge, pattern recognition, and the ability to translate between technical realities and human contexts. No amount of better tooling would have made them easier. The people who made them well were valuable long before vibe coding existed.

The Complexity Matrix No One Is Talking About

Part of why the popular narrative feels limiting is that it collapses a genuinely complex set of distinctions into a single claim.

Whether vibe coding meaningfully changes the engineering bottleneck depends entirely on where you are across multiple dimensions at once:

Internal tool vs. external product. An internal tool built for yourself has radically different tolerances — for rough edges, for edge cases, for performance under load. When I built the nature access map, it ran locally, served one user, and if it broke I fixed it. That’s a completely different problem than a SaaS product with uptime requirements, paying customers and users who aren’t you.

Prototype vs. production. The gap between something that works in a demo and something that works under real conditions — concurrent users, unexpected inputs, evolving requirements — remains enormous, and LLMs haven’t closed it. Prior no-code tools produced a high ratio of prototypes to production-ready products for real reasons. There’s an honest open question about whether this generation finally crosses that threshold. I think the answer is “maybe, in limited contexts, but we don’t fully know yet.”

Simple vs. complex use case. Some design patterns have been solved thoroughly enough that you can implement them at a raised level of abstraction and they’ll work reliably. Others haven’t. New interfaces — AR, voice, novel interaction paradigms — haven’t accumulated the solved patterns that web and mobile have built up over decades. What worked on desktop didn’t always translate to mobile. What works now won’t automatically translate to whatever comes next. Every new interface introduces substantial new variables and trade-offs.

Regulated vs. unregulated. Financial services, legal, healthcare — these domains introduce constraints that compound complexity in ways that aren’t just about technical difficulty. The cost of certain failure modes is qualitatively different. The judgment required to navigate those trade-offs is domain-specific in ways that can’t be substituted by general technical fluency.

Scale. A few users is categorically different from a few thousand or a few million. Engineering judgments that produce acceptable outcomes at small scale often produce catastrophic ones at large scale. Latency barely noticeable to one user becomes a UX crisis under load. Data pipelines that work fine on small datasets break in ways that are hard to predict without depth.

The claim that “coding is no longer the bottleneck” may be true in the simplest corner of this matrix — internal tools, prototype-grade, unregulated, small scale. It becomes progressively less true as you move in any direction.

Engineering Judgment Is Still Judgment

There’s a subtler problem buried in how “judgment” is being used right now: it’s being treated as a single thing when it’s actually several.

Product judgment — understanding user problems, prioritizing correctly, knowing when good enough is good enough — that’s one kind. Design judgment — taste in visual hierarchy, interaction patterns, information density — that’s another. Engineering judgment is another — and it doesn’t disappear because LLMs can write code.

Engineering judgment is knowing that a given API’s rate limits will create a bad user experience under realistic load, before you find out by watching it fail. It’s recognizing that a context window limitation will cap what’s possible for your use case, and structuring your approach accordingly. It’s knowing when the cost of using an LLM makes it the wrong tool for a particular subtask, even if it would technically work.

When I was building the nature access map and hit a frontend rendering bug that sent Claude into circles — each fix creating a new problem — what broke the loop wasn’t better prompting. It was recognizing that we needed browser developer tools to surface actual error information, rather than continuing to diagnose from assumptions. That was a judgment call, made by a human, that the LLM hadn’t arrived at on its own. And this was a simple, single-user, locally-run application. The engineering complexity scales from here fast.

You can’t decouple engineering decisions from user experience decisions, or from business decisions. The people who understood this — who made proactive choices about downstream needs before being asked, who proposed building test infrastructure before it became obviously necessary, who recognized when a technically feasible solution would create maintenance debt that wasn’t worth it — were always valuable because of those judgment behaviors. They just happened to express them through engineering work.

Vibe coding doesn’t make that less relevant. It reduces the coordination costs of separating judgment from execution across different people. That’s genuinely useful. It doesn’t synthesize the judgment.

The Economics of It

When the cost of development decreases, the value of complementary capabilities increases. This is basic economics — cheaper peanut butter raises demand for jelly — and it explains why judgment, taste, and domain expertise are getting more attention right now.

But the implied corollary — that the previous bottleneck disappears — doesn’t follow. It becomes less constraining for simpler cases. It remains fully present for complex ones. And new bottlenecks emerge in places previously masked by the old one.

When I built the nature access map, part of what made it worth building was that the search cost of finding a tool that answered my specific question exceeded the development cost of building it myself. That math used to reliably go the other direction. Now it sometimes flips. That’s a real shift — but one that made domain knowledge more load-bearing, not less. The development cost dropped. The judgment requirements didn’t.

There’s a second economic dynamic worth naming. When anyone can build, the supply of apps proliferates rapidly. At some point supply exceeds demand. And in a buyer’s market, buyers have two levers: pay less for the same thing, or demand higher quality. These are in tension — as quality requirements increase, fewer people can meet them, which changes pricing again. But the net effect is that “good enough” stops being sufficient in the same way it was when supply was scarce. When you were one of a few people who could ship anything at all, the bar was low by necessity. When thousands of people are shipping similar things, the baseline shifts. The judgment required to clear it doesn’t go away. It goes up.

The Open Question

We’ve had “low-code” and “no-code” tools for years. Their history is consistent: useful for prototypes, insufficient for production, high ratio of demos to shipped products. They lowered the floor without raising the ceiling. People built more things, most of which didn’t make it.

The open question is whether this generation of tools — Claude Code and its successors — finally crosses that threshold. Whether the production gap will actually be closed for a meaningful range of use cases in a way it wasn’t before.

My honest answer is: maybe. Given enough time, probably, for some categories of use cases and some levels of complexity. But we don’t fully know yet, and the people making the strongest claims — in either direction — are probably overstating their case.

What I do know is that the judgment question isn’t new. It was always the hard part. The fact that more people can now see it — because more people can now build things and see the results — doesn’t mean it recently emerged from new tools.

It was always there, hidden in the gap between shipping and shipping something worth using.

This is Article 5 in a series on building with AI tools as a non-native technologist — covering my real experiences and observations building products as someone without a formal engineering background. Previous articles covered financial planning with Claude Code, a framework for city selection, using LLM tools to help with a cross-country move, and building the Seattle Nature Access Map.

No one could answer it well.

Google Maps can route you between two points. LLMs can tell you that Seattle “has great parks.” Walk Score can give you a general walkability number. But none of them answered my actual question: for a given block within a given neighborhood, what’s the walking distance to a meaningful green space?

So I built it myself.

This is the story of how a personal frustration during my move from New York to Seattle became my first public app—built with Claude as a coding partner, shipped as an MVP over a handful of nights and weekends, and designed to answer one specific question that no existing tool answered well.

Why This Existed as a Problem

In my previous articles, I wrote about the framework I used to pick a city and how I used LLMs during the planning process. One of my core criteria was convenient walking access to nature—not “nature exists somewhere in the metro area,” but “I can walk to a real park in under 15 minutes from my front door.”

This mattered to me because of lived experience, not aspiration. Living in New York, I’d discovered that regular morning walks—first through Carl Schurz Park, then Central Park—were one of the highlights of my daily life. I felt measurably better on days I walked. Over time, I realized that a city with great restaurants and nightlife but mediocre park access would be a worse fit for my actual life than a city with fewer amenities but a large park within walking distance.

The problem was that once I had this criterion, I couldn’t evaluate it efficiently.

I did rounds of Google searches. I asked LLMs. I checked existing nature and hiking apps. Nothing quite fit. There are plenty of apps for finding trails or planning hikes—but I wasn’t looking for a weekend adventure. I wanted to know: if I live on this block, can I walk to a good park easily? And how does that compare to the next block over, or the next neighborhood?

That question turns out to be surprisingly hard to answer without building something.

When Finding Is Harder Than Building

This experience led me to a realization I hadn’t expected: for a sufficiently specific personal need, the search costs of finding the right tool can actually exceed the development costs of building one yourself.

Historically, that wouldn’t have been true. Building a map-based web app required professional software engineering skills and weeks or months of development time. The rational move was always to search harder, compromise on fit, or just do the research manually.

But something has shifted. Between vibe coding with LLM tools like Claude, publicly available geographic data, and open-source mapping libraries, the cost of building a simple, personalized tool has dropped dramatically. Meanwhile, the cost of searching—scrolling through apps that almost-but-don’t-quite solve your problem, configuring tools not designed for your use case, cobbling together manual research across multiple sources—hasn’t changed much.

For me, the math flipped. I’d already spent hours on manual research and still didn’t have what I wanted. Building the thing I actually needed took less additional time than continuing to search for it would have.

This has broader implications for anyone building with AI tools. As development costs decrease, the value of complementary capabilities increases—things like judgment about what to build, taste in how it should work and look, and access to the right data. The bottleneck isn’t “can I code this?” anymore. It’s “do I understand the problem well enough to build the right thing?”

What I Built

The concept was deliberately simple: a color-coded map of Seattle showing walking distance to parks, broken down by neighborhood and then by census block.

That’s it. One question, answered visually, at a glance.

Why a map and not a report? I actually tried the text-based approach first. I ran LLM queries and compiled natural language descriptions of each neighborhood’s park access. It wasn’t very useful. Geographic and spatial relationships are fundamentally hard to convey in text. “Wallingford is 0.8 miles from Woodland Park” is less informative than seeing Wallingford on a map, colored green, with a park icon nearby. The visual format communicates distance, relative position, and comparison across neighborhoods simultaneously—something that would take paragraphs to express in words and still wouldn’t land as intuitively.

This connects to a broader principle I’ve been thinking about: the right interface depends on the type of information. Natural language works well for conceptual questions and subjective reasoning. But for spatial data, structured comparisons, or anything where relationships between data points matter as much as the data points themselves—a visual interface is almost always better.

The flow works like this:

Start with the city overview. Every neighborhood is color-coded from green (excellent park access) to red (poor access), scored 0–100. You can see at a glance which areas of Seattle have the best walking access to nature. Hover over a neighborhood and you see its score.

Click into a neighborhood. Now you see the individual census blocks within it, each with its own color and score. This matters because neighborhood averages can be misleading—a neighborhood might score 65 overall, but the blocks closest to a park could be 85+ while blocks on the far edge might be 35.

Click a specific block. You see the address, the nature access score, the nearest park name, the distance, the block area, and a link to view it on Google Maps.

The whole thing follows a principle called progressive disclosure: show the minimum useful information first, then let people drill deeper based on what they care about. For someone just exploring neighborhoods, the city overview is enough. For someone evaluating a specific apartment, the block-level detail is what matters.

How I Built It (And What Was Harder Than Expected)

I used Claude as my coding partner for the entire build. The overall process followed what most people describe as vibe coding: provide a high-level plan, iterate step by step, and make trade-off decisions along the way.

The plan was conceptually simple. I needed three things: data (parks, neighborhoods, geographic boundaries), a frontend (the map interface), and backend logic (calculating distances, generating scores, connecting everything). This is essentially the model-view-controller pattern—data, interface, and logic working together.

Where it got interesting was the trade-offs.

The data layer was the biggest source of real decisions—not because getting data was impossible, but because every data choice had downstream consequences.

First, how do you define a “park”? Is a 2-acre playground sufficient? What about a botanical garden? A greenway? For MVP, I kept it simple: parks above a minimum size threshold, focused on the kind of green space you’d actually walk through for 20+ minutes. That meant leaving out some edge cases, but it kept the scope manageable.

Second, where does the data come from? Publicly available geographic datasets exist, but they come with trade-offs: API rate limits, licensing restrictions, varying data quality, and inconsistent naming conventions across sources. Park data uses one taxonomy. Neighborhood boundary data uses another. Census block data uses yet another. Making these work together required judgment calls—how to define neighborhood boundaries, how to handle edge cases where a block spans two neighborhoods, what to do when data sources disagree.

Third, how do you serve the data? Making live API requests for every user interaction would be slow and fragile—and could hit rate limits fast if multiple people use the app simultaneously. So I downloaded the data upfront and serve it locally. For an MVP with a small user base, this works well and keeps the experience fast. Scaling would require a different approach, but that’s a future problem.

The frontend was where taste mattered most. The color scheme, the scoring scale, the level of detail in the popup, the decision to show census blocks rather than arbitrary grid squares—these are all judgment calls that determine whether the app feels useful or confusing. No amount of coding skill replaces knowing what the user (in this case, initially me) actually wants to see.

The backend logic surfaced subtle complexity. Distance calculations between geographic points aren’t as straightforward as they appear—coordinate systems, projection methods, and the difference between straight-line distance and walking distance all matter. For MVP, I used a reasonable approximation that’s good enough for “is this a 5-minute walk or a 20-minute walk?” If I need higher precision later, I can refine it.

Where Vibe Coding Helped—and Where It Didn’t

Building with Claude as a coding partner was significantly faster than building alone would have been. The LLM was particularly good at generating boilerplate code for map rendering and data processing, suggesting library choices and architectural patterns, helping debug issues when I could describe the problem clearly, and handling repetitive tasks like data formatting and conversion.

Where it struggled was when problems required context that’s hard to express in a prompt. For example, I hit a rendering bug where the frontend wasn’t displaying blocks correctly. Claude tried multiple fixes, each of which introduced a new problem—we went in circles. What eventually broke the loop was me suggesting we use browser developer tools to inspect the actual errors, which gave us concrete diagnostic information to work with. In another case, a more capable model release solved a problem the previous model couldn’t crack.

This reinforced something I’ve come to believe through this process: understanding computer science fundamentals matters, even in a vibe coding world. Not because you need to write every line yourself, but because when something breaks—and it will—you need enough understanding to diagnose the problem, ask the right questions, and evaluate whether the LLM’s proposed fix makes sense. If you abstract away all understanding of how the system works, you’re stuck the moment something goes wrong.

That said, I want to be honest about the tradeoff: for a simple, single-user application like this, the fundamentals matter less than they would for a production system serving thousands of users. The stakes of a bug are low. The complexity is manageable. If you’re building something for yourself and learning as you go, vibe coding is a remarkably effective way to do both simultaneously. Just know that the further you scale—more users, more data, more features—the more that underlying understanding pays off.

Lessons Learned

Build for yourself first, then share. I built this because I genuinely wanted it. That meant I understood the problem deeply, had strong opinions about what the solution should look like, and could evaluate quality without user testing. Solving your own problem is the fastest path to a useful product.

Start with one question. The app answers one question: how walkable is the nearest park from this location? Every feature decision flowed from that. When I was tempted to add hiking trail data, transit overlays, or restaurant proximity, I asked: does this help answer the core question? If not, it’s post-MVP.

Data is the real bottleneck, not code. The most time-consuming part wasn’t writing code—it was finding reliable data sources, understanding their limitations, reconciling different taxonomies, and making judgment calls about quality. As vibe coding makes development easier, data access and data quality become key differentiators for applications.

The right interface matters more than the right model. I tried answering this question with LLMs (natural language), with manual research (spreadsheets and notes), and with a visual map. The map won—not because it has more information, but because it presents spatial information in the format humans process most naturally. Choosing the right interface for your information type is a design decision that no amount of better AI can substitute for.

Ship the MVP, then iterate. The current version has rough edges. The scoring could be more sophisticated. The data could be more comprehensive. The styling could be more polished. None of that matters as much as having a working tool that answers the core question. Everything else is refinement.

What’s Next

The immediate plan is to continue refining the app—improving the interface, exploring additional cities, and seeing whether other people find it as useful as I do.

But the bigger takeaway from this experience is about a shift that’s happening in who can build useful tools. A few years ago, building a map-based web application required a team of engineers. Today, a product manager with some CS background and an LLM coding partner can ship a functional MVP in weeks.

That doesn’t mean the engineering is trivial. It means the barrier to entry has moved. The scarce resource is no longer “can you code it” but “do you understand the problem well enough, and do you have the judgment to make the right trade-offs?”

For me, this started as solving my own problem. It turned into my first public app. And it taught me more about building products than years of managing them from the other side of the table.

The app is available here: nature-access-map.

This is the fourth article in a series. Article 1 covered using Claude Code for financial planning. Article 2 covered the framework I used to evaluate potential cities to move to. Article 3 covered where LLMs helped and failed during the planning process.

I asked ChatGPT for the best cities to live in matching my criteria.

It gave me Portugal.

I asked for mild weather.

It gave me South Carolina.

That’s when I realized: LLMs are powerful research tools, but they’re solving for the average person—not for you.

Over three months of planning a cross-country move, I learned exactly where AI tools help and where they fail. The key wasn’t finding the “right prompt.” It was understanding which parts of the problem are objective—and which are subjective—and treating them completely differently.

Here’s what I found.

The Real Problem: Your Criteria Aren’t as Clear as You Think

When I started this process, I had what I thought were well-defined criteria: mild weather, walkability, access to nature, reasonable cost of living.

But here’s the thing—most of those words don’t actually mean anything specific.

“Mild weather” means something different if you walk everywhere carrying groceries versus if you drive. I’ve done both. I’ve walked through Atlanta summers carrying grocery bags, and I can tell you that 85°F with humidity when you’re on foot for 20 minutes is a fundamentally different experience than 85°F when you’re stepping from an air-conditioned car into an air-conditioned store.

“Walkability” means something different if you work remotely and mainly care about parks and errands versus if you commute downtown daily.

“Access to nature” means something different if you want weekend alpine hikes versus a daily walk through a 50+ acre park within 15 minutes of your front door.

The hard part isn’t that these criteria are wrong. It’s that they contain hidden assumptions—and you often don’t realize it until you get results that feel off.

Part of the reason is that many preferences are internalized. You don’t actively think about them. They’re visceral. You know the Atlanta summer grocery run feels bad, but you might not think to tell an LLM, “I walk to the grocery store three times a week and I carry bags, so your temperature recommendation needs to account for extended pedestrian exposure in humidity.” That level of context doesn’t come naturally.

It’s similar to how a UX researcher has to coax out a user’s real workflow—people don’t volunteer the details that matter most because those details feel obvious to them.

The Insight That Changed Everything: Objective vs. Subjective

After weeks of getting well-intentioned but unhelpful results, I noticed a pattern.

LLMs performed very differently depending on whether my question was objective or subjective—and much of the frustration came from treating subjective questions as if they were objective.

Objective questions have verifiable, well-structured answers. Historical temperature data. Cost-of-living indices. Transit system maps. Average precipitation by month. These are previously solved, commonly asked, publicly available, and quantifiable. LLMs are excellent at these—often as good as or better than manual research because they can synthesize across sources quickly.

Subjective questions involve personal interpretation, context-dependent meaning, or ambiguous language. “Is it mild?” “Is this neighborhood walkable?” “Is there good nature access?” These feel like they should have clear answers, but they don’t—because the answer depends on who’s asking and why.

The mistake I kept making was asking subjective questions and expecting objective answers.

When I asked for cities with “mild weather,” the LLM had to interpret “mild.” Its interpretation was reasonable—but it wasn’t mine. South Carolina has mild winters. It also has summers that would make my grocery walks miserable. The LLM didn’t know that because I hadn’t connected the dots between “mild” and “I walk carrying bags in the heat.”

This distinction—objective vs. subjective—became the foundation for everything else I did.

Working With Objective Criteria: Let the LLM Do the Heavy Lifting

For objective criteria, LLMs were genuinely excellent.

Once I learned to ask for data instead of adjectives, the quality of answers jumped dramatically.

Instead of: “Which of these cities has mild weather?”

I asked: “For each of these 15 cities, what is the average, median, and range of monthly high temperatures over the last 5 years? Include humidity and precipitation.”

Instead of: “Is Raleigh walkable?”

I asked: “What is Raleigh’s Walk Score? What percentage of residents commute without a car?”

The principle is simple: if a question can be answered with publicly available, well-structured data, ask for the data directly. Don’t ask the LLM to interpret it for you.

This worked well for criteria like temperature ranges, cost-of-living comparisons, transit infrastructure, general park and green space data, and airport access and logistics.

The results weren’t always perfect—LLMs can be confidently wrong on specifics, and data can be outdated—but they were directionally reliable. Good enough to build a shortlist. Good enough to eliminate obvious misfits.

Working With Subjective Criteria: That’s Your Job (But LLMs Can Help You Think)

The subjective criteria were where things got interesting—and where I had to change my approach entirely.

For these, the LLM’s role shifted from “researcher” to “thought partner.” Instead of asking it to give me answers, I used it to help me figure out what my questions actually meant.

Technique 1: Benchmark against cities you know.

Rather than trying to define “mild” in the abstract, I used cities I’d actually lived in as reference points.

“I’ve lived in New York and Atlanta. Compare the summer pedestrian experience in Raleigh, Seattle, and Pittsburgh to those two cities—specifically for someone who walks daily and carries groceries.”

This forced specificity. The LLM couldn’t hide behind “mild climate.” It had to compare against my actual lived experience.

Technique 2: Ask yourself why—and tell the LLM.

When I caught myself using a vague word, I’d ask: why do I actually care about this?

“Mild weather”—why? Because I walk for exercise and errands daily. Because I don’t want to dread going outside for four months of the year. Because I don’t want to rely on AC or heating as a coping mechanism for a climate I fundamentally don’t enjoy.

Each of those “becauses” translates into a more specific, testable question. And once I fed that context to the LLM, the recommendations improved noticeably.

Technique 3: Pressure-test with specific examples.

When an LLM told me a city had “good nature access,” I’d follow up: “What specific parks are within 20 minutes walking distance of downtown? How large are they? Do they have trails or is it a playground?”

The specifics often revealed that “good nature access” meant something very different from what I needed. A city park with a playground and a walking loop is not the same as a 300-acre urban forest with trail networks. Both are technically “nature access.”

This is actually a pattern I recognized from a previous career chapter. Years ago, I managed machine learning annotation projects—labeling training data for AI models. Even with seemingly well-defined categories, annotators would apply labels subjectively. The fix was always the same: go through specific examples, compare them, explain your reasoning, surface the disagreements, and refine the criteria. The same principle applies when you’re “annotating” your own preferences for an LLM.

The Division of Labor

After a few weeks, a clear division of labor emerged:

I used LLMs for thought partnership to pressure-test my criteria, researching and summarizing candidates against those criteria, helping me narrow down on specific dimensions, drafting checklists (apartment tours, moving logistics, donation steps), and estimating logistics and surfacing things I might be missing.

I kept for myself defining what I actually want and how to prioritize it, calibrating subjective labels (”mild,” “walkable,” “quiet”), verifying anything that could cost money or time, and running real-world tests (walking neighborhoods, simulating daily life, trying errands).

The frame that made it click: LLMs are great at research, organization, and estimation. They’re not built for judgment—and personal decisions are mostly judgment.

Once I internalized this, the tool stopped being a frustrating “decision maker” and became a genuine force multiplier.

Top-Down First, Then Bottom-Up

A mistake I made early was trying to get the perfect answer in one shot.

What worked was a two-phase approach:

Top-down (cast a wide net): I asked for 20–40 candidate cities given my criteria. I deliberately wanted a long list with false positives, because false negatives are more costly—if you never consider a city, you never discover it’s a fit.

At this stage, I accepted messy, imperfect results. The goal wasn’t precision. It was coverage—and using the results to refine my own thinking about what mattered.

Bottom-up (go deep on the shortlist): Once I’d narrowed to a handful of cities, I shifted to targeted, specific queries. Neighborhood-level research. Specific parks, transit routes, grocery stores. Temperature data by month. This is where the objective/subjective framework paid off most—I knew which questions to ask the LLM (data) and which to answer myself (judgment).

The key insight: city-level averages hide neighborhood-level reality.

A city can be “unwalkable” overall and still have pockets that are perfect for a car-free lifestyle. Or the reverse. So I treated each promising city’s neighborhoods as a separate research problem—and that’s actually where the decision happened.

Where LLMs Consistently Fell Short

A few patterns emerged:

Ambiguous terms without operationalization. Every time I used a word like “mild,” “safe,” “quiet,” or “diverse” without defining it, I got generic results. The fix was always the same: replace adjectives with numbers, ranges, or specific examples.

Hidden assumptions. The LLM would assume I had a car, or that “affordable” meant a certain range, or that “city” meant metro area rather than a specific neighborhood. The fix: ask the model to list its assumptions, or preempt by stating constraints (”assume no car,” “budget is Y”).

Trade-off weighting. If you ask an LLM to optimize across 10 criteria simultaneously, you’re asking it to make value judgments about your life. It can’t do that. The fix: ask for a longer list with trade-offs noted, and do the weighting yourself.

Freshness and specificity. LLMs can be confidently wrong on details that change—pricing, availability, neighborhood dynamics. The fix: treat every output as a lead to verify, not a fact to trust.

The Limits of Research (And Why Real-World Testing Matters)

At some point, no amount of prompting eliminates uncertainty. You have to test.

I found that the best way to reduce uncertainty was simply visiting—or simulating a visit as closely as possible. Walking the actual routes I’d walk. Checking the actual grocery stores I’d use. Riding the actual transit I’d depend on.

This is also where I learned that not everything should be an LLM conversation. Some questions are better answered by a map. Some by a transit app. Some by walking around for an hour.

I actually started building a tool for one of these gaps—a neighborhood park access map that shows walking distance to parks at the block level—because it was a question I kept asking that no existing tool (including LLMs) answered well. I’ll share more about that in the next article.

The principle: LLMs generate hypotheses. Targeted tools and real-world experience verify them.

What I’d Do Differently

If I were starting this process from scratch, I’d do three things earlier:

First, separate objective from subjective criteria on day one. This alone would have saved me weeks of frustration. Objective criteria get delegated to the LLM immediately. Subjective criteria get a “define what I actually mean” session before any research happens.

Second, use benchmark cities from the start. Instead of trying to define preferences in the abstract, ground everything in places you’ve actually experienced. It’s faster and more reliable.

Third, go neighborhood-level sooner. City-level research is useful for elimination, but the decision lives at the neighborhood level. I spent too long comparing cities when I should have been comparing neighborhoods within the top 3–5.

Conclusion

Using LLMs for this move didn’t eliminate uncertainty. Moving is still a leap.

What it did was compress the process—from fuzzy to structured, from overwhelming to manageable, from “I don’t even know where to start” to “I have a clear shortlist and I know what to test.”

The trick was learning that the most important skill isn’t prompting. It’s knowing which parts of the problem are yours to solve—and using the tool to handle everything else.

LLMs are great at helping you think. They’re great at helping you research and organize.

They’re not great at being you.

Next article: I built a neighborhood park access map because no existing tool answered my question. Here’s what I learned about going from an internal tool to a public product.

I know this because I spent ten years living in a New York City – a city that looks great on paper.

It’s globally recognized, full of opportunity, and has every possible convenience. And yet, over time, it just didn’t feel right for me.

It wasn’t a bad experience. It just felt like I was in the wrong place for reasons I couldn’t quite articulate.

I’d had this feeling for a long time.

But given the range of possible cities to choose from, the uncertainty and ambiguity around them, and a wide range of experiences being reported…

Information overload and decision paralysis followed.

Combine that with psychological tendencies towards pain avoidance (the delightful moving process) and doubt avoidance (how do I even know the new city will be better?) – and slight variations of the status quo becomes the norm.

I.e. stay in the same city.

When I started planning a possible move late last year, I tried to address this by getting advice from both people who knew me (and of course, LLMs).

The advice was well-intentioned but mostly unhelpful.

Not that I felt like anyone was trying to mislead me—but because the advice wasn’t personalized to my life. People were giving me recommendations filtered through their definitions of what matters.

Someone says, “Don’t move to Seattle, it rains a lot.”

Someone else says, “You should move to Nashville or Austin.” When pressed, it’s because they took a two-day business trip, went to great restaurants, and enjoyed the vibe.

Even well-vetted advice can be wrong in context. When I lived in Atlanta for a few years, the default line was: you need a car. Maybe that’s true for many lifestyles. It wasn’t true for mine. What mattered more was how I actually spent my day: school, studying, errands, gym, basic routines.

I lived near where I needed to be, walked more than most people assumed, and used taxis and rideshares strategically. In Atlanta, monthly parking was $~150. Ubers averaged less than $20/week.

The city’s reputation didn’t match my lived experience—because my routine didn’t match the “average” routine implied by the reputation.

That’s the fundamental problem:

Most advice about choosing a city fails because it isn’t personalized.

The good news is: this is solvable.

It just requires a framework that starts with your real day-to-day life.

Step 1: Audit your day-to-day (the foundational move)

Before you research cities, ask a simpler question:

What does my life actually look like right now?

For a couple weeks, track your routine at a practical level. Not “I value health.” Not “I love nature.” The actual sequence:

• Do you work remotely or commute? If you do commute, how?

• How often do you leave the house on weekdays?

• Do you cook, buy pre-cooked meals, or mostly eat out?

• What errands happen weekly (groceries, pharmacy, gym, laundry)?

• What do you do on weekends when you have free time?

• What do you do for fun / to blow off steam?

This is where people get tripped up. They choose cities based on identity (“I’m a nature person”), aspiration (“I’m going to hike every weekend”), or aesthetics (“I want charm”).

None of those are inherently wrong.

But they’re unreliable inputs if they aren’t grounded in practical behavior and habits.

One of the biggest insights I had was realizing that some “preferences” were actually requirements for my mood, health, and productivity—and some things I thought mattered were mostly nice-to-haves.

Example: nature access. It didn’t become real for me because I wrote it down as a value. It became real when I started walking in the park regularly—early, even when it was cold, even when the weather wasn’t ideal. That behavior was evidence. It told me: this isn’t a nice-to-have or a fantasy. It’s a proven requirement for my wellbeing.

Step 2: Turn your routine into criteria (make it testable)

Once you understand your day-to-day, convert it into criteria that are specific enough to evaluate.

Start with broad categories (most people share these):

• Cost of living (true cost, not just rent)

• Housing (size, age, noise, amenities)

• Employment

• Food access (diet, convenience)

• Walkability and transportation

• Healthcare access (if relevant)

• Weather (what you enjoy, tolerate or dislike)

• Access to nature

• Safety (as you define it)

• Logistics (airports, shipping, family visits, errands)

• Density / stimulation (do you want energy or calm?)

Then make them concrete. “Access to nature” is not a criterion. It’s a label.

A criterion is something you can test, like:

• “Within a 15-minute walk: a park I’d actually use.”

• “I can do 80% of my weekly errands without a car.”

• “My grocery options support how I actually eat (not how I wish I ate).”

• “Rent + utilities + predictable ‘extras’ fit within my comfort range.”

• “Noise and stress in the home environment stay below a threshold.”

The point is not to create a perfect scoring system. The point is to stop making decisions based on vibes you can’t operationalize.

Step 3: Prioritize (must-haves vs nice-to-haves, reality vs aspiration)

Now the real work: prioritization.

People frequently confuse a preference with a requirement.

A must-have is a constraint you can’t realistically violate without breaking the move:

• budget reality

• commute requirements (if you have them)

• core health needs

• housing necessities (space, noise insulation, safety, etc.)

• logistics you can’t wish away

A nice-to-have is something you want, but could sacrifice if other wins are strong.

A second consideration matters just as much: Daily needs vs episodic needs.

Daily needs dominate your happiness because they repeat. A great restaurant scene matters less if you eat at home 90% of the time. A world-class airport matters less if you fly twice a year. On the flip side, if you travel constantly, the airport becomes a daily-life consideration.

Finally: Current reality vs future aspiration.

If you aren’t exercising now, don’t overweight “strong fitness culture” as the deciding factor. That doesn’t mean you can’t change—but it means you should be honest about the likelihood that a move alone will change you.

A useful test here:

• If this criterion disappeared, would I be mildly disappointed—or miserable?

• If I got “good enough” here, would that free me to prioritize other things?

Step 4: Apply the framework to your current city first

As with any experiment – having a benchmark or starting point is best.

Run the framework on where you currently live:

• What works well?

• What’s constantly annoying?

• What are you compensating for with money, time, or stress?

• Which friction points keep showing up?

This is how you learn what you’re actually trying to change.

For me, this is also helped flesh out interdependencies. The living environment affects stress. Stress affects sleep. Sleep affects diet. Diet affects health. Health affects productivity. Productivity affects earning power and confidence. Suddenly the city choice isn’t just lifestyle—it’s downstream of everything.

Step 5: Research cities—but evaluate at the neighborhood level

Here’s a simple truth that collapses a lot of noise:

Most cities aren’t one experience. They’re a bundle of neighborhoods.

Even in cities with strong overall characteristics, your day-to-day is determined by:

• where you live

• where you work

• what’s within walking distance

• how you move around

• what your immediate environment feels like

Atlanta has walkable areas. It has areas where you can live without a car. It also has areas where that would be miserable. “Atlanta” as a single recommendation is meaningless without the neighborhood.

Same for Seattle. Same for NYC. Same for almost anywhere that isn’t tiny.

A practical research method that helped me:

• List cities you think you’d like

• Also list cities you’re confident you wouldn’t

• Use the contrast to refine your constraints (e.g., “I don’t tolerate extreme heat” or “I don’t want months of sub-freezing temperatures”)

Step 6: Down-select to 3–10 options (then stop browsing)

At some point, “more research” becomes procrastination.

Build a shortlist. For me, it included places like Boulder, Denver, Nashville, Seattle, Santa Clara, San Francisco, Oakland, Chicago, Tampa, Raleigh, Bellevue.

Your list will differ. That’s the whole point.

Use a simple rule:

• Must-haves = pass/fail

• Nice-to-haves = 1-5 score

• Keep notes on trade-offs, not just ratings

Step 7: Experiment (visits prevent expensive mistakes)

If there’s one cheat code here, it’s this:

Visit. Simulate a normal week or at least a few days.

Don’t just do tourist stuff. Do your routine:

• walk to a grocery store you’d actually use

• try the transit route you’d actually take

• go to a gym you’d actually join

• walk in the neighborhood at the time you’d normally walk

• sit in a café and work for an hour if you work remotely

• pay attention to friction: noise, safety vibe, stress level, convenience

This is also where you learn things you can’t learn online.

I almost chose a building/neighborhood combination that looked perfect on paper—until I visited and noticed sketchy behavior from the building manager. The neighborhood and apartments were great. But the “day to day reality” felt off – ranging from spikes in traffic noise, hidden fees and a lot of “it depends” answers to my questions. That’s not a minute detail. That’s the kind of thing that turns a move into a chronic stress generator.

Step 8: Decide (don’t let perfect be the enemy of good)

There will always be trade-offs.

Some people try to optimize to a mythical “perfect city” and end up stuck, endlessly comparing. Instead, aim for:

Significantly better than your current state.

Then commit.

You can refine over time—sometimes within the same region, sometimes within the same city, often just by switching neighborhoods.

This is where diminishing marginal returns matters. Getting to 8/10 on your must-haves usually beats sacrificing multiple criteria to chase 10/10 on one. “Good enough” is not settling; it’s calculated decision.

Lessons learned (what I wish I could tell my past self)

1) Test aspirations vs reality.
If you don’t do it now, don’t overweight it. Validate needs through behavior.

2) Social life is portable.
“Seattle is less friendly” is too generic to be actionable. You can meet people anywhere and be lonely anywhere. Your habits and communities matter more than city stereotypes.

3) Experimentation prevents expensive mistakes.
Visits, neighborhood testing, and realism checks beat internet certainty.

4) Hidden costs matter.
Not just money—time and friction. Filters, locks, insurance, parking, delivery fees, and all the little things that add up. A rent number without lifestyle math is incomplete.

5) Diminishing marginal returns are everywhere.
You don’t need perfection. You need a strong baseline across key variables. Then you can make a decision regarding what to optimize and when/where to fit in nice to haves.

6) “Access to nature” is not a single preference.
A 5am hiker and a casual park walker are describing different lives. Same label, different requirement.

Closing Thoughts

I’ve walked in the Seattle rain. I’ve seen gray days. And I’m still glad I moved—because the decision wasn’t based on a city’s reputation. It was based on whether the city fit my real day-to-day life.

If you’re considering a move, don’t start with “What’s the best city?”

Start with:

1. What does my life actually look like?

2. What would make it meaningfully better?

3. Which places can realistically support that?

Then build your shortlist—and go test it.

I explored a few different approaches in parallel – a top-down type of workstream and a bottom-up workstream.

In this context – a top-down approach means larger scope: effectively I compiled all of the financial statements and transactions from a given period of time, say the last 3-5 years, and then used Claude Code and other tools to actually parse out the transactions, compile them, categorize them, and analyze them or enable my own analysis of them.

I tested a few different approaches that uses this sort of top-down workflow.

The original overall approach or pipeline was:

- Create an overall plan or outline

- Compile / download financial statements (PDFs)

- Drop into a local folder and have claude code work with them end to end:

• Provide initial context and refine plan using Claude Code’s plan mode

• Convert each PDF into Excel or csv

• Parse out the transactions

• Aggregate the transactions into a single Excel or csv

• Categorize the transactions

• Perform follow-on analysis

• Iterate

After a few hours it became clear that parsing out transactions from PDFs to Excel / csv is deceptively challenging (even for Claude), particularly when working with different data sources. So I replaced that specific step with using a third-party PDF to Excel conversion tool that I already had access to and was both much faster and more reliable.

Otherwise, the overall top-down approaches were the same.

My findings from the top-down approaches were as follows:

• First of all, handling large scope to start can be overwhelming in terms of the amount of the amount of necessary code (and of course corresponding context windows)

• Second of all, because of the volume of code and data corresponding to the scope, the human testing costs become much higher

  • In other words, the realm of possibilities of things that can go wrong are much broader, require more rigorous testing and the testing is more time consuming to perform

  • Unless an issue is glaring or there is some clear prioritization that can be done – for example identify what the largest transactions (such as rent payments), the transactions require a fair amount of individual review

  • However even when prioritizing these things there are all sorts of nuances that can get introduced – for example transactions being combined together, or amounts being off, or occasional exceptions that are difficult to spot (e.g. imagine a paycheck and a bonus being combined together or a rent payment being combined with a security deposit or a move from one apartment to another leading to different rent costs, etc.)

  • Oftentimes the context is situation-specific as well which makes more automated tests or handling this via the planning phase more difficult

• Third of all, in terms of the reliability and accuracy of the code – the top-down approach is decent at creating estimates but if exactness or precision is necessary, it is of limited utility as it would require a fair amount of manual quality control and testing. This may be okay as a starting point but usually requires complementing with other approaches to get more precise data

The bottom up approach

Given the limitations of the top-down approach, the question came up: is there a complementary or perhaps alternative approach to the top-down process that focuses on a smaller scope?

For example, starting with a specific category of spend, mastering that and then repeating the process for other categories.

The business analogy I would use here is how Amazon focused on a specific product category to start (books). There are similar best practices when starting a business – specifically to focus on an underserved niche rather than trying to explore too large of a market and risk more competition, expensive customer / user acquisition, etc.

So the bottom up approach has the benefit of targeting specific categories one at a time (e.g. rent, grocery spending). This in turn involves smaller volumes, similarly structured transactions and less corresponding complexity so it involves less code, simpler testing, less cognitive load and greater transparency into both the data and the code.

This has been my experience so far.

To summarize: the key benefit of the bottom-up approach is simplicity. Less categories, smaller transaction volume, less things that can go wrong, and less things to consider at any given time.

Now compare that with the top-down case where you have many statements across multiple categories. So you have a much larger volume of transactions of different types. You have different types of data structures for each. You have different types of ways in which things are expressed and net of those considerations there are many things that can go wrong potentially at each stage of the process, particularly when you’re parsing, aggregating, and analyzing the transactions.

An Open Question – Building Up to Complex Use Cases?

In addition, there is an additional potential benefit for more complex systems with a large volume of categories. My hypothesis is that if you build it up from the bottom step-by-step, category by category and each category is sufficiently constrained, over time you may be able to create an orchestration agent to actually coordinate each of these as if they were separate services.

For example you can have a grocery purchase analyzer as one service.

You can have a rent payment analyzer as another service, a medical appointment analyzer, a transportation payment analyzer, and so on.

You can do something similar for basically every category of items that is sufficiently standardized or constrained. Then, as you (and the system) develop a strong understanding of these you can then potentially automate the system entirely.

I need to pressure test the feasibility of this by running some experiments, but early indications are promising.

Conclusion

Overall, Claude Code has been very helpful in assisting me with financial planning and other use cases. So far – the top-down approach has been preferable to get a decent estimate of overall spend while the bottom-up approach has shown the most promise when exactness and detailed results are necessary.

We can see this tension everywhere. Google must balance its dominance in search with new bets on AI and conversational interfaces like Gemini. Amazon went from selling books to becoming an “Everything Store”, then turned the internal logistics and computing system that powered that store into an entirely new external business — AWS — one of the most successful divisions in corporate history.

But expansion carries risks. Venture too far from your strengths, and you risk competing where you are weakest.

Capital, time, and attention — all scarce resources — get spread thin and diverted from the highest impact areas, leading to underperformance and, in the extreme, existential risk.

Yet focus alone carries its own danger. When customer preferences shift or technology evolves, companies that stay too narrowly focused can be blindsided by simpler, cheaper, or more relevant competitors. Disruption thrives where incumbents refuse to experiment.

Shopify’s Fulfillment Bet

Shopify’s short-lived fulfillment business captures this tension perfectly.

The business – known as the Shopify Fulfillment Network – launched in 2019.

For years, Shopify’s value proposition was clear: provide small and medium-sized merchants with software tools to compete online. The platform handled websites, payments, and inventory management, but left physical logistics to merchants themselves or third-party providers.

This created a gap. Small merchants lacked the scale to negotiate favorable rates with logistics providers or build their own warehouses. Meanwhile, Amazon’s fulfillment network gave its sellers a powerful competitive advantage. If Shopify could offer similar infrastructure, its merchants could compete more effectively.

The idea was bold: extend Shopify’s software platform into the physical world by building fulfillment and logistics infrastructure for small merchants. If successful, it would give small sellers the same capabilities as giants like Amazon — affordable, reliable, fast delivery — while letting Shopify capture a larger share of the merchant ecosystem. It would enable a more seamless, end-to-end experience for merchants and their customers.

On paper, the strategy made sense. Merchants already used Shopify to run their online storefronts. Why not also help them manage inventory and shipping? It was a logical extension of the “all-in-one commerce platform” narrative.

In practice, though, it pulled Shopify into a fundamentally different business.

Running fulfillment centers, managing warehouses, and coordinating last-mile delivery is operationally complex and capital-intensive — the opposite of Shopify’s asset-light software model. Suddenly, Shopify was competing in Amazon’s backyard, a domain where scale and logistics expertise matter far more than software elegance.

Knowing When to Walk Away

With hindsight, it’s easy to label Shopify’s fulfillment expansion a misstep.

Shopify sold the business in 2023, ultimately deciding to partner with existing providers instead.

But that decision shouldn’t be viewed as simple failure — it’s better understood as an experiment that ran its course.

The willingness to run large experiments is a sign of an innovative culture. Shopify bet big on improving its merchants’ experience, mirroring Amazon’s customer-obsessed ethos. It saw an unmet need and tried to fill it.

The more impressive part, though, is the willingness to walk away.
When the economics didn’t justify continued investment — and when it became clear that the effort was drawing focus away from Shopify’s software and payments businesses — leadership made the call to exit. That takes discipline. Many companies hold onto sunk-cost projects far too long.

Changing Context, Changing Calculus

Part of the decision was driven by evolving company and industry dynamics. As Shopify’s merchant base expanded to include larger enterprises, the value proposition for an in-house fulfillment network weakened; big merchants already had established logistics partners, reducing the need for Shopify to build its own.

The context had shifted – the same strategic move that once seemed visionary now risked becoming a distraction.

Lessons Learned

There are three broader lessons from Shopify’s entrance into and exit from the fulfillment space:

1. Experiment boldly, but deliberately.
Growth often requires venturing beyond your comfort zone and running experiments with high-risk and high-reward. Shopify’s decision to make a large bet on a new business model was a testament to its experimental, high-agency builder culture.

2. Know when to quit.
Success isn’t about avoiding mistakes — it’s about reallocating time and capital quickly when experiments don’t pan out. Strategic focus and adaptability are competitive advantages.

3. Adapt as the world changes.
A good strategy at one moment can become a poor fit later. Shopify’s willingness to re-evaluate assumptions as context evolves is part of its disciplined execution and has helped sustain its momentum since.

The Bigger Picture

Whether at the level of a company, a team, or an individual career, the same logic applies. Experimentation is essential for learning and innovation — but so is focus. The goal isn’t to avoid failed experiments; it’s to make smart bets, learn fast, and redirect effort when the evidence points elsewhere.

Shopify’s fulfillment story isn’t just about logistics. It’s about the art of balancing curiosity with clarity: knowing when to build, when to explore, and when to let go.

From 2023 to 2024, Shopify’s revenues have increased 26% to over $8.8 billion while net income climbed to over $2 billion.

They have two main revenue components:

• Subscription Solutions (~26%) – recurring revenue from monthly subscriptions to access the Shopify platform, plus related activities like app sales and domain registrations.

• Merchant Solutions (~74%) – transaction-based revenue from payment processing, currency conversion, and other services linked to merchant success.

Subscription Solutions

Subscription solutions are mostly comprised of platform access, and are priced accordingly (mix of fixed costs and variable platform fees). There are different tiers of platform access with different corresponding pricing. The reasoning being that when entrepreneurs are starting out, simplified / entry-level platform access makes sense but as a business grows and has more complex needs it will likely need access to a broader range of features to navigate the complexity involved (for example managing multiple sales/marketing channels or managing more customer accounts). Shopify Plus is mentioned as their premium tier with significantly higher pricing than the more basic out of box offerings.

It’s noteworthy that for any given reporting period, no single merchant has accounted for more than 5% of their revenue (which speaks to how diversified the business is).

Also, there are some other recurring revenue sources such as, sale of apps, registration of domain names and sale of themes (online store/website templates). These do not appear to be significant contributors to revenue but help provide a one-stop-shop in some respects, enabling customers (merchants) to get their needs met without having to explore other alternatives, which provides convenience and an overall better user experience.

Merchant Solutions

Merchant solutions are priced on a transaction basis – as more transaction volume comes through the Shopify platform, Shopify collects payment processing and currency conversion fees, referral fees from partners, Shopify Capital (lending to / financing of qualified merchants), and Transaction Fees.

In addition, they generate revenue from services and products like sale of shipping labels, sale of point-of-sale (“POS”) hardware, advertising on the Shopify App Store, and Shop Campaigns (buyer acquisition offering).

While individually small, these involve similar dynamics as the subscription solutions segment of their business. Specifically, these offerings reduce friction and keep merchants operating within Shopify’s ecosystem – strengthening the user experience and retention.

Incentives and the Flywheel Effect

From an incentives standpoint – when merchants succeed, Shopify succeeds.

This happens in a few different ways:

• Subscription solutions

  • When a business is starting out – they don’t need access to premium platform features

  • Free trials (particularly during times of market slowdown) help incentivize entrepreneurs and up and comers to start using the platform. If they start gaining traction as a business and view the platform as helpful in doing so, they’ll start paying

  • If the business succeeds over time, it’s likely growing and will be a prime target to upsell/upgrade to the next tier

  • In addition, while most businesses pay month to month, some elect to pay annually or multi-year subscriptions (with longer terms likely incentivized by discounts, akin to volume-based pricing)

• Merchant solutions

  • The more business a customer (merchant) does through the platform, the more Shopify collects in transaction-based fees. This is even better aligned in terms of incentives as Shopify immediately benefits while the merchant collects most of the profit associated with the sale, creating a win-win situation

  • Referral fees from partner fees are more likely to be generated as Shopify captures more of the market in its core offerings and develops increased brand recognition, and allow all parties in the ecosystem - including customers (merchants), partners and Shopify itself - to benefit accordingly from the network effect

  • Through Shopify Capital, lending to qualified merchants with strong business prospects generates revenue in the short term (loan interest) while functioning as a form of customer acquisition for the core offerings. As these businesses grow, they are more likely to upgrade to premium subscription tiers and process higher transaction volumes.

  • The other offerings – while not significant revenue drivers help customers (merchants) focus on their core business by reducing search/switching costs as well as stay in the platform (possibly reducing risk of customer churn over time)

Conclusion

The genius of Shopify’s model lies in its dynamic, stage-appropriate pricing: it meets merchants where they are and grows with them. Free trials convert to paid plans, paid plans upgrade to premium tiers, and transaction fees scale with success—all while ancillary services reduce switching costs. It’s a masterclass in aligning company incentives with customer outcomes, and the 28% YoY revenue growth in 2024 provides strong evidence that it works.

Next time – I’ll explore Shopify’s profitability and cost structure.

Here are some things I learned from reading Shopify’s Recent Quarterly and Annual Filings:

1. Two Main Revenue Streams

Shopify’s revenue is divided into two broad categories:

• Subscription Solutions (~26%) – recurring revenue from monthly subscriptions to access the Shopify platform, plus related activities like app sales and domain registrations.

• Merchant Solutions (~74%) – transaction-based revenue from payment processing, currency conversion, and other services linked to merchant success.

Together, these two segments reflect Shopify’s hybrid model: predictable recurring revenue plus upside from merchants’ transaction volume.

2. Customers

Shopify’s customers are merchants—ranging from solo entrepreneurs to large enterprises.

3. How Shopify Measures Performance

Shopify highlights two key operating metrics in its filings:

• Monthly Recurring Revenue (MRR): the following month’s expected subscription revenue based on currently active subscriptions

  • up from $144 million in 2023 to $178 million in 2024, a ~24% increase.

• Gross Merchandise Volume (GMV): this is a “merchant success” metric corresponding to the total merchant transactions processed over a period of time, including shipping + taxes, net of all cancellations / refunds

  • up from $235.9 billion in 2023 to $292.3 billion in 2024, a ~24% increase.

It’s interesting to note that the growth rates are the same despite corresponding to separate parts of the business. This could be a coincidence but is worth noting when reviewing the results from different periods.

4. Growth and Pricing Strategy

• In 2022, Shopify introduced Paid Trial Incentives—low-cost access periods designed to attract new merchants after a post-COVID slowdown.

• Many of those merchants upgraded in 2023, boosting growth, followed by a more normalized growth rate in 2024.

• This illustrates how Shopify invests in the long term - providing increased value up front at a time when customers are more price-sensitive and then upselling when the time is right

5. Partnerships and the Payment Stack

• Shopify relies on third-party partners such as PayPal and Stripe to process some payments.

• However, most transactions run through Shopify Payments, the company’s own payment processing system.

6. Deferred Revenue Explained

• Because merchants often pay upfront for monthly or annual access, Shopify records those payments as deferred revenue (a liability) and recognizes them gradually over the service / delivery period.

• It’s a classic SaaS accounting principle that provides visibility into future recognized revenue.

7. Other Operational Notes

• App Store: Shopify operates its own App Store, where developers can build and sell plug-ins that extend store functionality.

• Sale of Logistics Business (2023): Simplified operations and improved margins, but makes year-over-year comparisons more complex.

• Seasonality: Q4 (holiday season) is historically the busiest quarter for Shopify merchants, boosting both GMV and Merchant Solutions revenue.

8. Open Questions Worth Exploring

1. How large is the Shopify App Store in terms of merchant adoption or developer revenue?

2. What impact could tariffs or trade policy shifts have on cross-border merchant sales?

3. How does Shopify Payments add value on top of partnerships with PayPal and Stripe?

4. Is Shopify Payments lower-margin per transaction but higher in operating margin as part of the overall business?

5. Why did Shopify switch from filing Form 40-F to Form 10-K? Does this mean it transferred its legal presence or main operating presence to the United States?

6. How does the logistics sale affect year-over-year comparability in Merchant Solutions revenue?