In this article I’ll share a practical trick to make your use of Claude, ChatGPT and other LLM tools more efficient while still getting strong responses.

Plan with strong model, execute with a lighter one

To reduce the cost of working with contracts, consider using a stronger model to plan and a lighter model to execute. This applies to agentic workflows more broadly as well.

The amount you can save will depend on the specific use case and models involved, but lighter models can cost 2-5X less (sometimes even 10X+) per token than more powerful but expensive ones.

For example, you could prompt: “can you help me plan out this workflow and decide which model I should use to balance performance with cost and efficiency? Here’s some context:…”

If the task is reasonably straightforward, a lighter model can often get the job done without issue and at a lower cost. The more powerful model can still help plan the workflow, break it into tasks, and orchestrate it as needed.

For example, let’s say you’re trying to:

• Find 10 agreements that meet certain criteria

• Identify several clauses or data points within those agreements

• Research additional information related to the results

A stronger model might be useful for determining the overall approach: how to identify the right agreements, what information should be extracted, which research sources to use, etc.

However, once that plan is established, most of the execution steps may be straightforward for a lighter model.

The Contract Signal is a reader-supported publication. Subscribe for free to receive new posts, or become a paid subscriber to support my work.

Subscribe now

In contrast, imagine you’re trying to understand whether an assignment clause is enforceable under unfamiliar governing law. In that example, using a stronger model may make more sense.

The core idea: use the stronger, more expensive model where it adds the most value.

Hope this is helpful. If there’s other content that you’d find useful, feel free to drop a comment or send me a note.

Thanks,
Leonid

AI is being used all around us.

Not just in the high-profile generative AI tools like ChatGPT, Claude and Gemini but also in applications ranging from Apple’s Face ID to Spotify music recommendations to Duolingo’s video call feature.

What makes these examples successful is that the AI is effectively incorporated into the user experience to solve real, valuable problems.

Contract AI, while perhaps less exciting for most people than listening to a rock concert or learning Spanish, can benefit from similar approaches to incorporating AI.

The question is how?

In this article, I’ll share my experiences on different techniques for getting the most out of AI models. I’ll focus on contract workflows but will note that there are commonalities with use cases across other domains (e.g. personal productivity, coding / software development, research, and various third-party tools).

High Performance in the Wild, Not Just in the Lab

A key insight is that regardless of the use case, there’s a difference between achieving high model performance in “the lab” (i.e. in a purely research and development context under strict guardrails) vs. achieving high performance in “the wild”, i.e. in practical settings where the models not only perform well from a technical standpoint but also help solve real problems and generate ROI relative to the time, money and other resources it takes to use the technology.

Generating this ROI requires operationalizing AI. When I say “operationalizing” AI, I just mean effectively incorporating AI models or their output (aka predictions) into concrete workflows to solve problems (preferably in a scalable fashion, though that will depend on the needs of the use case).

Before getting into that, I wanted to offer some thoughts on why getting ROI from contract AI (as well as legal AI and other AI use cases) can be challenging.

Why Getting ROI from AI is Challenging

There’s been many studies and headlines about the challenges with getting ROI from AI despite significant investments.

There are various explanations for why, but here are three categories of reasons I’ve observed:

1. Focusing on the right problem

2. Identifying and getting access to high-performing models relative to the problem’s needs

3. Using the AI tools effectively

In previous articles, I discussed category 1 (mapping your contract workflows to identify high-impact AI opportunities) and category 2 (the contract AI vendor diligence framework to understand how likely it is that a given set of capabilities can help support a given use case).

This is important context to understand as simply deploying AI models and throwing them at a problem without first addressing these questions increases the risk of either 1-working on problems that don’t matter and/or 2-running in technical feasibility barriers.

In this article, I’ll focus on category 3.

To start, it’s worth discussing why using AI tools effectively can be challenging, even after successfully identifying high-impact problems to solve and having access to the right capabilities.

There’s three main reasons I’ve observed:

1. AI is probabilistic, not deterministic

2. Important dependencies are missing: e.g. data, the right technical infrastructure, or intuitive UX

3. AI requires some level of technical know-how, which is often missing

Reason 1: AI is Probabilistic, Not Deterministic

One of the main reasons why AI tools can be tricky to use is that AI is probabilistic. This means that for the same input, the output isn’t always going to be same, let alone correct or reliable.

Contrast this with deterministic software. Historically, before AI-powered software applications became more prevalent, software was based on procedural rules or conditional logic (aka if-then rules and similar mechanisms). Essentially, this meant that if the software was working correctly, the same input would always yield the same output.

By way of analogy: the software worked like flipping a simple light switch: flip up to turn it on, down to turn it off. Much of the technology in the world has historically functioned in a similar way: e.g. traffic lights can be red, yellow or green based on specified time intervals, the previous state or other conditions, such as a pedestrian crossing button being pressed.

In contrast, using probabilistic output is a fundamentally different experience because some part of the time the output is going to behave as expected, but not always. E.g. 90% of the time it’ll be as expected (i.e. “correct” depending on the use case) and incorrect 10% of the time, or it’ll be correct 80% of the time, and incorrect 20% of the time.

Relatedly, you typically don’t know what that exact accuracy number is in any given situation until you actually run the models on the inputs. That’s why investing in rigorous testing and other techniques is important, as we’ll get into.

It’s also why AI systems involve more ambiguity and can cause apprehension from a user adoption standpoint.

Reason 2: Dependencies (AI Requires the Right Inputs)

Just as a car doesn’t run by itself (it requires fuel such as electricity or gasoline), AI also requires various inputs and complementary capabilities to work well.

Some of the most common examples include high-quality data, the technical infrastructure necessary to train, deploy and test models, and an intuitive user interface that makes it easy to surface predictions and actually consume the output.

Reason 3: AI Requires Technical Know-How

The third category is that AI requires some level of technical know-how.

This has been significantly less of a factor recently, as most AI tools powered by large language models have a natural-language interface that doesn’t require code in most cases. However, getting the best results from these tools still requires some understanding of how they work.

Ways to Operationalize AI

There are many different ways to help get the most out of AI tools and increase ROI.

Which practices will be most effective in a given situation will depend on the specific situation and use case.

Method 1: Transparency

• What it is: Including a breakdown of whether AI is used in the product or application, where and how it is used and related caveats regarding imperfect output or potential mistakes.

• Context / Why it Matters: Contracts involve high-stakes legal and business decisions, and relying on information requires that information to meet a very high quality bar. Relatedly, attorneys and other contracts professionals have a significant responsibility to allocate risk effectively, which requires a realistic understanding of AI performance in any given situation.

• Pros: This is a simple way of flagging to a user that the data or other output is AI-generated (i.e. imperfect and may contain mistakes) and signals to the user to double check it before making a consequential decision.

• Cons: Even if the product or application is transparent regarding its use of AI, this isn’t always effective at driving the right behaviors. For example, attorneys have been sanctioned by courts for citing cases that didn’t actually exist (they were hallucinations made up by LLM-powered tools). The ability to understand these risks is driven by the user’s mental model, their understanding of probabilistic systems, and the availability of mechanisms to use AI effectively, which we’ll get into below.

Method 2: Confidence Thresholds

• What it is: Only surfacing predictions that meet a certain confidence threshold, determined by the model’s estimated confidence that the prediction is correct. For example, only surfacing predictions with a certain confidence level.

• Context / Why it Matters: Not all predictions are equally likely to be correct. For example, you may have 1000 predictions for a certain data point in a contract, with 80% accuracy overall (800 are correct, 200 are incorrect). Only using predictions that meet a certain confidence level (e.g. 90%+) may reduce the total number of predictions used to 500-600, but increase their accuracy level.

• Pros: This enables AI predictions to be routed to different workflows: for example AI only (high confidence), AI + human (medium confidence), or human only (low confidence). Whether and how to implement this will depend on considerations such as the cost of errors.

• Cons: Confidence scores are not always reliable, and so neither are confidence thresholds based on them. One reason is that confidence scores can be calibrated based on specific training data sets. New data risks lower performance due to data drift (when the models run on new contracts with a different structure, such as those corresponding to new customers, business units, or document types). Frontier models tend to generalize better, but not perfectly, and the error rate can also rise with use case complexity. For example, drafting a contract includes considerations such as tone, flow, and audience. As a result this task is harder to calibrate than a simpler binary classification problem such as “does an assignment clause exist in this contract?”

There are six additional methods I’ve found helpful for operationalizing AI.

Subscribe for free to continue reading.

Subscribe now

The Contract Signal is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

In a previous article, I shared a framework I’ve found helpful for identifying high-impact opportunities to apply AI.

In this article, I do a deeper dive by applying that framework to a concrete contract workflow: reviewing and responding to customer redlines in a master services agreement.

Business Problem and Context

Business Problem: imagine…

Read more

But how? It turns out that many of the most common obstacles to getting ROI from contract AI solutions aren’t due to the models themselves.

There are three main categories of things that I’ve found helpful when conducting contract AI vendor diligence:

1. Mapping out your own workflows to identify the highest-impact opportunities to apply AI

2. Applying a structured diligence framework to evaluate whether an AI provider is capable of delivering against those opportunities

3. Knowing what to look for during diligence, which comes from experience in applying categories 1 and 2 above (i.e. judgment)

In previous articles I shared my perspective on categories 1 and 2 (links inline above). These are good starting points. However, making the frameworks actionable requires knowing how to apply them and what to look for.

With that said, for category 3, here are seven common issues that I’ve found helpful to watch out for, along with context and practical alternatives:

1. Data Can’t Leave the Vendor’s System (Vendor Lock-In)

   1. Context: sometimes the vendor does not allow data to leave the platform once it has been ingested, or makes it very difficult to export or migrate the data. This is increasingly less common in enterprise, as these types of use cases almost always require connecting to or using multiple systems, but it does continue to happen.

   2. Practical alternatives: test the ability to export, migrate or integrate data sets upfront (e.g. via proof of concept), pressure-test the scale / volume at which this can be done, and negotiate the contract to enable this as needed.

Evaluating a contract AI vendor or portfolio company or building your own team? I provide select diligence and advisory consulting covering product architecture, model performance, workflow fit, hiring and implementation risk.

Subscribe now

Phrased another way: just because you can apply AI to a contract or other type of workflow, doesn’t mean you should. For example, earlier this year, Amazon reportedly incurred a cost overage of 1.8 million dollars related to a menial coding task using Anthropic’s models.

I previously wrote about AI’s evolving role in decoupling judgment from execution in the enterprise contracts space, including historical context, and introduced the value of breaking down a contract or other legal workflow end to end.

In this article I’ll share a practical framework for mapping out workflows and identifying the high-impact opportunities to apply AI.

Terminology

Here’s how I define a few key terms for purposes of this article:

• Judgment refers to domain expertise, tacit knowledge or experience in dealing with specific types of problems.

• Execution refers to completing work, guided by judgment. Another way of thinking about it: judgment is knowing what work is important to do (and when and how to do it), while execution is actually doing it.

• A task refers to a unit of work that needs to be completed. For example, finding a contract or researching a legal issue.

• A workflow refers to completing a series of related tasks connected together to achieve some broader goal.

Couple call-outs:

1. These terms can blend a little bit. For example, contract drafting can be a described as an overall workflow (if planning, research, analysis is included in that scope), but drafting a contract can also refer to a single task if referring to the specific step of writing or updating the contract document after completing any other necessary parts of the process.

2. I’ve noticed that as of late, the market sometimes refers to a workflow as using agents or other AI capabilities to complete one or more tasks. For purposes of this article, a workflow may or may not involve AI.

Breaking Down a Contract Workflow

Breaking down a workflow involves mapping out the entire process from beginning to end: the key stages, inputs and outputs, decision points, and other necessary steps to successfully move from one stage to the end.

In a workflow such as drafting a contract, this may look like: receive trigger or notification, perform intake / scope out problem, create plan, define criteria, perform research, summarize research, perform analysis of research, draft contract, share contract draft, etc.

Framework / Questionnaire

We can use a more general framework to break down similar types of workflows and identify the higher impact opportunities to apply AI. For those that want to do so, here’s a starter set of questions that I’ve found helpful:

Subscribe to get the full framework (it’s free).

Subscribe now

Hope this was helpful – if you have any feedback or requests for other walkthroughs, send me a note.

The Contract Signal is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Subscribe now

A core theme that came up, decoupling execution from judgment, is something that I first heard about back in late 2015 as I was preparing for my first role out of law school. I started working on an emerging M&A projects business at Axiom as an early hire, focused on turning large volumes of enterprise contracts into structured data and analytics (rebranded within a few years as contract intelligence).

The market has evolved significantly since then. AI has played a role in this evolution in four main ways:

1. AI is now able to cover or assist with a greater proportion of execution

2. AI has improved to the point where the line between judgment and execution has shifted (for some workflows)

3. Access to high-performing AI models has increased significantly

4. AI’s capabilities continue to evolve at a fast pace

However, before getting into AI’s current capabilities in greater detail and where the opportunities are to incorporate them into legal workflows, let’s contextualize what’s going on in the market with a historical look back into how those capabilities evolved and what the impact was.

Judgment vs. Execution 10 Years Ago

The original value prop of Axiom’s contract projects business was using people, process and technology to help enterprise clients review high volumes of contracts with high quality and consistency. The legal and business driver was usually M&A diligence and integration as those workflows were time-sensitive and required both efficiency and scale, though there were other use cases as well that benefited from a similar approach.

So if we were to break the market (at that time) down into the problem space, the competitive dynamics and the state of the art of decoupling judgment vs. execution, it looked something like this:

• Problem — the problem being solved for was largely the same as it is today: getting better visibility into large volumes of enterprise contracts that were often hard to find, let alone understand.

• Competitive dynamics — AI capabilities at the time were limited so the few providers that were trying to use AI to review contracts were able to sell its potential far better than they were able to deliver value. Phrased another way: hype exceeded reality.

  • More commonly, the law firm memo (a qualitative summary of the main issues found) was the main type of deliverable at the time. It was created using a judgment-focused approach with limited execution capabilities (from a high-level: throw groups of smart law firm associates or other junior resources at the contracts, bill by the hour). In addition, in terms of utility, while it had its place, it wasn’t structured data: it couldn’t be searched or analyzed at scale, particularly on an aggressive M&A timeline.

• The state of the art — the best-in-class approach in separating judgment from execution involved a combination of people, process and technology. This referred to a combination of contract analysts, subject matter experts and project managers using written playbooks created through specialization and experience in these types of projects, as well as eDiscovery software that had been specifically configured for the purpose of delivering these projects.

Judgment vs. Execution (Example)

Let’s use a concrete use case to illustrate how judgment and execution are different, specifically in the context of contract review.

For simplicity’s sake I’ll define judgment (otherwise known as domain expertise or tacit knowledge) as experience or expertise in dealing with specific types of problems, and execution as completing the work, guided by the judgment. Another way of thinking about it: judgment is knowing what work is important to do (and when and how to do it), while execution is actually doing it.

The line between the two can be blurry, however. For example, with some training, a contract analyst could take a contract clause and identify it as being an assignment clause, an indemnification clause, a renewal clause, a governing law clause, etc. They could also likely categorize it further, for example:

• Whether the assignment clause included consent or notice requirements,

• Whether the indemnification is one-sided or two-sided,

• Whether the renewal clause is an automatic renewal, an optional renewal or something else, or

• What the specific geography of the governing law clause was.

However, it would become more challenging for the analyst, if (without any legal training) they were asked to opine on whether a given clause was enforceable. Or the overall risk profile of a clause or entire contract. Or whether a clause draft or position should be re-negotiated as a result.

So in that sense, while each of the tasks required some level of judgment and execution, the types and levels of judgment required varied depending on the stage in the process and the corresponding complexity of the problems involved. Concretely, the initial training would likely need to be done by a more experienced subject matter expert (in other words, the training would involve more judgment). Applying the training to identify, capture and categorize contract clauses would require some judgment as well though less so, and would involve more execution.

AI and Judgment

However, as different types of judgment are applied repeatedly (in other words executed, specifically for more common, standardized tasks or workflows), they can be codified as context (for example via playbooks or taxonomies) and data.

In that sense, a workflow that was at one time judgment-focused can become execution-focused over time as it becomes more frequently used and codified.

And once they’ve been codified, different forms of AI, while not having judgment in the human sense are able to identify and apply those groups of data patterns and context to new situations.

When this happened for the contract review use case, contract AI rapidly improved (initially the branch of AI called machine learning applied to contracts). This was driven both by increased availability of these data sets and context as well as increased availability of more powerful, open source AI models and architectures that could better leverage the data and context to solve natural language problems (for example Google’s BERT).

These AI capabilities continued to evolve into Large Language Models (which power many of the products available in the market today) as well as more recent related capabilities like agentic AI.

Judgment, Execution and AI’s Role

The lower the complexity or level of judgment required to solve a particular task (in other words the simpler or more standardized the task), the more likely it is that the task can be codified in the form of data and context. The more relevant data and context available to train the AI models to solve it, the more likely it is that AI can assist with (or even automate) that workstream.

As a result, in current state, a contract analyst with the exact same level of skill as a contract analyst 10 years ago could more easily be replaced with AI. This is why continuous learning and upskilling is so important: these help build new types of judgment. For example, knowing how to use AI effectively has emerged as a new type of judgment that the market demands.

Incorporating AI into more experienced attorneys’ workflows is the logical next step as those workflows have been commonly applied and model capabilities have evolved over the last year or two and are more accessible than ever before. As before, the proportion of a workstream that was “judgment” has been restructured or reclassified as execution as a result of the consistent use and standardization of those tasks along with AI’s evolving capabilities.

So with all of that context in mind, what are examples of how we can actually separate out judgment from execution and related, use AI tools effectively?

Applying AI to Legal Workstreams

I would frame this in three main categories, from lowest level of effort to highest:

1. Using AI as a thought partner / to help plan a workflow

2. Breaking down the existing end-to-end process for performing a legal workflow into separate but related tasks and identifying where AI can assist

3. Re-engineering the process entirely from the ground up / from first principles

The first two ways above are the market norm right now, but the third is emerging as an area of exploration as well. In addition, performing the breakdown of the existing process is often helpful for understanding opportunities for all three categories.

AI as Thought Partner

This is one of the simplest, most popular ways of using AI in legal workflows.

Simply enter a prompt into an LLM-powered tool such as ChatGPT or Claude, and the AI tool can help you plan out a workstream (for example drafting a contract or researching a legal issue). This is similar to augmenting judgment with data.

If desired, a person can iterate on the plan with the AI tool or simply make adjustments as needed and execute on it.

Breaking down the existing end-to-end process

However, to get a more comprehensive view of where AI can fit into the process, it’s often helpful to systematically break down the end-to-end workflow or process into tasks, subtasks and related.

Here’s an example of a concrete legal workflow (contract drafting) and how we can break down the related end-to-end process this way. Two caveats:

1. This is a high-level, non-exhaustive example but serves to illustrate the overall technique for doing so.

2. The process is iterative and is not meant to be perfectly linear or sequential (for example going back to earlier stages or jumping around may make sense).

Contract Drafting

• Trigger or notification — notification of an event that requires contract drafting (for example a new client or opportunity)

• Intake / Scope out problem — structure the initial notification or request and request additional context as needed

• Create plan — create a plan for executing the workflow, including defining the scope of what you’re looking for and why

• Define criteria — define the concrete search conditions to help you find what you’re looking for

• Perform research — apply the search criteria to locate the relevant information

• Summarize research — condense the most important results from the search into a relevant summary (e.g. relevant legal issues that may be applicable)

• Analysis — validate the initial search results and summary, incorporate nuances, exceptions, and additional criteria, and perform a final assessment before moving to drafting

• Draft Contract — either select and tailor an existing template or if dealing with a brand new issue, start from scratch (less common)

• Share contract draft / communicate — send a draft of the contract along with any relevant information (for example a cover letter or email) to the counterparty or other stakeholder

We could then add additional steps if we wanted the workflow to cover the execution / signature of the contract as well:

• Negotiate

• Edit / refine the contract as needed (redlining)

• Get approvals as needed (for example if agreeing to non-standard terms)

• Sign / Execute the contract

Note that there are commonalities with other legal workflows as well: for example, drafting a legal memo. We could abstract the “draft contract” workflow into something like “draft or create deliverable” and then re-use / re-purpose steps in the process. We could also replace one task with another as needed: for example in the case of certain litigation documents, instead of signing the document we could also file it with the courts.

Applying AI to each step of the process

Once we break the process down end to end, we can assess where it would potentially be helpful to apply AI tools to improve the efficiency or quality of the workflow.

Here are a few examples of how AI could assist with each step of the above process:

• Trigger or notification — proactively perform regulatory monitoring to help inform legal and business planning. Here’s an example of how Amazon’s legal team is doing this internally.

• Intake / scope out the problem — identify known information, assumptions and where additional information would be helpful. Here’s an example of a Claude / ChatGPT skill I put together to help do this.

• Act as a thought partner / help form a plan — help form an execution plan (per category one above).

• Research — perform research into common legal issues that come up during contract drafting exercises of this type.

• Draft — draft the contract, using whatever context or template that we want to provide to do so.

In other words…on paper AI can assist with every single part of the workflow.

So the relevant question becomes should AI be used rather than can it be used?

The answer is: it depends.

How to determine whether to use AI

To answer the question, the next step is translating the conceptual framework into practical, problem-solving considerations. Note that there’s no one-size-fits-all solution and usually our specific situation and context will inform the approach.

However, here are some considerations that will inform whether it makes sense to apply AI to a given workflow or individual task:

• Financial costs — for example, costs of AI software licenses (both fixed and variable depending on usage).

• Vendor diligence / related costs — there are many AI providers and they need to be vetted. I put together a diligence checklist (available here) that can be used to help with this.

• Setup / onboarding time — learning how to use the tools, configuring them, etc.

• Opportunity cost (cost of time / best alternative) — time that could be spent doing something else (e.g. working on different matters).

• Benchmark of performance / current state — without using AI, how efficient and valuable is the existing process?

• Incremental performance improvement — how much better is the AI relative to current state or the best alternative?

• Maintenance costs — how difficult is it to maintain performance?

• Other considerations — e.g. aligning incentives to make sure efficiency doesn’t conflict with time-based pricing models (one common one: the billable hour).

Creating an entirely new process

Finally, as AI and other related technological capabilities become more advanced, there’s an open question regarding whether we can go beyond breaking down or refining an existing process by applying AI to individual tasks within that process.

Specifically, it raises the question whether we should completely change how we think about things and whether certain tasks or even entire workflows should exist at all.

Here’s a historical analogy: physically going to the library to do legal research was a common practice at one point. As legal research became digitized and sufficiently reliable relative to research done using physical copies, going to the library was eliminated as a practice in most cases.

Imagine if in a future state, a key research area, such as those underlying certain types of contracts became sufficiently clear to the point where research was eliminated as a necessity when drafting contracts.

Or during negotiation, what if the data and processes supporting negotiation playbooks and cycles became so standardized and predictable that two AI agents could essentially negotiate against one another and come up with the optimal outcome for both sides, and then communicate the results.

Ultimately as with the other examples of applying AI, we’ll need to assess the trade-offs of doing so. But we can imagine that some re-engineered processes are very impactful and happen sooner rather than later.

The Contract Signal is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Subscribe now

In a previous article I discussed the value of starting with a concrete problem or use case (in that case, a contract renewal question) and then turning it into a skill. To recap: the main benefit of doing so is creating a reusable, specialized set of instructions that can be re-purposed for similar tasks or workflows in the future.

In this walkthrough, I’ll go through the process of creating skills to solve that problem.

Problem / Scenario

Let’s say that you’re an in-house counsel, contracts manager or account executive. You’ve received an email from a client or from an internal stakeholder asking about a renewal for a specific contract.

Here are four common things someone might want to know about a renewal:

1. Is there renewal language at all?

2. Renewal type (automatic vs. optional)

3. Renewal timing (when is the contract coming up for renewal)

4. Counterparty outreach necessary (do I need to provide notice or do anything else and if so, when)

We can create one or more skills to handle this type of analysis.

What the skills do

The skill is broken into three components:

1. Finding the relevant renewal language in a contract

2. Performing the analysis - including classification of the renewal type and identification of the renewal date

3. Drafting an outreach email with the relevant language

How to use the skill

First, confirm that using LLMs and skills on your contracts is allowed by your organization. Typically companies require enterprise licenses as these have more robust information security and related safeguards.

Assuming you are allowed under your company’s policies, here’s the high-level setup process (see README file for more details):

• Download the skill files

• Create a new folder on your hard drive for the relevant project or workstream

• Copy the skill bundle to a local folder on your hard drive

• Open up Claude Cowork or ChatGPT / Codex

• Create a new project and add the relevant folder and files

• Alternatively, create a new chat and upload the files directly

Claude

Download and quick start

To download the skill and get the quick start guide, subscribe (it’s free):

Subscribe now

Codex

To download the skill and get the quick start guide, subscribe (it’s free):

Subscribe now

The Contract Signal is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

In addition, send me a note if there are additional skills or walkthroughs you would find helpful or if you have any other feedback or questions.

Subscribe now

The last few weeks have brought more AI announcements both focused on and tangential to the contract AI space.

Some of these have been significant, including another Big Tech entrant into the contract AI space (Amazon), foundation model upgrades and launches by OpenAI, Anthropic, and Google, a push towards more capable open source models, and some interesting partnerships.

Two trends worth watching:

1. The proliferation of large language models and related capabilities has reduced the barrier to entry on the AI modeling side. As increasingly capable models are made available to all firms, there will likely be an increase in the value of capabilities complementary to the models. These include end-to-end product experiences and integrations, data, and distribution channels. This is even more likely to be the case if the open source push gains traction and we see open source models at or near the frontier level.

2. Related, many companies seem to be adopting a combination of build, buy and partner approaches in their AI programs, including Contract AI and related workflows. These create interesting “frenemy” type dynamics that parallel similar historical trends in the broader technology sector.

With that, let’s dive in.

New Market Entrants

• Amazon launched Quick for Legal - an AI assistant that helps with legal research, contracts, and compliance. It appears that this is the same core AI assistant (Quick) but directed at legal use cases. Contract use cases mentioned include contract monitoring, clause extraction, risk flagging, and obligation tracking across portfolios. Amazon is a highly diversified company that includes AWS, the market-leading cloud business that supports a variety of businesses, including legaltech companies, with its infrastructure and related capabilities. This creates interesting market dynamics as it functions both as a key vendor for these companies as well as a potential competitor. However, this dynamic has existed in other industries as well (e.g. Amazon provides Cloud services and infrastructure for Netflix despite also having their own Amazon Prime Video offering).

• Anthropic, Blackstone, Hellman & Friedman launched Ode, an enterprise AI services firm. While the announcements have not directly mentioned contract AI or legaltech, it appears that the firm will support enterprises across industries. Given Anthropic’s launch of Claude for Legal, and increasing demand for both in-house departments and service providers to incorporate AI into their workflows, it will be interesting to see how much investment the firm makes in contract AI and legaltech. My sense from the announcements so far is that the firm’s focus will be untapped AI opportunities in other sectors.

• Why it matters: contracts and other legal use cases are great applications for large language models given the business and legal information expressed in

  high volumes of natural language. The availability of increasingly capable LLMs has not only reduced the barrier to entry into the space but has increased the value of capabilities complementary to the models themselves. In Amazon’s case, its vast infrastructure and ability to serve up models securely through Amazon Bedrock has become a significant competitive advantage.

Foundation Model Updates

• OpenAI released the GPT 5.6 family of models, including Sol, Terra and Luna. The model capabilities have been well-received by the market and are comparable in performance to Anthropic’s flagship Fable model.

• Anthropic released Opus 5, a model it says comes close to the performance level of Fable and is twice as efficient. Fable continues to be available for Max and Enterprise plans and via separate usage credits for other plans.

• Google released updates to 3.6 Flash, 3.5 Flash-Lite and 3.5 Flash Cyber. While not as powerful as its flagship model Gemini Pro, each promises increased token efficiency. In addition, Flash Cyber has improved cybersecurity applications.

• Jensen Huang published an open letter on X discussing the importance of open models. The letter was co-signed by various companies, most notably OpenAI, Microsoft, and Perplexity, each of which provides access to their own proprietary AI models.

• Why it matters: these four announcements are the latest examples of increased foundation model performance and competition. This is likely to produce better outcomes for consumers of AI models over time and enable more, better applications. However, there’s also a lingering question regarding whether any of these models will replace more specialized AI or other software as the frontier models become more capable. So far the doomsday predictions of a “SaaSpocalypse” have been overblown. My sense is that in the near term, we’ll continue to see a range of successful SaaS software providers, but there will be increased pressure for vendors to provide more value relative to general purpose LLMs such as Claude and ChatGPT.

Product Announcements and Related Updates

• Litera has announced a re-launch of the company focused around a single AI agent. Based on the announcement, the goal is to provide a single entry point into multiple workflows and use cases. In addition, the company mentioned that it has been able to develop a more deterministic redlining tool, leveraging its years of experience.

• Microsoft’s partnerships with different contract AI and legaltech companies including Harvey and LexisNexis have been in the news lately. This is in line with Microsoft’s vast ecosystem and adoption by lawyers, including Microsoft Office 365.

• Harvey, the AI-native legaltech company, has acquired Benchmark, a decision-infrastructure startup. Benchmark provides tools for asset management firms to better leverage their institutional data in decision-making.

• Box has announced more rigorous security and governance controls for its enterprise-focused intelligent content management system. Over the last few years, Box has expanded its offerings to serve enterprise customers, including in the contracts space.

• Why it matters: there is ongoing pressure to offer an end-to-end product experience, whether through building in-house, partnering or acquiring start-ups or growth companies. Box provides an example of enhancing its own offerings to account for growing demand for responsible AI use and governance. However, building everything in-house is not a sustainable route for large companies, in part due to the many competing demands on company time. Acquisitions of complementary tools or channel partnerships between large incumbents with large user bases and start-ups or growth companies with new capabilities are common alternatives. Another example: Microsoft acqui-hired the tech team from Robin AI in early 2026.

Other Highlights

Funding

• Norm Ai, the agentic AI company, has raised $120M at a $1.2B valuation. While not focused on contract AI, it does support contract workflows as part of its offerings.

• Why it matters: legal AI continues to be a hot space in the investment community, even with advancements in foundation models.

The Contract Signal is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Subscribe now

An approach I’ve found helpful to address the information overload and FOMO is to focus on a smaller volume of high-impact concepts, one at a time, organized by concrete problems and use cases. Here’s an example.

Problem / Scenario

Let’s say that you’re an in-house counsel, contracts manager or account executive. You’ve received an email from a client or from an internal stakeholder asking about a renewal for a specific contract.

Here are four common things someone might want to know about a renewal:

1. Is there renewal language at all?

2. Renewal type (automatic vs. optional)

3. Renewal timing (when is the contract coming up for renewal)

4. Counterparty outreach necessary (do I need to provide notice or do anything else and if so, when)

Solving the Problem

One approach is to search the repositories you have access to for the contract using whatever criteria is available. Or if you don’t have access to all of the repositories, sending a note around internally to people who might know. If it’s a customer contract, sending a note to sales, if it’s a vendor contract, sending a note to procurement, etc.

For a single request, we also create a chat with an LLM to help us. E.g. ping Claude or ChatGPT, provide some context, and then move forward with the search.

There’s nothing wrong with these approaches for a single request, but they don’t scale well.

For many requests, for example if you get dozens or hundreds of similar requests, you would probably benefit from a more methodical approach.

One approach is to compile a rough estimate of the types of requests you receive in a given week (or month / quarter / year). Which are the common ones, the most cumbersome ones or the ones where it feels like you’re doing the most duplicative work?

For these we can create a skill, which is essentially a reusable instruction that reduces the need to repeatedly prompt an LLM-powered chatbot for the same thing.

• See my article here for a specific example that is useful for managing contract intelligence requests.

The prompt doesn’t need to be over-engineered: for example for our renewal language you can start by simply asking “can you help me put together a skill to find renewal language in contracts”? Then you can provide additional context as needed and check the AI tool’s work.

This works in both ChatGPT and Claude, two of the most popular LLM tools. You can also go to https://chatgpt.com/skills to manage skills there. Please follow your organization’s policies on AI use regarding which tools and context are appropriate.

Summary

• Don’t worry about learning every AI concept or about AI FOMO

• Start with a concrete problem or use case

• Determine whether it’s a recurring or duplicative task

• If it’s truly a one-off task, use a regular chat

• If it’s recurring or involves significant duplicative work, create a skill by prompting the LLM

• Check the LLM’s work

The Contract Signal is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Subscribe now

A former colleague told me this in 2020, describing a failed AI project. I was interviewing them for any wisdom they could share as I stepped into a product manager role for our machine learning program. They had engaged a third-party provider with best-in-class technical credentials and a great demo — and had virtually nothing to show for it beyond spent time, effort, and money.

Unfortunately, this is not an isolated incident in enterprise AI. I’ve seen the same themes surface again and again when doing diligence on third-party providers, working alongside teams of AI engineers and data scientists, running my own experiments with AI tools, and following the industry’s research and best practices.

The good news is most of these problems could have been prevented (or at least significantly reduced) with better diligence before a purchase or investment decision is made.

That is the guiding principle behind this questionnaire: help separate fact from fiction by examining the vendor’s end-to-end process. What problem it solves, how contract data is produced, how performance is measured, how results are verified, how users consume the deliverable and whether the underlying business and technology stack can support the promised outcome.

One call-out - when using the questionnaire, do not rely only on verbal answers. Wherever possible, request written evidence: sample outputs, citations, test methodology, data-flow diagrams, sandbox access, implementation plans, customer references and contractual quality guarantees or other commitments. These are often more substantive and more reliable than verbal responses alone.

Subscribe to get the full checklist (it’s free).

Subscribe now

In a previous article I discussed how every post-signature enterprise contract workstream, big or small, can be framed as the same underlying six-part process:

1. Trigger — an event occurs that requires contracts work

2. Define criteria — determine what you’re looking for and why

3. Find — apply the criteria to locate the target population of contracts

4. Analyze — go deeper on the population: validate the initial search results, incorporate nuances, exceptions, and additional criteria, and perform a comprehensive assessment

5. Decide — translate the analysis into recommendations, remediation paths, answers to stakeholder questions and other internal deliverables

6. Remediate — take action through counterparty outreach, internal communication, negotiation, and workflow tracking

This article focuses on creating a skill to support the first phase: Trigger / Intake.

• One call-out re terminology: the trigger is the event or request (usually outside of our control). Intake is the process of turning that event or request into enough structured context to begin planning the work.

Problem / Scenario

Imagine receiving this hypothetical request for M&A diligence:

Quick update: we signed an LOI to acquire ABC corporation and expect to close soon. Please start reviewing the top 100 customer contracts (will send a data-room link once available).

It can be tempting to get access to the repository and immediately start reviewing contracts. However, it’s usually better to scope out what the request is asking for in greater detail along with any available context to help inform execution.

What the skill does

The intake skill acts as a thought partner that helps clarify and understand an inbound request for contracts analysis in sufficient depth to effectively begin planning and execution.

If you provide context to the skill, it helps produce a structured intake brief that:

• Classifies the trigger or request

• Identifies the decision or deliverable that the contract analysis needs to support

• Translates business context into relevant scope (contract populations, data points, etc.)

• Flags missing inputs, assumptions, unknowns, and potential blockers

• Generates recommended next steps, including what can begin now versus what should wait and related scope-change risks

Two ways to use the skill

Option 1: Provide the underlying context directly

If your organization’s AI use, information security and related policies permit it, you can provide the relevant context directly to the model.

Option 2: Use the skill to generate an intake checklist

Where providing the underlying context directly is not appropriate, the skill can instead generate a structured checklist of the information that would be helpful to collect.

You can then work through that checklist separately and return only the information that is appropriate to provide. This is less direct, but it can still help structure an incomplete request and identify the inputs most likely to affect scope or execution.

Limitations

The skill’s goal is to create a better starting point for later stages: defining the final scope of the work, performing contract analysis, making decisions, and perform remediation actions. However, its output should be checked and confirmed as being correct by the relevant stakeholders.

Subscriber resources

Subscribers receive:

• The complete intake-contract-workstream skill bundle for each of Claude and Codex

• Installation instructions

• A practical M&A example (hypothetical)

• Future updates to the intake framework and request category modules

The clock starts running on the contracts workstream.

You’re juggling a handful of other tasks. You’re being told to use AI. Life outside of work doesn’t stop either.

So how do you proceed?

Almost every post-signature enterprise contract workstream, big or small, can be framed as the same underlying process. It doesn’t matter whether it’s M&A diligence, regulatory remediation, a backlog review or a one-off question from sales.

In this article I’ll break that process into six pieces and walk through each one, using M&A as the core example.

The six pieces are:

1. Trigger — an event occurs that requires contracts work

2. Define criteria — determine what you’re looking for and why

3. Find — apply the criteria to locate the target population of contracts

4. Analyze — go deeper on the population: validate the initial search results, incorporate nuances, exceptions, and additional criteria, and perform a comprehensive assessment

5. Decide — translate the analysis into recommendations, remediation paths, answers to stakeholder questions and other internal deliverables

6. Remediate — take action through counterparty outreach, internal communication, negotiation, and workflow tracking

The details may vary significantly from one project or task to another but the underlying process will remain fairly consistent.

Subscribe now

1. Trigger

There are all sorts of possible triggers, but almost all of them fall into three main buckets:

1. External events - something outside the normal course of business creates the need for contract work.

   1. Two of the biggest examples are:

      • M&A (buy-side, sell-side, reorgs, etc.)

      • Regulatory remediation

2. Internal initiatives - the business identifies an internal need.

   1. The classic example is a backlog or legacy review: getting better visibility into a large historical contract population, identifying risks and opportunities, or solving for one or more specific use cases

   2. These projects can be very valuable, but they are often perceived as nice to have and easier to defer than an acquisition or regulatory deadline

3. Business as usual - the steady stream of recurring and one-off questions from various business stakeholders. For example:

   • Sales is negotiating a renewal and wants to understand the existing agreement

   • Finance is asking about deliverable timing or pricing terms

   • Product wants to know what AI or data-use rights the company has

   • Legal wants to identify risk allocation terms such as indemnification

2. Define criteria

Before searching for contracts or specific clauses, you need to answer two interrelated but different questions:

1. Which contracts are in scope?

2. What do we need to know about them?

I think of these as search criteria (aka population criteria) and analysis criteria.

Search criteria: which contracts matter?

For an M&A transaction, examples of the relevant population include:

• The top 100 or 200 customer contracts by revenue

• All active vendor agreements above a certain spend threshold

• Agreements involving a particular legal entity

The initial scope or data may come from a variety of places: for example the finance team, the seller, the M&A team or a data room.

This is where metadata that is use-case-agnostic often becomes essential: document title, party names, dates, contract type and similar information.

Analysis criteria: what do we need to know?

The second question is more substantive and depends on the use case. For M&A, common areas of interest include:

• Assignment restrictions

• Change-of-control provisions

• Consent and notice requirements

• Termination rights

• Governing law

• Restrictive covenants

• Liability and indemnification provisions

These provisions help determine what to do with the contract as a result of the transaction, what counterparty outreach and other actions may be required, what risks the acquirer inherits, and the trade-offs of different approaches.

For data privacy or regulatory work, the criteria might instead include:

• Does the required privacy language exist?

• Does it satisfy the relevant regulatory standard?

• Where are the parties located?

• Where is data stored or processed?

• Are sub-processing or cross-border transfers permitted?

For business-as-usual questions, the search criteria and analysis criteria may merge. “Find the contract with Customer X and tell me whether they can terminate for convenience” already contains both the search criteria and the substantive question. In addition, sometimes simply finding the relevant agreement is enough.

There are also common secondary filters: these include things like active versus expired contracts, upcoming renewals, geography, contract value, business unit. Note that the relevance of each of these depends on the use case. An expired contract might be irrelevant to an assignability analysis but highly useful when someone asks, “What language did we agree to last time?” or “Do we already have an agreement with this counterparty?”

One final point: flagging missing information is often valuable in and of itself. In an M&A context, an inability to locate critical customer agreements or a lack of certain clause types can affect the risk profile of both the contracts and the deal.

3. Find

Once the criteria are defined, the next question is: How do you identify the relevant contracts?

The answer depends on both what you’re looking for and what information you have to work with. There are two common starting points: a list of contracts or a data dump.

List of Contracts

For example the M&A team or the seller provides the top customers by name.

In that case, finding the relevant contracts is a matching exercise: compare the list against the party names in the contracts. If there’s a match, it generally means the contract is in scope. The comparison can run at several levels of strictness but the core concept is the same:

• Exact legal-entity match

• Approximate name match / fuzzy match (e.g. Acme Corporation vs. Acme Corp.)

• Affiliate or corporate-family match

One reason the specific matching logic used is important is because contracts are entered into by legal entities, while the business often thinks about contracts in terms of broader customer or vendor relationships. So the key is finding the balance between matches that are exact enough to avoid too many irrelevant documents (false positives) while flexible enough to avoid missing any legitimate ones (false negatives).

Data Dump

Alternatively, you may receive a data room full of documents described as “top customer contracts,” with inconsistent naming conventions and organization. In this case, there are multiple paths forward.

You can first identify parties and other metadata to organize the population.

Or you can begin with substantive clause or language-based filtering, for example:

• Which contracts restrict assignment?

• Which require consent?

• Which require notice only?

• Which provide termination rights in connection with the transaction?

In practice, these approaches often work best together. The list helps define the initial population and the follow-on searches help pressure test, refine the scope of, and prioritize the population of interest, as needed.

4. Analyze

The line between finding a contract and analyzing it isn’t always clear. For example, flagging a contract for assignment restrictions is already a form of analysis.

But the analyze stage is where you go deeper, and it has four layers:

1. Validate the search results

2. Incorporate exceptions, carveouts and nuances

3. Expand the criteria

4. Synthesize

Validate

First, confirm that the results actually match your intent.

For example, if a system identifies 75 agreements as containing assignment restrictions, do they contain assignment restrictions relevant to the transaction?

A clause may contain the word “assignment” but address the counterparty’s assignment rights rather than yours. It may apply only to certain rights or obligations. It may not directly use the word “assignment” or “assign” at all. Or it may use the same keywords but be completely irrelevant – for example referring to a work assignment rather than the legal transfer of an asset.

Key point: the same keywords may have varying semantic meanings or significance.

Incorporate exceptions, carveouts and nuances

Next, examine the exceptions, carveouts, and situation-specific facts that can change the initial categorization.

Suppose a contract says consent is required for assignment, unless the agreement is assigned to an affiliate.

If the deal structure involves assignment to an affiliate, the remediation path for that agreement may move from “consent required” to a completely different remediation path such as “courtesy notice”.

Or consider a clause permitting assignment in connection with a merger, sale of substantially all assets, or similar transaction. Whether the exception applies depends on both the contractual language and the structure of the deal.

This is why the contract and the business context cannot be separated.

Expand

Finally, the analysis often expands beyond the original criteria. You may begin by looking for assignment restrictions. But once you understand the contract population and transaction structure, additional questions emerge.

For M&A, examples include:

• Change of control

  • Assignment and change of control belong to the same general umbrella of concepts and are often analyzed together, but they are not identical.

  • The specific transaction structure may trigger a change-of-control provision even where no contractual assignment occurs.

• Termination rights

  • Termination rights (especially for convenience or a result of corporate transactions) can significantly impact the risk/opportunity profile of a contract

    • If a customer can terminate a significant revenue contract at will, that may represent risk to the acquirer.

    • If the acquired company has a broad right to terminate a vendor agreement, that may create flexibility: the acquirer can potentially exit an unfavorable contract, consolidate vendors to capture deal synergies, or simply use the credible possibility of termination as leverage in renegotiating commercial terms.

• Limitation of liability and indemnification

  • Even assuming the same contractual terms, the significance of contractual risk can change after an acquisition. A smaller company may have operated for years under a contract with unusually broad liability exposure without a dispute ever arising. But when a much larger and better-capitalized company (i.e. a better litigation target if something goes wrong) acquires the business or assumes the agreement, the practical risk assessment may change.

  • That does not automatically mean every contract with unfavorable liability terms must be terminated or renegotiated. But it may change the priority assigned to the contract and the remediation decision.

• Restrictive covenants

  • Various commercial restrictions may conflict with the strategic rationale for the transaction. Examples include: exclusivity language, most-favored-nation clauses, non-competes, non-solicitation and subcontracting restrictions.

  • Imagine acquiring a company specifically to enter a new market, only to discover that a material contract limits the company’s ability to compete, distribute products, work with certain customers, or use particular subcontractors.

Synthesis

At the end of the analysis stage, the output should be more than extracted data.

Net of all of considerations, one or more internal deliverables is created: a spreadsheet, report or something else showing the analysis along with the suggested remediation path and reasoning. For example, if 100 contracts were in scope:

• 10 may be expired

• 20 may require assignment consent

• 40 may require notice

• 10 may require an amendment or negotiation due to risk concerns

• 20 should be left alone - perhaps because they’re undesirable contracts coming up for expiration before deal close or because they’re with related entities and don’t require additional action

5. Decide

Before anyone reaches out internally or externally, there’s a need for a decision.

For an M&A transaction, that often means classifying contracts into different remediation paths: assign, exit/terminate, renegotiate, renew, or leave alone.

In addition, more significant issues may also affect the transaction itself. Diligence and integration are often gray areas rather than cleanly separated phases. The significance of a contractual issue depends on when it is discovered, the magnitude of the exposure, the importance of the contract, and the broader rationale for the acquisition. A single problematic contract does not necessarily jeopardize a deal, but a population-level pattern might.

For example, perhaps a large percentage of the target’s revenue is tied to contracts that can be terminated for any reason. Or critical agreements cannot be transferred without consent. Perhaps restrictive covenants interfere with the strategic rationale for the acquisition or the company simply does not have enough durable customer contracts to provide the market beachhead the acquirer expected. Perhaps uncapped liability exists in critical places.

All of these considerations are ways that contract data can impact the overall deal.

6. Remediate (Take Action)

Once the decisions are made, it is time to act.

For M&A and other episodic projects, this often means contract remediation and any related counterparty outreach:

• Requesting consents

• Sending legally required notices or courtesy notices where appropriate

• Negotiating where counterparty consent is difficult to obtain

• Amending agreements as needed before assignment

• Terminating and/or entering into new contracts where necessary

All of this requires a significant workflow tracking component as well, including:

• Internal communication - legal, sales, finance, procurement, integration teams, business owners, and senior leadership may each need different views of the same underlying analysis and remediation.

• Tracking the remediation results on a contract by contract basis - which consents are outstanding, which notices went out, which contracts are stuck, what’s blocking or impacting the deal close, etc.

Recap

Almost every post-signature enterprise contracts project can be framed as the same underlying 6-part process:

1. Trigger — an event occurs that requires contracts work

2. Define criteria — determine what you’re looking for and why

3. Find — apply the criteria to locate the target population of contracts

4. Analyze — go deeper on the population: validate the initial search results, incorporate nuances, exceptions, and additional criteria, and perform a comprehensive assessment

5. Decide — translate the analysis into recommendations, remediation paths, answers to stakeholder questions and other internal deliverables

6. Remediate — take action through counterparty outreach, internal communication, negotiation, and workflow tracking

Hopefully the above helps illustrate that post-signature contract intelligence is a multi-faceted problem and that each step in the process has its own challenges and nuances.

That leads to an interesting question: how much of each stage can AI actually help with? Update: I wrote a follow-up article here that discusses this. Or more specifically: at each stage, from trigger to remediation, what can AI reliably do, what inputs does it need, what are the alternatives and trade-offs, and what are the complementary capabilities that need to exist around the model to get the best results?

If you found this article helpful, subscribe to follow along and forward along to anyone who might find this useful.

Subscribe now

Imagine you have a $1 million customer contract.

Let’s assume it has a five-year term and is worth roughly $200,000 per year.

The contract also has a renewal option, but the renewal window is missed and the agreement expires.

This is an enterprise customer, so continuing business with the customer requires getting a new contract in place. This in turn means re-entering the contracting cycle: budget approvals, procurement, information security and legal review, etc.

Let’s suppose the customer renews without major issues but the delay pushes out a quarter of revenue.

That equates to roughly $50,000.

Now imagine the root cause was not an unsuccessful negotiation, a missed email, or a mistake by the account team.

Imagine it was this: the contract system had the wrong renewal date because it relied on the wrong effective date. The effective date was wrong because the OCR misread the scanned contract.

This is a hypothetical but a plausible one. It’s something that becomes possible when post-signature contracts are converted into structured data at enterprise scale: hundreds of thousands of contracts spanning many contract types, use cases and internal systems.

For a deeper breakdown of post-executed enterprise contract intelligence and complexity, see the previous article here and video here.

There are three common examples of OCR failures. But before getting into them, we need to unpack why OCR matters for enterprise contracts to begin with.

Subscribe now

Why OCR Is Important: The model often does not see your contract

Enterprise contracts are often stored as PDFs. But a PDF is essentially just a container; the contents of the container (and how clean or machine-readable it is) varies widely.

Some PDFs are text-native. Some are image-only scans. Some have OCR layers that are incomplete or otherwise incorrect.

If the PDF is text-native, a system may be able to extract the embedded text directly. If it is a scan, OCR has to create the text layer. Once that text layer exists, many downstream extraction and classification workflows treat it as the relevant contractual text. This is crucial to understand because a large language model usually has no built-in way to know whether the extracted text is consistent with the original page.

In other words: OCR quality can determine what the AI system thinks the contract says. So when the OCR layer is wrong, the model’s input is wrong. And a scanned contract can be perfectly legible to a human reviewer and still be unreliable for a contract AI system.

With all that in mind, there are three OCR failure modes worth separating based on both the practical impact they have and how difficult they are to spot:

1. No text layer exists

2. Text layer exists but is damaged

3. Silent failure

Failure mode 1: there is no text layer

The simplest failure is when the document has no usable text layer at all.

This happens when the contract is an image-only PDF, when OCR was never run, or when some file-quality issue prevented OCR from producing usable text.

To a human reviewer, the document may look perfectly readable:

To a text-based extraction pipeline, it may be a blank page.

No usable text layer means no reliable text input — at least not without additional preprocessing.

This failure mode is annoying, but relatively straightforward. You can detect that there is no text, no extracted content, or no meaningful input for the model to use.

The harder cases come when OCR did run.

Failure mode 2: the text layer exists, but parts of it are damaged

The more common case is that OCR ran and got enough of the document right that the problem is easy to miss.

Two things make this difficult.

First, OCR quality is not always consistent within a single document or even within the same page. Some parts of the document can have clean, easily readable text and others can be damaged.

Second, whether any damage actually matters depends on the use case. A model may be able to classify a clause correctly even when the text contains small errors. But those same errors may be fatal if the task requires exactness. The challenge is that many contract use cases do require exactness, as we’ll see.

Here are excerpts from the same document with significantly different OCR results.

The indemnification language came through cleanly.

Screenshot:

OCR:

(C) Each Party agrees to defend, protect, indemnify and hold harmless each other Party from and against all claims and demands, including any action or proceeding brought thereon, and all costs, losses, expenses and liabilities of any kind relating thereto, including reasonable attorneys fees and cost of suit, arising out of or resulting from any construction activities performed or authorized by such indemnifying Party; provided however, that the foregoing shall not be applicable to either events or circumstances caused by the negligence or willful act or omission of such indemnified Party, its licensees, concessionaires, agents, servants, employees, or anyone claiming by, through, or under any of them, or claims covered by the release set forth in 5.4(D).

But a few pages away, the term language picked up OCR errors:

ARTICLE VII

TERM

Screenshot:

OCR:

7.1 Term of this OEA. This OEA shall be effective as of the date first above written and shan continue in full force and effect until 11:59 p.m. on December 31, 2049; provided, however, that the easements referred to in Article II hereof which are specified as being perpetual or as continuing beyond the term of this OEA shall continue in force and effect as provided therein. Upon termination of this OEA, an rights and privileges derived from and all duties and obligations created and imposed by the provisions of this OEA, except as relates to the easements mentioned above, shan terminate and have no further force or effect; provided, however, that the termination of this OEA shan not limit or affect any remedy at law or in equity that a Party may have against any other Party with respect to any liability or obligation arising or to be performed under this OEA prior to the date of such termination.

“Shan continue in full force and effect” is damaged text, but it is still recognizably term language.

So if the task is clause classification — does this document contain a term provision? — the OCR may still be good enough for the model to classify it correctly. The model has plenty of semantic context to work with.

But contrast that with data points where exactness is crucial: effective dates, expiration dates, signature dates, notice periods, renewal deadlines, monetary amounts and party names.

Here is a preamble where the date was handwritten into a blank:

The OCR output looked like this:

THIS AMENDED AND RESTATED OPERATION AND EASEMENT AGREEMENT (”OEA”) is made and entered into as of the a<g day of (j)Q.\uber, 1999, between DAYTON HUDSON CORPORATION, a Minnesota corporation (”Target”) and Northway Mall Associates, a New York partnership (”Developer”).

The handwriting distorted the OCR, and the date came out as junk. A date that cannot be parsed is not usable.

Interestingly, running the same page through a different OCR engine produced a completely different — and correct — result:

THIS AMENDED AND RESTATED OPERATION AND EASEMENT AGREEMENT (”OEA”) is made and entered into as of the 28 day of October, 1999, between DAYTON HUDSON CORPORATION, a Minnesota corporation (”Target”) and Northway Mall Associates, a New York partnership (”Developer”).

Same page. Same underlying contract. Two different OCR outputs. One incorrect, one correct.

This is not a rare edge case. Post-executed enterprise contract repositories often contain scans, handwritten dates, signatures or notes, and documents of widely varying age and quality. OCR quality varies by engine (the type of OCR software), by file type, by scan quality, by page, and sometimes by individual field.

That is a recurring theme regarding why post-executed enterprise contract intelligence is so hard – the multi-step, document-to-data pipeline. Model selection is just one step.

Still, in this second failure mode, the damage is at least reasonably easy to spot: “Shan” is not a word and “a<g day of (j)Q.\uber” is not a date.

A QC layer, parser diagnostics and observability, OCR confidence scores, or even a careful human glance at the extracted text may flag the problem.

Which brings us to the failure mode that is the most difficult to catch.

Failure mode 3: the text is plausible — and wrong (aka silent failure)

The most dangerous OCR failure is when the output looks clean.

No garbage characters. No obvious misspellings. No broken date format.

Just a value that happens to be incorrect.

A handwritten 7 gets read as a 1. A 3 gets read as an 8. A $100,000 value gets misread as $1,000,000 — or the other way around.

Here is a real example — a signature block with a handwritten date:

The OCR output appears clean:

AGREED AND ACCEPTED:

PLITT THEATRES, INC.

By: [Signature] Henry Plitt, Chairman of the Board

Date: Oct 12 1982

But look closely at the image. The day appears to be 13, not 12.

The likely reason is that the lower part of the handwritten digit overlapped with the signature line, causing the OCR to read the 3 as a 2.

This situation is much more dangerous than junk text.

Junk text tells you something went wrong so you can course correct as needed.

A date that is plausible but wrong does not throw up any red flags. In this case, “Oct 12 1982” is a valid date – it has the right format and is semantically plausible.

In other words, it is a silent failure.

And silent failures are the ones that create the most risk in contract AI systems.

Why small OCR errors become business problems

You might look at the example above and say – it’s only one day off, that won’t have much impact.

Maybe. But I’d call out two things:

1. Enterprise contracts often have more significant silent OCR failures such as a wrong month or wrong year (note that the example above is from a publicly available contract). This becomes more likely as hundreds of thousands of documents present many opportunities for silent OCR failures.

2. The broader point, however, is that a single, seemingly innocuous bad value rarely stays contained to one situation or use case and can have downstream consequences.

For example, a renewal date or expiration date may not be written explicitly in the document. It may be computed from other extracted fields: effective date plus initial term, potentially plus a renewal period minus a notice window.

If OCR silently corrupts the effective date, the computed expiration date may be wrong.

If OCR silently corrupts the term length, the renewal calculation may be wrong.

If OCR silently corrupts the notice period, the counterparty outreach or operational deadline may be wrong.

If OCR silently corrupts the contract value, the agreement may be routed to the wrong workflow, wrongfully excluded from a report, or treated as below a threshold that should have triggered additional review.

And the model may have no way to know, because every input it was handed looked valid on the surface.

That is how a small OCR problem becomes a contract intelligence problem.

This is also why scale changes the risk profile. In a small set of contracts, these errors may be annoying but manageable. In an enterprise repository with tens or hundreds of thousands of documents, even a low error rate can create meaningful downstream exposure.

OCR quality is not just a preprocessing issue. It is a contract data quality issue.

What can be done

There are various approaches to either reducing or mitigating issues that can come up from OCR or file quality issues. This discussion could easily be its own article.

At the OCR layer - some common approaches involve checking for existing OCR, re-running OCR software, testing different engines and capturing OCR confidence where available.

At the extraction layer - high-impact contracts or fields can be routed into more robust QC workflows.

At the QC or validation layer - rules can catch what is catchable: for example values outside plausible ranges.

In addition, evolving AI model capabilities may help with some of these as well.

Recap and the open question

To recap, there are three OCR failure modes worth separating:

1. No text layer at all — the document is readable to a human but may be blank or illegible to a text-based extraction system. This is usually relatively easy to detect by a well-designed system.

2. Visible OCR damage — the text layer exists, but contains obvious problems like non-words, junk dates, or corrupted phrases. This can also often be detected, and whether it matters depends on the use case.

3. Silent OCR damage — the text looks plausible, but is wrong. This is the most dangerous because it can flow through the pipeline undetected.

The open question is empirical: in a real population of post-executed enterprise contracts, how often does each failure mode occur? And how much does each one affect downstream contract AI accuracy?

That is a type of measurement I have not seen yet — and one that matters if contract AI is going to move from impressive demos to trusted enterprise systems.

Because the real issue is not simply whether an AI model is powerful enough.

It is whether the system has given the model the right contractual text to read in the first place.

If you found this useful, subscribe to The Contract Signal — and forward it to anyone who might find it helpful.

Subscribe now

1. Post-signature – these are contracts that have already been signed or executed, otherwise known as post-executed contracts, as compared to pre-execution or pre-signature contracts that have been drafted, negotiated or redlined but not signed by all parties.

2. Enterprise – at least one party to the contract is a very large business, such as a F500 company.

3. Contract Intelligence – enabling people to make legal and business decisions by turning contracts into structured data that can be searched, filtered, reported on, and analyzed at scale.

Let’s dig into this in greater depth – including how these concepts influence one another.

Prefer to watch? Here's the video version:

Subscribe now

Deeper Dive

Post-signature

First, we should dig into the implications of what it actually means for a contract to be post-signature.

Contracts are written agreements containing the terms of an exchange, including legal obligations on both sides.

Post-signature means the contract is already signed. This means we’re past the drafting stage, negotiation, and redlining — the language has been finalized and agreed to. In other words, pending an amendment to the original contract or another legal mechanism, the language is locked, and the rights and obligations in effect.

Enterprise

Being a large enterprise has many significant implications for contracts as well.

Range and volume of contract types

One implication is that the volume of contracts is much, much higher for large enterprises than it is for smaller businesses.

This happens for a few reasons.

First, enterprises are large, diversified companies, often with multiple business units, many customers and vendors, many products and services, and a more complicated supply chain. For example, everything ranging from the need to manufacture things to the need to distribute products and services at scale means that there are many more significant business exchanges and many corresponding opportunities for contracts across a range of contract types.

These may include manufacturing agreements, reseller or distribution agreements, licensing agreements, partnership agreements, leases, and other types of customer and vendor contracts.

Contract Families

Another implication is that because enterprises have been around for a long time, they have business relationships and contracts from many different periods.

This includes older contracts with longer lifecycles, often including multiple rounds of contracting between entities. These relationships may include more than just an initial master agreement. They may also include NDAs, amendments, statements of work, order forms, addenda, and other related documents.

As a result, large enterprises can have very large families of related contracts.

Complexity of individual contracts

Relatedly, the complexity of the contracts themselves tends to be very high for enterprises. Large enterprises have all sorts of complicated requirements built into their contracting practices.

This is partly due to being subject to different regulations as a result of their broad reach of business activity, often spanning geographies and business lines. Also, because they have been in business for a long time, different laws, rules and conventions may have applied over different periods of time.

It’s also because they are at greater risk of litigation, leading to more robust risk allocation measures. This is driven in part by increased opportunities for something to go wrong, given their higher levels of business activity and transaction volume. It is also driven by the fact that large enterprises often have more money and assets to pursue, making them more attractive litigation targets relative to smaller entities.

As a result, they often have to be more aggressive about risk allocation. This can lead to lengthier, more detailed provisions that limit liability exposure in different circumstances.

Net of these considerations, the population of contracts at a given enterprise is highly mixed. It is not homogeneous or standardized.

Even the same type of contract may have many different variables and complexity drivers baked in.

That gets compounded further by other complexity drivers, such as:

1. File quality. Many of these documents include emails, scans, pages with coffee stains, handwriting in the margins, or image-only PDFs. Each of these can reduce the quality and reliability of OCR (optical character recognition). Bad OCR means the text may not be properly recognized by AI models. So before any AI touches the contract, the file often has to go through a chain of preprocessing steps just to become usable. Example here: link

2. Multi-document files. One file will often contain multiple discrete parts. For example, you may have the core contract, cover pages, exhibits, addenda, additional terms and conditions, and other related documents all in one file. Determining which document or clause takes precedence can be a very challenging exercise as a result. Example here: link

3. No system of record. Enterprises are diversified, with many business units, siloed repositories and multiple custodians. As a result, there is often no central place where all contracts live. Companies frequently can’t even find their contracts, let alone analyze them. And when they do find them, they are often dealing with legacy systems that contain partially executed agreements, duplicates or overlapping documents.

Compounding Complexity

Each of these factors would create complexity on its own. Enterprises often deal with all of them at once, across hundreds of thousands of documents.

So there is a range of complexity drivers, and many of them are closely linked to large enterprises. That’s why the enterprise dimension is so important and why it has so many implications for contracts.

Net of these considerations, you do not have one big, centralized, clean dataset. You have many unstructured, complex, messy, difficult-to-find documents — subpopulations inside subpopulations, fragmented across many systems.

Subscribe now

Contract Intelligence

Collectively, this is what creates the need and opportunity for contract intelligence.

Even finding contracts in an enterprise can be very challenging, let alone knowing the substance of what is in those contracts and what obligations they contain.

And this matters because hidden within these already-executed contracts are collectively huge opportunities, both in terms of risk allocation and mitigation, as well as increasing business value.

For example, it is not unheard of for a contract to lack a limitation of liability clause, which could introduce significant liability for the business, particularly if it is an older contract without other safeguards in place.

Payment terms are often suboptimal, leaving value behind or creating misalignment with the company’s standards, internal finance systems, and processes.

There may also be untapped opportunities to increase prices, or rights and restrictions to assignment, termination or exiting agreements.

The goal of contract intelligence is to take all of these contract documents and be able to search, filter, report on, and analyze them at scale.

The data points of interest vary based on the use cases and the business or legal problems people are trying to solve.

For example, in M&A diligence and integration, you might be interested in assignment, change of control, governing law, and related provisions.

For a business-as-usual type of review where a question comes up from the commercial team, you may be interested in a specific clause, such as the indemnification language agreed to in previous similar situations, or how long a contract requires the company to support a product that is being sunset.

This is a very small sample of potential use cases. The relevant questions vary depending on the business line, industry, function, department, specific situation, and place within the larger supply chain.

Contract Intelligence vs. Contract Summarization

For sake of clarity, it is important to understand that the difference between contract intelligence and a contract summary. These are complementary use cases that serve different purposes.

Contract intelligence specifically relates to turning the natural language of contracts into a structured data asset at scale, so users can gain visibility into their contract data through search, filters, reports and analytics.

In contrast, with a contract summary, you are typically doing a deeper dive into one or more contracts, often at smaller volume, to better understand the underlying nuances.

The critical difference is that the output involves qualitative descriptions of text. These are not ideal for running searches, reports and analytics at scale.

However, summaries can be very helpful when creating a diligence memo or other report to support a decision across a contract population, such as whether to move forward with an acquisition or how to prioritize regulatory remediation.

Summary

To summarize, post-signature enterprise contract intelligence involves three main components:

1. Post-signature – contracts that have already been signed or executed, otherwise known as post-signature contracts, as compared to pre-execution or pre-signature contracts that have been drafted, negotiated or redlined but not signed by all parties.

2. Enterprise – at least one party to the contract is a very large business, such as a Fortune 500 company.

3. Contract Intelligence – enabling people to make legal and business decisions by turning contracts into structured data that can be searched, filtered, reported on, and analyzed at scale.

   Subscribe now

If you found this article helpful, subscribe to follow along and forward it to anyone you think might find it useful.

I’ll take the overlap as validation. The framing works in part because contracts are becoming more than documents to store, negotiate, and search. Once you parse through the dense language, contracts start to look like signals: of risk, leverage, governance, market dynamics, opportunity, and operational behavior.

But the naming overlap is also a small symptom of a larger market pattern.

The volume of contract AI news keeps accelerating. Every week brings another product announcement, research release, workflow claim, or market report. And in that stream of news, one word keeps coming up: benchmark.

Subscribe now

Crosby published a contract-redlining benchmark. Harvey has published benchmarks for contract understanding and legal agents. TermScout, LexisNexis, and others benchmark contract terms against market norms. CLM vendors benchmark cycle time and operational performance.

One word — benchmark — is taking on multiple meanings.

Sometimes “benchmark” means a test. Sometimes it means a dataset. Sometimes it means the results from a market survey or a dashboard metric summarizing the same to compare against industry peers.

It has quietly become a word that sounds precise but is often focused on marketing.

So when (or before) the next benchmark lands in your inbox, the first move is simple: ask what kind of benchmark you are actually looking at.

Benchmark of what? Against what? Scored by whom? Using whose data? And for whose benefit?

What the word actually means

Strip away the jargon and a benchmark is just a reference point. A fixed thing you measure against — a baseline, a starting line — so that any other number means something relative to that initial reference point.

You set the reference, then you compare: better than the benchmark, worse than the benchmark, by how much.

Contracts professionals already had a specific version of this long before AI showed up. In outsourcing and other long-term agreements, a benchmarking clause lets a customer bring in an independent third party to compare a supplier’s pricing and service levels against the wider market — and adjust the deal if the supplier has drifted off-market.

Note what made that mechanism trustworthy: an independent party, an agreed method, and a defined market to compare against.

In the AI era, the word has stretched to cover at least four genuinely different activities. They get announced with the same or similar vocabulary, which is exactly why they are hard to read.

The first evaluates models. The second evaluates contract positions (i.e. the values of given clauses or other data points). The third evaluates operational efficiency. The fourth evaluates AI or system improvement over time.

Separating these concepts can help provide clarity.

The four benchmarks hiding under one word

1. Performance benchmarks — “How good is the AI?”

This is the one generating the loudest headlines: take a task, have AI models do it, and score them against a human expert or a model answer.

One recent example is Crosby’s Redline Bench, which scored frontier models on realistic SaaS-contract redlining against attorney-authored “golden” responses. The reported results put the top model around 50% and the rest clustered below it — a narrow spread, with humans still better at finding new routes to resolution while models tended to anchor on their opening positions.

Conceptually, there is nothing wrong with this. Measuring model performance against domain experts in realistic conditions is a reasonable thing to do. To their credit, the better recent benchmarks in this category disclose a lot: the use cases, the scoring dimensions, the rubric, and in some cases the whole thing is open for other labs to run.

That is valuable as it helps practitioners assess actual performance.

The trouble starts with how the numbers get used downstream.

“Model X scored 50.5%” gets repeated far from the methodology that produced it. Without that context, the number is almost impossible to interpret.

It may be directionally interesting. It may tell you something about frontier-model behavior in a simulated negotiation. It may help compare models under one defined test design.

But it does not automatically tell you which product to buy, which workflow to automate, or whether the same system will work inside your contract population.

That distinction matters.

A multi-turn negotiation benchmark is not the same thing as a data extraction benchmark. A redlining benchmark is not the same thing as a playbook-compliance benchmark. A test of SaaS agreement negotiation is not the same thing as a test of high-volume vendor onboarding, procurement triage, or post-signature obligation extraction.

And in many contract AI workflows, the advanced task depends on the foundational layer being right first.

Before a system can negotiate from a contract, it often has to identify the agreement, extract the relevant provisions, understand the parties, map the clause to the right playbook position, and preserve the context across the workflow. If those earlier steps are noisy, the later benchmarked task may inherit the errors. Sometimes the problem is not only model performance. It is error propagation – the snowballing or compounding of small mistakes at different stages in the process until the end result is a far cry from the original performance findings.

That is why the headline score is not enough.

Did the model miss obvious issues? Did it identify the right concern but propose a weaker redline? Did it produce something commercially reasonable but different from the attorney-authored answer? Was it penalized for not matching the exact path the human took? Would two senior attorneys have agreed on the same “perfect” answer?

A performance benchmark can be useful. But the headline score is not the benchmark. The benchmark is a combination of the task, the dataset, the scoring method, and the judgment calls underneath the score.

2. Norm benchmarks — “Is my contract typical?”

A completely different activity wears the same label.

Here, you assemble a large body of contract data, derive what is standard for a given type of deal, and then compare a new agreement against that norm or standard.

LexisNexis’s Market Standards, TermScout’s reports, and various “state of contracts” reports live in this category. The output is not a model score. It is a reference set.

For example: for vendor agreements of this type, in this industry, at this deal size, mutual indemnification is standard; one-way indemnification is aggressive; this limitation of liability formulation is common; that AI-use restriction is becoming more frequent.

This is genuinely useful and very doable.

Most of it reduces to structured data. A clause like indemnification can be reduced to a small set of positions — both parties indemnified, one party indemnified, neither party indemnified — and once you have coded enough contracts and broken down each clause type sufficiently, you can say how common each position is. Control for industry, deal size, template source, whether the agreement is customer paper or vendor paper, and other criteria and you have a real reference point for negotiation and risk.

But norm benchmarks have a problem that rarely makes the announcement: representativeness.

Much of this analysis is built on public filings, and public contracts are a biased sample by construction. A contract gets filed with the SEC because it crosses a materiality threshold. That means the corpus skews toward large, heavily negotiated agreements that may look very different from the thousands of ordinary commercial contracts a business actually runs on.

Even the filed agreements are frequently redacted at the commercially sensitive points you would most want to benchmark.

I used SEC contracts myself in last week’s article on creating a contract data extraction benchmark, and the same caveat applied there: they are a convenient, legitimate starting point. They are not a stand-in for your private contract population.

A norm built on a skewed sample is a useful hint, not ground truth.

3. Process benchmarks — “How fast is my operation?”

A third meaning has nothing to do directly with model quality or contract terms at all.

It is operational: cycle time, turnaround, throughput, percentage of contracts on standard templates, time-to-signature, number of negotiation rounds, fallback frequency, legal touch rate, approval bottlenecks.

CLM vendors increasingly sell “benchmarking” dashboards that compare your contracting operation against peers.

This is often the management-consulting meaning of benchmark, applied to legal ops. It may also be the most actionable version for many teams.

If your average NDA takes 12 days and comparable teams complete theirs in three, that is useful information to have. By the same token, if 80% of your low-risk agreements still require legal review while similar companies route most of these through self-service or your contracting cycle time spikes in one region or business unit, or by contract type, that is useful too.

But it is worth separating this cleanly from the other meanings.

A vendor saying it can “benchmark your contracts” might mean it can compare clause language to market norms. It might mean it can compare your contracting process to peer operations. It might mean it can measure model performance. Those are different use cases with different products, solutions, and datasets.

Process benchmarks are valuable when the metric maps to a real operational decision.

They are less valuable when the metric becomes theater: a dashboard showing you are slower than “peers” without explaining who the peers are, what work was included, or whether the comparison reflects your risk profile, company stage, industry, deal size, or contracting model.

Subscribe now

4. Baseline benchmarks — “Is my system getting better?”

The fourth meaning is the one practitioners building AI systems use most, and the one that almost never makes the press release.

Here, a benchmark is an internal baseline.

You fix a starting level of performance — an out-of-the-box model, a first prompt, a v1 pipeline, an existing review process — and then measure every later change against it.

Did the new prompt help? Did the more expensive model actually earn its cost? Did retrieval improve accuracy or just add latency? Did fine-tuning move the metric or just the vibes? Did the new clause taxonomy improve consistency? Did the latest model version break something that used to work?

This is how serious AI and data teams actually work.

And it carries two critical ideas the headline benchmarks tend to gloss over but often have a huge impact on assessing the benchmarks and making them actionable.

1. The first is generalization. A model trained or tuned on one contract population can fall apart on a new population due to different drafting, file quality, document structures and conventions used.

2. The second is data drift. A system that scored well last quarter can quietly degrade as the documents flowing through it change.

A single published score is a photograph: one snapshot in time, taken under a defined set of assumptions.

Truly effective benchmarking is more like a video – multiple points in time, multiple snapshots, multiple angles, and enough continuity to tell whether the system is improving, degrading or staying flat.

And the video is what tells you whether something works in production.

Four meanings. Four different questions. One word.

The moment you hear “benchmark,” your first move should be to figure out which one you are actually being sold.

How to read any benchmark

Sorting the type gets you halfway there. The rest is actually performing the analysis.

There are three layers that rarely get discussed and almost always decide whether a number means anything: the use cases, the scoring and the incentives.

The use cases

First, what was actually tested?

“Contract review” can mean redlining a standard NDA against a clear playbook. It can also mean a no-playbook judgment call on a bespoke, heavily negotiated agreement with limited commercial context.

Those are wildly different tasks.

Then, the specific scenarios corresponding to the use cases.

Before you trust a performance number, you want to know the distribution of difficulty behind it. Was the benchmark mostly easy cases? Edge cases? Common issues? Rare but high-risk provisions? First-pass issue spotting? Full redlining? Negotiation strategy? Escalation judgment?

That is the difference between “the AI is good” and “the AI is good at the easy version.”

The scoring

How was “right” decided?

This is where legal benchmarks are genuinely hard, because you are scoring dense, qualitative prose, not simple multiple-choice.

Did they grade it as a simple classification problem — risk flagged correctly or not, one point each over the total possible? Or did they use a rubric that accounts for subjectivity, ambiguity, commercial reasonableness, drafting quality, and the context the model was given?

And what does 100% mean?

Does it mean the model matched the attorney answer exactly? If so, would two senior attorneys have agreed 100% of the time?

Often they would not.

Law is full of gray areas. That is why it needs lawyers. A scoring method that pretends the gray areas are black and white will produce a confident number that does not survive contact with reality.

In addition, watch for false precision.

A score reported to a tenth of a percent — 50.5, 45.1, 44.4 — signals rigor. But the decimal is only as meaningful as the rubric underneath it.

When the rubric is scoring subjective legal judgment, three-significant-figure precision is often decoration. The confident number is the easiest part to produce and the least informative.

The incentives

Who ran the benchmark or experiment, and what do they sell?

This is the layer that matters most and gets discussed least.

Charlie Munger is often quoted as saying: “Show me the incentive and I will show you the outcome.”

A private sector benchmark is almost never published for philanthropy.

A law firm has an interest in a benchmark that shows legal judgment remains hard to automate. A vendor has an interest in a benchmark that makes its workflow look measurable and differentiated. A model lab has an interest in a benchmark that defines progress around the capabilities its model performs well on.

None of that means the benchmark is dishonest.

It means the benchmark has a point of view.

Wherever judgment enters — which tasks to include, how to score gray areas, what to call 100%, which dataset to use, which comparison group to define — the benchmark will reflect the author’s theory of the problem.

Even a scrupulously transparent benchmark is shaped by what its author chose to measure in the first place.

Transparency lets you check the math. It does not remove the incentive that selected the problem.

How to spot a benchmark worth trusting

None of this is a reason to dismiss benchmarks.

They are one of the better things to happen to a field that ran on vibes and demos for years. But they need to be read like evidence, not like slogans.

To summarize, a benchmark you can actually lean on tends to disclose:

• Scope — the tasks tested, how hard they were, and whether the headline average hides meaningful variation.

• Method — how “correct” was scored, who judged, and what a perfect score would mean.

• Data provenance — where the corpus came from and how representative it is of the contracts you care about.

• Authorship and incentive — who built it, who funded it, and what they sell.

• Reproducibility — whether anyone else can run it and get the same answer.

• Business relevance — whether the metric maps to a decision you actually need to make or problem you need to solve.

Use that as a checklist.

A benchmark that discloses its scope, method, and data but is published by a party with an obvious stake in how it is used is not worthless. It just needs to be read with the stake in view.

A benchmark with a strong headline but no corpus, no rubric, no task design, no scoring explanation, and no discussion of limitations is not reliable. It is a claim dressed up as a benchmark, often for marketing purposes.

The through-line

Last week I made the case that contract AI looks magical on one document and gets hard across a population — that the demo and the workflow are different problems.

A benchmark is the same story one level up.

The score is the demo. The methodology is the workflow.

A number that travels without its context is a promise without the receipts.

That is why independent reading of this market matters.

Independent does not mean neutral or detached. I have a point of view, shaped by a decade in enterprise contracts, taxonomy design, structured data and extraction workflows, AI, and legaltech product work.

It means something narrower and more important: I do not have a platform, model, law firm service, or benchmark to sell inside the analysis.

When the company publishing the benchmark also benefits from the way the benchmark is framed, the right response is not cynicism. It is professional skepticism.

What was tested? Against what? By whom? On whose data? And who benefits from the answer?

More signals soon.

If you found this useful, subscribe to follow along, forward it to someone who is about to make a tooling decision off a benchmark headline, and tell me which contract AI terms or claims you want me to unpack next.

Subscribe now

Relative to what was possible 10 years or even five years ago, the experience feels almost magical.

However, this is a significantly easier problem to solve than organizing an entire contract population, which can span hundreds of thousands of agreements.

So as a middle ground I ran a first benchmark to test performance against a more challenging problem: 100 real contracts for five basic metadata fields. I used three different models, one fixed prompt and assumed publicly available, text-native PDFs (having clean OCR).

The results were interesting in part because performance ranged quite a bit not just by model or model version but by the criteria used (e.g. accuracy, session usage or token limits, speed) and the specific situation (e.g. document structure).

In addition, contract AI workflows are increasingly being built on top of specific models or even specific model versions. This dependency can introduce risk not just in terms of model performance but model access. Last week’s export-control disruption affecting Anthropic’s Fable 5 and Mythos 5 models is an extreme example: access can change suddenly for reasons outside of our control.

So this benchmark is meant to help answer a practical question: what happens when a capability that works in a demo or other tightly controlled set of conditions has to survive a realistic contract population workflow?

Subscribe now

What I tested

The scope of the experiment was deliberately narrow: extract five metadata fields from each document and return one CSV row per document with the relevant information.

The reason for this narrow scope as a starting point is that more substantive analysis (e.g. M&A diligence or renewals management) relies on the contracts being sufficiently organized. Specifically, for each document we need to know:

• Is the document a contract or an ancillary document of some kind?

• How is the document referred to or titled?

• What type of contract is it?

• Who are the parties involved?

• When did the agreement take effect?

• Is there any other procedural history we should take into account to contextualize the document?

The five fields:

• Document title — as stated in the heading or preamble

• Party names — full legal names as stated in or around the preamble

• Effective date — stated effective date, falling back to other best date available in the preamble

• Preamble — verbatim introductory opening paragraph usually containing the party names and date

• Recitals — verbatim background or procedural history

The documents

The sample consisted of 100 real contracts sourced from SEC EDGAR filings.

These are real, executed agreements, not synthetic examples, but they are not perfectly representative for real enterprise contract populations. For example, they skew toward higher-value, material agreements and generally arrive in cleaner condition (from an OCR standpoint) than the contracts sitting in a legacy enterprise repository.

However, they are useful for a first benchmark because they are real, publicly available agreements with structural similarity to enterprise contracts that are reasonably easy to annotate to create ground truth. As mentioned above, for this first run, I used clean text-layer PDFs only (controlling for the variable of document quality).

The models and prompt

I tested three models against the same 100 documents using the same exact prompt: Claude’s Sonnet 4.6 and Opus 4.8, and OpenAI’s GPT-5.5 (I used Claude Cowork and OpenAI’s Codex to run these).

The prompt included approximately one page of context that a knowledgeable contracts professional could plausibly draft in an hour with a clear understanding of the desired output, without requiring significant technical expertise. It was specific enough to define the extraction task and output format but not a version that was optimized through iterations of trial and error.

In terms of methodology, I’ve described the test documents, fields, scoring approach, and limitations so that the findings can be contextualized and evaluated.

How I scored the results

I scored the outputs using three metrics:

1. Field return rate. Did the model return a value for the field at all? I tracked this separately from accuracy because a blank output and a wrong output create different risks. In this run, it was less of a differentiator between models.

2. Accuracy. For free text fields (document title, party names, preamble, recitals) I assessed substantive match against the manually annotated ground truth. For effective date, I used exact match after date conversion to YYYY-MM-DD.

3. Time and usage. I tracked wall-clock run time and session usage consumed per model, per run.

Two notes on ground truth:

1. Objective vs. Subjective. Contract metadata is often less objective than it looks from the outside. For example, even a seemingly simple field like document title or effective date can have nuances baked in. For document title, sometimes the cover page says one thing and the opening paragraph says another. Sometimes the heading includes “Executed Version,” “Confidential,” “Exhibit 10.3” or party names that are not really part of the title. Sometimes the preamble characterizes the document differently from the header. Effective dates have similar dynamics. Across this sample, effective dates appeared in several forms: stated explicitly in the preamble, defined by reference elsewhere in the document, or contained in the signature block.

2. “Golden Set” misnomer. For contract populations, the term “golden set” often oversells the quality and consistency of the available data. The goal is a reasonably standardized reference set (even that can be challenging to build). The benchmark uses documented annotation rules applied as consistently as possible to handle different cases while flagging ambiguity. But where to draw the line between an edge case or sufficiently common example is ultimately a judgment call.

Results (Headlines)

The headlines from the experiments are:

1. some models performed better than others but there were errors across all three

2. there are trade-offs between speed, usage and accuracy across models

3. many of the errors were patterned

On the third point, the models struggled on the same kinds of contract complexity:

• ambiguous titles,

• party naming conventions,

• effective dates defined by reference,

• and multi-party agreements.

One of the learnings from the exercise is that knowing your contract population becomes key prior to, or in parallel with model selection.

A better model helps, but a better understanding of the document population may help more given the common types of errors across models.

Speed and usage

In addition, the three models diverged sharply on both accuracy as well as speed and session usage.

A few caveats.

These runs were performed using paid consumer tiers (Anthropic Pro, ChatGPT Plus), not enterprise licenses. I call that out because many enterprise use cases would likely require more stringent controls for data security, confidentiality, admin oversight, and related needs (e.g. via enterprise licenses for these products). Usage limits and economics would likely differ (perhaps materially) in those environments.

The GPT-5.5 usage figure also deserves a specific note. Current GPT-5.5 pricing appears unusually favorable and may be subsidized as part of a competitive user-acquisition strategy. This may be directionally accurate, but it should not be treated as a durable long-term cost benchmark and API usage at scale may tell a different story.

All that said, a single Claude Opus 4.8 run on 100 documents consumed roughly 71% of a paid consumer-tier session. That is manageable for one experiment but is less likely to be a scalable approach to routine extraction work.

At 1,000 documents, consumer-tier session limits would likely become a workflow design constraint before the task is complete.

Accuracy by field

Claude Opus 4.8 performed best in terms of accuracy both overall and for each specific field. GPT-5.5 and Claude Sonnet 4.6 performed similarly overall but varied significantly in terms of field-level performance.

n = 100 documents. Effective date scored on normalized exact match; free-text fields scored on substantive match.

The models did show different levels of accuracy. The broader question is around impact: how much did the corresponding errors matter for the workflow?

For example, if one model is five percentage points more accurate but consumes three times the session capacity, that trade-off will be worth it in some cases but not in others. Whether it does will depend on the use case..

As another example, for party names: more lenient criteria (including partial matches – such as where the model found one but not all of the parties) yielded the following results, with substantial improvements for both GPT-5.5 and Sonnet 4.6:

What the models did well

On clean text-layer PDFs with standard preamble structure, all three models were generally consistent. They identified well-structured document titles, party names, and expressly stated effective dates reasonably well.

Where the models struggled

There were some common areas where performance dipped across models.

Here are some examples:

1. Document title ambiguity and noise

Clean document titles (commonly found in the document header) were captured without issue.

However, others had various issues, including exhibit numbers, file stamps or party names.

In some cases, the heading and preamble had referred to the document in different ways. For example: “FOURTH AMENDMENT TO COMMON SECURITY AND ACCOUNT AGREEMENT” (doc header) vs. “Fourth Amendment” (preamble).

This led to inconsistencies in terms of the document title selected.

2. Party names - missing entities, exactness

Party name mistakes included missing relevant entity names (particularly in multi-party agreements) and returning party names that were substantively correct but not exact (including extraneous text in addition to a relevant party name or omitting legal suffixes or punctuation).

Even small variations matter when used for legal and business decisions, which often require exactness (for example: searching for agreements or using the data as inputs into legal analysis or operational processes).

3. Effective-date variation

The main reason that effective dates can trip up AI models seems to be that contracts state effective dates in various ways.

For example, some are explicit: “This Agreement is effective as of January 1, 2024.”

Others are indirect: “This Agreement is entered into as of the date first written above.”

When models only looked in the preamble for effective dates, they sometimes missed dates defined elsewhere. When models searched more broadly, they sometimes surfaced the wrong date: a prior agreement date, notice date, recital date, or signature date (and also consumed more tokens).

The silent failure problem

One of the most important findings was that there were wrong answers that looked correct at first glance.

At 100 documents, a human QC pass can catch many of these issues.

At 1,000 documents, manual QC starts becoming overly time-consuming and sampling needs to be built into the workflow.

At 10,000 documents, even a small systematic error becomes a significant data-quality problem.

At 100,000 documents, a 2% human QC rate means 2,000 human interventions.

What this means

The first lesson: model choice mattered, but document structure mattered more.

Across all three models, the lowest-accuracy documents were the tricky cases. Examples included documents where preambles were missing, effective dates were not clearly defined, more than two parties were mentioned, or the document heading included one title while the preamble included another.

That means knowing your contract population is key prior to model selection. If your population has a high concentration of those structures, prompt design, field definitions, and QC process may be more impactful than simply switching models.

The second lesson: there are trade-offs between accuracy, speed and usage.

Claude Opus 4.8 performed best on accuracy alone, but it was also the slowest and consumed the most session capacity. Claude Sonnet 4.6 was materially faster and used less session capacity. GPT-5.5 was fastest and cheapest by session usage, but with pricing and durability caveats.

That creates a practical trade-off. For example, if Claude Opus 4.8’s overall accuracy advantage is 22 percentage points, the decision is whether those extra correct answers are worth the added time and usage for the specific use case.

The third lesson: consumer-tier tools can perform the workflow at lower volumes, but they struggle at scale.

At 100 documents, they are usable. At 1,000, session limits and manual handling become serious constraints.

The fourth lesson: model failures require QC (especially silent ones).

If the output will be used for decisions with significant consequences, such as diligence or creating a centralized system of record across business lines, some QC process has to be designed into the workflow.

Takeaways

Prelim note: these thresholds are directional based on this first run.

10 documents or fewer

Use the chat interface as at this volume, setup overhead matters more than workflow optimization. Prompt with the contracts and instructions, copy the output into a spreadsheet, and manually check the results.

Model choice is probably less important than clear instructions and human review.

Around 100 documents

The chat interface still works, but it starts to become unwieldy.

You need a repeatable prompt, a consistent output schema, a place to store the results, and a QC pass before using the output downstream. Watch especially for ambiguous titles, effective dates, and multi-party agreements.

100 to 1,000+ documents

The workflow around the model starts to matter as much as the model itself: consistent prompting, QC sampling, tracking and logging errors, etc. In addition, scale starts to become a consideration.

10,000+ documents

The overall workflow and operating model starts to become more important than model selection.

The Contract Signal is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Subscribe now

Last year brought more of the same: LexisNexis announced Protégé™ General AI, Thomson Reuters announced agentic capabilities in CoCounsel Legal, OpenAI published a detailed account of building an internal contract data agent.

That’s not a coincidence.

The market is converging around the contracts space using ever-improving AI capabilities — from creating “contract intelligence” to building end-to-end “workflows.”

But there’s a lot missing in the conversation around these announcements. So let’s unpack what it means and why it matters.

Subscribe now

To start — contract intelligence

I’ve spent about a decade working on this problem.

I started in 2015, inside a small contracts business at Axiom that eventually became Knowable, a LexisNexis company. Back then, the deliverable was a spreadsheet containing structured data. The deliverable was created using a combination of people, process and technology: project managers, subject matter experts and analysts running a specially designed review process on large volumes of enterprise contracts using specially configured eDiscovery software.

There was no software to buy that did this reliably. Early contract AI vendors were able to sell and market the tools but struggled to deliver results. The concept of turning large volumes of contractual text into structured data and analytics to solve legal and business problems predated the term “contract intelligence”.

That is why the current wave is so interesting. The concept is old but the technology is new and continues to evolve.

Same space, different layers

These announcements are solving for different things in different ways, but they’re all filed under similar labels.

Harvey’s Contract Intelligence appears to be a portfolio-level playbook management system — surfacing negotiation patterns, fallback positions, and clause language from executed agreements to make future review faster. The goal is that every signed contract feeds back into and updates the playbook automatically. This looks like a workflow-compounding capability aimed at in-house teams doing high-volume, BAU contract reviews.

Claude for Legal is positioned as the intelligence that connects to other tools and enterprise systems, not a standalone contracts product. It includes practice-area plugins, first-pass review and redlining, and MCP integrations to help do so.

OpenAI’s contract data agent was announced as an internal tool for their own finance team: ingesting PDFs and scans, extracting structured data, and serving it up for human-in-the-loop review. The purpose of the tool is to meet the scale of a growing business with limited headcount.

DocuSign’s agents operate inside its Intelligent Agreement Management platform: checking agreements against company standards, flagging risks, tracking obligations, and allowing teams to build custom agents for deal and renewal workflows through Agent Studio. The use cases overlap with what others cover, but the emphasis is on a platform-ecosystem anchored in their e-signature capabilities.

While the use cases sound similar on the surface, these providers are addressing different workflows that most announcements don’t unpack.

And if you’re a contracts professional, in-house counsel, legal ops leader, or someone running a deal process, it’s helpful to break down the announcements:

• Which part of my actual problem does each one address?

• How well does it address each part of the problem?

• What parts of the problem are left unsolved?

• What trade-offs come with choosing one approach over another?

The Root Causes of Complexity

Those questions are hard to answer from the announcements alone because the announcements are built as much to impress as to inform.

The announcements rarely explain the assumptions, i.e. what must already be true for the product to work well, not just demo well. These assumptions include: the scope has been defined properly and the relevant contracts have been collected, organized, and analyzed in a way that the output can be trusted and used in the company’s legal and business processes.

Why I’m writing this

The goal of The Contract Signal is to give people who work with contracts an independent read on what the capabilities can and can’t do and how to use them to solve real problems. I come at this from a decade of working across ML and LLM product development, contract analysis, solution sales enablement, and end-user research, with cross-functional visibility into how enterprise contracts work actually gets sold and delivered.

I am not promoting any particular vendor or employer.

If you work with contracts, buy or evaluate AI tools for legal workflows, or build in this space, subscribe to follow along.

Subscribe now

I used a web tool that aggregated deal announcements and made them easily searchable: mergers, acquisitions, and similar types of corporate transactions. It allowed searching and filtering by various criteria, including the deal size, type, advisors, etc.

I scoured for large deals as part of a seemingly straightforward business strategy: a big deal gets announced, big deals mean lots of contracts to review, lots of contracts mean someone needs help reviewing them, so we call them and win the work.

On paper, it sounds like a reasonable approach. However, it has two main limitations.

Subscribe now

Limitation one: by the time it’s news, the opportunity is gone

The first limitation is that by the time a deal has been announced, the advisors have already been selected and the diligence scope has been set (if not completed).

Related, the contract review approach has either been decided weeks or months before the press release went out or is an add-on to the more complex advisory workstream. Typically some combination of an internal team, outside counsel, and preferred vendors will be used.

So the deal announcements that show up in your alert feed or reports are disproportionately the ones you can no longer help on this deal. The key decisions were already made, during a highly confidential planning period.

As a result, reactive monitoring is a lagging indicator. It isn’t an effective origination strategy or lead source. The real opportunities come from being known before the deal exists: proactive relationships, so the company comes to you when the deal is still a secret. However, you can complement these proactive relationships with news monitoring, using announcements to check in on those relationships or perhaps build new ones.

Limitation two: volume doesn’t mean need

The second failure is subtler: it turns out that the premise that larger deal size means more contract work isn’t reliable.

Deal structure determines whether contract review is important and different deals are structured in different ways. A few examples that look similar but are fundamentally different in terms of contract work:

• The acquihire. A handful of critical people command a large deal price: you can get to several hundred million, even half a billion, on a few names in the current AI talent market. The contract review necessary for these types of deal is minimal, often limited to things like employment agreements or restrictive covenants associated with those key employees.

  The asset purchase. One entity buys a discrete set of assets, for example a fleet of trucks. There is often a small amount of contract diligence that is helpful on the buy side, mostly to validate things like ownership. However, because the assets are owned outright there is often limited counterparty outreach necessary. However, the seller may need to do more expansive review to transition or exit any related agreements (for example maintenance, support or vendor agreements to keep operating the assets).

• Business-as-usual flying under the radar. The company enters into fifty thousand new agreements a year in the ordinary course of business. That’s a completely separate pool of potential contract work from anything M&A-related, and it never shows up in a deal feed because the vast majority of contracts are too small to require announcing publicly.

These scenarios illustrate that high deal volume is weakly correlated with contract-review demand, and in some cases the flashy transactions that dominate the news are often the least relevant for contract review work.

What actually predicts demand

The initial screen can still be helpful as a starting point. Big, diversified companies are likely serial transactors; their historical deal cadence tells you something. Working in an industry with more physical goods is a valid consideration as well: a business with a wider supply-chain footprint tends to have more contracts than a lighter software business (though very large software companies are exceptions to the rule). Company size, deal history, deal size, industry provide a decent starting point that’s reasonably simple to build.

To make this more valuable requires going one click down, however.

To do so requires layering in the things the deal aggregation tool can’t see: your relationship history, your sales pipeline and track record by type of deal and lead source, and nuances learned from industry experience such as the buy-side/sell-side asymmetry mentioned above. Feed that into a model and you’re asking 2 questions:

1. Who’s likely to transact in a way that actually generates contract work?

2. Where are we well-positioned to win it?

These two questions are more thoughtful and reliable than simply looking for the biggest deals.

Conclusion

As mentioned above, to generate the best insights requires internal context: your relationships, data, business history, experience and know-how. A third party tool can help you generate a first pass but it can’t replicate the part that makes the prediction more reliable, which is the proprietary context.

Proceed accordingly and be wary of anyone selling you something to the contrary.

Subscribe now

If you’re working on origination or capacity planning around contract work and any of this maps to (or contradicts) what you’re seeing, I’d love to discuss.

No one could answer it well.

Google Maps can route you between two points. LLMs can tell you that Seattle “has great parks.” Walk Score can give you a general walkability number. But none of them answered my actual question: for a given block within a given neighborhood, what’s the walking distance to a meaningful green space?

So I built it myself.

This is the story of how a personal frustration during my move from New York to Seattle became my first public app—built with Claude as a coding partner, shipped as an MVP over a handful of nights and weekends, and designed to answer one specific question that no existing tool answered well.

Why This Existed as a Problem

In my previous articles, I wrote about the framework I used to pick a city and how I used LLMs during the planning process. One of my core criteria was convenient walking access to nature—not “nature exists somewhere in the metro area,” but “I can walk to a real park in under 15 minutes from my front door.”

This mattered to me because of lived experience, not aspiration. Living in New York, I’d discovered that regular morning walks—first through Carl Schurz Park, then Central Park—were one of the highlights of my daily life. I felt measurably better on days I walked. Over time, I realized that a city with great restaurants and nightlife but mediocre park access would be a worse fit for my actual life than a city with fewer amenities but a large park within walking distance.

The problem was that once I had this criterion, I couldn’t evaluate it efficiently.

I did rounds of Google searches. I asked LLMs. I checked existing nature and hiking apps. Nothing quite fit. There are plenty of apps for finding trails or planning hikes—but I wasn’t looking for a weekend adventure. I wanted to know: if I live on this block, can I walk to a good park easily? And how does that compare to the next block over, or the next neighborhood?

That question turns out to be surprisingly hard to answer without building something.

When Finding Is Harder Than Building

This experience led me to a realization I hadn’t expected: for a sufficiently specific personal need, the search costs of finding the right tool can actually exceed the development costs of building one yourself.

Historically, that wouldn’t have been true. Building a map-based web app required professional software engineering skills and weeks or months of development time. The rational move was always to search harder, compromise on fit, or just do the research manually.

But something has shifted. Between vibe coding with LLM tools like Claude, publicly available geographic data, and open-source mapping libraries, the cost of building a simple, personalized tool has dropped dramatically. Meanwhile, the cost of searching—scrolling through apps that almost-but-don’t-quite solve your problem, configuring tools not designed for your use case, cobbling together manual research across multiple sources—hasn’t changed much.

For me, the math flipped. I’d already spent hours on manual research and still didn’t have what I wanted. Building the thing I actually needed took less additional time than continuing to search for it would have.

This has broader implications for anyone building with AI tools. As development costs decrease, the value of complementary capabilities increases—things like judgment about what to build, taste in how it should work and look, and access to the right data. The bottleneck isn’t “can I code this?” anymore. It’s “do I understand the problem well enough to build the right thing?”

What I Built

The concept was deliberately simple: a color-coded map of Seattle showing walking distance to parks, broken down by neighborhood and then by census block.

That’s it. One question, answered visually, at a glance.

Why a map and not a report? I actually tried the text-based approach first. I ran LLM queries and compiled natural language descriptions of each neighborhood’s park access. It wasn’t very useful. Geographic and spatial relationships are fundamentally hard to convey in text. “Wallingford is 0.8 miles from Woodland Park” is less informative than seeing Wallingford on a map, colored green, with a park icon nearby. The visual format communicates distance, relative position, and comparison across neighborhoods simultaneously—something that would take paragraphs to express in words and still wouldn’t land as intuitively.

This connects to a broader principle I’ve been thinking about: the right interface depends on the type of information. Natural language works well for conceptual questions and subjective reasoning. But for spatial data, structured comparisons, or anything where relationships between data points matter as much as the data points themselves—a visual interface is almost always better.

The flow works like this:

Start with the city overview. Every neighborhood is color-coded from green (excellent park access) to red (poor access), scored 0–100. You can see at a glance which areas of Seattle have the best walking access to nature. Hover over a neighborhood and you see its score.

Click into a neighborhood. Now you see the individual census blocks within it, each with its own color and score. This matters because neighborhood averages can be misleading—a neighborhood might score 65 overall, but the blocks closest to a park could be 85+ while blocks on the far edge might be 35.

Click a specific block. You see the address, the nature access score, the nearest park name, the distance, the block area, and a link to view it on Google Maps.

The whole thing follows a principle called progressive disclosure: show the minimum useful information first, then let people drill deeper based on what they care about. For someone just exploring neighborhoods, the city overview is enough. For someone evaluating a specific apartment, the block-level detail is what matters.

How I Built It (And What Was Harder Than Expected)

I used Claude as my coding partner for the entire build. The overall process followed what most people describe as vibe coding: provide a high-level plan, iterate step by step, and make trade-off decisions along the way.

The plan was conceptually simple. I needed three things: data (parks, neighborhoods, geographic boundaries), a frontend (the map interface), and backend logic (calculating distances, generating scores, connecting everything). This is essentially the model-view-controller pattern—data, interface, and logic working together.

Where it got interesting was the trade-offs.

The data layer was the biggest source of real decisions—not because getting data was impossible, but because every data choice had downstream consequences.

First, how do you define a “park”? Is a 2-acre playground sufficient? What about a botanical garden? A greenway? For MVP, I kept it simple: parks above a minimum size threshold, focused on the kind of green space you’d actually walk through for 20+ minutes. That meant leaving out some edge cases, but it kept the scope manageable.

Second, where does the data come from? Publicly available geographic datasets exist, but they come with trade-offs: API rate limits, licensing restrictions, varying data quality, and inconsistent naming conventions across sources. Park data uses one taxonomy. Neighborhood boundary data uses another. Census block data uses yet another. Making these work together required judgment calls—how to define neighborhood boundaries, how to handle edge cases where a block spans two neighborhoods, what to do when data sources disagree.

Third, how do you serve the data? Making live API requests for every user interaction would be slow and fragile—and could hit rate limits fast if multiple people use the app simultaneously. So I downloaded the data upfront and serve it locally. For an MVP with a small user base, this works well and keeps the experience fast. Scaling would require a different approach, but that’s a future problem.

The frontend was where taste mattered most. The color scheme, the scoring scale, the level of detail in the popup, the decision to show census blocks rather than arbitrary grid squares—these are all judgment calls that determine whether the app feels useful or confusing. No amount of coding skill replaces knowing what the user (in this case, initially me) actually wants to see.

The backend logic surfaced subtle complexity. Distance calculations between geographic points aren’t as straightforward as they appear—coordinate systems, projection methods, and the difference between straight-line distance and walking distance all matter. For MVP, I used a reasonable approximation that’s good enough for “is this a 5-minute walk or a 20-minute walk?” If I need higher precision later, I can refine it.

Where Vibe Coding Helped—and Where It Didn’t

Building with Claude as a coding partner was significantly faster than building alone would have been. The LLM was particularly good at generating boilerplate code for map rendering and data processing, suggesting library choices and architectural patterns, helping debug issues when I could describe the problem clearly, and handling repetitive tasks like data formatting and conversion.

Where it struggled was when problems required context that’s hard to express in a prompt. For example, I hit a rendering bug where the frontend wasn’t displaying blocks correctly. Claude tried multiple fixes, each of which introduced a new problem—we went in circles. What eventually broke the loop was me suggesting we use browser developer tools to inspect the actual errors, which gave us concrete diagnostic information to work with. In another case, a more capable model release solved a problem the previous model couldn’t crack.

This reinforced something I’ve come to believe through this process: understanding computer science fundamentals matters, even in a vibe coding world. Not because you need to write every line yourself, but because when something breaks—and it will—you need enough understanding to diagnose the problem, ask the right questions, and evaluate whether the LLM’s proposed fix makes sense. If you abstract away all understanding of how the system works, you’re stuck the moment something goes wrong.

That said, I want to be honest about the tradeoff: for a simple, single-user application like this, the fundamentals matter less than they would for a production system serving thousands of users. The stakes of a bug are low. The complexity is manageable. If you’re building something for yourself and learning as you go, vibe coding is a remarkably effective way to do both simultaneously. Just know that the further you scale—more users, more data, more features—the more that underlying understanding pays off.

Lessons Learned

Build for yourself first, then share. I built this because I genuinely wanted it. That meant I understood the problem deeply, had strong opinions about what the solution should look like, and could evaluate quality without user testing. Solving your own problem is the fastest path to a useful product.

Start with one question. The app answers one question: how walkable is the nearest park from this location? Every feature decision flowed from that. When I was tempted to add hiking trail data, transit overlays, or restaurant proximity, I asked: does this help answer the core question? If not, it’s post-MVP.

Data is the real bottleneck, not code. The most time-consuming part wasn’t writing code—it was finding reliable data sources, understanding their limitations, reconciling different taxonomies, and making judgment calls about quality. As vibe coding makes development easier, data access and data quality become key differentiators for applications.

The right interface matters more than the right model. I tried answering this question with LLMs (natural language), with manual research (spreadsheets and notes), and with a visual map. The map won—not because it has more information, but because it presents spatial information in the format humans process most naturally. Choosing the right interface for your information type is a design decision that no amount of better AI can substitute for.

Ship the MVP, then iterate. The current version has rough edges. The scoring could be more sophisticated. The data could be more comprehensive. The styling could be more polished. None of that matters as much as having a working tool that answers the core question. Everything else is refinement.

What’s Next

The immediate plan is to continue refining the app—improving the interface, exploring additional cities, and seeing whether other people find it as useful as I do.

But the bigger takeaway from this experience is about a shift that’s happening in who can build useful tools. A few years ago, building a map-based web application required a team of engineers. Today, a product manager with some CS background and an LLM coding partner can ship a functional MVP in weeks.

That doesn’t mean the engineering is trivial. It means the barrier to entry has moved. The scarce resource is no longer “can you code it” but “do you understand the problem well enough, and do you have the judgment to make the right trade-offs?”

For me, this started as solving my own problem. It turned into my first public app. And it taught me more about building products than years of managing them from the other side of the table.

The app is available here: nature-access-map.

This is the fourth article in a series. Article 1 covered using Claude Code for financial planning. Article 2 covered the framework I used to evaluate potential cities to move to. Article 3 covered where LLMs helped and failed during the planning process.

I explored a few different approaches in parallel – a top-down type of workstream and a bottom-up workstream.

In this context – a top-down approach means larger scope: effectively I compiled all of the financial statements and transactions from a given period of time, say the last 3-5 years, and then used Claude Code and other tools to actually parse out the transactions, compile them, categorize them, and analyze them or enable my own analysis of them.

I tested a few different approaches that uses this sort of top-down workflow.

The original overall approach or pipeline was:

- Create an overall plan or outline

- Compile / download financial statements (PDFs)

- Drop into a local folder and have claude code work with them end to end:

• Provide initial context and refine plan using Claude Code’s plan mode

• Convert each PDF into Excel or csv

• Parse out the transactions

• Aggregate the transactions into a single Excel or csv

• Categorize the transactions

• Perform follow-on analysis

• Iterate

After a few hours it became clear that parsing out transactions from PDFs to Excel / csv is deceptively challenging (even for Claude), particularly when working with different data sources. So I replaced that specific step with using a third-party PDF to Excel conversion tool that I already had access to and was both much faster and more reliable.

Otherwise, the overall top-down approaches were the same.

My findings from the top-down approaches were as follows:

• First of all, handling large scope to start can be overwhelming in terms of the amount of the amount of necessary code (and of course corresponding context windows)

• Second of all, because of the volume of code and data corresponding to the scope, the human testing costs become much higher

  • In other words, the realm of possibilities of things that can go wrong are much broader, require more rigorous testing and the testing is more time consuming to perform

  • Unless an issue is glaring or there is some clear prioritization that can be done – for example identify what the largest transactions (such as rent payments), the transactions require a fair amount of individual review

  • However even when prioritizing these things there are all sorts of nuances that can get introduced – for example transactions being combined together, or amounts being off, or occasional exceptions that are difficult to spot (e.g. imagine a paycheck and a bonus being combined together or a rent payment being combined with a security deposit or a move from one apartment to another leading to different rent costs, etc.)

  • Oftentimes the context is situation-specific as well which makes more automated tests or handling this via the planning phase more difficult

• Third of all, in terms of the reliability and accuracy of the code – the top-down approach is decent at creating estimates but if exactness or precision is necessary, it is of limited utility as it would require a fair amount of manual quality control and testing. This may be okay as a starting point but usually requires complementing with other approaches to get more precise data

The bottom up approach

Given the limitations of the top-down approach, the question came up: is there a complementary or perhaps alternative approach to the top-down process that focuses on a smaller scope?

For example, starting with a specific category of spend, mastering that and then repeating the process for other categories.

The business analogy I would use here is how Amazon focused on a specific product category to start (books). There are similar best practices when starting a business – specifically to focus on an underserved niche rather than trying to explore too large of a market and risk more competition, expensive customer / user acquisition, etc.

So the bottom up approach has the benefit of targeting specific categories one at a time (e.g. rent, grocery spending). This in turn involves smaller volumes, similarly structured transactions and less corresponding complexity so it involves less code, simpler testing, less cognitive load and greater transparency into both the data and the code.

This has been my experience so far.

To summarize: the key benefit of the bottom-up approach is simplicity. Less categories, smaller transaction volume, less things that can go wrong, and less things to consider at any given time.

Now compare that with the top-down case where you have many statements across multiple categories. So you have a much larger volume of transactions of different types. You have different types of data structures for each. You have different types of ways in which things are expressed and net of those considerations there are many things that can go wrong potentially at each stage of the process, particularly when you’re parsing, aggregating, and analyzing the transactions.

An Open Question – Building Up to Complex Use Cases?

In addition, there is an additional potential benefit for more complex systems with a large volume of categories. My hypothesis is that if you build it up from the bottom step-by-step, category by category and each category is sufficiently constrained, over time you may be able to create an orchestration agent to actually coordinate each of these as if they were separate services.

For example you can have a grocery purchase analyzer as one service.

You can have a rent payment analyzer as another service, a medical appointment analyzer, a transportation payment analyzer, and so on.

You can do something similar for basically every category of items that is sufficiently standardized or constrained. Then, as you (and the system) develop a strong understanding of these you can then potentially automate the system entirely.

I need to pressure test the feasibility of this by running some experiments, but early indications are promising.

Conclusion

Overall, Claude Code has been very helpful in assisting me with financial planning and other use cases. So far – the top-down approach has been preferable to get a decent estimate of overall spend while the bottom-up approach has shown the most promise when exactness and detailed results are necessary.