The barrage of legal AI announcements continues, many of them with applications to contracts.

Yet I continue to hear from contract professionals that, while they’re being told that AI can do almost everything, results vary dramatically when applied in practice.

This has been my experience as well. Part of the disconnect seems to stem from treating contract AI as a single thing that applies across situations. In practice, it’s much more like an umbrella term covering a range of capabilities and use cases, such as finding a contract, managing upcoming expirations or renewals, or conducting M&A diligence and integration.

Even the strongest AI models have a range of performance depending on the use case and situation. They can be very useful in many situations, somewhat useful in others, and counterproductive in still others.

So how do you know whether a contract AI system performs well enough on your actual problems and use cases to add value, and where it might need refinement?

To help make those evaluations more concrete, I’ve developed a contract AI benchmarking kit. It’s designed to help users evaluate a prospective or existing contract AI system, compare results, and make more informed decisions about where and how to use it.

You can explore the kit here.

It contains:

• 192 publicly available contract documents filed with the SEC, curated to capture a variety of practical scenarios and challenges: different contract and document types, drafting approaches and other sources of complexity.

• 24 field definitions spanning common legal, business, and operational use cases, with examples organized into standard, atypical and out-of-scope situations.

• A benchmark dataset consisting of quality-controlled annotations created by applying those definitions to the documents.

• A guide that explains the approach and how to use the materials.

Preview: why even a document title can be tricky for AI

To get a sense of the scenarios the kit helps you test, consider one seemingly simple field: document title.

After all, how can a document title be complex? It seems like you (or an AI system) should simply grab the title from the top of the document, and move on.

Sometimes that works. But it turns out that document titles can be deceptively tricky to capture consistently.

For example, sometimes there are file stamps, such as those added by e-signature tools:

Sometimes there is additional text near the top of the document, or a title that spans multiple lines:

Sometimes there is a hybrid document title / preamble that combines dates, party names, and reference numbers:

Sometimes the information appears in a table, with no standalone document header:

Sometimes a separate cover page contains what appears to be a document title, but the substantive agreement begins later in the file (in this example, on page 5):

Page 1:

Page 5:

Conclusion: Test AI to inform decisions

These examples don’t necessarily mean that an AI system will get the title wrong. AI has its place and can add significant value.

However, they do show why even a seemingly simple field like document title often requires more precise structure and instructions than might appear necessary at first glance.

Should we include the header? When? How much of the header? Do we include sub-headings? Reference numbers? Dates?

Given these sources of complexity, we also need a systematic approach to testing that helps inform when and how to use contract AI.

Compiling and curating documents, defining fields, and reviewing annotations is a time-consuming, cumbersome process. I’ve brought those materials together so you can spend less time preparing a test and more time evaluating the results and using them to drive value.

Explore the Contract AI Benchmarking Kit here.

Years ago, a young Steve Jobs illustrated the power of contract data.

It was in the early days of Apple. He and Steve Wozniak were working on the Apple I.

Jobs realized that if he could get paid immediately upon delivery of Apple’s product but pay his component suppliers on 30-day credit terms, it would give Apple the lead time necessary to bu…

Read more

In this article I’ll share a practical trick to make your use of Claude, ChatGPT and other LLM tools more efficient while still getting strong responses.

Plan with strong model, execute with a lighter one

To reduce the cost of working with contracts, consider using a stronger model to plan and a lighter model to execute. This applies to agenti…

Read more

AI is being used all around us.

Not just in the high-profile generative AI tools like ChatGPT, Claude and Gemini but also in applications ranging from Apple’s Face ID to Spotify music recommendations to Duolingo’s video call feature.

What makes these examples successful is that the AI is effectively incorporated into the user experience to sol…

Read more

In a previous article, I shared a framework I’ve found helpful for identifying high-impact opportunities to apply AI.

In this article, I do a deeper dive by applying that framework to a concrete contract workflow: reviewing and responding to customer redlines in a master services agreement.

Business Problem and Context

Business Problem: imagine…

Read more

But how? It turns out that many of the most common obstacles to getting ROI from contract AI solutions aren’t due to the models themselves.

There are three main categories of things that I’ve found helpful when c…

Read more

Phrased another way: just because you can apply AI to a contract or other type of workflow, doesn’t mean you should. For example, earlier this year, Amazon reportedly incurred a cost overage of 1.8 million dollars related to a meni…

Read more

A core theme that came up, decoupling execution from judgment, is something that I first heard about back in late 2015 as I was preparing for my first role out of law school. I started working on an emerging M&A projects business…

Read more

In a previous article I discussed the value of starting with a concrete problem or use case (in that case, a contract renewal question) and then turning it into a skill. To recap: the main benefit of doing so is creating a reusable, specialized set of instructions that can be re-purposed for similar tasks or workflows in the future.

In this w…

Read more

The last few weeks have brought more AI announcements both focused on and tangential to the contract AI space.

Some of these have been significant, including another Big Tech entrant into the contract AI space (Amazon), foundation model upgrades and launches by OpenAI, Anthropic, and Google, a push towards more capable open source models, an…

Read more

An approach I’ve found helpful to address the information overload and FOMO is to focus on a smaller volume of high-impact concepts, one at a time, organized by concrete problems…

Read more

A former colleague told me this in 2020, describing a failed AI project. I was interviewing them for any wisdom they could share as I stepped into a product manager role for our machine learning program. They had engaged a third-party provider with best-in-class technical credentials and a great de…

Read more

In a previous article I discussed how every post-signature enterprise contract workstream, big or small, can be framed as the same underlying six-part process:

1. Trigger — an event occurs that requires contracts work

2. Define criteria — determine what you’re looking for and why

3. Find — apply the criteria to locate the target population of contracts

4. A…

Read more

The clock starts running on the contracts workstream.

You’re juggling a handful of other tasks. You’re being told to use AI. Life outside of work doesn’t stop either.

So how do you proceed?

Almost every post-signature enterprise contract workstream, big or small, can be framed as the same underlying process…

Read more

Read more

1. Post-signature – these are contracts that have already been signed or executed, otherwise known as post-executed contracts, as compared to pre-execution or pre-signature contracts that have been drafted, negotia…

Read more

I’ll take the overlap as validation. The framing works in part because contracts are becoming more than documents to store, negotiate, and search. Once you parse through the dense language, contracts start to look like signals: of risk, le…

Read more

Relative to what was possible 10 years or even five years ago, the experience feels almost magical.

However, this is a significantly easier problem to solve than organizing an entire contract populati…

Read more

Last year brought more of the same: LexisNexis announced Protégé™ General AI, Thomson Reuters announced agentic capabilities in CoCounsel Legal, OpenAI published a detailed account o…

Read more

I used a web tool that aggregated deal announcements and made them easily searchable: mergers, acquisitions, and similar types of corporate transactions. It allowed searching and filtering by various criteria, including the deal size, type, advisors, etc.

I scoured for large deals as part of…

Read more