Hi everyone,
The barrage of legal AI announcements continues, many of them with applications to contracts.
Yet I continue to hear from contract professionals that, while they’re being told that AI can do almost everything, results vary dramatically when applied in practice.
This has been my experience as well. Part of the disconnect seems to stem from treating contract AI as a single thing that applies across situations. In practice, it’s much more like an umbrella term covering a range of capabilities and use cases, such as finding a contract, managing upcoming expirations or renewals, or conducting M&A diligence and integration.
Even the strongest AI models have a range of performance depending on the use case and situation. They can be very useful in many situations, somewhat useful in others, and counterproductive in still others.
So how do you know whether a contract AI system performs well enough on your actual problems and use cases to add value, and where it might need refinement?
To help make those evaluations more concrete, I’ve developed a contract AI benchmarking kit. It’s designed to help users evaluate a prospective or existing contract AI system, compare results, and make more informed decisions about where and how to use it.
It contains:
192 publicly available contract documents filed with the SEC, curated to capture a variety of practical scenarios and challenges: different contract and document types, drafting approaches and other sources of complexity.
24 field definitions spanning common legal, business, and operational use cases, with examples organized into standard, atypical and out-of-scope situations.
A benchmark dataset consisting of quality-controlled annotations created by applying those definitions to the documents.
A guide that explains the approach and how to use the materials.
Preview: why even a document title can be tricky for AI
To get a sense of the scenarios the kit helps you test, consider one seemingly simple field: document title.
After all, how can a document title be complex? It seems like you (or an AI system) should simply grab the title from the top of the document, and move on.
Sometimes that works. But it turns out that document titles can be deceptively tricky to capture consistently.
For example, sometimes there are file stamps, such as those added by e-signature tools:
Sometimes there is additional text near the top of the document, or a title that spans multiple lines:
Sometimes there is a hybrid document title / preamble that combines dates, party names, and reference numbers:
Sometimes the information appears in a table, with no standalone document header:
Sometimes a separate cover page contains what appears to be a document title, but the substantive agreement begins later in the file (in this example, on page 5):
Page 1:
Page 5:
Conclusion: Test AI to inform decisions
These examples don’t necessarily mean that an AI system will get the title wrong. AI has its place and can add significant value.
However, they do show why even a seemingly simple field like document title often requires more precise structure and instructions than might appear necessary at first glance.
Should we include the header? When? How much of the header? Do we include sub-headings? Reference numbers? Dates?
Given these sources of complexity, we also need a systematic approach to testing that helps inform when and how to use contract AI.
Compiling and curating documents, defining fields, and reviewing annotations is a time-consuming, cumbersome process. I’ve brought those materials together so you can spend less time preparing a test and more time evaluating the results and using them to drive value.