Hi everyone,
AI is being used all around us.
Not just in the high-profile generative AI tools like ChatGPT, Claude and Gemini but also in applications ranging from Apple’s Face ID to Spotify music recommendations to Duolingo’s video call feature.
What makes these examples successful is that the AI is effectively incorporated into the user experience to solve real, valuable problems.
Contract AI, while perhaps less exciting for most people than listening to a rock concert or learning Spanish, can benefit from similar approaches to incorporating AI.
The question is how?
In this article, I’ll share my experiences on different techniques for getting the most out of AI models. I’ll focus on contract workflows but will note that there are commonalities with use cases across other domains (e.g. personal productivity, coding / software development, research, and various third-party tools).
High Performance in the Wild, Not Just in the Lab
A key insight is that regardless of the use case, there’s a difference between achieving high model performance in “the lab” (i.e. in a purely research and development context under strict guardrails) vs. achieving high performance in “the wild”, i.e. in practical settings where the models not only perform well from a technical standpoint but also help solve real problems and generate ROI relative to the time, money and other resources it takes to use the technology.
Generating this ROI requires operationalizing AI. When I say “operationalizing” AI, I just mean effectively incorporating AI models or their output (aka predictions) into concrete workflows to solve problems (preferably in a scalable fashion, though that will depend on the needs of the use case).
Before getting into that, I wanted to offer some thoughts on why getting ROI from contract AI (as well as legal AI and other AI use cases) can be challenging.
Why Getting ROI from AI is Challenging
There’s been many studies and headlines about the challenges with getting ROI from AI despite significant investments.
There are various explanations for why, but here are three categories of reasons I’ve observed:
Focusing on the right problem
Identifying and getting access to high-performing models relative to the problem’s needs
Using the AI tools effectively
In previous articles, I discussed category 1 (mapping your contract workflows to identify high-impact AI opportunities) and category 2 (the contract AI vendor diligence framework to understand how likely it is that a given set of capabilities can help support a given use case).
This is important context to understand as simply deploying AI models and throwing them at a problem without first addressing these questions increases the risk of either 1-working on problems that don’t matter and/or 2-running in technical feasibility barriers.
In this article, I’ll focus on category 3.
To start, it’s worth discussing why using AI tools effectively can be challenging, even after successfully identifying high-impact problems to solve and having access to the right capabilities.
There’s three main reasons I’ve observed:
AI is probabilistic, not deterministic
Important dependencies are missing: e.g. data, the right technical infrastructure, or intuitive UX
AI requires some level of technical know-how, which is often missing
Reason 1: AI is Probabilistic, Not Deterministic
One of the main reasons why AI tools can be tricky to use is that AI is probabilistic. This means that for the same input, the output isn’t always going to be same, let alone correct or reliable.
Contrast this with deterministic software. Historically, before AI-powered software applications became more prevalent, software was based on procedural rules or conditional logic (aka if-then rules and similar mechanisms). Essentially, this meant that if the software was working correctly, the same input would always yield the same output.
By way of analogy: the software worked like flipping a simple light switch: flip up to turn it on, down to turn it off. Much of the technology in the world has historically functioned in a similar way: e.g. traffic lights can be red, yellow or green based on specified time intervals, the previous state or other conditions, such as a pedestrian crossing button being pressed.
In contrast, using probabilistic output is a fundamentally different experience because some part of the time the output is going to behave as expected, but not always. E.g. 90% of the time it’ll be as expected (i.e. “correct” depending on the use case) and incorrect 10% of the time, or it’ll be correct 80% of the time, and incorrect 20% of the time.
Relatedly, you typically don’t know what that exact accuracy number is in any given situation until you actually run the models on the inputs. That’s why investing in rigorous testing and other techniques is important, as we’ll get into.
It’s also why AI systems involve more ambiguity and can cause apprehension from a user adoption standpoint.
Reason 2: Dependencies (AI Requires the Right Inputs)
Just as a car doesn’t run by itself (it requires fuel such as electricity or gasoline), AI also requires various inputs and complementary capabilities to work well.
Some of the most common examples include high-quality data, the technical infrastructure necessary to train, deploy and test models, and an intuitive user interface that makes it easy to surface predictions and actually consume the output.
Reason 3: AI Requires Technical Know-How
The third category is that AI requires some level of technical know-how.
This has been significantly less of a factor recently, as most AI tools powered by large language models have a natural-language interface that doesn’t require code in most cases. However, getting the best results from these tools still requires some understanding of how they work.
Ways to Operationalize AI
There are many different ways to help get the most out of AI tools and increase ROI.
Which practices will be most effective in a given situation will depend on the specific situation and use case.
Method 1: Transparency
What it is: Including a breakdown of whether AI is used in the product or application, where and how it is used and related caveats regarding imperfect output or potential mistakes.
Context / Why it Matters: Contracts involve high-stakes legal and business decisions, and relying on information requires that information to meet a very high quality bar. Relatedly, attorneys and other contracts professionals have a significant responsibility to allocate risk effectively, which requires a realistic understanding of AI performance in any given situation.
Pros: This is a simple way of flagging to a user that the data or other output is AI-generated (i.e. imperfect and may contain mistakes) and signals to the user to double check it before making a consequential decision.
Cons: Even if the product or application is transparent regarding its use of AI, this isn’t always effective at driving the right behaviors. For example, attorneys have been sanctioned by courts for citing cases that didn’t actually exist (they were hallucinations made up by LLM-powered tools). The ability to understand these risks is driven by the user’s mental model, their understanding of probabilistic systems, and the availability of mechanisms to use AI effectively, which we’ll get into below.
Method 2: Confidence Thresholds
What it is: Only surfacing predictions that meet a certain confidence threshold, determined by the model’s estimated confidence that the prediction is correct. For example, only surfacing predictions with a certain confidence level.
Context / Why it Matters: Not all predictions are equally likely to be correct. For example, you may have 1000 predictions for a certain data point in a contract, with 80% accuracy overall (800 are correct, 200 are incorrect). Only using predictions that meet a certain confidence level (e.g. 90%+) may reduce the total number of predictions used to 500-600, but increase their accuracy level.
Pros: This enables AI predictions to be routed to different workflows: for example AI only (high confidence), AI + human (medium confidence), or human only (low confidence). Whether and how to implement this will depend on considerations such as the cost of errors.
Cons: Confidence scores are not always reliable, and so neither are confidence thresholds based on them. One reason is that confidence scores can be calibrated based on specific training data sets. New data risks lower performance due to data drift (when the models run on new contracts with a different structure, such as those corresponding to new customers, business units, or document types). Frontier models tend to generalize better, but not perfectly, and the error rate can also rise with use case complexity. For example, drafting a contract includes considerations such as tone, flow, and audience. As a result this task is harder to calibrate than a simpler binary classification problem such as “does an assignment clause exist in this contract?”
There are six additional methods I’ve found helpful for operationalizing AI.
Subscribe for free to continue reading.
The Contract Signal is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.