Hi everyone,
In a previous article I discussed why even a seemingly simple field like document title can be surprisingly complex. That complexity comes from both how the field is defined and how contracts are drafted and structured.
I explored this in the context of evaluating contract AI. If you’re working through that process, I also put together a curated dataset and kit to help test different solutions.
In this article I want to explore three of the underlying questions:
What makes a contractual data point “correct”?
How is that correctness impacted by how clearly it’s defined?
Regardless of “correctness”, does the definition produce an answer that’s useful for the problem we’re trying to solve?
These questions are interrelated. For example, a contract data point or answer to a contract question may seem to be “incorrect” at first glance, but that incorrectness is driven more by ambiguity in how the question is asked or the schema or data model (the formal way of structuring the information) is defined.
This applies beyond contracts. An ambiguous question can have several reasonable answers, depending on how it’s interpreted. Adding specificity to the question can reduce its ambiguity and make it easier to answer objectively.
To illustrate, let’s start with a basic question: “What is the biggest country”?
This may seem more objective than a question like “What is the best country?”, but still contains a fair amount of ambiguity. Do we mean the biggest based on population size? By physical size? By inhabitable square acreage? By total gross domestic product?
And the question doesn’t have to be about the biggest country. It could be “What is the richest country?”, “What is the hottest country?”, etc. In each case, adding specificity to the question or criteria helps establish what is considered a more objectively correct answer.
An effective date can be defined in different ways
Now let’s take an example specific to contracts: what is the effective date?
I asked several high-performing LLMs (including Opus 5.5, Sol 6 and Gemini 3.1 Pro) to answer the question on a sample of documents and they repeatedly came back with a range of different answers depending on how the question was asked, what additional constraints were provided, the specific document, and other relevant context.
If I didn’t specify those constraints and context, the LLMs made assumptions. Depending on the use case, some of those assumptions could be reasonable.
Compare these three examples (excerpts shortened to focus on the relevant language):
“This MASTER SERVICES AGREEMENT (“Agreement”) dated September 1, 2022, and effective August 30, 2022 (“Effective Date”)…”
THIS AMENDMENT NO. 2 (“Amendment No. 2”)…is effective as of the date of last signature below.
This Seventh Amendment to Lease (“Amendment”) is entered into, and dated for reference purposes, as of July 24, 2018 (the “Execution Date”) by…
The first clearly defines the effective date and includes the calendar date directly.
The second defines the effective date indirectly by reference to the signature date.
The third doesn’t mention the effective date at all but does reference a date with similar phrasing that may be usable in some contexts (“entered into”…”dated for reference purposes”…“execution date”).
In my testing, the LLMs sometimes selected what appeared to be the best available date. Whether that was “correct” depended on the field definition and how it was applied to the specific document.
Out of a sample of 192 documents I looked at, less than half actually had the effective date expressly defined and included the calendar date directly (81, or about 42%). A handful of others were defined in a different way. The full breakdown is below:
Effective Dates are used in different ways
Different stakeholders have different problems that they’re trying to solve, which impacts how field definitions are used.
Some users want to know the effective date to check whether it is expired or coming up for renewal, which are sometimes but not always calculated using the effective date.
Others might want to know the effective date to help inform the template that the document is likely to use and whether it needs updates following regulatory changes.
Still others may need to know when it became effective as part of resolving a legal dispute or understanding risk exposure.
For some of these goals, knowing the effective date is critical. For others, the effective date is either a means to an end or less relevant than it may initially seem. In some cases (such as when trying to understand the template a document uses) a best available date may work just as well.
But in all cases, clear definitions that are aligned with the actual use cases are necessary.
Otherwise, an AI system can get the answer wrong or users with completely different experiences, mental models and context may interpret the same data differently.
And what you’re left with is a real-life manifestation of the old joke: ask 10 different lawyers the same question and you get 10 different answers.