The Evolving Role of AI In Legal Judgment and Execution

A 10-year retrospective along with a practical framework for breaking contract workflows down and determining where to apply legal AI

Last week I attended a webinar hosted by Priori Legal. As I was listening to the sessions, I had a surreal feeling of déjà vu.

A core theme that came up, decoupling execution from judgment, is something that I first heard about back in late 2015 as I was preparing for my first role out of law school. I started working on an emerging M&A projects business at Axiom as an early hire, focused on turning large volumes of enterprise contracts into structured data and analytics (rebranded within a few years as contract intelligence).

The market has evolved significantly since then. AI has played a role in this evolution in four main ways:

1. AI is now able to cover or assist with a greater proportion of execution

2. AI has improved to the point where the line between judgment and execution has shifted (for some workflows)

3. Access to high-performing AI models has increased significantly

4. AI’s capabilities continue to evolve at a fast pace

However, before getting into AI’s current capabilities in greater detail and where the opportunities are to incorporate them into legal workflows, let’s contextualize what’s going on in the market with a historical look back into how those capabilities evolved and what the impact was.

Judgment vs. Execution 10 Years Ago

The original value prop of Axiom’s contract projects business was using people, process and technology to help enterprise clients review high volumes of contracts with high quality and consistency. The legal and business driver was usually M&A diligence and integration as those workflows were time-sensitive and required both efficiency and scale, though there were other use cases as well that benefited from a similar approach.

So if we were to break the market (at that time) down into the problem space, the competitive dynamics and the state of the art of decoupling judgment vs. execution, it looked something like this:

• Problem — the problem being solved for was largely the same as it is today: getting better visibility into large volumes of enterprise contracts that were often hard to find, let alone understand.

• Competitive dynamics — AI capabilities at the time were limited so the few providers that were trying to use AI to review contracts were able to sell its potential far better than they were able to deliver value. Phrased another way: hype exceeded reality.

  • More commonly, the law firm memo (a qualitative summary of the main issues found) was the main type of deliverable at the time. It was created using a judgment-focused approach with limited execution capabilities (from a high-level: throw groups of smart law firm associates or other junior resources at the contracts, bill by the hour). In addition, in terms of utility, while it had its place, it wasn’t structured data: it couldn’t be searched or analyzed at scale, particularly on an aggressive M&A timeline.

• The state of the art — the best-in-class approach in separating judgment from execution involved a combination of people, process and technology. This referred to a combination of contract analysts, subject matter experts and project managers using written playbooks created through specialization and experience in these types of projects, as well as eDiscovery software that had been specifically configured for the purpose of delivering these projects.

Judgment vs. Execution (Example)

Let’s use a concrete use case to illustrate how judgment and execution are different, specifically in the context of contract review.

For simplicity’s sake I’ll define judgment (otherwise known as domain expertise or tacit knowledge) as experience or expertise in dealing with specific types of problems, and execution as completing the work, guided by the judgment. Another way of thinking about it: judgment is knowing what work is important to do (and when and how to do it), while execution is actually doing it.

The line between the two can be blurry, however. For example, with some training, a contract analyst could take a contract clause and identify it as being an assignment clause, an indemnification clause, a renewal clause, a governing law clause, etc. They could also likely categorize it further, for example:

• Whether the assignment clause included consent or notice requirements,

• Whether the indemnification is one-sided or two-sided,

• Whether the renewal clause is an automatic renewal, an optional renewal or something else, or

• What the specific geography of the governing law clause was.

However, it would become more challenging for the analyst, if (without any legal training) they were asked to opine on whether a given clause was enforceable. Or the overall risk profile of a clause or entire contract. Or whether a clause draft or position should be re-negotiated as a result.

So in that sense, while each of the tasks required some level of judgment and execution, the types and levels of judgment required varied depending on the stage in the process and the corresponding complexity of the problems involved. Concretely, the initial training would likely need to be done by a more experienced subject matter expert (in other words, the training would involve more judgment). Applying the training to identify, capture and categorize contract clauses would require some judgment as well though less so, and would involve more execution.

AI and Judgment

However, as different types of judgment are applied repeatedly (in other words executed, specifically for more common, standardized tasks or workflows), they can be codified as context (for example via playbooks or taxonomies) and data.

In that sense, a workflow that was at one time judgment-focused can become execution-focused over time as it becomes more frequently used and codified.

And once they’ve been codified, different forms of AI, while not having judgment in the human sense are able to identify and apply those groups of data patterns and context to new situations.

When this happened for the contract review use case, contract AI rapidly improved (initially the branch of AI called machine learning applied to contracts). This was driven both by increased availability of these data sets and context as well as increased availability of more powerful, open source AI models and architectures that could better leverage the data and context to solve natural language problems (for example Google’s BERT).

These AI capabilities continued to evolve into Large Language Models (which power many of the products available in the market today) as well as more recent related capabilities like agentic AI.

Judgment, Execution and AI’s Role

The lower the complexity or level of judgment required to solve a particular task (in other words the simpler or more standardized the task), the more likely it is that the task can be codified in the form of data and context. The more relevant data and context available to train the AI models to solve it, the more likely it is that AI can assist with (or even automate) that workstream.

As a result, in current state, a contract analyst with the exact same level of skill as a contract analyst 10 years ago could more easily be replaced with AI. This is why continuous learning and upskilling is so important: these help build new types of judgment. For example, knowing how to use AI effectively has emerged as a new type of judgment that the market demands.

Incorporating AI into more experienced attorneys’ workflows is the logical next step as those workflows have been commonly applied and model capabilities have evolved over the last year or two and are more accessible than ever before. As before, the proportion of a workstream that was “judgment” has been restructured or reclassified as execution as a result of the consistent use and standardization of those tasks along with AI’s evolving capabilities.

So with all of that context in mind, what are examples of how we can actually separate out judgment from execution and related, use AI tools effectively?

Applying AI to Legal Workstreams

I would frame this in three main categories, from lowest level of effort to highest:

1. Using AI as a thought partner / to help plan a workflow

2. Breaking down the existing end-to-end process for performing a legal workflow into separate but related tasks and identifying where AI can assist

3. Re-engineering the process entirely from the ground up / from first principles

The first two ways above are the market norm right now, but the third is emerging as an area of exploration as well. In addition, performing the breakdown of the existing process is often helpful for understanding opportunities for all three categories.

AI as Thought Partner

This is one of the simplest, most popular ways of using AI in legal workflows.

Simply enter a prompt into an LLM-powered tool such as ChatGPT or Claude, and the AI tool can help you plan out a workstream (for example drafting a contract or researching a legal issue). This is similar to augmenting judgment with data.

If desired, a person can iterate on the plan with the AI tool or simply make adjustments as needed and execute on it.

Breaking down the existing end-to-end process

However, to get a more comprehensive view of where AI can fit into the process, it’s often helpful to systematically break down the end-to-end workflow or process into tasks, subtasks and related.

Here’s an example of a concrete legal workflow (contract drafting) and how we can break down the related end-to-end process this way. Two caveats:

1. This is a high-level, non-exhaustive example but serves to illustrate the overall technique for doing so.

2. The process is iterative and is not meant to be perfectly linear or sequential (for example going back to earlier stages or jumping around may make sense).

Contract Drafting

• Trigger or notification — notification of an event that requires contract drafting (for example a new client or opportunity)

• Intake / Scope out problem — structure the initial notification or request and request additional context as needed

• Create plan — create a plan for executing the workflow, including defining the scope of what you’re looking for and why

• Define criteria — define the concrete search conditions to help you find what you’re looking for

• Perform research — apply the search criteria to locate the relevant information

• Summarize research — condense the most important results from the search into a relevant summary (e.g. relevant legal issues that may be applicable)

• Analysis — validate the initial search results and summary, incorporate nuances, exceptions, and additional criteria, and perform a final assessment before moving to drafting

• Draft Contract — either select and tailor an existing template or if dealing with a brand new issue, start from scratch (less common)

• Share contract draft / communicate — send a draft of the contract along with any relevant information (for example a cover letter or email) to the counterparty or other stakeholder

We could then add additional steps if we wanted the workflow to cover the execution / signature of the contract as well:

• Negotiate

• Edit / refine the contract as needed (redlining)

• Get approvals as needed (for example if agreeing to non-standard terms)

• Sign / Execute the contract

Note that there are commonalities with other legal workflows as well: for example, drafting a legal memo. We could abstract the “draft contract” workflow into something like “draft or create deliverable” and then re-use / re-purpose steps in the process. We could also replace one task with another as needed: for example in the case of certain litigation documents, instead of signing the document we could also file it with the courts.

Applying AI to each step of the process

Once we break the process down end to end, we can assess where it would potentially be helpful to apply AI tools to improve the efficiency or quality of the workflow.

Here are a few examples of how AI could assist with each step of the above process:

• Trigger or notification — proactively perform regulatory monitoring to help inform legal and business planning. Here’s an example of how Amazon’s legal team is doing this internally.

• Intake / scope out the problem — identify known information, assumptions and where additional information would be helpful. Here’s an example of a Claude / ChatGPT skill I put together to help do this.

• Act as a thought partner / help form a plan — help form an execution plan (per category one above).

• Research — perform research into common legal issues that come up during contract drafting exercises of this type.

• Draft — draft the contract, using whatever context or template that we want to provide to do so.

In other words…on paper AI can assist with every single part of the workflow.

So the relevant question becomes should AI be used rather than can it be used?

The answer is: it depends.

How to determine whether to use AI

To answer the question, the next step is translating the conceptual framework into practical, problem-solving considerations. Note that there’s no one-size-fits-all solution and usually our specific situation and context will inform the approach.

However, here are some considerations that will inform whether it makes sense to apply AI to a given workflow or individual task:

• Financial costs — for example, costs of AI software licenses (both fixed and variable depending on usage).

• Vendor diligence / related costs — there are many AI providers and they need to be vetted. I put together a diligence checklist (available here) that can be used to help with this.

• Setup / onboarding time — learning how to use the tools, configuring them, etc.

• Opportunity cost (cost of time / best alternative) — time that could be spent doing something else (e.g. working on different matters).

• Benchmark of performance / current state — without using AI, how efficient and valuable is the existing process?

• Incremental performance improvement — how much better is the AI relative to current state or the best alternative?

• Maintenance costs — how difficult is it to maintain performance?

• Other considerations — e.g. aligning incentives to make sure efficiency doesn’t conflict with time-based pricing models (one common one: the billable hour).

Creating an entirely new process

Finally, as AI and other related technological capabilities become more advanced, there’s an open question regarding whether we can go beyond breaking down or refining an existing process by applying AI to individual tasks within that process.

Specifically, it raises the question whether we should completely change how we think about things and whether certain tasks or even entire workflows should exist at all.

Here’s a historical analogy: physically going to the library to do legal research was a common practice at one point. As legal research became digitized and sufficiently reliable relative to research done using physical copies, going to the library was eliminated as a practice in most cases.

Imagine if in a future state, a key research area, such as those underlying certain types of contracts became sufficiently clear to the point where research was eliminated as a necessity when drafting contracts.

Or during negotiation, what if the data and processes supporting negotiation playbooks and cycles became so standardized and predictable that two AI agents could essentially negotiate against one another and come up with the optimal outcome for both sides, and then communicate the results.

Ultimately as with the other examples of applying AI, we’ll need to assess the trade-offs of doing so. But we can imagine that some re-engineered processes are very impactful and happen sooner rather than later.

The Contract Signal is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.