The Contract AI Vendor Diligence Checklist

46 questions to test the data, models, workflow, usability and business behind a contract AI product.

“We were losing more than one million dollars per year.”

A former colleague told me this in 2020, describing a failed AI project. I was interviewing them for any wisdom they could share as I stepped into a product manager role for our machine learning program. They had engaged a third-party provider with best-in-class technical credentials and a great demo — and had virtually nothing to show for it beyond spent time, effort, and money.

Unfortunately, this is not an isolated incident in enterprise AI. I’ve seen the same themes surface again and again when doing diligence on third-party providers, working alongside teams of AI engineers and data scientists, running my own experiments with AI tools, and following the industry’s research and best practices.

The good news is most of these problems could have been prevented (or at least significantly reduced) with better diligence before a purchase or investment decision is made.

That is the guiding principle behind this questionnaire: help separate fact from fiction by examining the vendor’s end-to-end process. What problem it solves, how contract data is produced, how performance is measured, how results are verified, how users consume the deliverable and whether the underlying business and technology stack can support the promised outcome.

One call-out - when using the questionnaire, do not rely only on verbal answers. Wherever possible, request written evidence: sample outputs, citations, test methodology, data-flow diagrams, sandbox access, implementation plans, customer references and contractual quality guarantees or other commitments. These are often more substantive and more reliable than verbal responses alone.

Subscribe to get the full checklist (it’s free).