A former colleague described the unsettling experience of losing a million dollars a year to a contract AI vendor run by industry-leading PhDs.
But how? It turns out that many of the most common obstacles to getting ROI from contract AI solutions aren’t due to the models themselves.
There are three main categories of things that I’ve found helpful when conducting contract AI vendor diligence:
Mapping out your own workflows to identify the highest-impact opportunities to apply AI
Applying a structured diligence framework to evaluate whether an AI provider is capable of delivering against those opportunities
Knowing what to look for during diligence, which comes from experience in applying categories 1 and 2 above (i.e. judgment)
In previous articles I shared my perspective on categories 1 and 2 (links inline above). These are good starting points. However, making the frameworks actionable requires knowing how to apply them and what to look for.
With that said, for category 3, here are seven common issues that I’ve found helpful to watch out for, along with context and practical alternatives:
Data Can’t Leave the Vendor’s System (Vendor Lock-In)
Context: sometimes the vendor does not allow data to leave the platform once it has been ingested, or makes it very difficult to export or migrate the data. This is increasingly less common in enterprise, as these types of use cases almost always require connecting to or using multiple systems, but it does continue to happen.
Practical alternatives: test the ability to export, migrate or integrate data sets upfront (e.g. via proof of concept), pressure-test the scale / volume at which this can be done, and negotiate the contract to enable this as needed.
Evaluating a contract AI vendor or portfolio company or building your own team? I provide select diligence and advisory consulting covering product architecture, model performance, workflow fit, hiring and implementation risk.