This post is for CTOs, procurement leads, and founders writing an RFP for an AI build. It gives you 12 questions to demand written answers to, what a good answer looks like for each, the red flags that should disqualify a vendor, and a simple rubric for scoring the responses.
Most AI development RFPs read like generic software procurement templates with the word "AI" sprinkled in. They ask for company history, team size, project lists, and a fixed-price quote against a one-paragraph scope. They do not ask the questions that actually predict whether the engagement will ship. The result is procurement teams selecting on price and brand recognition, and CTOs inheriting failed engagements six months later.
We have written and audited dozens of AI vendor RFPs across the US, UK, Canada, and Australia in the last three years. The 12 questions below come from those audits. Each one separates vendors who have shipped production AI from vendors who have read about it. Demand written answers, not slide decks. If a vendor cannot answer a question in writing inside 48 hours of receiving the RFP, the engagement will be in discovery phase for the rest of its life.
1. Who exactly will write the production code?
Ask for named engineers, their years of LLM production experience (not generic software), and a written commitment that the named engineer will be on every standup and write the merged PRs. Bait-and-switch staffing is the single largest source of failed AI engagements: a senior on the sales call, a junior pool on the build. Anchor the contract on names, not headcount.
2. Will an eval harness exist in week one, or week ten?
Eval-first means the test set is built before architecture choices are finalized. Vendors who build the eval set in week one anchor every subsequent decision on measurable accuracy. Vendors who defer it to phase two have a reason: they do not want their early prompt iterations measured. By the time the eval lands, the architecture is locked in and the regressions are silent. Our post on building an eval harness from scratch covers the structural pattern.
- Ask: in which sprint does the eval set get drafted, reviewed, and frozen?
- Ask: how many test cases will the v1 eval set contain, broken down by category (factual, refusal, format, latency, cost)?
- Ask: which member of our team will sign off on the eval set as representative of production traffic?
- Ask: is the eval gated in CI so prompt or model changes that regress accuracy cannot merge?
3. Where does inference run, and which provider sees the data?
The deployment architecture is the single most consequential decision in an AI engagement, and it gets made in the first two weeks. Ask whether inference will run inside your VPC, on a shared SaaS provider, or on the vendor's infrastructure. Ask which LLM provider receives the prompts and what data residency commitments are in writing. Ask how secrets, embeddings, and audit logs are isolated per tenant.
A vendor with real deployment experience will distinguish between hosted LLM with a DPA, hosted LLM with a BAA, dedicated tenancy, and self-hosted Llama or Mistral inside the customer VPC, and will recommend the right pattern for the workload's regulatory posture. See our healthcare AI development page for the HIPAA decision points and our UK GDPR AI development page for the UK ones.
4. Who owns the IP, the prompts, and the eval set?
Standard work-for-hire language covers the source code. AI engagements have three other assets that frequently fall outside the contract: the prompt library, the eval set, and the fine-tuned model weights. Vendors who quietly retain ownership of these assets keep the customer locked in long after the engagement ends.
What good looks like: IP assignment in the contract explicitly enumerates source code, system prompts, eval datasets, fine-tuned weights, retrieval indexes, and runbooks, all assigned to the customer on milestone payment. Red flag: "vendor retains rights to general knowledge and methodology" applied broadly enough to cover the prompt library.
5. What are the takeover terms?
The takeover audit checklist is the single best predictor of a healthy engagement. Ask whether the vendor will deliver, as part of the standard deliverable set: a versioned prompt library with model and retrieval config tags, the eval set checked into the customer's repo, observability instrumented and dashboards exported, runbooks for incident response, secret rotation procedures, and a written handoff plan for the customer's team to operate the system without the vendor.
A vendor that ships these as standard has thought about what happens when the engagement ends. A vendor that treats them as a phase-2 retainer upsell is selling lock-in. Our post on vendor takeover audit signs covers the symptoms that surface when this question is dodged.
6. Compliance posture, beyond the certification badges
Every vendor says they are SOC 2, HIPAA-ready, GDPR-compliant, and ISO 27001 attested. The question is what those certifications actually cover. Ask for the SOC 2 Type II report under NDA and read the scope section: does it cover the development environment that will write your code, or only the corporate SaaS the company uses internally?
- US: SOC 2 Type II scope must cover AI development practices, code review, and secret handling. HIPAA BAA available if PHI is involved. CCPA and the growing patchwork of state privacy laws documented.
- UK: UK GDPR and a documented DPIA workflow; ICO guidance on automated decision-making referenced in writing.
- Canada: PIPEDA, Quebec Law 25, and OSFI E-23 for federally regulated banks.
- Australia: Privacy Act 1988, APP-compliant data flows, and APRA CPS 234 and CPS 230 if financial services.
7. What real metrics do you ship, not vanity stats?
Every AI vendor's pitch deck has the same five metrics: "50% productivity gain," "10x faster," "90% accuracy," "ROI in 3 months," and a customer quote. Almost none of them are measured against a real eval set. Ask for production metrics with categories: citation accuracy (for RAG), tool-call success rate (for agents), refusal rate, p95 latency, cost per request, and category-level eval scores. Ask for them on a recent engagement, not the lighthouse from 2023.
Concrete examples from our own production stack: 98.4% citation accuracy on a medical-inquiry RAG agent at a 6% refusal rate, 68% L1 deflection on a telco SMS bot handling 110k conversations a week, 1,400 hours a month saved on an EU insurance back-office build, a 47% completion lift on an adaptive interview agent, and sub-2-second voice latency on an outbound voice agent making 4,000 calls a day. All measurable, all reproducible against the eval sets that produced them, and all documented in our case studies.
8. What does observability look like on day one?
Observability is not an afterthought for AI systems; it is the operating manual. Without per-call audit logs, token-cost tracking, latency p50/p95/p99, refusal-rate dashboards, and tool-call success rates, the system is opaque the moment users hit it. Ask the vendor which observability stack they instrument by default and what the day-one dashboard looks like. Our agent observability post has the full inventory.
A vendor with production experience will name Braintrust, Langfuse, Arize Phoenix, or an equivalent, and will have an opinion on which is the right call for your traffic profile. A vendor without production experience will say "we will instrument observability as part of the production hardening phase," which means it does not exist yet.
9. What is the eval cadence after launch?
Production AI drifts. Model providers update versions, prompt edits creep in, retrieval indexes age, and the eval set itself needs expanding as new query categories emerge. A vendor that ships once and walks away leaves you with a frozen system in a moving world. Ask what the post-launch eval cadence is: weekly automated runs against the golden set, monthly review against drift detection, quarterly expansion of the eval set itself.
The honest answer involves either a retainer with a defined eval cadence, or a written handoff plan for the customer's team to run the eval cadence themselves. Either is acceptable. "We will check in if you have issues" is not.
10. How does escalation work when something breaks?
Every production AI system will have incidents: a model provider outage, an embedding index corruption, a prompt regression that slipped past the eval gate. The question is who answers the page at 2am. Ask for the on-call coverage windows, the response-time SLA, the named engineers in the rotation, and the communication channel (Slack Connect, a dedicated PagerDuty integration, or email only).
What good looks like: a written SLA with defined response and resolution windows per severity tier, named engineers in the rotation, Slack Connect or an equivalent direct channel, and a published incident review template. Red flag: "escalation through your account manager during business hours."
11. Cost transparency: what changes the bill?
AI engagements have moving cost components that traditional software does not: per-token LLM costs, embedding generation, vector store hosting, voice STT/TTS billing, observability platform fees, and managed-eval-platform subscriptions. Ask the vendor for a written cost-per-component breakdown: per LLM call, per 1k tokens of context, per voice-agent minute, per RAG ingestion run.
The same vendor should be able to project the monthly run rate for your expected traffic volume, broken down by component. A vendor who quotes a single all-in monthly number is either masking the cost structure or has not done the math. For a starting point on what drives the numbers, see our AI development cost guide.
12. What is the contract structure, and what risk is each party taking?
The contract structure determines the incentive structure for the rest of the engagement. Time-and-materials transfers schedule and scope risk to the customer; the vendor has no incentive to finish. Fixed-price transfers risk to the vendor, but only if the scope is well defined and the vendor stands behind overruns. Hybrid milestone-based contracts can work if the milestones are real acceptance criteria, not progress reports.
- Fixed price with written acceptance criteria and a vendor-absorbs-overrun clause: best for v1 scoped builds.
- Milestone-based with eval-set acceptance gates: best for multi-phase deliveries with regulatory review.
- Time-and-materials with weekly burn caps and signed weekly scope: acceptable for genuinely exploratory work.
- Pure T&M with no cap and no fixed scope: avoid unless the engagement is research.
How to score the responses
Send the 12 questions to four to six vendors and score the written responses on a simple rubric: 2 points for a substantive answer with a named example, 1 point for a hedge, 0 points for a dodge or a checklist response. Vendors who score 18 or more out of 24 have shipped production AI. Vendors who score under 12 are pitching slideware. The narrowing happens before the first sales call, which is when the unfit vendors are most expensive to engage with.
The 12 questions also act as the spine of the SOW. The vendor's written answers become contractual commitments: named engineers, eval cadence, deployment architecture, IP assignment, takeover deliverables. If the vendor's actual delivery later diverges from the written RFP response, you have explicit grounds for escalation or termination.
The vendor you want is the one who answers fast
A vendor that has shipped dozens of AI systems can answer the 12 questions in writing inside 48 hours. The answers exist as standard language in their pre-sales materials because they have written them many times before. A vendor that needs three weeks to draft answers is working them out for the first time during your RFP. That delta, the speed of a substantive response, is one of the cleanest signals you will get in the entire selection process.
If you are scoping an AI engagement and want a 30-minute call that runs the 12-question rubric against your specific build, book a discovery call. We will return a written one-pager inside 72 hours with the SOW shape, the eval-set proposal, the deployment architecture, and the fixed-price number. It is the same artifact your other shortlisted vendors should be able to produce.
Frequently asked questions
How quickly should a vendor answer an AI RFP?
In writing, inside 48 hours. A vendor that has shipped production AI already has these answers in its pre-sales material. One that needs three weeks is working them out for the first time on your RFP.
Which contract structure is best for an AI build?
Fixed price with written acceptance criteria and a vendor-absorbs-overrun clause for a scoped v1; milestone-based with eval-set acceptance gates for multi-phase or regulated deliveries. Uncapped time-and-materials is only appropriate for genuine research work.
What should the IP clause in an AI engagement cover?
Source code, system prompts, eval datasets, fine-tuned weights, retrieval indexes, and runbooks, all assigned to the customer on milestone payment. A broad 'vendor retains methodology' clause that covers the prompt library is a red flag.