Enterprise Software Vendor Selection: 2026 CTO Checklist (38 Questions)
Most software vendor selections happen through a tender process: collect three to five quotes, pick the cheapest (or the one with the best pitch deck), sign a contract.
This method fails for three reasons:
The quote represents only 30-40% of the real total cost. The rest surfaces after the contract is signed (scope creep, bug-fix contracts, missing handover materials).
Pitch quality ≠ delivery quality. The sales engineer who runs the demo will not be on your project.
Eighty percent of reference calls return positive feedback. Because vendors list only their happy customers.
The 38-question checklist below was distilled from 16 years of custom software work — both projects we built and projects we were brought in to rescue. Most questions push the vendor; the answers teach you a lot more.
For each category, we cover the questions, why they matter, how to test, and the red flags.
Category 1: Technical Capability & Stack Fit (6 Questions)
The goal here is not "do they know it?" but "do they know it fits this project?"
1. Can you show three production case studies at similar scale?
Why it matters: Anyone who can say "we can do it" is not saying "we did it."
How to test: Ask for live URLs or a client-acknowledged live phase. Don't count every "under NDA" case as real.
Red flag: Three out of three cases "under NDA" → likely either nonexistent or failed.
2. Can you name the top three technical risks of this project right now?
Why it matters: A vendor who can name specific risks before discovery is speaking from experience.
How to test: Check if the answer is categorical (generic) or specific. "Scalability matters" is weak; "Stripe Turkey side-by-side payments are still partial in 2026; we'd need a PayU + Iyzico fallback" is strong.
3. Why this stack choice? What was the alternative?
Why it matters: Vendors usually recommend what they know, not what's right.
How to test: Pressure-test with alternatives: "Why Node and not .NET?" A vendor who immediately retreats to "our team knows it better" is not optimizing for you.
4. Can you estimate cloud and infrastructure cost right now?
Why it matters: A good vendor thinks beyond code to total cost of ownership.
How to test: Ask for AWS/GCP/Azure monthly estimate plus scaling thresholds. Vendors who can't have never gone past dev.
5. Walk me through CI/CD, automated testing, and monitoring from scratch
Why it matters: Half of vendors with "modern stacks" have no production monitoring.
How to test: Ask for specific tool names (GitHub Actions / GitLab CI / Argo / Jenkins; Datadog / Grafana / New Relic / Sentry). Vague answers = not doing it.
6. What was the biggest technical mistake in a previous project?
Why it matters: Single most powerful question. A team that can name its mistake has learned from it.
Red flag: "We don't make mistakes" → either lying or inexperienced.
Category 2: Process & Methodology (6 Questions)
"We work in Agile" tells you nothing. Ask for the specific process.
7. How long is your discovery / inception phase, and what does it deliver?
Expected: 2-4 weeks plus concrete deliverables (technical spec, user stories, risk registry, milestone plan).
Red flag: "We can start without discovery" → scope creep is guaranteed.
8. How are sprints planned? Who tracks velocity?
Expected: 1-2 week sprints, sprint planning + review + retro, scrum master or tech lead tracking burndown.
9. How does code review work?
Expected: Every PR gets at least one senior review, automated linting + test gates, blocked merges.
Red flag: "The PR author can self-merge" → no quality control.
10. QA split: is developer testing enough, or do you have a separate QA team?
Expected: Mixed model — unit tests with devs, integration + UAT + regression with QA + automated tests.
11. What's your SLA when a bug appears? Response time for a production-critical issue?
Expected: Severity 1 (P0) - under 2 hours response, under 24 hours resolution.
12. When a milestone slips, when and how do we hear about it?
Expected: As soon as the slip is forecast (1-2 sprints ahead), with reason and remediation plan.
Red flag: "We'll tell you if there's a problem" → you'll hear about it at the last minute.
Category 3: Team Structure & Stability (5 Questions)
The pitch team is not the delivery team. This is the category most CTOs skip.
13. Can I see CVs of the team that will work on this project?
Expected: Name, role, years of experience, prior project list, profiles of real engineers on staff.
Red flag: "We'll share once the team is allocated" → no allocation yet.
14. What's the senior / mid / junior ratio?
Expected: Senior 25-40%, mid 40-50%, junior 15-25%. All-junior teams are cheap but expensive (rework + handover overhead).
15. What's your annual team turnover rate?
Expected: 15-25%. Above 40% → no team stability, your project will suffer handover damage.
16. How many other projects are these engineers allocated to?
Expected: 1.0 FTE (dedicated) is ideal; 0.5-0.75 FTE is acceptable; below 0.25 FTE → they're not actually giving you capacity.
17. Which roles are onshore vs. nearshore/offshore, and is that split clear?
Why it matters: Some vendors front a small team locally, while the actual build happens overseas. Timezone and communication issues surface after the contract.
Category 4: Transparency & Reporting (6 Questions)
"It's going well" is the most expensive sentence in software. Without visibility, there's no control.
18. Can I get repo access (GitHub/GitLab/Bitbucket) on day one?
Expected: Yes, read access at minimum.
Red flag: "We hand it over at project end" → the code is theirs, not yours.
19. At sprint demos, do I see live code in a real environment, or just slides?
Expected: Working environment (staging/dev) demo, every sprint.
20. Can I see actual hours spent or story points consumed?
Expected: Jira / Linear / ClickUp visibility.
Red flag: "No need for that detail, we're fixed-price" → you lose every scope argument later.
21. Do you report velocity, bug rate, deployment frequency?
Expected: Monthly report with trend graphs.
22. After discovery, how do you handle scope changes?
Expected: Written change request, impact analysis (effort + cost), approval before continuing.
23. Do you report quality metrics (test coverage, technical debt)?
Expected: Test coverage above 70%, SonarQube or equivalent code quality report.
Category 5: Contract, Pricing & IP (8 Questions)
Most expensive problems are hidden here.
24. Pricing model: fixed price, T&M (time & material), outcome-based, which and why?
Expected: Decided after discovery - discovery itself is usually T&M or fixed-fee; main build is T&M or hybrid.
Red flag: A fixed-price quote for the entire project before discovery → either inflated with buffer, or you'll be billed extra for changes.
25. Is the IP (intellectual property) clause clear? Who owns the code?
Expected: All code and documentation belong to you, on payment completion.
Red flag: "License model" or "right to use" language.
26. If the vendor's own libraries / frameworks are used, are they removable?
Why it matters: Vendor lock-in is the largest hidden cost. If internal frameworks can't be transferred to you, leaving the vendor becomes impossible.
Expected: All utility libraries are either MIT/Apache-licensed or transferred to your ownership.
27. Can I exit after discovery + design phase? Are there phase gates?
Expected: Continuation option at end of each major phase (discovery, MVP, scale), with penalty-free exit rights.
28. Are there penalty / SLA breach clauses in the contract?
Expected: Discounts on missed milestones, rejection rights on quality gates.
29. How often is documentation (architecture, deployment, runbook) updated?
Expected: Continuously (Confluence / Notion), with every major change.
30. What's in the handover package at project end?
Expected: All code + repo access + deployment runbook + architecture diagram + 30 days of support + knowledge transfer sessions (2-4 sessions).
31. What's the payment schedule? What's the upfront percentage?
Expected: 20-30% upfront, rest milestone-linked. Above 50% upfront → red flag.
Category 6: References, Domain Expertise & Sustainability (7 Questions)
32. In this vertical (e-commerce / fintech / health / logistics / ERP), how many production cases do you have?
Expected: At least 2-3, with domain-specific regulatory knowledge (GDPR, HIPAA, SOC 2, PCI DSS, FDA, etc.).
33. As a reference, which was the hardest (most difficult client/project)?
Why it matters: Vendors list only happy clients. The "hardest" question pulls out real stories.
34. Have you lost clients? Why?
Expected: An honest answer. A vendor claiming zero churn is either lying or new.
35. Can I do reference calls with clients I choose from your list?
Why it matters: Vendor-selected references are cherry-picked. Self-selected references are more objective.
How to test: Ask for the client list, then you pick 2-3.
36. What's the current state of past clients' ownership? Are they running it themselves, or still on your support?
Why it matters: Good vendors don't leave clients dependent. Bad vendors live on perpetual support fees.
Expected: Mixed - some clients self-manage (handover succeeded), others continue on managed services (by choice).
37. Can we embed 2-3 of our engineers into your team?
Why it matters: Embed transfers knowledge fastest. Refusal usually means IP paranoia.
38. What's the vendor company's financial stability? Growth/contraction over the last 3 years?
Why it matters: If the vendor shrinks mid-project, you lose your team.
Expected: Growing (10-30% YoY), profitable, no mass layoffs in the last 12 months.
Quick Decision Matrix
Have each vendor answer all 38 questions. Score each answer 0-3:
3 — Excellent: Specific, with numbers, with concrete examples
2 — Good: General but reasonable, they can do it
1 — Weak: Vague, cliché, no proof
0 — Red flag: Evasive, contradictory
Total score interpretation:
Total Score Meaning
95+ Outstanding vendor - rare
75-94 Reliable partner - move to contract
55-74 Acceptable - follow up on weak categories
35-54 Risky - re-evaluate
<35 Do not work with this vendor
If Category 5 (Contract & IP) averages below 1.5, do not work with this vendor regardless of total score. A bad contract makes everything else irrelevant.
Three Extra Rules to Avoid Mistakes
Rule 1: Run discovery on a separate contract. Decide the whole project after discovery. Walking away from a vendor after discovery is 10x cheaper than walking away after the main contract.
Rule 2: Don't pay a single invoice until the code repo + documentation belong to you. Take read access on day one.
Rule 3: Don't work with vendors who refuse embed. Good vendors welcome knowledge transfer; bad vendors sell black boxes.
Next Step
Use this checklist. Send it to your vendors. If they refuse to answer, they've already answered you.
Vendor selection is one piece. Once you've decided, calculate your project's ROI: Custom Software ROI Calculation Framework (2026).
If you're still deciding between custom build or SaaS: Custom Software vs SaaS: 9-Question Decision Framework (2026).
If you're trying to rescue a failing project or considering a vendor switch, our Software Graveyard rescue process is built exactly for this.