Ask About the Outage: How to Vet a Software Vendor's Real Engineering Ability

The material a software vendor shows you before a contract cannot tell you whether they are good, because a weak firm can produce the same material as a strong one. Two things do separate them: how they talk about what went wrong, and whether they can hand over evidence that was created during real work instead of for the sales process. This article is a practical procedure for collecting both.

Why the standard evidence proves so little

Open ten agency websites and you will see the same parts: success stories with a percentage in bold, headshots, a wall of technology logos and a diagram of a process in four or five stages.

None of this is dishonest. It just cannot be tested. A success story is written by the party it flatters, about a client who is often unnamed. A list of technologies says the team has used a tool, not that they have kept it running through a bad night. A process diagram describes an intention.

Be especially careful with headline metrics. "Cut processing costs by 70 percent" means nothing without the starting point, the period measured, what else changed at the same time and a client you can call. A number that cannot be checked is not evidence of results. At most it tells you the vendor is comfortable publishing numbers that cannot be checked.

Engineering depth shows up somewhere else. Anyone can tell a convincing story about a project that went well. Only someone who was in the room can describe, in detail, the night the database ran out of connections.

The interview: three areas, seven questions

Run this conversation with the people who would do the work, not only with the account lead. If you have an engineer on staff, bring them. You are not grading for a correct answer. You are listening for detail: names of components, times, trade-offs, the thing they changed afterward.

Area one: do they operate software, or only deliver it?

"Describe your most recent production incident." You want an event, not a policy. How was it detected, and how long after it started? Who got the alert? What was the immediate fix and what was the real cause? What is different now because of it? A team that runs systems has this story ready and tells it without notes. Without that experience, the reply you get is their incident management process, recited in the abstract.

"How does your on-call work, and who is on it this week?" A firm that builds and hands over has no rotation, and that is a legitimate business model if it is what you want. A firm that runs what it builds can describe the schedule, the escalation path and what counts as a page, with no preparation at all.

"Walk me through how a change gets from a developer's laptop to production." Listen for automated tests, review by a second person, a staging environment, a way to roll back, and how often they release. If releasing is a tense, scheduled event that happens monthly, expect slow fixes. If it is boring and happens several times a week, that is a strong sign.

Area two: how is their judgment?

"Which of your projects failed, or came close, and what do you do differently because of it?" Every team that has shipped real work has one. The good answer names a decision they made, explains why it looked reasonable, says what it cost and describes what they do differently now. The worrying answers put all the blame on the client or a third party, or describe trouble that was apparently nobody's doing. If they say they have never had a bad project, they are either new or not being candid.

"What happens when a requirement turns out to be wrong halfway through?" This always happens, so the question is really about their contract model and their honesty. A workable answer: we flag it as soon as we see it, we estimate the change, and you choose. An answer to distrust: our discovery is thorough, so that does not come up. That claim will be tested within the first two months.

"What would you decline to build?" A firm with no answer will build anything a client asks for, including the feature that should never exist or the architecture that will not survive its second year. Which thing they refuse matters less than the fact that they have a line.

Area three: who are you really hiring?

"Name the engineers you would put on our project. Which other clients share their week?" The senior people in the pitch being replaced by juniors after signing is the oldest complaint in services. Ask for names, seniority, the share of each person's week you will get, and what happens if one of them leaves. Then write it into the agreement.

How answers tend to sound

Notice also how the vendor reacts to being asked. Good teams tend to enjoy these questions, because the questions let them show what sets them apart. A vendor who keeps steering back to the case study page is giving you an answer too.

Evidence that is hard to produce after the fact

A portfolio is selected and polished. The following items are much harder to stage, and I have ordered them from the cheapest for you to obtain to the most expensive. Work down the list and drop vendors as you go, so the costly steps are spent only on finalists.

1. Their answer on production data access. Ask which people at the firm would be able to open your live data, what the procedure is for giving and withdrawing that permission, and whether a log records every use of it. If every engineer holds permanent access, then no real control exists, regardless of what their website says about security. Good answers involve named individuals, time-limited grants and an audit trail.
2. A description of their release pipeline. This is the deployment question again, in writing or as a screen share. Seeing the real pipeline run takes 15 minutes and is very hard to fake.
3. A postmortem they wrote. Ask for one with client details removed. You learn whether failures are analyzed for causes or just survived. Look for a timeline, contributing factors and follow-up actions with owners, and the absence of blame aimed at individuals.
4. A sample of real code, sanitized. Have a technical person read it. Are there tests? Do comments explain why something is done, instead of repeating what the code says? Can a stranger follow the structure? You are judging habits, not style.
5. A reference from a client who left. Current clients are happy by selection. A former client tells you how the firm behaved at the end: whether they handed over documentation and credentials cleanly, and whether the system could be maintained by someone else. Most of the risk in these relationships sits at the end.
6. A small paid engagement. More on this below.

A single item from this list tells you more than a stack of case studies, because each was produced for a real purpose before you asked.

Deal-breakers: when to walk away

A few signals are sufficient by themselves, with no need to weigh them against anything else.

• A firm quote issued before anyone has looked inside your systems. Either the number contains a large cushion or the scope will be reopened later. Both outcomes are paid by you.
• Roles instead of people. If nobody will name the engineers, you are buying hours from a pool. That can be acceptable, but only if it is what you were told you were buying.
• No view on your architecture. After a serious conversation about your systems, a competent engineer has opinions and concerns. Neutral agreement with everything you say means your problem has not yet received any real thought from them.
• "We can handle compliance." For regulated data, such as health information under HIPAA, the useful answer is concrete: we have done this under signed agreements, here is how we control access, here is what we log. Vague confidence is not experience. (General information, not legal advice.)
• Reluctance to put assumptions in writing. Written assumptions are how a scope survives its first surprise. A vendor who avoids them wants room to maneuver later.
• Pressure to skip steps. A discount that expires on Friday is a sales tactic and has nothing to do with engineering.

A simple scorecard

If you are comparing several firms, keep it mechanical so charisma does not decide. Score each area from 0 to 2: 0 for no answer or a red flag, 1 for a plausible general answer, 2 for specific, checkable detail.

• Operations: incident, on-call, deployment
• Judgment: failed project, wrong requirement, refusals
• Staffing: named people and allocation
• Artifacts: data access, postmortem, code, departed reference

Any deal-breaker from the previous section removes the vendor regardless of the total. Among the rest, low scores in operations matter most if the vendor will also run the system.

The best test is a small real project

Interviews and artifacts reduce risk, but nothing replaces watching a team do paid work on your actual problem. Before committing to the full build, commission something small with a real deliverable. A discovery phase of about two weeks is the natural form.

You then judge the output on its merits. Did they understand how your systems really work? Did they find something your own team did not know? Did they name risks, including ones that are inconvenient for their own proposal? Is the plan one you would follow, and could another firm follow it? A disappointing result at this stage is cheap knowledge, bought with a small fraction of the budget.

Two conditions make this test honest. The deliverable must belong to you, and it must be usable without the vendor who wrote it.

Keep geography as a separate question

Buyers often mix "can this team build it" with "where is this team located". These are different questions. Strong and weak teams exist in every country. Location affects overlap hours, coordination cost and, for regulated data, who is allowed to touch what. Evaluate depth first, then evaluate the delivery model.

Which of these questions would you most like answered?

Put all seven to Trivo. We will answer each one, name the engineers who would be on your project and tell you directly if the work is not a good fit for how we operate.