"Our agents will handle everything for you. No team needed. Just plug us into your stack and we will automate your entire funnel in 30 days."
I watched a founder nod along to that pitch last month. He was 20 minutes from signing a six-figure contract.
I felt my stomach drop. Not because the vendor was lying. Because the pitch was so familiar I could finish their sentences before they did.
Ten years ago, this was the dev shop pitch. Five years ago, it was the no-code pitch. Three years ago, it was the chatbot pitch. Now it is the agentic AI pitch.
Same script. New buzzword on the brochure. Same founders, walking into the same room, about to make the same mistake.
I want to write down what I have learned from watching that mistake happen, and from making versions of it myself, because the cost of repeating it with agentic AI is going to be higher than it was with dev shops. The agent will be making real operational decisions inside your business. The dev shop just shipped you code.
This is the playbook I use inside Chykalophia when I evaluate AI vendors, and the one I push founders to use before they sign a contract.
I am not anti-AI
I want to be clear about this before I go further.
We use agentic AI at Chykalophia for things I would not have believed possible 18 months ago. I am writing parts of this article inside one of those tools. The productivity gains are real. The teams that figure out how to use these systems are going to outpace the teams that do not, by a significant margin.
This article is not "be skeptical of AI." It is "be skeptical of vendors who are using AI as a magic word to bypass the operational questions you would normally ask a tech partner."
There is a difference between buying a tool and buying a partnership. AI vendors are trying to sell you the second one. Treat it like the second one.
Why this pattern repeats every cycle
Founders get burned by the same pattern over and over. It is not because founders are dumb. It is because the pattern is engineered to bypass the part of your brain that asks operational questions.
Here is how the pattern goes, in three acts.
Act 1: The magic moment
The vendor leads with the magic. "Our agents will handle X for you." The demo looks great. The senior person on the sales call is genuinely impressive. They show you a customer logo wall. They promise speed.
The room fills up with possibility. You start picturing the operational headache this is going to solve. Your team is excited because they would also like to stop doing that headache.
The questions you would normally ask about a tech partner never come up. Not because they were dodged. Because the room never goes there. The energy is wrong for it.
Act 2: The signature
You sign. The senior person disappears. You get handed to an account manager who has read your contract but not your business.
Implementation goes slower than the demo suggested. There are integration issues that were not in the slide deck. The vendor explains them as edge cases. You start to wonder if your operational complexity is unusual. It is not. It is just that the demo was a happy path.
You are now six weeks in. You have already paid. The sunk cost has started to do its work.
Act 3: The 2am page
The first time the agent does something wrong at 2am, nobody answers the page until 9am the next day. The vendor's first instinct is to explain why the wrong thing was actually the right thing. There is a back-and-forth that takes two weeks and ends with a workflow tweak.
You realize you are now operating an AI vendor on top of operating your business. You did not sign up for that.
This is the dev shop story. It is also the agentic AI story.
The packaging is different. The failure mode is identical.
The three conversations to have before you sign
When I sit down with a founder who is evaluating an AI vendor, I push for three specific conversations before they sign. Not three slide decks. Three conversations, with the answers in writing.
These three conversations work because they are too specific to fake. A vendor who has done the operational work can answer them in a way that makes you confident. A vendor who has not done the work will use adjectives, deflections, and "we can get back to you on that." The deflections are the data.
Conversation 1: References that actually look like you
Most vendor evaluations stop at the customer logo wall. The logos are not references. The logos are decoration.
Real references are two or three conversations with clients who are about the same size as you, in roughly the same industry, with roughly the same operational maturity. You should be able to ask the reference client uncomfortable questions and get honest answers.
When I push a vendor for these references, I am listening for two signals.
Can they produce comparable references at all? A vendor with no comparable clients is selling you a science experiment. That might be fine if you are pricing it like a science experiment. It is not fine if you are pricing it like a finished product. If their entire reference list is enterprise and you are a 30-person team, those references are not for you. They are for the next enterprise client they are courting.
What do the references actually say about the operational side? Not the demo side. Did the implementation timeline match what the vendor promised? When the agent did something wrong, how fast did the vendor respond? Who at the vendor actually owns the relationship six months in?
If the references all say "the demo was great and onboarding was smooth," that is not a reference. That is an early honeymoon report. The vendor probably handpicked clients who are still in month one.
If the references say "they were responsive when the agent flagged a refund incorrectly and they ate the cost," that is a reference. That client has lived through an actual failure with this vendor and is still willing to vouch for them.
The reference question I always ask
"What did you wish you had asked them before you signed?"
This question almost always produces a real answer. The reference client has now lived through the contract and they know what they would do differently. Their answer is your shortlist of pre-signature asks.
Conversation 2: Architecture, not demo
This is the conversation most founders skip because they feel underqualified to evaluate the answer.
You do not need to be a CTO to ask the architecture question. You need to be able to listen for whether the vendor has thought it through. If they have, they will walk you through it with specifics. If they have not, they will give you adjectives. Adjectives are a tell.
Here are the questions I push founders to ask. None of them require a technical background to ask. All of them require a technical background to answer well.
Where does our data live?
On your servers, on theirs, somewhere else? Who has access to it? What happens to it if we churn?
A vendor with a good answer will know the answer immediately and they will be able to show you a data flow diagram. A vendor with a bad answer will need to "check with the team and get back to you."
If your business is in a regulated industry, this question is not optional. Healthcare, financial services, legal, education. The vendor's answer here determines whether you can even sign with them. Get the answer before you get to the contract.
What is the data flow when the agent makes a decision?
Inputs, processing, outputs, logs. Can you show me the diagram?
The diagram is the test. A vendor with a clear architecture will hand you a diagram. A vendor with a fuzzy architecture will describe it verbally and you will leave the meeting not really knowing how the system works.
Verbal descriptions are not architecture. They are marketing.
What are the failure modes?
When the agent gets it wrong, what is the path from "wrong outcome" back to "fixed workflow"? Is there a human in the loop? Who?
This question is the most important one in the architecture conversation. Every AI agent will get something wrong. The question is what happens next.
A mature vendor has thought about this carefully. They will tell you about confidence thresholds, escalation paths, human review queues, audit logs. They will probably show you the dashboard their internal team uses to monitor agent decisions.
A less mature vendor will tell you the agent is highly accurate. Accuracy is not the question. The question is what happens during the inaccurate moments. If the vendor cannot answer that, they have not been in production with anyone yet, or they have been in production and they are hoping you do not ask.
What does the human review process actually look like?
Not in theory. In practice. Show me the dashboard.
The dashboard is the proof of concept. If they can show you a dashboard, the process exists. If they describe a dashboard without showing it, the process is on a roadmap.
Conversation 3: Who carries the pager at 2am
This is the question that separates a vendor from a partner.
A vendor sells you software. A partner is on the hook when the software causes a problem. Most agentic AI vendors are selling themselves as partners. Treat them like one in your evaluation.
The questions
If the agent makes a wrong decision tonight, who do I call? What is the response SLA?
What does your on-call rotation actually look like? Where do they live, what time zones are covered, what is the path from "we just paged you" to "an engineer is looking at it"?
When the wrong outcome costs us money, who pays for that money? Is it in the contract? Where?
When the workflow needs to change because the world changed, what is the request process? How long does it take? Who has authority to approve it?
What the answers tell you
A real tech partner has answers ready. They have lived through 2am pages. They have a runbook. They have insurance. They have probably eaten the cost on a wrong outcome at least once and they will tell you the story.
A vendor who has not lived through that yet will tell you "we have not had any major incidents." Translate that to "we have not been in production at scale with anyone yet." That might be fine. Just price it that way. Pay them less. Get out clauses. Run a smaller initial deployment.
The contract terms I push for
When I am helping a founder evaluate, I push for a few specific contract terms that turn the operational conversation into a binding commitment.
A defined response SLA for incidents, in writing, with consequences for missing it.
A defined data deletion clause, with timeline, that activates on churn.
A defined human review process, with the vendor's commitment to staff it, baked into the price.
A defined change request process with a maximum turnaround time for workflow updates.
If the vendor balks at putting these in writing, that is the negotiation surfacing what was already true. They are not ready to be your partner. They just want to sell you software and walk away.
The line that changes the meeting
Here is the single line I want more founders to use this quarter. It is a small reframe and it does a lot of work.
In your next vendor meeting, say this out loud:
"Pretend you are our long-term tech partner, not a magic box. What would you insist we understand before we sign?"
The right partner leans into that question. They start walking through the operational stuff. They tell you the things they wish their other clients had asked.
The wrong partner goes back to the demo.
The line works because it does two things at once.
It signals to the vendor that you are not going to be impressed by magic alone. The salesperson recalibrates. The senior person on the call shifts posture. They stop performing and start consulting.
And it gives the vendor permission to drop the sales posture and have an actual conversation about how the relationship will work. Most vendors have a much more useful conversation in them than the one they are running by default. They just learned, from a thousand demos, that founders do not want to have that conversation. So they stopped offering it.
You being the founder who wants to have it is a gift. They will remember.
A short story about how I learned this
Years before I started Chykalophia I made the dev shop mistake myself. We hired an agency to build a chunk of product. The demo was great. The senior person on the sales call was sharp.
We signed. The senior person disappeared. Six months later we had software that was 80 percent of what we needed and a relationship that was 20 percent of what we needed.
The lesson was not "do not use agencies." The lesson was "I did not ask the operational questions because I was excited about the demo, and I paid for that in cash and time."
Every time I see a founder evaluating an agentic AI vendor right now, I see myself in that conference room watching the demo. The excitement is the same. The questions I am not asking are also the same.
The reason I am writing this article is because I do not want a generation of founders to make this mistake at AI prices. AI is going to be more expensive to get wrong than dev shops were. The dev shop just shipped you code. The AI is going to make decisions inside your business.
What I am not saying
I am not saying do not use agentic AI vendors. We use them. They are real.
I am not saying every vendor is bad. Most of them are trying to do good work. Some of them are very good and worth signing.
I am not saying you need to be a CTO to evaluate this. You need to be a founder who treats this purchase like a long-term tech partnership, because that is what it is.
I am saying the failure pattern is repeating, and the cost of repeating it is going to be higher than it was with dev shops. So slow the decision down by exactly three conversations.
References that look like you. Architecture, not demo. Who carries the pager at 2am.
Same due diligence as you would do for a dev agency. New buzzword on the brochure.
What to do this week
If you have an AI vendor evaluation in progress, take the three conversations above into the next call. Ask the questions out loud. Get the answers in writing. Watch how the vendor responds to the second and third conversation, not just the first.
If you do not have an evaluation in progress but you are about to start one, write the three questions on the back of the meeting agenda before you walk in. Or better, send them to the vendor 48 hours before the call so the senior person has time to prepare a real answer instead of a polished one.
If you are months past signing and you are starting to feel the dev shop pattern repeat, the questions still work. They are uncomfortable to ask after the fact. They are less uncomfortable than the alternative, which is realizing in month nine that you have been operating a vendor on top of operating your business.
How Chykalophia can help
We help founders evaluate AI vendor decisions before they sign and we help teams clean up after the ones who did not.
If you are mid-evaluation and you want a second set of eyes on the architecture conversation, get in touch through piotrkrzyzek.com. I will walk through your specific vendor situation, ask the questions you would benefit from someone else asking, and tell you honestly whether I think the partnership is going to hold.
If you have already signed and the dev shop pattern is starting to show, we run a 30-day vendor diagnostic. We look at the actual operational reality, document the gaps, and either help you renegotiate the contract or help you build the internal capacity to take the work back.
Either way, the goal is the same. Make the next vendor decision a partnership, not a magic box.
That is the difference that will compound for the next decade of your business.
The bravest thing you can do this quarter is not signing the fastest contract. It is having the three conversations before you sign.
Same due diligence. New buzzword.