AI has sped up work, but few companies can point to organization-wide return on investment (ROI). Everyone is calling this a paradox. This three-part series names the three things buying the tools never fixes on its own, the people, the organization, and the foundation, and what to do about each. Read all three to find where your own gap is hiding.
Part 1: The People | Part 2: The Organization | Part 3: The Foundation
You have a few AI pilots that have worked. The results are promising, and the business case makes sense. You spin up a dedicated AI team, give them a mandate, and hand off the work.
Six months later, nothing has shipped to production. The business side says the AI team doesn’t understand the constraints. The AI team says the business doesn’t understand what’s possible. Meanwhile, the productivity gains that looked obvious in the pilot have vanished.
It’s the most common failure pattern we see, and it’s rarely a technical one. It’s organizational.
After working through this with clients across finance, manufacturing, and operations, we’ve landed on three structural mistakes that kill AI initiatives before they scale. Here’s what they look like, and what we’ve found works.
Mistake 1: Centralizing AI Decisions Away from the Business
The error: You create an AI Center of Excellence or a dedicated AI team, and suddenly every AI decision routes through them. They’re the gatekeepers. The business waits.
Why it fails: The people closest to the problem, the ops manager, the finance lead, the customer service director, can’t move fast. They hand requirements to the AI team, the AI team interprets them, requirements get misunderstood, timelines slip, and by the time something ships the business context has already shifted.
What works instead: Push AI decisions closer to the business. You still need shared standards — how AI workflows get evaluated, how decisions get logged, how risk is governed — but the business unit lead, who owns the outcome, should decide when an AI workflow is ready to go live in their function.
We made the same call inside our own delivery organization. Instead of standing up one central AI practice that every client team has to route through, we organized around small cross-functional pods, each aligned to a specific client, each deciding for itself how AI gets used in that engagement. A pod answers to shared standards, a hiring bar, a set of proficiencies, not to a gatekeeper reviewing its every move.
Mistake 2: Treating Upskilling as Training, Not Proficiency Building
The error: You run a one-week training program. Everyone learns to use Claude, Copilot, or ChatGPT. Then you expect productivity to jump.
Why it fails: Training teaches how to use a tool. It doesn’t build organizational proficiency. The operations manager learns Claude syntax, goes back to their desk, and does the same job exactly the way they used to, just a little faster. Productivity gains are marginal, executives see no ROI, and everyone quietly concludes AI didn’t work.
What works instead: Build proficiency through doing, not classrooms.
Start by asking what this person stops doing, what new responsibilities they take on, and how their relationship to the work changes. Answer those honestly and you’ve effectively restructured how that function operates, at which point upskilling stops being a training event and becomes proficiency built through doing the work.
A client we worked with had an immediate unlock the first time we had them use an orchestrated workflow we built for their forecasting process. The response was emphatic, “This will replace 90% of the meetings we have, I can simply ask AI questions about my forecast, and it knows my entire book.”
Zero training involved, they got it immediately. That is powerful. This frees them up people to do what they do best, build relationships to close deals.
Mistake 3: No Feedback Loop Between Operations and AI
The error: The AI team builds an AI workflow, the business puts it to work, and six months later nobody can say whether it’s moving the outcomes that matter.
Why it fails: Without real feedback you can’t improve: performance drifts, edge cases pile up, the business loses confidence, and the AI team never learns what’s needed.
What works instead: Build operational feedback into the rollout from day one. Who’s watching how it’s performing against the outcomes you care about? Who surfaces problems? How fast can you respond?
On another engagement, we skipped the weekly-sync approach entirely and went tighter. The day two account managers first tried the assistant live, on their own real data, their feedback went straight into a ranked list for engineering before the day was over. Four fixes made the build before the end of the same day the feedback was received. A handful of other requests got logged as legitimate but not urgent. A couple of ideas got an explicit “not this round,” with the reasoning written down so nobody had to relitigate it later.
The Pattern Underneath
These aren’t technology problems, they’re proficiency problems: companies think they’re buying an AI system when what they’re building is organizational capability.
The fix comes down to three structural shifts, and they reinforce each other.
Push AI decisions closer to the business, because feedback only flows fast when the people doing the work own the decision to change it.
Redefine roles and workflows through doing instead of training, because software that keeps evolving forces people to build proficiency in real time, and that’s where the actual transformation happens.
Close the feedback loop, then keep it tight, because iteration speed is the discipline: every cycle, the system improves, and the organization learns alongside it.
When those three things hold, AI proficiency becomes the muscle of how the organization operates day to day, and the productivity gains stick instead of fading out after the pilot. None of it requires a heroic AI team. It requires distributed decision-making with clear guardrails around it.
If your AI tools are live but proficiency, and the productivity gains that are supposed to come with it, still haven’t shown up, this is the first place to look.
Questions to take back
- What did your last AI pilot leave behind that the next one can reuse, or are you paying to start over?
- Of the AI you’re running today, which of it can someone prove is working?
Up next in this series, Part 3: Foundations
Fixing the organization gets you further, but it does not answer the harder question underneath it. Once the model itself becomes a commodity, what should you actually own? In Part 3, we lay out the five-part foundation every company needs, whether it builds its own AI or simply buys it. Read Part 3: Own the foundation, rent the model.
About the Author
Brendan Flynn is SVP, Strategist at Robots & Pencils where he heads industry strategy within the Generative & Agentic AI Studio.
Key Takeaways
- Most AI initiatives fail for organizational reasons, not technical ones. The common pattern is a pilot that succeeds, a dedicated AI team that takes over, and six months later nothing has reached production.
- Centralizing AI decisions in one team slows everything down. The business unit that owns the outcome should decide when a specific AI workflow goes live, inside shared standards.
- Training teaches a tool. It does not build proficiency. Real capability comes from changing how the work gets done, not from a one-week class on a chatbot.
- Without a feedback loop between operations and the team building the AI, nobody can say months later whether a workflow is actually improving the outcomes that matter.
- The fix is three structural shifts that reinforce each other: push AI decisions closer to the business, build proficiency through real work, and close the feedback loop early, then keep it tight.
FAQs
Why do AI pilots that work so well often fail to scale?
Because the failure is usually organizational, not technical. If AI decisions route through one central team, if training substitutes for real proficiency, or if there is no feedback loop back to the business, the gains a pilot proved rarely carry into production.
Should a company centralize its AI decisions in one team?
No. Centralizing every AI decision in one gatekeeping team creates a queue the business waits behind. Shared standards should be centralized. The decision to put a specific AI workflow into production should sit with the business unit that owns the outcome.
What is the difference between AI training and AI proficiency?
Training teaches someone to use a tool. Proficiency changes how they do the job. A short training session on a chatbot rarely moves productivity. Asking what a role stops doing and what it takes on instead, then building that into daily work, does.
What happens without a feedback loop between the business and the AI team?
Performance drifts, edge cases pile up, and nobody can prove months later whether the AI workflow is helping. A tight feedback loop, where real users test the workflow on real data and issues go straight into a ranked list for the team building it, catches problems while they are still cheap to fix.
What three changes help AI initiatives scale past the pilot stage?
Push AI decisions closer to the business unit that owns the outcome, build proficiency through real work instead of classroom training, and close the feedback loop between operations and the team building the AI, then keep it tight.
