321. Why 80% of AI projects fail — and it's not the AI's fault

Contrast two AI projects.

One cost $600,000 to build, $500,000 per year to maintain and was shut down as functionally unusable.

The other cut energy waste, reduced equipment damage, and returned 35.8% on investment.

Same tech. Different outcomes.

The difference had nothing to do with the AI — and everything with how the project was set up and managed.

In this episode:

• The New York City AI chatbot that confidently told people to break the law
• The Korean steel plant that used AI to solve one specific measurable problem — and returned 35.8% on its investment
• The five reasons AI transformations fail and how to avoid every one of them
• Why AI is execution and product thinking is judgment — and why you cannot out-execute a wrong decision

This episode is for you if:

• You are a business or political leader with an AI budget and want to make sure it is not wasted
• You are a corporate innovator trying to make the case for an AI initiative internally
• You want a clear framework for evaluating AI projects before you commit to them

Timestamps:

• 00:00 – The $600K AI chatbot that failed vs. the AI that worked
• 02:11 – NYC's government AI chatbot disaster
• 07:01 – Korean steel company's successful AI project
• 09:23 – Why the steel company's AI project succeeded
• 10:45 – Why 80% of enterprise AI projects fail (RAND research)
• 11:45 – 5 reasons AI transformation projects fail
• 14:06 – 2 solutions to fix failing AI initiatives
• 15:12 – Why product thinking matters more than AI technology 

Follow and Review:

We’d love for you to follow us if you haven’t yet. Click that purple '+' in the top right corner of your Apple Podcasts app. We’d love it even more if you could drop an honest review on Apple Podcasts. Simply select “Ratings and Reviews” and “Write a Review” then a quick line with your favorite part of the episode. It only takes a second and it helps spread the word about the podcast.

Listen to our podcast on:

• Apple
• Spotify
• YouTube
• Audible
• Pandora

Transcript:

[00:00] There were two AI projects. One cost $600,000 to build and was shut down as functionally unusable by the mayor of New York. The other cut energy waste, reduced equipment damage, and returned almost 36% on investment. Both used the same technology, but got completely different outcomes. That's all we're covering today — you'll learn why that happened.

Welcome to Tech for Non-Techies. This is a podcast for business leaders and non-technical founders building the future in the age of AI. Whether you've been in business for a hundred-plus years and you're looking to modernize, or whether you're building something new, this is the show for you. You'll learn how to come up with new ideas and make them come to life, no matter the size of your organization. I've taught tech and innovation at Oxford University, advise companies like Microsoft, and written for the Harvard Business Review. You're going to hear the frameworks and the thinking that I've built, tested, and taught at the highest level. And now it's your turn. Let's get started.

Hello, smart people. How are you today? I'm very excited, because tomorrow I'm going on my first business trip since my baby was born. The talk is called "Leadership in the Age of AI and the Power of Product Thinking." My audience will be business leaders and political leaders — people with big budgets, who constantly have innovators coming to them saying, "Hey, we've got this idea, can you fund us, can you allow this project to happen in this region or this company?" These leaders want to invest in AI, they're curious about it — some already have major AI initiatives and have spent quite a lot of money on it, and they want to make sure they get a good investment, because ROI in AI is what all the leaders are getting worried about now. That's what the class I'm teaching them is going to help them do.

Today I'm going to give you the beginning of that class — we don't have time for the whole thing, but I'll give you the opening.

[02:11] And actually, with just the opening, you can do quite a lot — you can already take action with it. And if you're listening and thinking, "My organization needs this," let me know on LinkedIn or email us, and we'll make it happen — I'd love to come and give you your own version of this class.

Okay, so we're going to start with two case studies — a good one and a bad one. Let's start with the bad one, because it'll be more recognizable. New York City launched its own government AI chatbot in 2023. The aim was to help small business owners understand city government regulations. It was built on Microsoft's cloud platform, part of New York City's "MyCity" digital services push — maybe some of you have even used it, I know I've got quite an audience in New York. It cost $600,000 to build, and another $500,000 to run. That sounds like a lot, but as someone who's built tech products, it's not massive. Still, as a taxpayer, over a million dollars going on something nonsensical — wouldn't you rather that money went to a school, or a hospital?

Anyway, this chatbot was giving wrong answers to high-stakes legal questions. Basically, this government chatbot was telling people to do illegal stuff — which is kind of funny when you think about it. People would ask, "Can I refuse to take cash in my shop?" It's actually illegal to refuse cash — but the chatbot said, "Yeah, you can absolutely do that." So it wasn't telling people to go kill people, but it was telling people to break the law — and the fact that it was a government tool made that especially problematic.

It launched in 2023, and journalists caught it telling people to do illegal stuff in March 2024. How did the city respond? In a very typical way — they added legal disclaimers. Do we ever read legal disclaimers? Basically it was like, "This thing might tell you a bunch of lies, but if you follow them, it's your fault, not ours." They also narrowed its scope. But two years later, when Mayor Mamdani came to power, he called it "functionally unusable" and shut it down. So this thing ran for two years and wasn't really that useful.

[07:01] Now let's have a good example — this one's from Korea. A Korean steel producer, referred to as "Company P" in the research. It's a major Korean steelmaker, putting out forty million tons of crude steel, which is apparently a lot — I don't know much about steel output, although I did work on the IPO of a steel company once and actually visited steel mills. Anyway.

The problem this company had — which I think all steel companies have — is that operators were setting furnace power based on their experience, judgment, and intuition. Super experienced people running these furnaces would decide what temperature to set, and once set, there's basically no way to measure the temperature inside the furnace directly while it's running. It's basically like the gates of hell — once they're closed, you can't get in there to see how hot it is.

The result of this problem: only 55% of production runs hit the target tap temperature range. The rest were running too hot or too cold, wasting energy and wearing out equipment — which is expensive to replace and not great for the environment. So what did this company do? They built a machine learning model to predict the temperature inside the furnace in real time. They tested six different algorithms and selected the best-performing one. Then they tested it on real production for five months before saying it was ready. And this algorithm worked alongside people — experienced operators with over 20 years' experience — who could always manually override the system.

[09:23] This is basically magic — this is why I wanted you to have this steel example, even though I'm really not a steel expert, because the way they handled the situation is just a work of art. So what was the result? Temperature deviation was cut by 17%, power use was down 282 kilowatts per batch, and the investment returned almost 36% IRR.

Why did this happen, in contrast to what happened in New York City? You already saw some of it in the case study. First, the company tested the algorithm — they created six different products, which meant they weren't falling in love with just one. When you create just one product, you tend to fall in love with it, even though you shouldn't, and irrationally try to protect it because it's your creation. When you have six, you're much less attached to any individual one — you're like, "Okay, if one of these works, I'll be happy, I don't mind which."

Then, once they settled on one, they tested it on real production for five months before rolling it out fully. Not just a one-time demo with five people testing it — a proper production test over time. And crucially, they kept humans in the loop — operators with proper experience could constantly override the system. This actually reminds me of our episode a few weeks ago with Rana Gujral, about relying on AI for our judgment. This Korean steel company basically said, "No, the humans know what they're doing — we trust the experts to keep checking on the algorithm," rather than saying, "You're no longer useful, the algorithm will do everything" — which probably also helps with staff retention, though that's a whole other topic.

What was also great about this problem is that it was measurable — temperature deviation, power cost, ROI. You can literally see: what was the metric before, what was it after, yes this was a good idea. Success could be proven, not disclaimed.

[10:45] So these are very contrasting case studies. I told you less about the New York one because it's generally harder to find out about negative case studies — but what we see is that this thing was released and basically lying to people for two years. Two years, come on guys.

But let's not be too hard on the New York City people, because 80% of enterprise AI projects fail, according to RAND — a very serious research organization, funded by the US government, a think tank that agencies like the Ministry of Defense take very seriously. I read the research on how they conducted this, and they did it very thoroughly.

So what did they tell us? 80% of enterprise AI projects fail — twice the failure rate of IT projects without AI. Pretty miserable. My aim for this episode, and for the bigger masterclass I'm giving tomorrow, is to help you not end up in that 80%. And by the way, that means you, my dear smart listener, are getting thoughtful, well-researched, high-quality content for free — you're welcome. So in return, I ask for some karma points — would you like to leave this podcast a rating and a review? Yes, I'm assuming you would. Thank you in advance — it really helps me and my team keep doing good stuff for you.

[11:45] So what does RAND tell us about why these AI transformation projects fail? Five reasons.

Number one: wrong problem and wrong metric. People aren't identifying the problem correctly and aren't measuring what they're trying to improve. Because they're not clear on the problem or what success looks like, nobody really knows what they're doing — the tech people work on one thing, the domain experts want another, and by the time it comes out, it's not what anybody wanted.

Number two: not enough data, or bad data. We hear about this a lot — organizations want to go through AI transformation, but their data is all over the place and needs lots of time to clean. If you ask a data scientist about bad data, you'll become their therapist — it's a very painful topic for them.

Another reason RAND cites: organizations are chasing the technology, not the need. We've talked about this before on the show — people or shareholders get excited and say, "We must have this AI thing, we must have our own AI agent." Why do you really need an agent? Maybe you do, but not in every part of the organization.

Another issue: when organizations try to do an AI transformation, they actually need proper infrastructure — data infrastructure — to make it work, and a lot of organizations don't have it.

The last point RAND raises, which I thought was really interesting: sometimes the problem is just too difficult for AI to solve. AI is great, really good — but there are just some things it cannot solve, and you need human expertise. AI shouldn't be applied to every problem.

[14:06] So what solutions does RAND suggest? A few things, but I've picked two of the most important. Number one: get the technologist and the domain expert speaking the same language, so they actually work together to solve the problem. Go back to our Korean steel company — the people making the algorithms and the domain experts worked together, testing together, and the domain experts were treated with respect, not as people who were about to be replaced by an algorithm. Get the two working together collaboratively and respectfully — this is literally the problem we solve at Tech for Non-Techies.

Number two: focus on the problem, not the technology — which is what I was just talking about. That's literally product thinking — focusing on the problem, understanding the user, understanding whether the problem is really a priority, because there are lots of problems that exist but that we're not willing to pay to solve, because while they exist, they're not that important.

[15:12] So the bottom line: AI is execution, product thinking is judgment. And product thinking is literally what you're learning on this show — which is why I asked you for that rating and review. Here's the thing: because AI is execution, if you have poor judgment and lay execution on top of that, you just get lots and lots of poor judgment, fast — kind of like the New York chatbot shows us. Instead of one person giving bad advice, it's this thing constantly awake, telling people to do illegal stuff over and over.

So next time you're talking to somebody about an AI initiative, don't get confused, over-excited, or over-enamored by frontier technology. Go back to basics — what you're learning here. What problem are you solving, and for whom? Are the technologists properly collaborating with the subject matter experts, and are they keeping the human in the loop? Are you constantly testing new versions of the product? Or are you doing what the New York City team did — releasing something and leaving it for two years to do God knows what?

This is why I'm so into teaching you product thinking — I genuinely think it's the answer, the foundation, whether you're doing an AI initiative, creating a new venture, launching a new advertising campaign, whatever you're doing. As a business leader, a non-technical founder, or an investor — this is the thing to learn. This is why we're learning it here.

And on that note, my dear smart people, I've got to go pack — I'm super excited. I'll be back from my business trip next week, and I'll be back in your ears then. Have a wonderful day. Ciao.