Profitable AI: How to Build AI That Actually Pays Off

But let's take a look at the numbers of such a use case. And I've been working with media companies before. So the numbers of this kind of use case are actually pretty brutal because just building a simple system like that is easily 50k development cost because you need to set up the text-to-speech engine you need to integrate it into your CMS you need to host a development and a production environment and that's just a build cost on top of that you have the run cost and if you collect everything even for a small publisher or medium-sized publisher that maybe has 150 articles a day per day five minutes per article. So that makes about 750 minutes per day for the whole audio feature. You get easily 30k per year on API cost alone plus infrastructure and logging another 20k plus the maintenance overhead. That's another 50k. So you're looking at 100k per year just to keep the lights on for that use case.

And for a news publisher that's a lot of money. 100k per year translates roughly to a thousand subscribers. Now, if you're the Wall Street Journal, well, maybe you can afford that just to put that online. But if you're a smaller publisher, maybe that's not the best investment of your money. So, what I would like to say with that is that even the simple cases, the easy ones, they tend to be very difficult from a business perspective.

But maybe we're just looking at the wrong place. Maybe we have to look at the bigger picture, right? So, let's take a look at the company that kind of like started this whole AI craze and they should know how to make it profitable, right? And I'm talking about of course OpenAI. So let's take a look at OpenAI. On the one hand side they are running the business of their life. We have 800 million weekly ChatGPT users. They are selling as many tokens as never before. So great business or not? Well not exactly because to be honest there is no business behind that. The Financial Times made an estimation that projected the profitability of OpenAI and with some goodwill, we're looking at 2030 where this company is profitable. Before that, it's all just negative revenue. So they have revenue, but they produce cost and cost and cost.

But okay, maybe we're looking at this just from an infrastructure perspective. Maybe we have to look at the application level. So let's take a look at an application that OpenAI actually recently launched. Who of you here in this room has heard about Sora? Maybe hands up. I see a few. All right. So, for those of you who missed Sora because it's already gone again. It was basically a video generation app that OpenAI launched last year, end of last year. It was super viral and everyone loved it because you could put your own character in it and it would put you like in all these crazy situations and like you had a ton of fun videos there.

But at some point, OpenAI realized two things. First, the user growth was kind of like stagnating. And the other thing is like running this application cost them a million dollars per day. A million dollars per day just to have a video app running, an AI native video app where everyone would say, well that's actually exactly what we all should be doing. So, they shut it down in the end because it was just too expensive to run. And the official wording on that was they are refocusing their priorities. But of course cost made a big impact here as well.

But now you could also say, oh man Tobias, why are you looking at these cases? These are US startups. They are not meant to be profitable. Uber was not profitable for 15 years. And I'm saying okay so then let's take a look at the quote unquote normal businesses out there. What are they doing with AI and what impact are they seeing? There was a really interesting and recent Ember study that found out that 70% of companies are using AI right now and they surveyed executives around different regions in the US, in the UK, in Germany, in Australia and they found out 70% are using AI actively but 80% are not seeing any impact on productivity at all. Can you imagine that? 70% using AI and then 80% not seeing any impact.

Why is that the case? And the case is that this is actually not really a new phenomenon. It's actually quite known. There's a really interesting paper from the 1990s by the economist Paul David called the dynamo and the computer. And he explored something really interesting. And he explored the impact that the electrical engine had in factories when it replaced the steam engine in the early 1900s. And his result, the impact of that was zero. There was no impact replacing steam engines with electrical engines for 30 years. It took 30 years for the industries to adapt and actually enhance productivity by switching to the electrical engine. Why? Well, because at the beginning they just put the electrical engine in the same system, in the same layout, in the same processes as the steam engine. And this is very similar to what's happening with AI today. We just plug in AI on very old processes on inefficient processes hoping that productivity will go up. We'll come back to that later on.

But there's a second thing that I would like to bring to your attention. Another study by Bark. So Bark has been like researching the impact of AI and data on companies for many years and they produced the latest report that is called lessons of the leading edge where they surveyed enterprises that are actively using AI and also asking them what is the main obstacle for AI success. So why are you not achieving success? And the key driver here, the biggest obstacle, and that shouldn't be surprising to you, is data, right? Data quality.

That's the biggest obstacle even before skills and expertise and integration issues. But what really surprised me is that this number rose from 19% to 44% last year. There was an explosion of people saying that data quality is my bottleneck. And why is that the case? Well, because today data quality does not only mean to have nice data cleanly ordered in a data warehouse. In the age of AI agents and large language models, data quality means that we need to deliver data at the right time in the right context in the right format to the right model. And that's a completely different challenge compared to putting data in a nicely organized data warehouse where we maybe run some dashboards on top. So that's a big challenge that is in here. And these are the main factors causing that AI is rare.

But let's assume for a second we found a good AI use case. And if you found a use case that had really a good business impact, would that use case be stable? And I realized no. If you have a good use case, even that is not stable. In fact, it's super fragile. What do I mean by that? If you ask people how they imagine an AI solution, they think about this. They think about the bright future, AI doing the work for us and we can just walk in the park and chill a little bit and AI is doing the work. So that's essentially what the picture is in their head when many business leaders are sold on the idea of using AI in their business. But when you actually get working on these things you find out there's so much work to do right. So much work. We need to bring the data quality up to a certain level. We need to integrate these systems. We need to upskill our employees. So many things to do. And that's also the reason why once we launch or bring AI projects into production there's a big gap between what was expected on the one hand and what can be delivered on the other hand.

The reason for that is what I call the 80% fallacy. Because the 80% fallacy means it's very easy with AI these days to get to a goodish prototype or solution in a very short amount of time. Couple of days, maybe even a couple of hours, you get to a point where this thing looks good and you think oh just a little bit more effort and we're done. But that little bit more effort turns out to be the actual work. The last 20% consume the 80% of the work. And actually each percentage point that you try to increase your solution or enhance your solution is getting more and more challenging, exponentially more challenging. And this also leads to these high failure rates that we are seeing in AI projects. And you've probably seen them, this famous MIT study which was a little bit debatable when you look at how it was produced but anyway I can also confirm for my own consulting business that a lot of expectations are not met in AI projects and especially because of what I showed you right. We have those super high expectations and in order to get to those expectations you need to put in a lot of effort to actually meet those expectations. And to understand the effort you need to put into an AI project you need to understand that AI projects work differently financially compared to classical IT projects.

So this is in a very simplified way how classical or traditional IT projects happen. Typically when you let's say you build a new CRM or you build a new ERP or you do some migration to your new system the costs for that are mostly front-loaded. So that means most of the implementation cost and most of the budget happens up front and then you have budgets that are slowly decreasing over time or staying stable after a certain period. At some point the marginal cost of adding another user is virtually zero.

But with AI you have a completely different set of economics here because if you launch an AI project the initial costs are really shallow. It's really easy and really simple to build a first prototype. But as you scale these solutions, as you bring them to more users or as you increase the usage of these solutions, you will find out that the costs are actually going up because someone has to pay for these expensive GPUs and pay Nvidia so their share can go up again or someone has to pay the memory bills for that. It's not just cheap compute and cheap network. We are using high resource intensive processes here. And the second phenomenon, and this is like this oscillation of the curve that you see there, is happening because you always have work with these AI systems. It's not like set and forget and plug and play and walk away. Even to keep features alive, not improving them, just to keep them alive, you need to constantly monitor, you need to constantly see, oh do we have data drift, are people suddenly using our chatbot to try to answer different questions, can our AI handle that, and from certain periods you need to update your solution or bring the new model in or see if the quality is still there.

So you have these constant maintenance cycles that of course need to be budgeted and accounted for as well and that has a very important implication and that is that your AI project needs to deliver and show value basically all the time. It needs to show recurring value in order to justify the recurring cost because otherwise at some point people will just forget the original business case and say why are we paying 100k per year for this feature here? What are we getting from this? So you need to show that benefit.

Let me show that to you on a practical example. Who of you have heard about the famous Klarna case study from the customer support? Oh no one, perfect. So let me tell you about this. So flashback to 2024, there was this big Swedish payment provider Klarna and they made a lot of noise in the AI community because they announced that they virtually replaced their customer service with AI and replacing their customer service happened on a really big scale. They had an AI that was doing the equivalent work of 700 full-time human agents in customer support. Two-thirds of the customer support by Klarna was handled through the AI service. And of course, this was a huge case study in the AI community. Everyone was waiting for something like that. Finally, we have a case that is profitable and that is super successful. And everyone was looking at that. Yeah.

But then one year later, they had to roll back a little bit. They had to adjust a little bit and say, "Oh yeah, we realize customers they might prefer the human answers and human support." But that was only one part of the story because the other part of the story is, and of course they didn't make this public, is that the cost of this project must just have eaten them up. If you build an AI that is doing the work of 700 full-time equivalents, you don't get that free of charge. Like that cost a lot of money to run. It's easily a seven figure budget every year. So, and if you have that seven figure budget every year and on the other hand side like people complaining that they still want to talk to a human, well, there's no economics in that, right?

So, they scaled that use case back and essentially they're still using AI in customer support right now, but at a very much lower scale and much shallower. So, this is how profitable AI is fragile. Even if you have something running, there's no guarantee that it will continue like that.

And the third obstacle that I would like to share with you today is that profitable AI is misunderstood and especially in organizations this is really the killer criterion because very often if you just drop the phrase AI then it gets treated the same way, same expectations, same budget, same development timelines. But that actually is a recipe for disaster because depending on what AI solution you build there are very different factors and also mechanics behind that.

So the framework I use for classifying AI solutions, and maybe if you're in a developer role here that might be helpful for you as well, is to say that there are basically four rough archetypes of what AI can do in AI solutions out there. So the most simple one is this assistant type. It has low integration and low automation. It's basically AI that you can ask something and it gives you an answer, right? Then we have the co-pilot type. This is AI that is sitting somewhere in your services, in your systems, like AI in Salesforce, AI in Outlook, AI in some systems. You can ask them or interact with them and they are integrated but they only suggest stuff right, they don't do things for you automatically. And then we have those autopilots. They run at low integration but high automation, point-to-point workflow automations where AI is part of that process, like for example an incoming email triage system. And then we have those AI agents, what everyone wants to get in the end, AI that figures out a problem and just gives you the results. So, these are highly integrated and highly automated solutions.

So, if you're able to map your use case around those four quadrants here then you can also see what kind of ROI approach you need for that because it turns out there are basically two approaches to ROI based on that framework. On the left hand side is what I call productivity AI. So, these are all AI solutions or AI services that don't do the work for you. They help you do the work. So they help you maybe to summarize an email or to work a little bit faster or to update an Excel spreadsheet, but they don't do the work for you.

And why is that so important? Well, because it has a huge impact on the economics. There's always a human in the loop. If you try to budget your co-pilot license with an Excel spreadsheet and saying, "Okay, how much productivity gains do we get from that?" You have already lost. Because it's not the case that just because 10 people save 20% of their time, you get a new full-time equivalent back. That's not how it works, unfortunately. So, you have to kind of like accept that the investment you make in that area is really hard to track. But on the other hand side, the investment is pretty low because you have like recurring cost for licenses mainly. You're not building your own solutions here typically. So, the focus here must be on adoption and behavior. Are people actually using that? What are they using this for? And then trying to make a qualitative estimate whether it makes sense to run that.

So that's the part on the left, the productivity AI spectrum. And then we have in my opinion these days even more exciting area which is the engineered AI, that's how I call it, spectrum. And what we have here are solutions that are designed for specific outcomes. So you build these solutions to solve a particular business problem for you and that also means there needs to be a business case before. If you don't have a business case for these solutions there's no point in running them. Because at some point you need to evaluate, for the given investment that I make, is this AI solution able to deliver a certain outcome? If yes, you have a business case. If not, well then there's no point in running that service for you.

If you try to measure adoption here, that would be the recipe for disaster here because you don't want to have adoption here. Literally, you want to get people out of the loop. If you measure adoption in AI agents, it's very hard to prove ROI then in the end because you have high cost and even a high human loop factor in here. So the misunderstanding that really happens is if organizations mix up those two things and try to for example budget those productivity AI solutions with an engineered AI approach or when they build engineered AI solutions and measure the wrong things or don't do a budget at all, right? So this is really important to separate those two tracks.

And now the trick is to kind of like combine those two in your organization because in the end you will have a mix, right. You will have productivity AI cases and you will have engineered AI cases. The way I approach that is that I try to find opportunities in the productivity AI quadrant and then see which of these can be more automated, which of these can be more scaled to a higher impact. And if you make that decision, which is by the way a decision you need to make yourself, this is nothing a vendor can do for you, you need to define what is the metric that I'm tracking, how much should I automate this workflow, what makes sense for my business. But once you enter the right track there to engineered AI, you need to track this, because you have real cost, you have real accountability, you have a real investment.

And the budget principle here that I use is don't plan for the best thing because the best thing is always the agent. It gets a ton of effort to get you there, but plan for what works. What's the baseline you're trying to beat? What does it take to beat this baseline? And then make a decision whether you want to do this or not. And this is also how you separate what I call motion from progress. Right? Because if you're not able to bring those four things in alignment, you will create a lot of motion. There will be a lot of activity in your organization. You buy tools and licenses and there are lots of trainings but you can't really see a big impact. On the other hand side progress means you have managed to bring those into an order. So by the end you get to a better business result, to a higher business outcome.

And this is also why I prefer to work in AI road maps versus AI projects. A single simple AI project is very often ad hoc. It has a certain outcome. You do it but it's not really connected to anything else. AI road maps on the other hand give you more of a structured plan. What are the different phases we want to push an AI use case through? What is the final vision that we have? What are the interconnected steps toward that vision? And what other elements do we have in our organization that is paying towards achieving that goal? So this is essentially how you clear up the misunderstanding and making sure that everyone is committed to that road map.

Let's make this practical. I brought an example for you today, a walkthrough of a previous client example of mine, of how to actually apply that in business in real life. And the example that I brought for you today is from a private equity firm. So for those of you who are not so deep in private equity let me give you a quick recap on how the business works. So essentially they get a bunch of investment proposals sent every week. So the firm we were working with here got about 100 investment proposals per month and one investment proposal looks something like that. So you have a nice PowerPoint deck with some investment criteria and some financials and it can be something between 20 up to 50 pages as a PDF or a PowerPoint presentation.

So what happens now is you have people, investment managers, scanning these incoming proposals. Which of them make sense, which don't really make sense, and then if something makes sense then they request even more information. So in this particular case we had five managers spending about three hours every week just reviewing those proposals. So in total that was about 700 to 800 hours per year spent on reviewing proposals. And you can imagine that it did not take long until the first person had an idea that AI could potentially be helpful here.

And the first idea that we had was that someone said, "Hey, I uploaded this proposal into ChatGPT." And it gave me a pretty good summary, right? I asked it to give me an executive summary of that proposal and it was pretty good. So the initial idea there was, hey, let's take all the incoming proposals, generate an executive summary for that, and we save a lot of time because we don't need to read the proposals, right? Sounds like a simple and elegant solution and old me would have probably just built a prototype right away because it's such a good use case. But now after some learnings in that space, I wouldn't do this and wouldn't build this out directly.

Instead I would take this idea and push it through different stages. And this is essentially the mental model I'm using for developing AI use cases. Everything starts with an idea at the beginning. Then we put it to a concept. If the concept gets really complicated and large, we create the proof of concept. Then we do the prototype, pilot, and finally to production.

Why are we doing this? Well, we're doing this because I learned that the implementation risk at the beginning is super high, 80%. If someone approaches me with an AI idea, 80% chance it won't work like that. So, we don't skip to building or to prototyping right away. Instead, we're trying to put a little bit more brain effort into understanding this overall idea. What is the problem we're trying to solve? Like what do we need for that? What is the AI capability we need for this?

And as we evolve this use case over time, this leads to the fact that the exit probability, and this might be a bit counterintuitive, right, so the more you work on that use case the higher the chance that this use case will not go into the next stage because you learn more. You understand more about the problem. You understand more about your data, about the AI. And the exit probability actually spikes here in the prototyping phase. Why? Well, because before it was all just words and theory, but in the prototyping phase, you first see it in real life. You build the solution. You understand what it can and can't do.

And now the goal of this whole process is, and this is why I'm showing you this, it makes a huge difference what you spend in order to get to this point. If you spend a whole year or lots of workshops or meetings on getting to this prototype stage and doing the concept before, then this kind of J-curve needs to be super aggressive at the end because this is the only place in the production phase where your use case will actually produce any return. Before it's only cost. But if you want to have the return you need to get to the production phase and you need to make sure that this curve at the beginning is as lean and as close to zero as possible. And in order to do this, you need to work a little bit on the concept phase, right?

And this was also what we did in this case study that I wanted to show you because when we had these proposals coming in and reviewing or summarizing these proposals, we actually didn't need to build anything to understand that this idea had at least three problems. So the first problem was that the incoming proposals were essentially marketing speech, right? So what happens if you do AI summary on marketing speech? Well, you get summarized AI marketing speech, right? So, it's not really that helpful.

Second, you can't really trust AI with financial data. So, there might always be the case that the AI maybe interprets a currency wrong or it misses a footnote or all these little small details that are in there, but that can have a huge impact. So, that was also kind of like a criteria against this.

And most importantly, this was also something that they really worried about. They thought okay if we have now AI summarizing these incoming proposals what happens to the skill that our employees build over decades of work filtering out good from bad proposals. We're kind of like outsourcing or delegating this core skill to the AI and this is not something we want to do, right. And we didn't have to build something in order to understand that. So what we did in the concept phase was that we pivoted this whole use case. Instead of summarizing documents at the beginning we said, "Let's not take a look at what's in the proposal. Let's take a look at what's wrong with the proposal."

Because it turned out that every investment manager had kind of like a hard key decision criteria, like a checklist they were going through to figure out if this use case is actually relevant for them or not. Like for example, is this investment in the EU region? Is this company older than 5 years old? Is this company profitable? Whatever. There was a certain list of criteria that we applied. So the solution now became that we skipped this whole summary part and instead we implemented a red flag filter. So incoming documents got automatically scanned against this hard criteria list and then at the top of each email there would be like a small summary as you can see here in this mockup with some deal screening criteria. So what is the hard criteria that led to for example deprioritizing or escalating this use case. And this was something really tangible we could now also estimate in terms of business impact.

So this was the solution we were looking at. But even here now before building it, we ran the numbers. And one number that I like to run in the beginning always is what's the theoretical maximum value of this use case. Because I have this rule of thumb, if a use case can't make at least 10k per year in business impact, either revenue up, cost down or any other value that you can quantify as 10k per year, it's just not worth building it. You can use copilot or ChatGPT, that's fine, but if it's not making at least 10k per year there's no point in building and maintaining that solution.

So we looked at the maximum value of that solution here and we ran the numbers. So we have about 100 proposals per month, 40 minutes on average to screen those proposals. That means about 700 to 800 hours depending on the overall work amount per year. With a blended hourly rate we got to a theoretical annual value of $100,000. That would be the maximum value if we had an AI that would fully automate this process. But of course, we didn't have an AI that is fully automating this process. We're looking at this red flag filter.

So, what we did then was some kind of napkin math here and say, what do we think, what kind of value share can our AI capture from that? And we estimated that the AI red flag filter will lead to 40% less time because all the quote unquote prep proposals got filtered out and they can focus on the more relevant ones. So we assumed that 40%, so that is 40k per year, as the value we were looking at. And this was also defining the parameters for our use case. If our use case has a value of 40k per year then of course it must cost less than this to run this whole thing.

And we were really conservative. We had a big buffer built in and we said okay the actual running cost of that solution including the maintenance, API cost and everything must not be higher than 5k per year. And from there we ran the numbers and said okay we want to see an ROI or payback for this use case of six months up to 12 months. So we were a little bit aggressive on that end and said six months. And essentially that gave us a build budget of 17 and a half k. That was our build budget for this and this was a super clear business case.

So now we were not going out and seeing okay what AI solutions exist on the market but we were looking at a case, can we build an AI that costs less than 5k per year, we can develop for 17 and a half k, and that can lead to a certain criteria acceptance rate of screening out the bad proposals. That was a really tightly scoped case and this is something we then pushed into the prototyping phase and this is something where you can actually test and prototype against because we now knew that in order to get to this 40% reduction the proposals need to be at least 90% accurate. So nine out of 10 proposals need to be classified correctly and when in doubt it should let one proposal through.

So we built a prototype for this solution. We took 25 historical proposals that were accepted and 25 that were rejected. We applied the AI. We compared different models and here we didn't train anything from ourselves. We just used different AI models out of the box and compared how they were working on this case. And it turned out especially with those new vision models that can actually visually process every slide or every page of a document, we got pretty good results. So we ended up with a solution that was able to beat this 90% accuracy threshold here and that became then the decision criteria for us to move to the next phase which was in this case of course to build this solution.

So the verdict here was that we said, "Okay, let's build it. We have clear economics. We can run an AI solution like that. We don't need to reinvent the whole wheel. We maybe don't have like the super flashiest AI solution later on in production, but it's something that stands on its own and something that is valuable on its own because it's viable and there's a clear business case attached to that. So yeah, in short, we can say the summary idea that we had at the beginning was exciting, but the red flag filter here was the most profitable. And that's also something I would like to leave you with today.

Whenever you get the first AI idea, we tend to be mentally wired to look for the cool ideas, for the ones that are most flashy or maybe most exciting. But take a step back and just think about it for a second. What problem are you actually trying to solve? What other ways are there to solve this problem? And most importantly, let's assume for a second that everything is working out as expected. The AI is working. You have the data. There's the organizational readiness. What would be the impact of that? If you're building something, you need to look for a certain outcome. What would the outcome be in that case? Can you quantify it?

Because at some point people will come back to you and ask, oh, I have this AI bill. Why are we paying this for? And if you can't show an outcome, you don't have a decision criteria to keep this case alive because it's really not just about building the AI and putting it into production. It's to continuously keep it there. So in our case, we had a really clear decision criteria of saying if we really filter out 40% of all the incoming proposals, we have a case for this AI solution. But let's say in half a year or so we would revisit the case and we find out that maybe only 20% or 10% get filtered out. Well, that would be a clear indication that this business case is maybe not worth running anymore. We need to increase the value of that by maybe adding some other information in there that would enhance the value.

So that's a design decision you need to make all the time. People are completely overlooking that they are buying some AI solution hoping it will do the work for them and then in the end they wonder why the bill is so high for this. So make that really a deliberate decision that you take for the AI use cases you put into production.

So to wrap up I have said profitable AI is rare, fragile and misunderstood. How do we overcome this? First we need a metric. We need a metric to identify the rare opportunities, to identify what is worth building. In our case study here, that was the 700 or 800 hours per year. That was kind of the core value we were looking at. You need to have a metric to go for. And second, you need a threshold to filter the noise, or you could also say to keep the use case alive. A threshold both for the cost side of the use case and also for the impact side of the use case. If you're not able to evaluate that and tell what the cost and the impact of that use case is, there's a high chance it won't survive the first, second or at least the third year.

And finally a road map to commit. So as you have seen there are lots of iterations. Once you start with an AI idea, there's no straight path forward to success. There will be a bumpy road and you will have to address a lot of questions and iterate on your use case. And the clearer that road map is from the beginning and the more alignment you have between different people in the group of saying okay this is for example the stages we are going through, like we are going from idea to concept to prototype, and it might be that we need to roll back at a certain point. If I budget AI solutions I assume 1.5 or two concepts per prototype per production so you can actually assign budgets to each of these stages. And by just making sure that you have this expectation management right at the beginning it's much easier to then go back to your business partners or stakeholders or customers and say hey let's not do a summary, I have something else, right. So they are on the same track.

So these are the key criteria or the key methods for finding profitable AI and putting it into production, profitable AI that works and that eventually pays for itself. If you are interested in this feel free to subscribe to my newsletter. I'm publishing weekly articles around that topic. On Saturday there will be the article that is summarizing this talk that I gave today and if you sign up there for the first time you'll also get two chapters for my book The Profitable AI Advantage for free so you can check that out. It contains all the methodologies, all the frameworks, so you can actually get profitable AI into production from your end. I hope this will be as useful for you as it has been for me. I wish you the best of luck for your AI journey and thank you so much for your attention today.