Cut AI agent costs by 80%

AI is changing the world, and as a business owner, you need to be using it to automate all sorts of tasks. The only problem is it's expensive as fuck, right? LLM tokens are not cheap, but there is a solution. For most businesses, you guys are probably using way too many expensive models, and you're not properly routing them. And if you just follow the advice I'm going to give you in this one video, you could probably cut your AI agent costs by up to 80%. So, if you're spending something like, I don't know, a thousand dollars a month on LLM tokens, you could probably bring that down to just 200. And the way we do that is a little something called model routing. So let me explain how this works. Basically, when I say model routing, I mean routing user requests to different LLMs based on costs, prompt and complexity, and all that other stuff, right? The problem with most businesses and how they integrate AI is they're just using one frontier intelligent model to do all their tasks, right? Which is just unnecessary. And that's costing you a lot of LLM tokens because the frontier models, the most intelligent ones, they are expensive, right? Like, sure, in an ideal world, you would send all of your requests to frontier models, right? You would let the most intelligent model handle every single request. But we don't live in that world. In the real world, your company is going to be constrained by limited resources that you need to decide how to use efficiently, right? And so you need a model router. And the way that works is the user is going to send a prompt like we always do, but there's going to be a router in between over here that's going to decide which model should handle this request, right? We could select a cheaper model, we could select a standard, and we could select the frontier model, the most intelligent one we've got in our stack, right? Right now, that would be something like Fable 5 from Claude or GPT-5.6 or whatever, right? And you might think, well, this guy's telling me to use a dumb, less intelligent model. That's probably not good advice. No, it is good advice because most common business tasks can actually be handled by these smaller and cheaper models, models like Gamma 4, Qwen 3.6, 27B. I've personally been using these models in all sorts of business tasks, right? Obviously, not going to use them for coding or for really complex reasoning tasks, but for most business tasks, these small models are amazing. Like, for formatting output, if you want to turn a messy meeting note into, I don't know, clean executive summaries with action items, you could do that. If you want to extract something like structured data, right, pulling all sorts of data from incoming PDF invoices, putting them into your database or JSON schema so that you can analyze them more easily later. Classifying inputs like support tickets and all that, summarizing short documents, you can do all of these things. Answering questions, you could have a chat agent to support your users. Like, these aren't very complicated tasks. You're probably using a frontier intelligent model to do this for you. You don't need that. You're paying for intelligence that you're not even using. 90% of that is just going to waste because these smaller models can do all of these tasks very effectively and at a much, much lower cost. Okay, so now that we've gone through the why we need to do this, let's actually discuss how to build a model router. This is going to involve a lot of manual work, so just be ready for that. The very first thing you need to do is you need to grab a sample of at least 500 user prompts, at least 500 actual prompts that your users have sent to your AI agent. And then you need to categorize them by complexity and task types. And you need to do this manually, all right? Because you'll find that most of your users are sending these prompts to accomplish the same kind of tasks, right? So you should be able to organize them by complexity and by task type. And once you've done this process manually for 500 user prompts, that's when you're going to find out that most of these tasks, at least more than 50, can be done by cheaper models. And that's the big win that I'm trying to discuss in this video. If you can get rid of these majority of tasks and just send them to cheaper models, that's the biggest cost savings that you'll get from your AI agent. So once you've done step one, the next thing you need to do is to create three tiers, right? You need to create three tiers where you're going to route these requests. You could have a simple tier for extracting, formatting data, for simple Q&A. You could have a standard tier for typical writing stuff. You could include some coding over here as well if that's the kind of business. And then you would have a frontier tier, which is the most high intelligence model you got for the most complex, high-reasoning tasks, right? I wouldn't recommend creating more than three tiers because that just gets way too complicated. Three is just the best number to have over here. Speaking of which, step three is to create a routing strategy. You're going to need some sort of a strategy, some rules on how you're going to actually select which tier should serve which request, right? Now, there's a couple ways to do it. There's rule-based routing, which basically means looking at the prompt length, if it's a long prompt, if it has certain keywords, etc. I wouldn't recommend this because this is just way too brittle and fragile. There's semantic routing, which is pretty good. It uses semantic analysis to figure out which model to select, but it can get kind of complicated. And for most business owners, you don't need to step into that kind of complexity. And finally, we have LLM routing strategies, and that is the one that I'm going to recommend. So here's how that's going to work, right? The user is going to send a prompt, right? And then you're going to have a router in between that's going to intercept that request. This is the first step in the chain, right? That router is going to send the prompt over to a small LLM, right? This would be a small, cheap model that understands language well. You would send it the prompt, and it would basically just decide which tier should serve this request. And then you would simply route that request to the simple tier or the standard tier or the frontier tier, depending on how complicated the user's request is, right? And then finally, step four is to simply track, iterate, and improve. After step three, you have your model router. You can deploy this and start using this, but you're not gonna get everything right on the first try, right? So you need to have a process to iterate and improve on this. The very first thing, and the most important thing you need to do with this, is to track costs because if you're not saving money off of this, what are we even doing, right? The whole point of this model router was to save you money. So actually track that. Track how many LLM tokens you're using, track how much these AI models are costing you, and see if you're actually saving money off of this because that's the whole point over here, right? Secondly, implement quality checks because just because we using cheaper models, we don't want to give users, you know, low-quality output. So implement things like human review. That means you or some employee is going to manually review the AI's output and ensure that users are getting quality answers. And use LLM as a judge, which basically means you would send the AI-generated answer along with the user prompt to another LLM that's going to decide whether that was a suitable answer or not. If it's not a good-quality answer, it would return a low score, and you would know that you need to fix something, right? And ideally, you should do this to every single response. But if you're getting a ton of traffic, like thousands and thousands of requests every single day to your AI agent, you can probably just sample a random batch, like a couple hundred of these. Then, if a model can't accomplish a task, you need to make sure you escalate it to a stronger model. If your router somehow sent a complex request to a cheap model and it can't execute it properly, make sure that it gets escalated to a stronger model. And then also track your escalation rate. Escalation rate basically means how many requests are you escalating to stronger, more intelligent models. And the neat thing about this is, if you find that you're escalating, let's say, 20% or more of the tasks to a stronger model, you know your routing logic is just not good anymore. Like, if most of your tasks keep getting sent to stronger models, then that just means you're routing complicated tasks to cheap models, and we don't want that, right? So track the escalation rate. It is a very useful metric, and it'll tell you these kinds of things. And that's how you build and improve your model router. And if you do this right, you're going to find that out of all the tasks that your AI is currently handling for your users, only a small portion of them are really complicated tasks that require a strong frontier intelligent model. And most of them are just simple tasks, simple queries that can be solved by cheaper models. The problem right now is that you're using a frontier model to serve all the requests, even simple ones, and that's skyrocketing your LLM token costs. If you just take this back over to here and you use these simple, cheaper models to serve most of the user requests, you've basically just saved 80% of the money you were currently spending. Anyway, hope that was helpful.