Auto-generated transcript AI is changing the world and as a business owner you need to be using it to automate all sorts of tasks. The only problem is it's expensive as fuck, right? LLM tokens are not cheap, but there is a solution. For most businesses, you guys are probably using way too many expensive models and you're not properly routing them. And if you just follow the advice I'm going to give you in this one video, you could probably cut your AI agent costs by up to 80%. so if you're spending something like i don't know a thousand dollars a month on llm tokens you could probably bring that down to just 200 and the way we do that is a little something called model routing so let me explain how this works basically when i say model routing i mean routing user requests to different llms based on costs prompt and complexity and all that other stuff right the problem with most businesses and how they integrate ai is they're just using one frontier intelligent model to do all their tasks, right? Which is just unnecessary. And that's costing you a lot of LLM tokens because the frontier models, the most intelligent ones, they are expensive, right? Like sure, in an ideal world, you would send all of your requests to frontier models, right? You would let the most intelligent model handle every single request. But we don't live in that world. In the real world, your company is going to be constrained by limited resources that you need to decide how to use efficiently right and so you need a model router and the way that works is the users going to send a prompt like we always do but there's gonna be a router in between over here that's going to decide which models should handle this request right we could select a cheaper model we could select a standard and we could select the frontier model the most intelligent one we've got in our stack right right now that would be something like fable 5 from cloud or gpd 5.6 or whatever right and you might think well this guy's telling me to use a dumb less intelligent model that's probably not good advice no it is good advice because most common business tasks can actually be handled by these smaller and cheaper models models like gamma 4 quen 3.6 27b i've personally been using these models in all sorts of business tasks right I obviously not going to use them for coding or for really complex reasoning tasks but for most business tasks these small models are amazing like for formatting output if you want to turn a messy meeting note into i don't know clean executive summaries with action items you could do that if you want to extract something like structured data right pulling all sorts of data from incoming pdf invoices putting them into your database or json schema so you can analyze them more easily later classifying inputs like support support tickets and all that summarizing short documents you can do all of these things answering questions you could have a chat agent to support your users like these aren't very complicated tasks you're probably using a frontier intelligent model to do this for you you don't need that you're paying for intelligence that you're not even using 90 of that is just going to waste because these smaller models can do all of these tasks very effectively and at a much much lower cost okay so now that we've gone through the why we need to do this let's actually discuss how to build a model router this is going to involve a lot of manual work so just be ready for that the very first thing you need to do is you need to grab a sample of at least 500 user prompts at least 500 actual prompts that your users have sent to your ai agent and then you need to categorize them by complexity and task types and you need to do this manually all right because you'll find that most of your users are sending these prompts to accomplish the same kind of tasks right so you should be able to organize them by complexity and by task type and once you've done this process manually for 500 user prompts that's when you're going to find out that most of these tasks at least more than 50 can be done by cheaper models and that's the big win that i'm trying to discuss in this video if you can get rid of these majority of tasks and just send them to cheaper models that's the biggest cost savings that you'll get from your ai agent so once you've done step one the next thing you need to do is to create three tiers right you need to create three tiers where you're going to route these requests you could have a simple tier for extracting formatting data for simple q a you could have a standard tier for typical writing stuff you could include some coding over here as well if you that kind of business and then you would have a frontier tier which is the most high intelligence model you got for the most complex high reasoning tasks right i wouldn recommend creating more than three tiers because that just gets way too complicated three is just the best number to have over here speaking of which step three is to create a routing strategy you're going to need some sort of a strategy some rules on how you're going to actually select which tier should serve which request right now there's a couple ways to do it there's rule based routing which basically means looking at the prompt length if it's a long prompt if it has certain keywords etc i wouldn't recommend this because this is just way too brittle and fragile there's semantic routing which is pretty good it uses semantic analysis to figure out which model to select but it can get kind of complicated and for most business owners you don't need to step into that kind of complexity and finally we have llm routing strategies and that is the one that i'm going to recommend so here's how that's going to work right the user is going to send a prompt right and then you're going to have a router in between that's going to intercept that request this is the first step in the chain right that router is going to send the prompt over to a small llm right this would be a small cheap model that understands language well you would send it the prompt and it would basically just decide which tier should serve this request and then you would simply route that request to the simple tier or the standard tier or the frontier tier depending on how complicated the user's request is right and then finally step four is to simply track iterate and improve after step three you have your model router you can deploy this and start using this but you're not gonna get everything right on the first try right so you need to have a process to iterate and improve on this the very first thing and the most important thing you need to do with this is to track costs because if you're not saving money off of this what are we even doing right the whole point of this model router was to save you money so actually track that track how many llm tokens you're using track how much these ai models are costing you and see if you're actually saving money off of this because That's the whole point over here, right? Secondly implement quality checks because just because we using cheaper models we don want to give users you know low quality output so implement things like human review that means you or some employee is going to manually review the ai output and ensure that users are getting quality answers and use llm as a judge which basically means you would send the ai generated answer along with the user prompt to another llm that's going to decide whether that was a suitable answer or not if it's not a good quality answer it would return a low score and you would know that you need to fix something, right? And ideally, you should do this to every single response. But if you're getting a ton of traffic, like thousands and thousands of requests every single day to your AI agent, you can probably just sample a random batch, like a couple hundred of these. Then if a model can't accomplish a task, you need to make sure you escalate it to a stronger model. If your router somehow sends a complex request to a cheap model and it can't execute it properly, make sure that it gets escalated to a stronger model and then also track your escalation rate. Escalation rate basically means how many requests are you escalating to stronger, more intelligent models. And the neat thing about this is if you find that you're escalating, let's say 20% or more of the tasks to a stronger model, you know your routing logic is just not good anymore. Like if most of your tasks keep getting sent to stronger models, then that just means you're routing complicated tasks to cheap models and we don't want that right. So track the escalation rate it is a very useful metric and it'll tell you these kinds of things and that's how you build and improve your model router and if you do this right you're going to find that out of all the tasks that your AI is currently handling for your users only a small portion of them are really complicated tasks that require a strong frontier intelligent model and most of them are just simple tasks, simple queries that can be solved by cheaper models. The problem right now is that you're using a frontier model to serve all the requests, even simple ones, and that's skyrocketing your LLM token costs. If you just take this back over to here and you use these simple cheaper models to serve most of the user requests, you've basically just saved 80% of the money you were currently spending. Anyway, hope that was helpful.