Semantic Caching: Stop Paying for the Same Answer

your users keep asking the same questions. Like, how do I reset my password? I forgot my password, how do I reset it? Help, I can't reset my password. And all of these have the exact same answers, right? Your chatbot is just going to give the user some instructions on how to go and reset your password on your website or your platform. But every single time someone sends a duplicate question like that, your LLMs have to recompute all of those tokens, generate the exact same answer every single time, and cost you a heck of a lot of money. We can fix all that by using a little something called semantic caching. Semantic caching helps you answer repeat questions without asking the LLM at all. So let's talk about how semantic caching works and how to use it to reduce your LLM token cost. Right off the bat, let me tell you, this is mostly applicable for knowledge-based agents, chatbots, support agents, FAQ agents, stuff like that, right? agents where your users are bound to send lots and lots of repeat questions. If you have agents like that running in your business, this video is going to help you a lot because it's going to teach you how to reduce your LLM costs by a lot. And we all know how expensive these things can get. So anything you can do to reduce your token cost, I'd say that's very worth it. Now, if you're not using knowledge-based agents and customer support agents and all the other stuff I mentioned, you can still watch this video. It's going to be a very great learning experience. It just might not be as immediately applicable as if you were using those agents. So let's say you've got an AI support agent. It's going to say something like, hey, how can I help you today? Users are going to send some questions or some queries, and it's going to do some processing, maybe run some tool calls, and then return an answer, right? Problem is that a lot of the queries that users send are going to be very repetitive. Like we just went over over here, right? resetting passwords, users can send like 100 different queries asking the exact same thing, just in slightly different words. And maybe they add a question mark, or maybe they somehow forget the question mark. And it's just slightly different, but it's the exact same question, right? This is where semantic caching comes in, it's going to sit between your user queries and your AI chatbot. And it's going to return cached answers to repetitive questions. So we have the exact same three people over here who keep forgetting their passwords. They ask the support agent how to reset it. The first query, since this is the very first query, is going to go through the vector cache over here. It's going to see that the answer doesn't yet exist in the vector cache because, again, this is the first query, right? It's going to trigger a cache miss, and it's going to route that request to your LLM provider, to the support agent, which is then going to generate an answer for this particular question over here. Then that answer gets cached over here in the vector cache. So future queries like this one over here are going to send a request to the vector cache and they're going to get the cached response back. Same thing over here in the third query. It's going to send a request and it's going to get the cached answer back over here right So in effect what we done basically is we reduced three LLM calls in this example to just one And of course you can scale this way the heck up And depending on what kind of questions your users are asking your support agents, you could probably reduce like hundreds or maybe even more requests just by adding a semantic caching layer between your chatbot and your user queries. Now, here's how this actually works under the hood, right? If you were using a normal traditional cache, it would have to cache on exact queries, right? Like, how do I reset my password? If the user sends a question that is even slightly different from this question, maybe even he forgets the question mark, exact word caching would completely fail, right? Because the prompt changed slightly, right? Semantic caching isn't like that. It's going to cache your user queries based on the semantic similarity, based on how similar two sentences are, not exactly on the wording, just the meaning behind the sentences. So let's talk briefly about semantic similarity embeddings and finding similar sounding sentences, which is the whole point of semantic similarity. So how semantic similarity works is it's going to take words like random English words that we use and it's going to turn them into vectors which is basically just these really complicated numbers you see over here. Every single word has a vector form and it's going to just use that vector form. The reason it does this is so it can do some crazy math to figure out if two words are similar to each other. You can see that king and queen over here in these examples are pretty similar to each other right. If you were to check these numbers you'll see that both of them are pretty similar to each other, right? As opposed to a word like Mercury over here, you can see that its numbers are way different than King and Queen over here, right? In this example. And so what that means is the words King and Queen are related to each other in some way. They're similar, right? Whereas the word Mercury, it has nothing to do with any of these words at all. So what does this have to do with prompt caching? Well, apply the same concept to the user queries that we saw above, right for resetting your passwords right all of these queries get turned into these vector forms right and by the way these numbers they're called embeddings right the vector form these vector numbers for any text or any word are called embeddings so if you look at the embeddings of all three of these queries you'll see that most of them are pretty similar right that's how semantic similarity works it's going to compare all these different queries and it's going to to generate some sort of a confidence score, right? A confidence similarity score of how related or how similar these three queries are. As you can see in this example, it says they are 95% similar, right? So almost the exact same queries, right? That's how we know that we can just return the exact same answer for all three of these queries, and that's going to be completely fine. So now let's talk about how semantic caching actually works under the hood. What exactly happens in your application to facilitate semantic caching We got a user over here he going to send a query that query lands in your application your application then sends the query over to an embedding model An embedding model is an AI model whose purpose is to literally just take text or images or any other piece of data, let's just consider text for now, its purpose is to take some text and generate the vector embeddings for it. So it's literally going to take a bunch of texts like this over here. And it's going to convert it into a bunch of fancy numbers like this over here that is what an embedding model is supposed to do you're going to need one of those it's going to take that user query like how do i reset my password or how do i change my email address or whatever and it's going to convert it into a vector into really complicated numbers that we don't even need to know what they're doing those numbers get stored in a vector database and they are used to tell whether this question or query from the user is repetitive or not? Have we already received a bunch of queries for the exact same question? And we already have an LLM generated answer for it? If so, then we're just going to return that answer, right? That's going to be called a cache hit. And we're going to return that cached response back to the user without having to call the LLM API at all. Now, if that answer doesn't already exist, then that's called a cache miss. And we're going to send a request to the LLM providers like OpenAI or something, we're going to regenerate the entire response, then we're going to send that back to the user. And we're also probably going to cache that response in our vector database, so that in the near future, when someone asks the exact same question again, someone asks the exact same query again, we can just return the answer from the vector database without having to call the LLM at all. And so that's how we actually build a semantic caching layer in your application. Get an embedding model. You can use one from the AI providers. Every single one of them has an embedding model. Gemini does, OpenAI does, or you could run an open source model locally on your machine or something. You can go over to Hugging Face and search for embedding models. And there are a ton of really good open source options over there as well. Get one of those. You're going to need them to convert your queries into vectors. Then get a vector database. You can use pgVector, which is a way to just store vectors inside Postgres. You can use other options like Pineco and ChromaDB and all the other ones as well. Just Google vector databases and you'll find like the top 10, 5 database options that you should go for. Store the vector forms of the user queries that you get in your vector database. Store the LLM's generated response for those queries and then just run cosine similarity or confidence similarity over here. See if two queries are the exact same or they're like 90% same or 95% same, they're probably the same question, just phrased differently. In that case, just return the LLM's response for one of those cache queries. Simple. Now, this is semantic caching, but you can also use another strategy called prompt caching. And you can use both of them together to really dramatically reduce your LLM costs. And I've made a video on prompt caching before as well. So I not gonna go into too much detail over here I just probably link that video and you can go watch that and learn prompt caching as well But the point is you should be using both of these things together And the way you do that is because it basically you have a user it going to send a query that query gets into your application, the application will first check the vector database, right? Since that's the first caching layer, it's going to run semantic caching, it's going to check the vector database, if the response already exists, it's going to send that back to the user. If the response doesn't exist, it's going to send that request over to the LLM. That LLM will likely have that prompt cached, which means it can return the response at a fraction of the cost. Then that response gets cached into your vector database, and it gets sent back to the user. And if, for instance, the prompt hasn't been cached in the LLM, it's obviously going to recompute all of those tokens and, you know, cost the entire cost of the query. But the next time you run that same request, the prompt is going to be cached in the LLM. And also, you'll have the questions and the answers stored in your vector database. So you'll be able to return answers way faster. And when you use both of these strategies together, your prompts, your LLM API requests, they become like they cost you like a fraction of their of what they should actually cost you. So go watch my previous video on prompt caching. So you know how that works. And make sure you're using both of them together to reduce costs and stuff. Now, lastly, let's talk about the best places to use semantic caching, because believe it or not, as great as the strategy is, it doesn't exactly work every single, like in every single use case. For instance, if I was building a coding agent, semantic caching would be the last thing I ever implement, because it is extremely useless and horrible in that particular use case. But there are certain use cases where it is extremely valuable. And I mentioned some of these in the beginning of the video as well. But basically, those would be things like support bots on your website. It would be something like an FAQ agent or a knowledge base assistant, like an agent that is trained on a large amount of documents on your knowledge base and supposed to answer questions based on those documents. Stuff like that, like QA agents, knowledge base agents, support and FAQ agents. These are the best places to use semantic caching. This is where you'll find you'll generate the most amount of cost savings by using semantic caching. So if you're using something like that, go ahead and implement semantic caching, and you should see a much less cost to your LLM tokens. So if your app gets a lot of repeat questions, that's the best place to use semantic caching. If you're not getting a lot of repeat questions, probably don't use semantic caching. It's not going to provide a very high ROI. But if your agents get a lot of repetitive questions, that is the best kind of agent to use semantic caching. Just embed the question using an embedding model. Look for a similar question in your cache. If it exists, return that answer. If it doesn't exist, you know, generate the answer with the LLM and then cache that answer. It's as simple as it gets. Now, I'm going to wrap that up over here. Hopefully, this helped you and hopefully you implement this and your costs go down and you love me for it. I'm going to wrap up the video over there. Like, comment, subscribe. And if you want to see more like this, let me know in the comments below. And thank you for watching.