How semantic search works

Have you ever wondered how if I search for fantasy movies on Google, it's going to recommend me things like Lord of the Rings, Fantastic Beasts, Harry Potter or something like that. Even though the names of these movies don't necessarily have the word fantasy in them, how does Google know that these movies are related to fantasy or adventure or stuff like that, right? You obviously can't use keyword search for this because none of the keywords would match, right? Who says that these are fantasy movies, right? You can't tell them from their titles alone, right? So how does Google know what fantasy even means? What do I mean when I say fantasy movies? And how does Google know that these movies match that criteria, right? Well, it's something called semantic search. Let's talk about that. Semantic search is the complete opposite of keyword search. In keyword search, let me just type all of these things out as well so you can talk about them. In keyword search we're basically you know matching keywords right does this word or does this title a movie or text have or contain the words in my search query if yes then we return those results right but obviously keyword search is not going to match things like fantasy movies and lord of the rings right that's where semantic search comes in semantic search is going to match the meaning behind the words right the actual meaning what the words mean right so as an example if you have two words over here like if you have two words like king and queen notice how much similar they are to each other right and that is why i placed them similar over here and notice the word solar system you can probably tell king and queen are very similar to each other but the word solar system that's that's just a completely different concept right and so that is why it's so far away from these two words that's what semantic search does it clusters together similar words like this it clusters together words with similar meaning so that they're you know closer together and as for the words that aren't really similar to each other it's going to move them far apart like this solar system word over here this solar system is going to have similar words like i don't know mars or earth or the sun it's going to have those kinds of words similar to it right over here close to each other, right? But it's going to be far away from words like king and queen, right? Because they're just not similar to each other. That's how semantic search works. It basically just clusters together similar words so that we can actually infer the meaning of those words. So that if I search for something like fantasy movies, movies like Harry Potter, Lord of the Rings, Fantastic Beasts, they're going to be similar to fantasy movies, the words, right? And that is why I get them in the search results. Now that's a very general high-level overview, right? That's something that even Google could tell you. How does this thing work under the hood, right? Let's talk about that. And the big thing here is basically vectors, right? If you've studied vectors like maths and physics before, you know what I'm about to talk about over here. But basically vectors are quantities that have both a direction and a magnitude, right? So for instance, you can have a value with just a magnitude like six feet tall or six figures income or you know stuff like that but when you have a vector you have both a quantity that's the magnitude right that's how big this thing is right now this could be six inches this could be 12 inches this could be nine inches and it also has a direction so right now it's pointing over here or it could point over here or wherever right vectors are used everywhere in the machine learning and you know natural language landscape right every single machine learning algorithm you use there's a high chance that they're using vectors right all the neural networks all the machine learning algorithms and models out there they're using vectors to model their data basically so you could have a vector over here that says king you can have a vector over here it says king queen something like that. Let me just label these as well for you. So this is how you would represent the same thing I drew over here for you in terms of vectors, right? You can see that the king and queen vector are very similar. They have a very similar direction. They have similar magnitudes as well. As for the solar system vector over here, it has a much different magnitude and it has a completely different direction as well That is why we use vectors in machine learning and neural networks because of their direction property right you can see how similar two vectors are by their direction since king and queen are two similar words they have almost the same direction over here right they're pointing in the same direction that is not the case for the word solar system it's pointing in a completely different direction because these words are not similar right so we use vectors for their direction property because the vector direction allows us to tell how similar two vectors are so i want to know how google is able to implement semantic search to tell you that lord of the rings is a fantasy movie this is how it has something like the word fantasy over here and then it has something like a lot of the rings over here and because these two vectors have almost the same direction google knows that they're similar so google knows that when you're searching for fantasy movies you're asking for lord of the rings and other movies like that right so now the question becomes how do you convert words into vectors right because fantasy solar system these are just english words right how do you convert these english words into their mathematical vector representation that's where embedding models comes in so let's talk about that let's talk about embedding models embedding models are a special kind of machine learning model whose entire purpose is to convert text or images or other kinds of data into vectors right let me actually show you an example of a model that i often use this is i don't even know how to pronounce this but ten leper something slash gte large right this is an embedding model that's going to convert any text that you give it into a bunch of vectors, right? And this is on Hugging Face. Hugging Face is basically just a website that has a bunch of AI models, data sets, and other things. It's a really great website to be on if you're developing AI and machine learning stuff, right? If you're working in this field, you need to be on Hugging Face or Kaggle or other websites like that. But basically, embedding models are just a bunch of machine learning models that take some sort of text. Let me actually draw something out over here, like a flowchart, so you guys understand. so this is what embedding models do you have some sort of input text you can also have images by the way we'll get to that you have some sort of input text or images that you want to convert to vectors you pass those in to an embedding model which is just a machine learning model that's used to convert text or images into vectors into their mathematical representation and then this model is going to output a bunch of embeddings embeddings is the word we use to represent these vectors so anytime someone says embeddings they mean vectors right they just mean vectors just like what we have over here but they just mean it in the context of embedding models and semantic search right that's what embeddings mean they just mean a bunch of vectors all right now once we have these vectors we've converted our input text into a bunch of embeddings into a bunch of vectors that's when we can actually check their vector direction and we can tell whether this is similar or not right We can tell if we search for fantasy whether that is similar to Lord of the Rings or not because now we have their direction Right and then we can return the appropriate search results. That's how semantic search works Okay, so now we know how to check whether two things are similar right now. How do we check how similar they are? How do we rank these search results? Um, so there is a visual way of doing this Let me add a couple more arrows over here. Let's say something like like, I don't know, Harry Potter, right? That's a kind of a fantasy movie as well. How do you rank these search results? Which one of these should come first, right? Which one? If I search for fantasy movies on Google, should Lord of the Rings come first or Harry Potter? Well, obviously, Lord of the Rings, because that's a better movie. But how would Google know that, right? Well, again, the benefit of representing these things as vectors, as mathematical, you know, quantities, is that we can do all sorts of mathematical formulas and computations on it. One such formula is called cosine similarity and I'm not going to go too deep on this but basically it's a formula a mathematical formula that you can use on vector quantities. You can run the numbers on this and you can tell how similar two vectors are right. So if you calculate this then let's say Lord of the Rings is more similar to fantasy the fantasy vector over here and so that going to be the top result of the search page right and then let say harry potter is not as similar as lord of the rings so it's going to be the second um search result on the page right that's how cosine similarity works and this is the i guess the most popular way to calculate vector direction and rank the results right there are others as well but we're not going to go too deep on them what you to just understand is that we have a bunch of vectors these are just mathematical quantities and since they're mathematical quantities we can run all sorts of formulas and calculations on them to tell which vectors are similar to each other and how similar they are we can then use these mathematical formulas to rank the search results right so for instance this cosine similarity is going to give us a search score a ranking score between 0 and 1 and whichever of these search results has the higher score obviously we're going to put that on top of the search results right that's how that works so on a high level overview that is how semantic search works right you have a you have a bunch of text you embed that text using some sort of an embedding model over here and you generate the embeddings which are just vectors under the hood right then you use mathematical formulas like cosine similarity like euclidean distance and like a bunch of other formulas to tell which vectors are more similar than others and then you use those scores those formula answers to rank the search results and show the most similar search results at the top right that's how semantic search works and i also have a little you know flow chart over here to really describe the entire process by the way we're going to implement this in code as well in a future video probably the next one after this i'm going to implement semantic search in code so you can actually see how it works but first of all this is like an entire overview of how to handle semantic search in your own database right so let's say you have a big database of text or documents that you want to be able to search through using semantic search using meaning right this is how you would do it you would first of all prepare the model that's stage one and to do that you load your data set which is a bunch of documents or databases or whatever you convert it into text you know you can't just pass in a database table or some you know other stuff like that you need to convert it into text because these embedding models are designed to are designed to take text as input right so you convert your data set into a bunch of text you pre-process it and that just means cleaning up the data a little bit you break it into chunks and the reason you break it into chunks and i forgot to discuss this is because each of these embedding models has some sort of a token limit right they have a token limit over here and this is different for every single model right so some models have 1024 token limits some have a 512 token limit some have even less than that and basically what this means is how many words can you input into this model and get some really high quality embeddings generated by that if you put too many tokens in you're gonna to get some really horrible results from that right so every single model has a token limit and you can read about this in the model's documentation page right wherever you download the model from from hugging face or something like that they have this information in their documentation and you can read that but yeah every single model has a maximum token limit and you need to stick in that token limit and that is why you need to break your text into chunks that can fit into the model's token limit right and then you use the ai embedding model to encode your text and generate the embeddings for it right once you've generated the embeddings you have the vectors then what you're going to do is you're going to store that embedding data alongside the data item that you've got right so if you have some sort of a document that you're embedding you want to store the embeddings that you've generated alongside that data so that you can reference them easily later right and also embedding is a very expensive process computationally It takes a lot of time to generate embeddings for text, especially if you're using a good decent model, right? So you need to make sure that you store the generated text, so store the generated embeddings in some sort of a cache file or a vector database so that you don't have to recalculate them. The best place to store this is because, look, embeddings are just vectors, right? There are vector databases you can use out there Things like Pinecone things like ChromaDB things like PGVector which is a plugin in postgres things like vv8 these are vector databases and they are specifically designed to store vector quantities right so you should use something like that or you can even literally store them in some sort of a binary file on your disk right we're going to go through that as well in the next video but you want to store them in some sort of a database or a file so you don't recalculate them every single time you run your program because this is a very computationally expensive process i literally generated like 50 000 embeddings yesterday for one of my projects and it took like four hours right so you need to make sure you cache them somewhere right so you don't recalculate them once you've done that you're basically done with the preparing the model right you're basically done with stage one which is just preparing the model now you can actually handle search queries and the way you do that is you take the user search query like let's say i i search for something like fantasy movies again or something like that you take that query and use the same embedding model that you used to generate an embedding for my search query as well right you're going to convert my search query like fantasy movies into its embedding form into its vector representation so now you have your search query and you also have your data set both of them as vectors right as embeddings then you're going to compare the cosine similarity of the query embedding which is what i search with each item's embedding in your data set right and again you can use cosine similarity you can use other formulas but cosine similarity is generally the best one right in most cases you're going to do that and you're going to get a bunch of similarity scores right you're going to see the score of how similar each document or each item in your data set is against the search query that i give you and based on those similarity scores you're going to see how well those items match with my search query and then you're going to rank them right and then you're going to display let's say the top 10 or the top 20 search results you're going to sort them by the similarity scores that you generate from the cosine similarity over here and then you're going to just rank them by that score and display the top 10 or 20 results to me right and that is how you implement semantic search right so that's a very high level architectural overview of how this works right the entire process and the algorithm you follow in the next video i'm going to actually implement this in code i'm going to build a very simple implementation of semantic search we're going to be able to search data through simple text right we won't need to match the keywords we'll just need to match the intent or the meaning behind the words right so we're going to do that in the next video or in some future video but for now i just wanted to teach you how semantic search works under the hood so when you search for something on google and even though the keywords don't match but google still gives you some pretty decent results this is how it does that and also when you talk to chat jibbiti and chat jibbiti is able to understand what you're trying to say even though you don't match the specific keywords this is how it's doing that it's using vectors it's using embeddings right it's using semantic search to tell what What exactly you mean, even if you don't say the exact right words, right? This is how AI and chat models and search is working under the hood. So I hope you're enjoying the work I'm doing over here and the videos I'm making. If you are, thank you and leave a like, please, and subscribe. This is a very fascinating topic that even I just recently learned like a month or two ago. And it's just amazing how beautiful technology can be when you actually understand it. So yeah, if you enjoyed this video and if you want to see more content like this, please leave a like, subscribe, share the video, comment down below what you think about the video. If you have any questions or something, I'll make sure to answer them. And yeah, thank you for watching. Lastly, quick self-promotion. This is my web agency's website and I build web apps and mobile apps and other software for people and businesses. If that is something you're interested in or if you know somebody who's interested in that guys Top link in the description is going to take you to book a call with me, which is completely free We can talk about your project. We can discuss it and we can get started working on it This is what I do for a living when I'm not recording videos, right? I build stuff for other people So if you want to hire me to do that for you as well, then top link in the description book a call completely free Let's get started. Other than that. Thank you for watching the video and I will see you in the next one