Building a Redis clone in Go - Part 2 (GET, SET, Writing to client)

Hey guys, welcome back. We're gonna continue to build this Go Redis clone thingy. But first of all, what I want to do is I want to recap how RESP works, Redis Serialization Protocol. We talked about this a bit in the last part as well, but I don't think I explained it as well as I could. So, over here I have a bunch of examples of RESP, different messages in RESP. And I'm just gonna go through each of them and just show you what each of them means, alright? And then hopefully this will give you a much better understanding of resp and how it what it looks like how it works and all That and how we can then parse it and just use it right so Let me just make some space over here So Let's go through this. Okay message over here because this is the simplest example I have right now so First of all this plus symbol this means a string so different data types in rest have a different like symbol to represent them right I think I'm again here we go so we have the string type over here represented by a plus sign and then after that we have the actual string message right so right now I just have an OK message over here and then after that to signal the end of this of this message we have these new line characters backslash r and backslash n what backslash r does is it just takes the cursor back to the beginning of the line and backslash n will move it to the new line right so you don't need to go too deep into those characters you just need to know that these are new line characters all right they're just going to signal the end of the message or the end of this part of the message right so that's how strings are represented in rest all right next we can look at errors now errors are going to be represented by this dash sign all right and we can add that over here as well I can add a new error data type over here represented by this dash symbol all right so to send an error or to receive an error in rest you have to use this dash symbol and then this or keyword over here and then after that you can have an error message all right then once you've added the error message you can then add these new line characters to signal the end of the message right that how Redis will know that the that this is the end of the message and it can stop reading the data after this right now moving on from that this dollar sign represents the bulk string and this is what we talked about in the last part of the video as well this bulk string right over here alright so when it comes to bulk strings we also need to know the length of the string why because let me explain in in regular strings right you You don't need to know the actual length of the string because you know that this string will continue until we reach these new line characters. So you can just continue reading the entire data until you reach these new line characters. So you don't need to know the length of the string. But when it comes to bulk strings, bulk strings can contain multiple words and stuff. So you do need to know the length of the string so you can read the exact number of bytes and know more than that. It's a performance thing over here. which is why you know in to use both strings because use this data this dollar sign symbol to represent a bulk string and then we add the length of the bulk string so right now over here I have six which is the length of this string right so you have the length of the bulk string and then you add these new line characters and then you add the actual string value and then you again add these new line characters to signal the end of this bulk string all right so that all the data types that we're going to use finally the array data type which we looked at in the last video as well I'll quickly go through this again the the Redis client sends all of its messages all of its commands as an array of bulk strings all right so it's important that we cover this one as well and we learn how this works properly. So arrays are represented by this star symbol and then after that symbol we have the length or the number of elements in the array right the length of the array and you can see I have this data type over here as well we we build this in the last part of the video so you can tell that this is an array of two elements based on this right once you add the length then you can add another new line character and then you can add the actual bulk strings over here, right? So you can see over here, if I just format this a bit better, this is the exact same format that we discussed in these bulk strings over here, right? So nothing has really changed, right? All we need to know is that this is an array, which we can tell from the star symbol, and this is an array of two elements. which we can tell from this two over here, right? After that, we just add the actual elements of the array in REST format. And that's about it, right? There's nothing else to discuss over here. These are just simple bulk strings, right? So that's how REST works. That's what it looks like. That's how we can represent and work with different data types in REST. And hopefully now you have a much better understanding of how this works, right? so I can actually close this now and write some code and continue building this project right so in this part of the video we're going to be building command handlers like get set retrieve exists delete all of these other things right so the actual database data handling part of this project that's what we're going to build in this part of the video first let's just quickly recap what we've got so far we have a value struck we have some value types we read the array that we get from the client and this is just to read the bulk string this method and yeah that's about it right so let's let's actually run this server now and I actually also add a little log statement over here we say listening on port 6 3 7 9 just so we know that these servers actually running right and over here as well I can add connection accepted right just some basic logs over here let's run this and Oaks yeah I forgot to shut down the original ready server so let's do that as well there we go and now let's run this there we go so it's running now I can run the client and let's just send a bunch of commands right so this is what we have so far now we need to parse these and just use them to actually build all of the data logic right so let's close all of this and start writing the command handlers first let's create let's go down here and let's create a type for these handlers so I'm gonna call it handler and what this is going to be is a function that takes a pointer to a value and also returns a pointer to a value I explain how this works but basically what you gonna have is you gonna send in the actual value that is the commands that the Redis client sends right which we read from this read array method right over here. We'll pass these into the handler and then the handler will read all of the commands and arguments and everything and then you know do all of the data stuff over there and then it will return a new value that will be a reply that we can send back to the client right like an okay success message or something like that right now let's also create a handler's map now this will be a string and it will hold handler objects right let's initialize this as well and now let's create a handle function over here this is going to be this is just going to take the array that the client sends and it will just select the appropriate handle handler and let it do its thing right so this is going to take the actual connection and also the value all right and there we go now the actual command is going to be in the first element right of the of the array so let do V dot array zero and we know that this array holds a bunch of bulk string elements right so we just get that value over here so that's the actual command right now let's grab the handler from the handlers list by doing this there we go I forgot to actually declare this variable okay now let's handle this okay error as well we need to make sure that this handler actually exists obviously and that's what I'm doing over here if this doesn't exist then let's just log dot fatal or not even log dot fatal we don't need to actually shut down the server just because the client sends a wrong message right let's just print this right invalid command and then just print the actual command and then return because we don't want to continue executing if the handler is wrong right and then we just call the handler with the value right that we get over there now before we go any further we also want to be able to reply to the client right so So the way to do that is we're using con.write, right? That's what we're doing up here to send a message back to the client. But this is kind of like a bit too low level, right? I want another abstraction over here. So let's create a writer struct. I'm going to call this writer. This is going to be a struct and it will have a writer field, which will be an io.writer, not reader, writer. And then let's create a new writer function. this is going to return a pointer to the writer and it's going to take an io.writer this will be the connection that we pass in all right because the problem over here is sure we can reply to the client like this right but we have to construct the reply messages in resp format ourselves right with this plus symbol with the new line characters and all that i don't want to do that every single time I want to reply to the client right so this we're going to create this writer struct over here and what this will do is it will just serialize all of our value structs over here into a REST format right so what I going to be able to do is I going to be able to create a value struct give it a type give it some bulk string data or something and it will just and then I can just pass that into the writer and the writer will convert it to resp format on its own right. So that just saves me a lot of work. So let's return a pointer over here and let's add the bulk field. Now I'm going to use buff.io to create a new writer from this io.writer. Now with that done let's create a write method. This will be a pointer receiver to the writer struct and we'll call this write and it will take a value. There we go. The actual reply will be a string obviously and then we can just use switch a switch statement over here on v dot type and then we can just handle each of the data types here now before i do that i want to add a couple more error types so we already have an error type i want to also add a null value type over here and this is not going to have any like um symbol over here right But we do want a null data type because what we can do is in so let's say you try to get a key that doesn't exist in the database right we want to we want to be able to return a null value in that case so let's do this first let's handle strings right now strings are just going to be let's do this there we go uh strings will need the data symbol which is the plus sign right so let's add that and then they just need the actual string message right after that they just need backslash r and backslash and the new line characters right so let's add the type here and then the actual string all right now let's handle the bulk in this case the reply should be the symbol first of all and then the length of the bulk string right so this will be a digit a number let's add both of these first the type and then the length of the bulk string there we go and then after that we add the new line characters and then after that again we have to add the actual bulk string right there we go and then then finally new line characters again And that how you represent bulk strings Next let do I was going to do array but let's not bother with that because we're not really going to send arrays back in any response in this project. So let's not even bother with that. Let's represent the errors as well. errors are going to be just like strings so we want to have the value type and then the actual error message i don't have an error field over here we need to add that and then finally the new line characters right let's add that value that error field over here this will just be a string there we go and then finally let's handle the null value so to send a null message what we can do is we can just send a bulk string with a length of negative one right and then just the new line characters there we go so now we have a reply message over here now we need to actually send this to the client. So let's grab the writer over here which is which will be the connection that we pass in. Then we need to convert this reply into a a byte slice instead of a string and then we just pass that in and boom that's all we need to do uh it's giving me an error not even an error but a warning over here because we don't need to use this method so let's just turn this into a string literal there we go and now we have a writer over here now we can go back up to this handle function so each of these handlers is going to return uh a value pointer, right? So we can grab that, call it reply. We can initialize a new writer by saying new writer and then pass in the connection over here. And then we can say w dot write and we can pass in the reply over here. There we go. Now up here, we want to actually handle this we want to actually call this function I realize sometimes I just say the most weirdest things if that happens I'm sorry I'm trying to explain this as best as I can but anyway now we handle this we want to pass in the connection and we want to pass in the value that we get over here so a pointer to the value this will first we read the array that we get from the connection right from the client and then we just pass that into this handle function and that about it then this handle function will take care of everything else so now we can actually create a bunch of commands right so let's start with get we're going to this is going to be a handler right so this will take a pointer to the value and And we'll return a new pointer to a new value. First of all, let's get the arguments for this command, right? Now, these are going to be in the array. Now, we know that the first element in the array is the command, right? So, everything after that is going to be the arguments, right? And in the case of get, we only need one argument. So if the arguments are more than one, that's an error, right? So we return that. We return an error message by just creating a value struct with the type of error. And the error message over here can be first the error keyword right Remember this is important This is how Redis knows that this is an error this error keyword and the dash symbol right so make sure you include that now we can just say invalid number of arguments for get command right now assuming that we get the appropriate number of arguments which is just one now we can get the name over here the name of the key that we're trying to fetch from the database this will be the first argument right because there's only one argument in this list and now we need to actually create a database right so for now i'm just going to create a map over here we'll just say this is a string map right as such and we can just use this to grab the value from over here right and of course this argument is going to be a value type so we should grab the bulk string over here and then let's grab the value as well from using from the database as such and make sure to use this ok coma okay idiom so we can make sure that there is a value over here and if there isn then we return a null type just to indicate that there is no value over here and if there is a value then we can return the correct value over here this will be a bulk string and we'll assign the value into the bulk string over here as such awesome so we have the get function now let's also create a set function just so we can test both of them together right again this will take a value and it will return a new value which will be the reply now set is going to have at least two arguments right or exactly two arguments the key that you want to set and the value that you want to set it to right so let's make sure that we're only receiving two arguments over here and if not then we can return an error and the message can be invalid number of arguments for set command now what we can do is we can assign this into the database right so let's grab the key which will be the first argument and and the value which will be the second argument and then we can assign that into the database so this key will be assigned to this value there we go and now we've set the value in the database correctly now we can return a success message like this we'll just give it a type of string and the str can be okay right so the same success message that we use over here all right so now we have the set and the get commands right let's add these to the handlers list up here we'll say get equals get and set equals set there we go now we've added these we've added these handlers over here and the handle function will call the appropriate handler right so now we can actually test this out and see how it works let's run the server it's running let run the Redis CLI there we go and you can ignore this little invalid command error over here you can see that in the handlers list we only handling get and set right now not this command and these dog stuff right so every time the Redis client starts up it sends this command message to the to the server and what this does is it basically returns a bunch of information about the server right we haven't created a handler for that yet which is why we're getting this invalid command error so just ignore this you don't need to worry about it right now let's actually try and get name now nothing at all and actually actually before before we proceed uh i forgot to where is it i forgot to remove this where's the main function there we go we don't need this uh con dot right anymore right because the handle function is going to return the replies for us we don't need it we don't need to do it it like that. So remove that con.write from the main function and then let try this again Run the client Now it not running That weird Oh oh never mind never mind let also handle the command over here because I forgot to I forgot how this works we actually do need to handle this command message over here first because if I don't send a reply to this then the client is just not going to start up right you saw that the client just doesn't start up anymore so let's create a handler for this as well and we don't need to do anything fancy over here. This is just going to return an OK message, right? We don't need to actually return anything other than that. So let's create a handler and just return a new value of type string and the message will just be OK. There we go now the command will actually run now the server can actually run properly and the client can actually start up properly so let's run this again let's clear this and run the cli again and it still not starting up i wonder why okay so we still have one tiny issue to fix um we are sending the replies over here by writing them to the connection but the problem is that sometimes they don get sent to the client because we don flush them out right So let's do that as well. We need to get the writer interface and convert it to a buff.io.writer as such and then just call the flush command right the flush function. What this is going to do is it's going to write any buffer data to the underlying writer right so how how writing data works is it sometimes stores the the data you write into a buffer and then it just delays writing it to the actual file or the actual you know connection and the way to actually force it to write it to the connection or the file or whatever is to just call this flush method right this is going to send it out of the buffer and into the whatever connection or file or whatever resource you have open in this writer right so let's make sure to do that as well now save everything run the client and Run the Client and server there we go now. Let's try to get name we get nil why because we don't have any key stored over here Right now let's try setting name to hassan and it gives me a message message okay right so that's the success message message now let's see if the name got assigned and it did awesome right so yeah we built a key value store over here obviously this is very basic and everything but this works right i can set the age or something to i don't know 21 i can get the age and there we go we get the age right awesome so this is a complete data val data store right a complete key value store just like redis yes the implementation is pretty basic but it gets the job done right now there's two more things we need to add and we're not going to do this in this part of the video because it's getting really long honestly i'm going to end the video over here we've built the get and set commands and even the this command command over here right we made a ton of progress over here, we built a little database object over here. Now, there's two more things we want to implement in this project, which we'll do in the next video. And that is concurrency. We want to make sure that this server is concurrent, we want to make sure that getting data, setting data, handling connections, all of this stuff is concurrent. And we'll use go routines and mute Texas to implement that concurrency. We need this because in a real world application, you're not gonna have just one client and one server right you can have one server and multiple clients from different locations or whatever right accessing the same database accessing the same server right so we want to make this whole thing concurrent we going to do that in the next video And I also want to build some sort of data persistence right We want to be able to store this data on our disk somehow, right? Because Redis is originally just a database stored in memory, right? Which is what we're doing over here. This is just going to be stored in memory. And as soon as I shut down the server like this, all the data gets destroyed, right? It just gets destroyed so we want to build a data persistence method as well there's two major ways that Redis uses the that Redis implements data persistence and we'll talk about both of them in the next video so that's what we're going to build in the next video data persistence making sure that the data we store in this database also gets stored in disk so we don't lose it and concurrency making this whole server concurrent so that multiple clients can connect to it and interact with it without like stepping over each other right that's what we're going to do in the next video and in this video we just built a bunch of command handlers and just built a basic key value store over here so yeah that's all that's all for this video I hope you learned something I hope you are enjoying the series so far we're making good progress here building this Redis clone I hope you're learning new things just as I am stay tuned for the next part and I hope you enjoyed this part if you did like subscribe all of those wonderful things just drive up the algorithm for me and yeah that's all thank you so much for watching and supporting this channel and everything thank you so much and I will see you in the next video bye bye